A focal stack out-of-focus region image depth estimation method

Abstract

The application belongs to the technical field of depth estimation, and particularly relates to a focal stack out-of-focus region image depth estimation method, which inputs a focal point stack into a DfF branch, extracts focal stack features through an encoder of a monocular relative depth estimator, and outputs a boundary expansion focus probability body through a multi-scale feature conversion and cross-image difference processing by a DfF decoder, performs probability regression based on the boundary expansion focus probability body, and obtains an initial depth map and an uncertainty map, wherein the boundary expansion focus probability body comprises a focus probability corresponding to actual focal stack image pairs and a virtual focus probability corresponding to two virtual boundary images located outside an effective focus range; meanwhile, at least one image in the focal point stack is input into the monocular relative depth estimator to obtain a relative depth map, which is aligned in a global scale and used as a structure prior, and the initial depth map and the uncertainty map are input into a refining module together to obtain a depth estimation map.

Description

Technical Field

[0001] This invention relates to the field of depth estimation technology, and in particular to a method for depth estimation of images in the out-of-focus region of a focal stack. Background Technology

[0002] Image-based depth information acquisition is a core foundational technology in the field of 3D reconstruction, widely applied in scenarios such as autonomous driving, virtual reality, and augmented reality. Compared to depth acquisition solutions such as LiDAR and multi-camera networks, depth estimation based on a single RGB camera offers significant advantages, including low cost, flexible deployment, and ease of use. However, a single RGB camera cannot directly collect depth data and must rely on various visual cues to perform indirect depth estimation. Currently, commonly used depth estimation cues in the industry are mainly divided into three categories: image content cues based on perspective relationships, object size, texture brightness, and lighting changes; motion cues based on camera pose changes and pixel parallax shifts; and physical optics cues based on optical imaging principles and imaging focusing characteristics. Among these, image content-based depth estimation methods are easily affected by scene type and lighting conditions, and are prone to overfitting. Motion-based depth estimation methods are highly dependent on camera motion; if the camera's motion amplitude is insufficient, the motion direction is singular, or the scene has few dynamic changes, and the geometric constraints are insufficient, the depth estimation accuracy will be significantly reduced. Depth estimation schemes based on optical physical properties have gradually become a research hotspot in the industry due to their advantages such as strong applicability and low implementation cost. The mainstream implementation form is the focus stack depth estimation method.

[0003] Based on the principle of image focusing, existing depth estimation methods based on focus characteristics are mainly divided into two major technical routes: Depth from Defocus (DfD) and Depth from Focus (DfF). Combined with the thin lens imaging model, it is known that a spatial target point can only form a clear image on the imaging plane at a corresponding focusing distance. Deviating from this distance will result in a circle of confusion phenomenon that is strongly correlated with depth. Both methods address depth estimation based on this optical imaging law. Specifically, the DfD method constructs a mapping model between the degree of defocus blur and scene depth, calculating the corresponding pixel depth based on the actual blur level of the image. Traditional DfD methods rely on the point spread function (PSF) for modeling and solving, often defaulting to a Gaussian kernel distribution as a fixed prior. However, this idealized assumption is difficult to adapt to complex real-world scenes, especially in object edge regions, resulting in limited overall generalization ability of the algorithm. Subsequent studies, while combining deep learning to build end-to-end mapping networks between fuzzy features and depth values, and continuously optimizing model performance through the introduction of camera parameters, self-supervised training, and image patch optimization, remain constrained by fixed imaging priors, resulting in inherent limitations in underlying physical modeling. The DfF method, on the other hand, follows the variation law of pixel image sharpness for depth estimation. The image sharpness of the same pixel exhibits a trend of first increasing and then decreasing with the focusing distance, and the focusing distance corresponding to the peak sharpness is the true depth of the scene point matched to that pixel. This method does not rely on the idealized point spread function prior required by DfD, making it more universally applicable. Traditional DfF methods rely on fixed-focus metric operators such as gradient, Laplacian, and discrete cosine transform to calculate sharpness pixel-by-pixel, and then traverse the focus stack to determine the optimal sharp frame to complete the depth calculation. Not only are the metric operators not universally applicable across all scenes, but they also rely on a large number of high-density sampled focus stack images, resulting in low inference efficiency. Current mainstream research often uses convolutional neural networks to adaptively extract focal stack focusing features, and combines them with optimization strategies such as probabilistic regression, differential focusing volume, multi-scale feature fusion, and Transformer non-local feature capture to improve depth estimation performance. At the same time, improved schemes that integrate single-image depth priors have also emerged.

[0004] Although existing DfF-related algorithms have achieved significant performance improvements, there are still core technical shortcomings that have not been resolved: existing DfF depth estimation models can only accurately infer the depth of scene regions within the effective focus range, and can only cover the depth interval between the minimum and maximum focus distance of the focus stack. For distant and near scenes outside the effective focus range, the model cannot use defocus blur features to complete effective depth inference. At the same time, it does not make sufficient use of global image context information, and is prone to outputting depth prediction values ​​with high confidence but incorrect results. The fused depth prior information is also difficult to play a corrective role, ultimately causing the depth estimation of areas outside the focus range to fail. This greatly restricts the practical application scope and overall practicality of monocular focus stack depth estimation methods.

[0005] In summary, existing image depth estimation methods have several limitations. They can only achieve effective prediction within the effective focal range, cannot reliably estimate the depth of distant scenes outside the range, and do not make sufficient use of global context information, which can easily lead to high-confidence erroneous predictions. As a result, their overall applicability and practical effectiveness are limited. Summary of the Invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the problems of existing image depth estimation methods, which can only achieve effective prediction within the effective focusing range, cannot reliably calculate the depth of near and far scenes outside the range, and do not make sufficient use of global context information, are prone to generating high-confidence erroneous predictions, and have limited overall applicability and practical use effect.

[0007] To address the aforementioned technical problems, this invention provides a method for estimating the depth of an image in the region outside the focus stack, comprising: Obtain the focus stack and its true depth label map from the training set; Input any image from the focus stack into the monocular relative depth estimator and output a relative depth map; Input the focus stack into the DfF branch, and output the initial depth map and its uncertainty map; The relative depth map and the initial depth map are input into the global scale alignment module in the thinning subnetwork for coarse alignment, and the target relative depth map is output. The target relative depth map, the initial depth map and its uncertainty map are input into the thinning module in the thinning subnetwork, and the depth to be updated of the initial depth map is output, thereby updating the initial depth map and obtaining the depth estimation map. A depth estimation loss function is constructed based on the real depth label map, the initial depth map and its uncertainty map, the target relative depth map, and the depth estimation map. The DfF branch and the thinning sub-network are then iteratively trained until the value of the depth estimation loss function converges. Based on the monocular relative depth estimator, the trained DfF branch, and the thinning sub-network, a trained image depth estimation model is obtained.

[0008] Preferably, any image in the focus stack is input into the monocular relative depth estimator, which outputs a relative depth map, including: Randomly select one image from the focus stack as the relative depth estimation image; The relative depth estimated image is sequentially input into the encoder and decoder in the monocular relative depth estimator, and the relative depth map is output.

[0009] Preferably, the focus stack is input into the DfF branch, and the output is an initial depth map and its uncertainty map, including: Each image in the focus stack is input into the encoder in the monocular relative depth estimator for feature extraction, and the focus stack features of each image are output. The focal stack features of each image are input into the feature transformation module for multi-scale feature transformation, and the multi-scale features of each image are output. Cross-image difference processing is performed on the multi-scale features at the same scale to obtain the difference focus volume at each scale. Differential focus volumes at all scales are input into a DfF decoder for parsing, outputting boundary expansion focus probability volumes for each scale. A softmax activation function is applied to the boundary expansion focus probability volumes at each scale, and the focus degree corresponding to each pixel within the corresponding scale is calculated. Thus, an initial depth map and its uncertainty map at each scale are generated using a probabilistic regression method. The boundary expansion focus probability volume includes the focus probability corresponding to the actual focus stack image and the virtual focus probability corresponding to the two virtual boundary images located outside the effective focus range. The initial depth map with the highest resolution and its uncertainty map are used as the output of the DfF branch.

[0010] Preferably, multi-scale features Represented as: , in, Represents multi-scale feature transformation; Indicates the focal stack feature; Differential Focusing Represented as: , in, Represents the multi-scale features of the i-th image; This represents the multi-scale features of the (i+1)th image; This indicates the number of images in the focus stack.

[0011] Preferably, the initial depth and uncertainty of each pixel in the initial depth map and its uncertainty map are represented as follows: , in, This represents the initial depth of the j-th pixel; This represents the probability that the i-th image is the sharpest frame with the j-th pixel. This represents the focal distance of the i-th image; This represents the uncertainty of the initial depth of the j-th pixel; when i=0 and i=n+1, the i-th image is a virtual boundary image.

[0012] Preferably, inputting the relative depth map and the initial depth map into the global scale alignment module in the thinning sub-network for coarse alignment includes: using the initial depth map to perform least-squares optimization on the relative depth map.

[0013] Preferably, the target relative depth map Represented as: , in, This represents least squares optimization; Indicates the scaling factor; Represents the initial depth map The depth values ​​are sorted in ascending order for the region between 10% and 90%. Relative depth map The j-th pixel in; Indicates translation offset; Represents the initial depth map The j-th pixel in; Depth estimation map Represented as; , in, Represents an uncertainty graph; This represents a refined module indicating a relatively deep level of guidance; This indicates the depth is yet to be updated.

[0014] Preferably, a depth estimation loss function is constructed based on the true depth label map, the initial depth map and its uncertainty map, the target relative depth map, and the depth estimation map, including: Based on the Smooth L1 error between the real depth label map and the initial depth map, an initial depth loss function is constructed. Based on the initial depth map, its uncertainty map, and the depth estimation map, a refined loss function guided by the first relative depth is constructed. Based on the depth estimation map and its gradient, as well as the target relative depth map and its gradient, a refined loss function guided by the second relative depth is constructed. The depth estimation loss function is obtained by summing the initial depth loss function, the refined loss function guided by the first relative depth, and the refined loss function guided by the second relative depth.

[0015] Preferably, the initial depth loss function Represented as: , in, Indicates scale; , , , ; Indicates the number of valid pixels; Represents the Smooth L1 error function; This represents the j-th pixel in the initial depth map at scale s; This represents the j-th pixel in the true depth label map at scale s; First relative depth-guided refined loss function Represented as: , in, Represents the natural exponential function; Represents an uncertainty graph; Represents a depth estimation map; Represents the initial depth map; Represents the L1 norm; The second relative depth-guided refined loss function Represented as: , in, Represents the gradient in the x-direction; This represents the gradient in the y-direction; This represents the relative depth map of the target.

[0016] Preferably, the depth estimation loss function Represented as: .

[0017] The depth estimation method for out-of-focus regions provided in this application designs a boundary-extended depth from focus (BE-DfF) network that integrates relative depth map structural priors. BE-DfF consists of an MRD (Monocular Relative Depth) branch, a DfF branch, and a thinning subnetwork. In the MRD branch, an image is randomly selected from the focus stack, and a relative depth map is estimated using a pre-trained monocular relative depth estimator as a structural prior. In the DfF branch, features are first extracted from the focus stack image using a monocular relative depth encoder. After transformation, the features are fed into the DfF decoder, and an initial depth map and an uncertainty map are calculated using a boundary extension probability regression mechanism. This mechanism additionally estimates the probabilities of two virtual boundary images (images whose focus distance is outside the effective focus range of the focus stack) in the probability modeling, used to determine the possibility that the pixel depth falls outside the effective focus range, thus overcoming the focus range limitations of existing methods. In the thinning subnetwork, a global scale alignment model is used... This paper completes scale matching and information fusion between the relative depth map and the initial depth map. Utilizing the global contextual structure information carried by the relative depth map as a guiding basis, it specifically performs depth optimization correction on areas outside the focus range without clear focus extremum clues. It fully mines and utilizes the global contextual features of the image, effectively avoiding the problem of the model outputting high-confidence erroneous depth values. This application solves the existing technical problems of existing focus depth estimation methods, which can only make effective predictions within the effective focus range, are difficult to reliably estimate depths for near and far scenes outside the range, and suffer from insufficient utilization of global contextual information, easily generating high-confidence erroneous predictions, resulting in a narrow applicability and poor practical application performance. Attached Figure Description

[0018] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 A flowchart of the depth estimation method for the out-of-focus region of the focal stack provided in this application; Figure 2 This application provides an architecture diagram for depth estimation of images outside the focal stack. Figure 3 A schematic diagram illustrating the depth distribution of the NYUv2 test dataset provided in this application; Figure 4 A schematic diagram illustrating experimental results with different numbers of images in the stack on the NYUv2 dataset provided in this application; Figure 5Schematic diagram of experimental results at different feature levels on the NYUv2 dataset provided in this application; wherein, Figure 5 (a) in the diagram is a schematic of the root mean square error. Figure 5 (b) in the diagram is a schematic diagram of absolute relative error. Figure 5 (c) in the diagram is a schematic of the squared relative error. Figure 5 (d) in the figure is a schematic diagram of the predicted depth accuracy; Figure 6 A schematic diagram illustrating the test results of the method provided in this application and other focus-based methods on the FoD500 dataset; wherein, Figure 6 (a) in the image is the original image. Figure 6 (b) in the image is the relative depth map obtained from the MRD branch. Figure 6 (c) in the image is the depth prediction map using the DDFF method. Figure 6 In the image, (d) represents the depth prediction map using the DefocusNet method. Figure 6 In the image, (e) represents the depth prediction map using the DFV method. Figure 6 In the diagram, (f) represents the depth prediction map using the HybridDepth method. Figure 6 In the image, (g) represents the true depth map. Figure 6 (h) in the figure represents the depth prediction map obtained by the method provided in this application; Figure 7 A schematic diagram illustrating the test results of the method provided in this application and other focus-based methods on the NYUv2 dataset; wherein, Figure 7 (a) in the image is the original image. Figure 7 (b) in the image is the relative depth map obtained from the MRD branch. Figure 7 (c) in the image is the depth prediction map using the DDFF method. Figure 7 In the image, (d) represents the depth prediction map using the DefocusNet method. Figure 7 In the image, (e) represents the depth prediction map using the DFV method. Figure 7 In the diagram, (f) represents the depth prediction map using the HybridDepth method. Figure 7 In the image, (g) represents the true depth map. Figure 7 (h) in the figure represents the depth prediction map obtained by the method provided in this application; Figure 8 A schematic diagram illustrating the test results of the method provided in this application and other focus-based methods on the Mobile Depth dataset; wherein, Figure 8 (a) in the image is the original image. Figure 8 (b) in the image is the depth prediction map using the DDFF method. Figure 8 (c) in the image represents the depth prediction map using the DefocusNet method. Figure 8 In the image, (d) represents the depth prediction map using the DFV method. Figure 8 In the image, (e) represents the depth prediction map using the HybridDepth method. Figure 8 (f) in the figure represents the depth prediction map obtained by the method provided in this application; Figure 9 Schematic diagrams illustrating depth estimation results of the method provided in this application and other focus-based methods in outdoor scenes; wherein, Figure 9 (a) in the image is the original image. Figure 9 (b) in the image is the depth prediction map using the DDFF method. Figure 9 (c) in the image is the depth prediction map using the DFV method. Figure 9 In the image, (d) represents the depth prediction map using the HybridDepth method. Figure 9 (e) in the figure is the depth prediction map obtained by the method provided in this application. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0020] Please see Figure 1 , Figure 1 The diagram shows the flowchart of the depth estimation method for the out-of-focus region image provided in this application. The method specifically includes steps S10 to S50: S10: Obtain the focus stack and its true depth label map in the training set.

[0021] S20: Input any image from the focus stack into the monocular relative depth estimator and output a relative depth map.

[0022] S30: Input the focus stack into the DfF branch and output the initial depth map and its uncertainty map.

[0023] S40: Input the relative depth map and the initial depth map into the global scale alignment module in the thinning sub-network for coarse alignment, and output the target relative depth map; input the target relative depth map, the initial depth map and its uncertainty map into the thinning module in the thinning sub-network, output the depth to be updated of the initial depth map, and thus update the initial depth map to obtain the depth estimation map.

[0024] S50: Construct a depth estimation loss function based on the real depth label map, the initial depth map and its uncertainty map, the target relative depth map, and the depth estimation map. Then, iteratively train the DfF branch and the thinning sub-network until the value of the depth estimation loss function converges. Based on the monocular relative depth estimator, the trained DfF branch, and the thinning sub-network, a trained image depth estimation model is obtained.

[0025] like Figure 2 The diagram shows the image depth estimation architecture for regions outside the focus stack provided in this application. This application designs a BE-DfF network model to improve the depth estimation accuracy for regions outside the effective focus range. Specifically, the BE-DfF network model consists of three parts: an MRD (Monocular Relative Depth) branch, a DfF branch, and a thinning sub-network. In the MRD branch, an image is randomly selected from the focus stack, and then a monocular relative depth estimator with good generalization is used for feature encoding and decoding to obtain a relative depth map. It is important to note that to ensure the generalization of the MRD branch, the encoder and decoder of the monocular relative depth estimator are frozen during training. In the DfF branch, the encoder, feature transformation module, DfF decoder, and boundary expansion probability regression mechanism in the monocular relative depth estimator are used to predict the initial depth map and the corresponding uncertainty map from the focus stack. In the thinning sub-network, the thinning module uses the globally aligned relative depth map as a structural prior and adaptively selects regions in the initial depth map where the depth estimation is unreasonable, based on the uncertainty map, for further thinning adjustments.

[0026] It should be noted that, in the method provided in this application, each image in the focus stack is pre-aligned by optical flow, homography matrix transformation or other methods.

[0027] Furthermore, this application uses a pre-trained Base version of DepthAnythingV2 as the monocular relative depth estimator, with DINOv2 as the encoder and DPT as the decoder. Depending on the task requirements, this part can use other existing monocular relative depth estimation methods to achieve performance improvements. To maintain the generalization ability of this depth estimator, the encoder and decoder are frozen during training.

[0028] Specifically, any image in the focus stack is input into the monocular relative depth estimator, which outputs a relative depth map, including S200~S201: S200: Randomly select an image from the focus stack as the relative depth estimation image.

[0029] S201: Input the relative depth estimation image into the encoder and decoder in the monocular relative depth estimator in sequence, and output the relative depth map.

[0030] Furthermore, the focus stack is input into the DfF branch, which outputs the initial depth map and its uncertainty map, including S300~S303: S300: Input each image in the focus stack into the encoder in the monocular relative depth estimator for feature extraction, and output the focus stack features of each image.

[0031] Specifically, to efficiently and fully utilize the pre-trained monocular relative depth estimator, this application uses the encoder of the monocular relative depth estimator as the feature extractor for the focus stack, denoted as the input focus stack (containing n images). , , These represent the height and width of the image, respectively; the extracted focal stack features are: , , This represents a 1 / 4 scale feature layer. This represents a 1 / 8 scale feature layer. This represents a 1 / 16 scale feature layer. This represents a 1 / 32 scale feature layer.

[0032] S301: Input the focal stack features of each image into the feature transformation module for multi-scale feature transformation, and output the multi-scale features of each image; perform cross-image difference processing on the multi-scale features at the same scale to obtain the difference focus volume at each scale.

[0033] Since the encoder of a monocular depth estimator extracts features at a single resolution, while current DfF depth estimation methods typically use multi-scale features at four scales, this application introduces a feature transformation module for multi-scale processing.

[0034] Specifically, multi-scale features Represented as: , in, Represents multi-scale feature transformation; This represents the focal stack feature. Among them, .

[0035] Furthermore, to enable the model to more easily find the extreme values ​​of pixels in the sharpness variation curve, this application performs differential processing on multi-scale features. Specifically, the differential focusing volume... Represented as: , in, Represents the multi-scale features of the i-th image; This represents the multi-scale features of the (i+1)th image; This indicates the number of images in the focus stack.

[0036] S302: Input the differential focus volume at all scales into the DfF decoder for parsing, output the boundary expansion focus probability volume at each scale, apply the softmax activation function to the boundary expansion focus probability volume at each scale, calculate the focus degree corresponding to each pixel within the corresponding scale, and then use the probabilistic regression method to generate the initial depth map and its uncertainty map at each scale; wherein, the boundary expansion focus probability volume includes the focus probability corresponding to the actual focus stack image and the virtual focus probability corresponding to the two virtual boundary images located outside the effective focus range.

[0037] Furthermore, during the decoding process, the DfF decoder resolves multi-scale (1 / 32, 1 / 16, 1 / 8, 1 / 4 of the original image resolution) boundary extended focus probability volumes from the differential focus volume.

[0038] In existing Depth-for-Focus (DfF) methods, depth is typically obtained by probabilistically modeling the sharpness response of each pixel in the focus stack along the focus distance dimension and then regressing it within the effective focus range. When the true pixel depth is outside the effective focus range of a given focus stack, the peak of its sharpness change curve no longer falls within this range. In this case, the model based solely on probabilistic regression within the effective focus range cannot accurately estimate the depth of that pixel. In contrast, the depth estimation range of the DfD method is not strictly limited to the effective focus range. Based on this observation, this application argues that within a certain distance outside the effective focus range, the focus stack image at the boundary does not completely lose focus cues, and the DfF model can still use this residual focus information for depth inference. Therefore, this application proposes a boundary-extended probabilistic regression mechanism to extend the depth estimation capability of the DfF model beyond the effective focus range.

[0039] Specifically, during the decoding process, the model additionally predicts the virtual focus state of two virtual boundary images based on the features of the focal stack image, thus forming a boundary extension focus probability volume. Finally, the corresponding focus degree is calculated through the softmax activation function (the value ranges from 0 to 1, with a larger value indicating greater clarity). Furthermore, initial depth maps and their uncertainty maps at different scales are generated through probabilistic regression. During the inference phase, only the initial depth map and its uncertainty map with the highest resolution are selected. It is important to note that the virtual focus state can not only represent the degree of focus of a pixel on the corresponding virtual image, but also be understood as the probability that the depth of that pixel deviates from the effective focus range: the greater the probability, the farther the pixel's true depth is from the focus boundary, thus enabling the model to extrapolate and model depths beyond the original focus range. Furthermore, the focus distance corresponding to these two boundary images can be physically interpreted as the nearest and farthest depth boundaries that the model can effectively infer under the given focal stack focus cues.

[0040] Specifically, the initial depth and uncertainty of each pixel in the initial depth map and its uncertainty map are represented as follows: , in, This represents the initial depth of the j-th pixel; This represents the probability that the i-th image is the sharpest frame with the j-th pixel. This represents the focal distance of the i-th image; This represents the uncertainty of the initial depth of the j-th pixel; when i=0 and i=n+1, the i-th image is a virtual boundary image.

[0041] It should be noted that the method proposed in this application introduces probability estimation of two boundary virtual images in probabilistic modeling, thereby extending the depth estimation range of the model to... This allows the model to estimate the depth of regions outside the effective focusing range. Meanwhile, The probability sum also represents the probability that the depth of pixel j is within the effective focusing range, which is closer to the actual meaning of depth probability.

[0042] S303: The initial depth map with the highest resolution and its uncertainty map are used as the output of the DfF branch.

[0043] Furthermore, although the boundary extension probabilistic regression mechanism can extend the depth estimation range of DfF beyond the effective focusing range, in these regions, since pixels are blurred in all focused images and the focusing response lacks a significant peak, relying solely on probabilistic regression often fails to obtain stable and accurate depth estimates. To address this, this application further introduces monocular relative depth as a structural prior, and a refinement subnetwork geometrically constrains and refines the initial depth map. Specifically, the refinement subnetwork mainly includes a global scale alignment module and a relative depth-guided refinement module.

[0044] Since monocular relative depth can provide reliable geometric structure information, but its output only has a relative scale and cannot be directly fused with the absolute depth result obtained from the DfF branch, this application introduces a global scale alignment module before performing structure-guided refinement. This module maps the relative depth to a scale space consistent with the initial depth output from the DfF branch. Specifically, the coarse alignment of the relative depth map and the initial depth map into the global scale alignment module in the refinement subnetwork includes: performing least-squares optimization on the relative depth map using the initial depth map.

[0045] Specifically, the target relative depth map Represented as: , in, This represents least squares optimization; Indicates the scaling factor; Represents the initial depth map Sort the depth values ​​in ascending order to the region between 10% and 90%, which helps filter out unreliable regions, such as outliers that are extremely far or extremely close. Relative depth map The j-th pixel in; Indicates translation offset; Represents the initial depth map The j-th pixel in the array.

[0046] Furthermore, after obtaining the target relative depth map, this application introduces it as a structural prior into the refinement module to constrain and optimize the initial depth map. The goal of this module is not simply to regress the numerical depth, but to correct structural errors in the DfF (Depth-for-Flight) at boundaries and outside the effective focusing range by forcing the final depth to be geometrically consistent with the target relative depth map. Especially outside the effective focusing range, due to the high uncertainty of the DfF, the structural prior plays a dominant role in the refinement process, ensuring that the depth propagates reasonably along the object boundaries and hierarchical relationships in the relative depth map.

[0047] Specifically, depth estimation map Represented as; , in, Represents an uncertainty graph; This represents a refined module indicating a relatively deep level of guidance; This indicates the depth is yet to be updated.

[0048] Furthermore, a depth estimation loss function is constructed based on the true depth label map, the initial depth map and its uncertainty map, the target relative depth map, and the depth estimation map, including S500~S503: S500: Construct an initial depth loss function based on the Smooth L1 error between the real depth label map and the initial depth map.

[0049] To maintain the performance of the DfF branch, this application retains the original initial depth map-related loss of the DfF model. Specifically, the initial depth loss function... Represented as: , in, Indicates scale; , , , ; Indicates the number of valid pixels (non-zero values); This represents the Smooth L1 error function, which combines L1 loss and L2 loss; This represents the j-th pixel in the initial depth map at scale s; This represents the j-th pixel in the true depth label map at scale s.

[0050] S501: Based on the initial depth map, its uncertainty map, and the depth estimation map, construct a refined loss function guided by the first relative depth.

[0051] Specifically, the refined loss function guided by the first relative depth. Represented as: , in, Represents the natural exponential function; Represents an uncertainty graph; Represents a depth estimation map; Represents the initial depth map; This represents the L1 norm.

[0052] S502: Based on the depth estimation map and its gradient, as well as the target relative depth map and its gradient, construct a refined loss function guided by the second relative depth.

[0053] Specifically, the refined loss function guided by the second relative depth. Represented as: , in, Represents the gradient in the x-direction; This represents the gradient in the y-direction; This represents the relative depth map of the target.

[0054] S503: Summing the initial depth loss function, the first relative depth-guided refinement loss function, and the second relative depth-guided refinement loss function yields the depth estimation loss function.

[0055] Specifically, depth estimation loss function Represented as: .

[0056] To verify the effectiveness of the method provided in this application, this application conducts extensive qualitative and quantitative experiments on multiple synthetic and real datasets using the BE-DfF model and presents the experimental results.

[0057] In the experiment, the entire network was implemented using PyTorch, with the Adam optimizer used and an initial learning rate set to... And gradually decrease during training. The maximum norm of gradient clipping is 0.5. This application trains the model on a single NVIDIA RTX 4060 GPU. It should be noted that the parameters of the monocular relative depth estimator are frozen in all experiments.

[0058] 1. Dataset Selection (1) Synthetic dataset FoD500 The FoD500 dataset is a synthetic dataset created by the Blender Cycles renderer. It contains 500 focus stacks, each with five defocused images at different focus distances (0.1m, 0.15m, 0.3m, 0.7m, 1.5m) and a corresponding depth map. The image resolution is [resolution missing]. The maximum depth is 3m. This dataset simulates the blurring phenomenon in areas outside the focus range of real-world scenes through complete synthesis, avoiding interference from other influencing factors and providing the true depth information corresponding to the images. In the experiment, the entire dataset was divided into training and validation sets in a 4:1 ratio.

[0059] (2) Real dataset NYUv2 The NYUv2 dataset is a monocular depth dataset captured by real cameras. Each sample contains a fully focused image and a corresponding depth map. We used a subset containing 795 training samples and 654 test samples, covering multiple different scenes. All images have a resolution of 640×480, and the maximum depth of the depth map is 10m. This dataset provides rich images of real-world scenes and objects. By blurring the fully focused images, we can simulate the blurring process outside the focus range in real-world applications, obtaining focus stack data in real-world scenes. This data, in complex scenes and diverse object environments, helps to more accurately evaluate the model's performance under real-world conditions. To obtain the focus stack for training and evaluation, a thin lens model and blurring methods were used to blur the fully focused images in the dataset. During the blur rendering process, the lens focal length f = 50mm and f-number N = 8 were set, generating 5 defocused images at different focusing distances: [1m, 1.5m, 2.5m, 4m, 6m]. During the training phase, the images are randomly cropped to a size of 256×256 and a flipping effect with a probability of 0.5 is applied.

[0060] (3) Real dataset Mobile depth The Mobile Depth dataset is a handheld focus stack dataset captured by a mobile phone for the DfF task. In this dataset, each sample consists of a series of images (i.e., focus stacks) taken at different focus distances. These images were captured sequentially on a Samsung Galaxy S3 phone in autofocus mode, with each stack containing 14–33 frames, corresponding to different focus length variations. The image resolutions range from 360×640 to 518×774, covering a variety of everyday scene categories such as plants, bottles, fruits, metal objects, windows, and telephones, demonstrating real-world focus response under various material and surface conditions. This dataset was used in testing to validate the effectiveness and robustness of the proposed depth estimation method on real mobile device data. Since ground truth depth values ​​are not provided, this application only uses this dataset for qualitative experiments.

[0061] 2. Selection of evaluation criteria This application selects Root Mean Square Error (RMSE), Logarithmic Root Mean Squared Error (log RMSE), Absolute Relative Error (AbsRel), and Square Relative Error (SqrRel). , ,and As an evaluation criterion, all the pixels involved in the calculation are valid pixels in the depth map.

[0062] (1) , Where n represents the number of true depth maps in the dataset. This represents the number of valid pixels in the i-th true depth map. Valid pixels are the non-zero pixels in the true depth map. This represents the predicted depth value of the j-th valid pixel in the i-th image. This represents the true depth value of the j-th valid pixel in the i-th image.

[0063] RMSE focuses on the magnitude of the global error and is sensitive to larger errors, thus effectively evaluating the model's performance in predicting large errors.

[0064] (2) , Among them, log RMSE is more balanced in handling large and small depth values, which can reduce the excessive influence of large depth values ​​on error measurement and is suitable for measuring the accuracy of large-scale depth changes.

[0065] (3) , AbsRel measures the relative error between the predicted depth and the true depth. It can adapt to data at different depth scales and is especially sensitive to errors in shallow depth regions.

[0066] (4) , SqrRel, similar to AbsRel, focuses on relative error, but penalizes larger errors more severely.

[0067] (5) , Where num() represents the number of valid pixels that meet the conditions in the i-th predicted depth map. Let be the number of valid pixels in the i-th true depth map.

[0068] The accuracy of the model's predicted depth within a certain threshold range was measured. , yes Further expansion of this model allows for a more comprehensive evaluation of its performance under different error tolerances, especially for scenarios with significant depth variations.

[0069] (6) , (7) .

[0070] 3. Ablation and Parameter Experiments (1) Ablation test This application conducted ablation experiments on the BE-DfF model. The modules used in the experiments included a boundary expansion probability regression mechanism, a relative depth-guided refinement module, and a global scale alignment module. The dataset used in the experiments was NYUv2.

[0071] Table 1 shows the results of the ablation experiments. 0 indicates that none of the above modules were used; 1 indicates that the boundary extension probability regression mechanism was used; 2 indicates that the relative depth-guided refinement module was used; and 3 indicates that the global scale alignment module was used. ↓ indicates that the smaller the value of the evaluation metric, the better the model performance; ↑ indicates that the larger the value of the evaluation metric, the better the model performance. Regular indicates that the pixels participating in the evaluation are non-zero pixels in the true depth map. 1.0m-6.0m indicates that the pixels participating in the evaluation are pixels with a depth between 1.0m and 6.0m in the true depth map, i.e., pixels within the effective focusing range; <1.0m or >6.0m indicates that the pixels participating in the evaluation are non-zero pixels with a depth outside the 1.0m-6.0m range in the true depth map, i.e., pixels outside the effective focusing range. The best-performing pixels in different evaluation intervals are highlighted in bold. As can be seen from Table 1, the boundary extension probability regression mechanism plays a significant role in the region outside the effective focusing range (<1.0m or >6.0m), greatly improving the model's depth estimation accuracy in this region. However, assuming the number of model parameters remains constant and no additional information constraints are introduced, this mechanism leads to a slight performance degradation within the effective focusing range. This is because the model's representational power is redistributed to cover a wider depth range, diluting the accuracy of feature modeling originally targeting the high-confidence interval (effective focusing range). The relative depth-guided refinement module can improve the overall depth estimation accuracy. Due to the scale difference between the relative depth map and the initial depth map, prioritizing global alignment of the relative depth map helps enhance the guiding refinement effect of the relative depth-guided refinement module.

[0072] Table 1

[0073] (2) Parameter change experiment This application conducted parameter variation experiments on several important parameters that affect model performance. The parameters tested included the extended focus distance of the virtual image, the focus distance of each image in the focus stack, the number of images in the stack (INum), and the multi-scale architecture of the relative depth-guided refinement module (Scale).

[0074] (2.1) Focusing distance of the boundary virtual image Let the Focus Distance be (1.0m, 1.5m, 2.0m, 4.0m, 6.0m), INum=5, Scale=4 (1 / 4, 1 / 8, 1 / 16, 1 / 32), and the candidate value ranges for the two Extended Focus Distances be set to [0.97* 0.95* 0.92* 0.9* ], [1.1* , 1.2* , 1.3* ], among which This represents the minimum focusing distance in the focal stack image. This represents the maximum focusing distance in the focal stack image. Experiments were conducted on the FoD500 and NYUv2 datasets, and the results are shown in Tables 2 and 3. It can be observed that on the FoD500 dataset, when the Extended Focus Distance is (0.95*... , 1.2* When the Extended Focus Distance is 0.9, the model achieves the best overall performance across all evaluation metrics; however, on the NYUv2 dataset, the Extended Focus Distance is only 0.9. , 1.3* The model performance reaches its optimal level when the scene scale distribution of different datasets differs. This indicates that the optimal Extended Focus Distance is chosen based on the distribution of scene scale across different datasets.

[0075] Table 2

[0076] Table 3

[0077] (2.2) Focusing distance of the focal stack image Set the Extended Focus Distance to (0.95*). , 1.2* The dataset used was NYUv2, with INum=5, Scale=4 (1 / 4, 1 / 8, 1 / 16, 1 / 32), and Focus Distance ranging from [(1.0m, 3.0m, 5.0m, 7.0m, 9.0m), (1.0m, 1.5m, 2.0m, 4.0m, 6.0m)]. Table 4 shows the experimental results. The results indicate that the focus distance setting has a certain impact on depth estimation. With the same number of samples, when the focus distance matches the depth distribution in the data, the model can generate a more accurate depth map. The depth distribution of the NYUv2 test dataset is shown in Table 4. Figure 3 As shown, the depth is mainly distributed in the range of 1m-6m.

[0078] Table 4

[0079] (2.3) Number of images in the stack Set the Extended Focus Distance to (0.95*). , 1.2* The focus distance was (1.0m, 1.5m, 2.0m, 4.0m, 6.0m), the scale was 4 (1 / 4, 1 / 8, 1 / 16, 1 / 32), and the value range of INum was [2, 3, 4, 5]. The experimental dataset was NYUv2, and the experimental results are as follows. Figure 4 As shown in the diagram, when INum=4, images with focusing distances of (1.0m, 2.0m, 4.0m, 6.0m) are selected; when INum=3, images with focusing distances of (1.0m, 2.0m, 6.0m) are selected; and when INum=2, images with focusing distances of (1.0m, 6.0m) are selected. It can be observed that the more images available, the more depth information is provided, and the more accurate the depth estimation, but the higher the requirements for image acquisition become.

[0080] (2.4) The hierarchy of refined modules in relative depth guidance Set the Extended Focus Distance to (0.95*). , 1.2* The focus distance is (1.0m, 1.5m, 2.0m, 4.0m, 6.0m), INum=5, and the scale value range is [1,2,3,4]. Scale=1 indicates that one scale (1 / 4) is used, Scale=2 indicates that two scales (1 / 4, 1 / 8) are used, Scale=3 indicates that three scales (1 / 4, 1 / 8, 1 / 16) are used, and Scale=4 indicates that four scales (1 / 4, 1 / 8, 1 / 16, 1 / 32) are used. The experimental dataset is NYUv2. Figure 5 The diagram shows experimental results at different feature levels on the NYUv2 dataset provided in this application; where, Figure 5 (a) in the diagram is a schematic of the root mean square error. Figure 5 (b) in the diagram is a schematic diagram of absolute relative error. Figure 5 (c) in the diagram is a schematic of the squared relative error. Figure 5 (d) in the figure illustrates the accuracy of depth prediction. The figure shows that the more scales, the better the model performance. This is because higher scales have larger receptive fields and a stronger ability to process contextual information. Currently, most existing technologies set the number of scales to 4.

[0081] (3) Experimental comparison and analysis This application first compares the model with other focus-based methods on different datasets, FoD500 (a synthetic dataset) and NYUv2 (a real dataset), as shown in Tables 5 and 6. Then, the generalization ability of the model is tested on the unseen dataset Mobiledepth, and the test results are as follows: Figure 6 and Figure 7 As shown.

[0082] Table 5 compares the proposed method with several focus-based depth estimation methods on the FoD500 dataset. DefocusNet represents the DFD method, while DDFF, DFV, HybridDepth, and the proposed method represent the DfF method. DFV is the baseline model of this application, and HybridDepth is the state-of-the-art (SOTA) model of DfF. Experimental results show that the proposed method achieves the best performance across all evaluation metrics. Compared to the baseline model DFV, BE-DfF reduces the RMSE error by 6.33% in regions outside the effective focus range and by 3.64% in all regions. Compared to the state-of-the-art HybridDepth model, BE-DfF reduces the RMSE error by 5.97% in regions outside the effective focus range and by 3.64% in all regions. These results demonstrate that the proposed BE-DfF has stronger depth estimation capabilities in regions outside the effective focus range.

[0083] Table 5

[0084] Table 6 compares the method provided in this application with several focus-based depth estimation methods on the NYUv2 dataset. DefocusNet is the DFD method, while DDFF, DFV, HybridDepth, and the method provided in this application are DfF methods. As shown in Table 6, the method provided in this application also exhibits excellent performance on the real dataset. Compared to the baseline model DFV, BE-DfF reduces the RMSE error by 35.4% in the region outside the effective focus range and by 25.38% in all regions. Compared to the state-of-the-art model HybridDepth, BE-DfF reduces the RMSE error by 22.64% in the region outside the effective focus range and by 14.53% in all regions. Since both HybridDepth and the method provided in this application introduce additional refinement modules, their inference time is slightly increased compared to methods without refinement modules, but they bring significant gains in accuracy, especially on the NYUv2 dataset.

[0085] Table 6

[0086] like Figure 6 The diagram shows the test results of the method provided in this application and other focus-based methods on the FoD500 dataset. Figure 6 (a) in the image is the original image. Figure 6 (b) in the image is the relative depth map obtained from the MRD branch. Figure 6 (c) in the image is the depth prediction map using the DDFF method. Figure 6 In the image, (d) represents the depth prediction map using the DefocusNet method. Figure 6 In the image, (e) represents the depth prediction map using the DFV method. Figure 6 In the diagram, (f) represents the depth prediction map using the HybridDepth method. Figure 6 In the image, (g) represents the true depth map. Figure 6 In this context, (h) represents the depth prediction map obtained by the method provided in this application; for example... Figure 7 The diagram shows the test results of the method provided in this application and other focus-based methods on the NYUv2 dataset. Figure 7 (a) in the image is the original image. Figure 7 (b) in the image is the relative depth map obtained from the MRD branch. Figure 7 (c) in the image is the depth prediction map using the DDFF method. Figure 7 In the image, (d) represents the depth prediction map using the DefocusNet method. Figure 7 In the image, (e) represents the depth prediction map using the DFV method. Figure 7 In the diagram, (f) represents the depth prediction map using the HybridDepth method. Figure 7 In the image, (g) represents the true depth map. Figure 7 (h) in the figure represents the depth prediction map obtained by the method provided in this application.

[0087] from Figure 6 It is evident that the depth map predicted by the method provided in this application is smoother and has clearer boundaries. In areas outside the effective focusing range, such as... Figure 7 In the top left corner of the third row, the method provided in this application can infer the depth of the region outside the effective focus range based on the probability of the estimated boundary virtual image.

[0088] To evaluate the model's generalization ability, this application pre-trains the model on the FoD500 and NYUv2 datasets and directly tests it on the Mobile Depth dataset. Since this dataset does not provide true depth annotations, this application... Figure 8 Qualitative experimental results are presented and compared with various focus-based depth estimation methods, such as... Figure 8The diagram shows the test results of the method provided in this application and other focus-based methods on the Mobile Depth dataset. Figure 8 (a) in the image is the original image. Figure 8 (b) in the image is the depth prediction map using the DDFF method. Figure 8 (c) in the image represents the depth prediction map using the DefocusNet method. Figure 8 In the image, (d) represents the depth prediction map using the DFV method. Figure 8 In the image, (e) represents the depth prediction map using the HybridDepth method. Figure 8 (f) in the figure represents the depth prediction map obtained by the method provided in this application. It can be observed that at relatively far distances (e.g., objects near the wall in the second row), the method provided in this application can still estimate relatively reliable depth information. This is due to the combined effect of the boundary expansion probability regression mechanism and the relative depth structure prior, enabling the model to maintain reasonable geometric consistency outside the effective focusing range. However, at even greater distances (e.g., distant areas in the third row), because these areas are consistently in a highly out-of-focus state throughout the focus stack, containing almost no discernible focusing cues, and the structural constraints of relative depth gradually weaken, all methods exhibit significant performance degradation. This phenomenon indicates that when the pixel depth far exceeds the effective focusing range perceptible by the focus stack, depth estimation will inevitably be fundamentally limited by physical imaging conditions.

[0089] like Figure 9 The diagram shows the depth estimation results of the method provided in this application and other focus-based methods in an outdoor scene. Figure 9 (a) in the image is the original image. Figure 9 (b) in the image is the depth prediction map using the DDFF method. Figure 9 (c) in the image is the depth prediction map using the DFV method. Figure 9 In the image, (d) represents the depth prediction map using the HybridDepth method. Figure 9(e) in the figure represents the depth prediction map obtained by the method provided in this application. In large-scale outdoor scenes, the depth map estimated under sparse focal stack conditions exhibits a significant layered structure in some areas. The fundamental reason for this is the structural limitation brought about by the discrete sampling of the focal length dimension. When the sampling interval is large, there are significant depth gaps between adjacent focal planes, making it difficult to fully cover the continuously changing real scene depth. Pixels located between sampling focal lengths usually lack significant sharpness peaks, making depth inference only selectable or approximate between a limited number of discrete focal lengths, thus producing an effect similar to "depth quantization". This quantization effect manifests as a stepped or segmented distribution in the spatial domain, weakening the continuity and detail representation of the depth results. Therefore, when the sampling density of the focal stack cannot match the complexity of scene depth changes, the layering phenomenon is difficult to completely avoid. This problem stems from insufficient sampling of focal length dimension information, rather than a defect in the model inference mechanism itself, constituting an inherent limitation of the DfF method under sparse sampling in large-scale scenes.

[0090] In summary, the proposed BE-DfF (Boundary-Extended Depth from Focus) network model integrates a boundary extension probability regression mechanism with relative depth information constraints. The boundary extension probability regression mechanism additionally estimates the probabilities of two virtual boundary images in probabilistic modeling, allowing the depth representation of the original probability distribution to transcend the limitations of the effective focus range, thus solving the inapplicability of the original probability regression mechanism in regions outside the effective focus range. Simultaneously, the BE-DfF model uses the relative depth map as a structural prior, providing additional contextual constraints for regions outside the effective focus range lacking sharpness extrema cues, thereby improving the depth estimation accuracy in these regions. Furthermore, experimental results on multiple datasets demonstrate that the BE-DfF network model exhibits significant advantages not only in regions outside the effective focus range but also in regions within the effective focus range, showing good estimation accuracy. In regions outside the effective focus range, compared to current state-of-the-art methods, BE-DfF reduces the RMSE error by 5.97% on the FoD500 dataset and by 22.64% on the NYUv2 dataset.

[0091] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0094] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for estimating the depth of an image outside the focal stack's focus area, characterized in that, include: Obtain the focus stack and its true depth label map from the training set; Input any image from the focus stack into the monocular relative depth estimator and output a relative depth map; Input the focus stack into the DfF branch, and output the initial depth map and its uncertainty map; The relative depth map and the initial depth map are input into the global scale alignment module in the thinning sub-network for coarse alignment, and the target relative depth map is output. The target relative depth map, the initial depth map, and its uncertainty map are input into the refinement module in the refinement sub-network, and the output is the depth to be updated of the initial depth map, thereby updating the initial depth map and obtaining the depth estimation map. A depth estimation loss function is constructed based on the real depth label map, the initial depth map and its uncertainty map, the target relative depth map, and the depth estimation map. The DfF branch and the thinning sub-network are then iteratively trained until the value of the depth estimation loss function converges. Based on the monocular relative depth estimator, the trained DfF branch, and the thinning sub-network, a trained image depth estimation model is obtained.

2. The image depth estimation method for the region outside the focus stack according to claim 1, characterized in that, Input any image from the focus stack into the monocular relative depth estimator, and output a relative depth map, including: Randomly select one image from the focus stack as the relative depth estimation image; The relative depth estimated image is sequentially input into the encoder and decoder in the monocular relative depth estimator, and the relative depth map is output.

3. The image depth estimation method for the region outside the focus stack according to claim 1, characterized in that, Input the focus stack into the DfF branch, and output the initial depth map and its uncertainty map, including: Each image in the focus stack is input into the encoder in the monocular relative depth estimator for feature extraction, and the focus stack features of each image are output. The focal stack features of each image are input into the feature transformation module for multi-scale feature transformation, and the multi-scale features of each image are output. Cross-image difference processing is performed on the multi-scale features at the same scale to obtain the difference focus volume at each scale. Differential focus volumes at all scales are input into a DfF decoder for parsing, outputting boundary expansion focus probability volumes for each scale. A softmax activation function is applied to the boundary expansion focus probability volumes at each scale, and the focus degree corresponding to each pixel within the corresponding scale is calculated. Thus, an initial depth map and its uncertainty map at each scale are generated using a probabilistic regression method. The boundary expansion focus probability volume includes the focus probability corresponding to the actual focus stack image and the virtual focus probability corresponding to the two virtual boundary images located outside the effective focus range. The initial depth map with the highest resolution and its uncertainty map are used as the output of the DfF branch.

4. The image depth estimation method for the region outside the focus stack according to claim 3, characterized in that, Multiscale features Represented as: ， in, Represents multi-scale feature transformation; Indicates the focal stack feature; Differential Focusing Represented as: ， in, Represents the multi-scale features of the i-th image; This represents the multi-scale features of the (i+1)th image; This indicates the number of images in the focus stack.

5. The image depth estimation method for the region outside the focus stack according to claim 4, characterized in that, The initial depth and its uncertainty of each pixel in the initial depth map and its uncertainty map are represented as follows: ， in, This represents the initial depth of the j-th pixel; This represents the probability that the i-th image is the sharpest frame with the j-th pixel. This represents the focal distance of the i-th image; This represents the uncertainty of the initial depth of the j-th pixel; when i=0 and i=n+1, the i-th image is a virtual boundary image.

6. The image depth estimation method for the region outside the focus stack according to claim 1, characterized in that, The coarse alignment of the relative depth map and the initial depth map into the global scale alignment module in the thinning subnetwork includes: using the initial depth map to perform least-squares optimization on the relative depth map.

7. The image depth estimation method for the region outside the focus stack according to claim 6, characterized in that, Target relative depth map Represented as: ， in, This represents least squares optimization; Indicates the scaling factor; Represents the initial depth map The depth values ​​are sorted in ascending order for the region between 10% and 90%. Relative depth map The j-th pixel in; Indicates translation offset; Represents the initial depth map The j-th pixel in; Depth estimation map Represented as; ， in, Represents an uncertainty graph; This represents a refined module indicating a relatively deep level of guidance; This indicates the depth is yet to be updated.

8. The image depth estimation method for the region outside the focus stack according to claim 1, characterized in that, A depth estimation loss function is constructed based on the real depth label map, the initial depth map and its uncertainty map, the target relative depth map, and the depth estimation map, including: Based on the Smooth L1 error between the real depth label map and the initial depth map, an initial depth loss function is constructed. Based on the initial depth map, its uncertainty map, and the depth estimation map, a refined loss function guided by the first relative depth is constructed. Based on the depth estimation map and its gradient, as well as the target relative depth map and its gradient, a refined loss function guided by the second relative depth is constructed. The depth estimation loss function is obtained by summing the initial depth loss function, the refined loss function guided by the first relative depth, and the refined loss function guided by the second relative depth.

9. The image depth estimation method for the region outside the focus stack according to claim 8, characterized in that, Initial depth loss function Represented as: ， in, Indicates scale; , , , ; Indicates the number of valid pixels; Represents the Smooth L1 error function; This represents the j-th pixel in the initial depth map at scale s; This represents the j-th pixel in the true depth label map at scale s; First relative depth-guided refined loss function Represented as: ， in, Represents the natural exponential function; Represents an uncertainty graph; Represents a depth estimation map; Represents the initial depth map; Represents the L1 norm; The second relative depth-guided refined loss function Represented as: ， in, Represents the gradient in the x-direction; This represents the gradient in the y-direction; This represents the relative depth map of the target.

10. The image depth estimation method for the region outside the focus stack according to claim 9, characterized in that, Depth estimation loss function Represented as: 。

Images