An occlusion-aware depth estimation and edge-aware depth completion method and system

Abstract

The application relates to a method and system for occlusion-aware depth estimation and edge-aware depth completion, belonging to the technical field of computer vision. The left view and the right view are input into an initial disparity estimation network to obtain a second feature map, an initial first feature map and a disparity map, and the view corresponding to the first feature map is taken as a reference view; the obtained result is input into a cross-epipolar line occlusion-aware iterative refinement network to obtain a refined disparity map, a refined occlusion mask and an uncertainty estimation map; the refined disparity map, the refined occlusion mask, the uncertainty estimation map and the reference view are input into an edge-aware depth completion network to obtain a complete depth map. The cross-epipolar line attention reduces the calculation complexity; the differential refinement strategy improves the disparity refinement precision of the occlusion area; the edge information constraint propagation direction makes the reliable depth recursively diffuse to the occlusion area along the surface of an object without crossing the object boundary, and high-precision and edge-accurate depth estimation of the occlusion area is realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method and system for occlusion-aware depth estimation and edge-aware depth completion. Background Technology

[0002] In existing binocular stereo matching methods, due to the deviation of the left and right camera perspectives, some areas in the scene can only be imaged from a monocular perspective, which cannot complete the effective correspondence matching of left and right pixels, greatly reducing the reliability of binocular depth estimation.

[0003] Current mainstream solutions mainly rely on parallel disparity-occlusion estimation modules and occlusion perception global aggregation modules to handle occlusion problems, but they still have three shortcomings: First, the computational complexity of global self-attention is high, making it difficult for the model to meet the requirements of real-time applications; second, existing methods rely on global correlation to refine the occluded region, ignoring the important geometric prior that the depth information of occluded pixels usually has a stronger correlation with its spatially neighboring non-occluded pixels; third, the ability to preserve edge details is lacking. No special optimization design is made for edge structure preservation during the depth completion process, which easily leads to the problem of blurred object boundaries, resulting in blurred object boundaries, distorted details, and damage to the structural integrity of the depth map.

[0004] Therefore, there is an urgent need to design a depth estimation method that can accurately handle scene occlusion while reducing model computational overhead and preserving depth map edge details, thereby meeting the high-precision and real-time depth estimation requirements of real-world scenarios. Summary of the Invention

[0005] The main technical problem this invention addresses is how to efficiently achieve high-precision and edge-fidelity depth estimation of occluded areas.

[0006] According to the first aspect, one embodiment provides an occlusion-aware depth estimation and edge-aware depth completion method, including:

[0007] Get the left and right views;

[0008] The left view and the right view are input into the initial disparity estimation network to obtain the second feature map, the initial first feature map and the initial disparity map. The first feature map and the second feature map each correspond to a view. The view corresponding to the first feature map is used as the reference view and the pixel in the first feature map is used as the reference pixel.

[0009] The initial first feature map, the second feature map, and the initial disparity map are input into a cross-epipolar occlusion perception iterative refinement network. Multiple iterations of iterative refinement are performed on the initial first feature map and the initial disparity map. In each iteration: based on the second feature map and the first feature map updated in the previous iteration, a cross-epipolar attention distribution map corresponding to different candidate disparities is obtained; based on the cross-epipolar attention distribution map corresponding to different candidate disparities, an aggregated feature map corresponding to the current iteration is obtained; based on the aggregated feature map and the first feature map updated in the previous iteration, the network refines the network. The disparity map is used to obtain the first feature map updated in the current iteration round. An occlusion mask corresponding to the current iteration round is obtained based on the cross-epipolar attention distribution map corresponding to different candidate disparities. The refinement method for each reference pixel is determined based on the occlusion mask corresponding to the current iteration round to obtain the disparity map updated in the current iteration round. The next iteration round is updated based on the second feature map, the first feature map updated in the current iteration round, and the disparity map, until a preset iteration round is reached, resulting in a refined disparity map, a refined feature map, and a refined occlusion mask. Uncertainty estimation is performed based on the refined feature map to obtain an uncertainty estimation map.

[0010] The refined disparity map, refined occlusion mask, uncertainty estimation map, and reference view are input into the edge-aware depth completion network to guide the occluded region for depth completion based on the edge information in the reference view, thereby obtaining a complete depth map.

[0011] In some embodiments, determining the refinement method for each reference pixel based on the occlusion mask corresponding to the current iteration round to obtain the disparity map updated in the current iteration round includes:

[0012] Based on the mask value of each reference pixel in the occlusion mask corresponding to the current iteration round, determine whether each reference pixel is occluded, so as to determine the refinement method of each reference pixel.

[0013] For any reference pixel: if the reference pixel is not occluded, perform residual refinement on the reference pixel to obtain the corrected disparity value of the reference pixel; if the reference pixel is occluded, perform support aggregation refinement on the reference pixel to obtain the aggregated disparity value of the reference pixel.

[0014] The disparity map updated in the current iteration is obtained based on the corrected disparity value or aggregated disparity value of each reference pixel.

[0015] According to a second aspect, one embodiment provides an occlusion-aware depth estimation and edge-aware depth completion system, comprising:

[0016] The data acquisition module is used to acquire the left and right views;

[0017] An initial disparity estimation module is used to input the left view and the right view into an initial disparity estimation network to obtain a second feature map, an initial first feature map, and an initial disparity map. The first feature map and the second feature map each correspond to a view. The view corresponding to the first feature map is used as a reference view, and the pixels in the first feature map are used as reference pixels.

[0018] A cross-polar occlusion perception module is used to input the initial first feature map, the second feature map, and the initial disparity map into a cross-polar occlusion perception iterative refinement network to perform multiple rounds of iterative refinement on the initial first feature map and the initial disparity map. In each iteration round: based on the second feature map and the first feature map updated in the previous iteration round, a cross-polar attention distribution map corresponding to different candidate disparities in the current iteration round is obtained; based on the cross-polar attention distribution map corresponding to different candidate disparities, an aggregated feature map corresponding to the current iteration round is obtained; based on the aggregated feature map and the first feature map updated in the previous iteration round... A first feature map is obtained by combining a feature map and a disparity map, and then updated to the first feature map for the current iteration. An occlusion mask for the current iteration is obtained based on the cross-epipolar attention distribution map corresponding to different candidate disparities. The refinement method for each reference pixel is determined based on the occlusion mask for the current iteration to obtain the disparity map updated for the current iteration. The next iteration is then updated based on the second feature map, the first feature map updated for the current iteration, and the disparity map, until a preset iteration is reached, resulting in a refined disparity map, a refined feature map, and a refined occlusion mask. Uncertainty estimation is performed based on the refined feature map to obtain an uncertainty estimation map.

[0019] The edge-aware depth completion module is used to input the refined disparity map, refined occlusion mask, uncertainty estimation map, and reference view into the edge-aware depth completion network, so as to guide the occlusion area to perform depth completion based on the edge information in the reference view, and obtain a complete depth map.

[0020] According to the above embodiments, an occlusion-aware depth estimation and edge-aware depth completion method and system replace the global self-attention mechanism with a cross-epipolar attention mechanism, significantly narrowing the increase in computational complexity with image resolution, and significantly improving computational efficiency while maintaining disparity refinement accuracy. By deriving the occlusion mask based on the cross-epipolar attention weight distribution, joint optimization of occlusion detection and disparity refinement is achieved without the need for an additional independent occlusion prediction branch. A differentiated refinement strategy is adopted, using residual refinement for non-occluded pixels and deformable neighborhood support aggregation for occluded pixels. This fully utilizes the geometric prior of the stronger correlation between the depth of occluded pixels and spatially neighboring non-occluded pixels, effectively improving the disparity refinement accuracy of occluded regions compared to the global indiscriminate aggregation method. By using uncertain estimates, the depth propagation process can adaptively avoid high uncertainty regions. At the same time, by constraining the propagation direction with edge information in the reference view, reliable depth is recursively diffused along the same object surface to the occluded region without crossing the object boundary, solving the problem of blurred depth values ​​at object boundaries in the depth completion process of existing methods, thereby efficiently achieving high-precision and edge-preserving depth estimation of occluded regions. Attached Figure Description

[0021] Figure 1 A flowchart of a method for occlusion-aware depth estimation and edge-aware depth completion;

[0022] Figure 2 A flowchart illustrating the method for iterative refinement in each iteration round;

[0023] Figure 3 This is a schematic diagram of a system module for an occlusion-aware depth estimation and edge-aware depth completion system. Detailed Implementation

[0024] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of the invention. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to the present invention are not shown or described in the specification. This is to avoid obscuring the core parts of the invention with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0025] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0026] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this invention, unless otherwise specified, include both direct and indirect connections (linkages).

[0027] Currently, deep learning-based binocular stereo matching algorithms, with their powerful feature extraction and matching modeling capabilities, have been widely studied in fields such as 3D reconstruction, autonomous driving, and intelligent perception. However, during binocular imaging, due to the fixed baseline distance and different viewing angles of the left and right cameras, there are many occluded and weakly matched areas in the scene that can only be captured by one camera. These monocular visible pixels cannot find effective matching points in the other frame, easily leading to problems such as cost calculation failure, disparity solving anomalies, and depth abrupt distortion. Ultimately, this results in a significant decrease in overall depth estimation accuracy, severely impacting the algorithm's effectiveness in complex real-world scenarios.

[0028] Existing methods, such as the Global Occlusion Aware Transformer (GOAT) model, accurately identify occluded regions by designing a parallel disparity-occlusion estimation module and building an occlusion-aware global aggregation module. This model leverages a global attention mechanism to model long-distance pixel associations, significantly improving matching stability in complex occluded scenes. However, it still has the following shortcomings: First, this model uses a global self-attention mechanism to perform global association modeling on the entire image, resulting in a computational complexity as high as O(H²W²), where H and W represent image size. As image resolution increases, the computational load increases significantly, leading to slow inference speeds and difficulty meeting real-time application requirements. Second, regarding the occlusion region refinement strategy, this model uses global correlation to complete disparity optimization. However, in real-world scenes, the depth distribution of occluded pixels has spatial continuity, and the correlation between their depth values ​​and nearby non-occluded effective pixels is much higher than that of distant pixels. Global indiscriminate modeling not only introduces invalid noise features but also significantly reduces the refinement efficiency and accuracy of occluded regions. Finally, the model does not have a dedicated constraint and preservation strategy for object edges and contour structures. In scenes with foreground and background interspersed and sharp object boundaries, it is prone to problems such as excessive edge smoothing, boundary offset, and blurred details, which destroy the structural integrity of the depth map.

[0029] Furthermore, patent CN111105451A discloses a binocular depth estimation method for driving scenarios to overcome occlusion effects, focusing on the occlusion interference problem in complex road conditions for autonomous driving. Although it involves occlusion mask prediction and designs a mask loss function adapted to the occlusion scenario, the optimization dimension of this method is relatively simple. It relies on traditional convolutional features for matching and optimization, and does not introduce a Transformer global modeling and iterative refinement mechanism. It cannot achieve multi-round iterative optimization of occlusion parallax and is difficult to handle large-area, complex-shaped occlusion regions. Moreover, this method does not build a dedicated edge constraint and depth propagation mechanism, and cannot perform targeted optimization for key structures such as object edges and road contours. The depth map after occlusion region completion is prone to structural distortion.

[0030] To address the aforementioned issues, this invention proposes a binocular occlusion-aware depth estimation and edge-aware completion method based on Transformer iterative refinement, comprising three stages: an initial disparity estimation stage, a cross-epipolar occlusion-aware iterative refinement stage, and an edge-aware depth completion stage. In the cross-epipolar occlusion-aware iterative refinement stage, iterative cross-attention is performed along the epipolar direction. The occlusion state is determined based on the cross-attention weights, and a differentiated refinement strategy is adopted: non-occluded pixels adjust disparity through residual learning, while occluded pixels adaptively aggregate disparity information from adjacent non-occluded regions through deformable attention. In the edge-aware depth completion stage, anisotropic recursive propagation is performed, guided by image edges, to propagate reliable depth along the edge structure to the occluded region while maintaining clear object boundaries. Compared to existing global occlusion-aware Transformers (such as GOAT), this method can reduce computational complexity by two orders of magnitude while further improving the accuracy and edge integrity of occluded region depth estimation.

[0031] Please refer to Figure 1 Some embodiments provide an occlusion-aware depth estimation and edge-aware depth completion method, including:

[0032] Step S100: Obtain the left and right views; input the left and right views into the initial disparity estimation network to obtain the second feature map, the initial first feature map, and the initial disparity map.

[0033] The initial disparity estimation stage in this embodiment includes a feature extraction module, an initial matching volume construction module, an initial matching volume regularization module, and an initial disparity map acquisition module. The initial disparity estimation network includes a weighted twin feature pyramid network and a 3D hourglass regularization network. The 3D hourglass regularization network consists of an encoder-decoder architecture, containing three 3D convolutional downsampling layers and three 3D deconvolutional upsampling layers, with an embedded channel attention module to adaptively recalibrate the importance of feature channels.

[0034] After obtaining the left and right views and inputting them into the initial disparity estimation network, the feature extraction module first performs multi-scale feature extraction on the left and right views respectively. For example, the feature maps corresponding to the left and right views at different scales (such as 1 / 4, 1 / 8, 1 / 16) can be extracted by a twin feature pyramid network with shared weights, thereby obtaining the multi-scale feature map corresponding to the left view and the multi-scale feature map corresponding to the right view.

[0035] The initial matching body construction module constructs the initial matching body based on the multi-scale feature maps corresponding to the left and right views. In this module, feature map fusion is first performed on the multi-scale feature maps corresponding to the left and right views respectively. High-resolution features preserve local edges, while low-resolution features provide a wider range of context, resulting in cross-scale fused feature maps, namely the first feature map and the second feature map.

[0036] For example, a 1x1 convolution projects the number of channels at each scale onto the same dimension, and then adjusts the feature maps at each scale to the same scale. For instance, the feature maps at each scale are uniformly adjusted to the feature map corresponding to the 1 / 4 scale, that is, the feature map corresponding to the 1 / 8 scale is upsampled by 2 times, and the feature map corresponding to the 1 / 16 scale is upsampled by 4 times. Then, feature fusion is performed on the feature maps after scale unification. For instance, the feature maps after scale unification are concatenated along the channel dimension, and then fused through a convolution operation. The fused feature map can then be represented as:

[0037] F fus =Conv 1×1 [F 1 / 4 ,up2(F 1 / 8 ), up4(F 1 / 16 )]

[0038] Among them, F fus Represents the fused feature map; Conv 1×1 F represents a 1×1 convolution; 1 / 4 This represents the feature map corresponding to a 1 / 4 scale; F 1 / 8 F represents the feature map corresponding to a 1 / 8 scale. 1 / 16 1 represents the feature map corresponding to a 1 / 16 scale; up2 represents upsampling by 2 times; up4 represents upsampling by 4 times.

[0039] It should be noted that, at this time, the size of both the first feature map and the second feature map is 1 / 4 of the original view, and the first feature map and the second feature map each correspond to a view. The view corresponding to the first feature map is used as the reference view. That is, if the first feature map is obtained by fusing the multi-scale feature map corresponding to the left view, then with the left view as the reference view, when the candidate disparity is d, the reference pixel with coordinates (x, y) in the first feature map has coordinates (x-d, y) in the second feature map. If the first feature map is obtained by fusing the multi-scale feature map corresponding to the right view, then with the right view as the reference view, when the candidate disparity is d, the reference pixel with coordinates (x, y) in the first feature map has coordinates (x+d, y) in the second feature map. Furthermore, the pixels in the first feature map are called reference pixels; that is, in this embodiment, the reference pixels are the pixels in the first feature map at 1 / 4 resolution. This embodiment uses the left view as an example for explanation.

[0040] Then, an initial matching body is constructed based on the obtained first feature map and second feature map, wherein the initial matching body includes a similarity distribution map between the first feature map and the second feature map under different candidate disparities.

[0041] For example, based on the first and second feature maps at the current scale (i.e., 1 / 4 scale), similarity maps corresponding to them under different candidate disparities are calculated through group correlation. This process iterates through all candidate disparities, stacking the similarity maps corresponding to each candidate disparity sequentially to form an initial matching body. Specifically, when the candidate disparity is d, the similarity map between the first and second feature maps can be represented as:

[0042]

[0043] Among them, C init (d, x, y) represents the similarity map between the first feature map and the second feature map when the candidate disparity is d; G represents the total number of groups after dividing all channels in the feature dimension into channels. For example, let the total number of channels in the feature dimension be C, and divide all channels into G groups on average. At this time, each group contains C / G channels; g represents the g-th channel group. This represents the feature submap formed by the channels contained in the g-th channel group in the first feature map; The second feature map is represented by the feature submap formed by the channels contained in the g-th channel group; angle brackets indicate inner product operations or other correlation operations.

[0044] In this embodiment, the range of candidate disparity values ​​is determined by the binocular camera calibration parameters and the target working distance range. For the stereo-corrected binocular camera, let the camera focal length be f, the baseline length be B, and the scene's allowable closest depth be Z. minThen the maximum value of the candidate disparity at full resolution can be set to fB / Z. minx If the upper bound of the candidate disparity obtained at this time is not an integer, it is rounded up. Since the initial matching volume in this embodiment is constructed on the 1 / 4 scale feature map of the original view, the upper bound D of the candidate disparity needs to be adjusted at this scale. max Set to 1 / 4 of the maximum candidate disparity at the full resolution; in addition, when the left view is used as the reference view, for the reference pixel (x, y) near the left boundary in the first feature map, it should be ensured that the horizontal coordinate of the corresponding pixel in the second feature map is greater than or equal to 0, that is, x﹣d≥0, and the out-of-bounds disparity candidates are filtered by a mask.

[0045] It should be noted that if the working distance is not specified in advance, the upper bound of the candidate disparity can be determined by the maximum effective disparity of the training set, but this value is only used as an implementation parameter, not a fixed global parameter applicable to all scenarios.

[0046] The initial matching body is regularized by the initial matching body regularization module. In this embodiment, the initial matching body is input into the 3D hourglass regularization network for spatial context information aggregation and regularization to calculate the disparity probability distribution and obtain the regularized matching body.

[0047] The initial disparity map acquisition module obtains the initial disparity map based on the regularized matching body. In this embodiment, the regularized matching body is transformed into a disparity probability distribution map through a Softmax operation. At this time, the same reference pixel corresponds to a probability value for each candidate disparity. Based on the obtained disparity probability distribution map, the discrete disparity index is expected and weighted by a differentiable soft-argmin form of expectation regression to obtain the initial disparity map. At this time, each reference pixel in the first feature map corresponds to a disparity value in the initial disparity map. Taking the reference pixel (x, y) as an example, its disparity value in the initial disparity map can be expressed as:

[0048]

[0049] Among them, D init (x, y) represents the disparity value of the reference pixel in the initial disparity map; P init (d|x, y) represents the probability value of the reference pixel when the candidate disparity is d; D max This represents the maximum candidate disparity.

[0050] This expectation regression method can circumvent the non-differentiability defect of the argmin operator (i.e., obtaining the disparity index corresponding to the minimum matching cost), thereby outputting a continuously differentiable disparity estimate with sub-pixel precision, ensuring that the gradient can be backpropagated normally, thus realizing end-to-end training of the network.

[0051] Step S110: Input the initial first feature map, the second feature map, and the initial disparity map into the transpolar occlusion perception iterative refinement network, perform multiple rounds of iterative refinement on the initial first feature map and the initial disparity map to obtain the refined disparity map, the refined feature map, and the refined occlusion mask; obtain the uncertainty estimation map based on the refined feature map.

[0052] In the cross-polar occlusion perception iterative refinement stage, it can be divided into an iterative refinement module and an uncertainty estimation module. The iterative refinement module is used to perform multiple rounds of iterative refinement on the initial first feature map and the initial disparity map to obtain a refined disparity map, a refined feature map, and a refined occlusion mask. The uncertainty estimation module is used to obtain an uncertainty estimation map based on the refined feature map to determine the confidence level of the final disparity value obtained by each reference pixel.

[0053] The cross-epipolar occlusion perception iterative refinement network CEO-Former used in this embodiment includes a backbone network and multi-branch task heads, which include residual prediction task heads, offset prediction task heads, weight prediction task heads, and uncertainty estimation task heads. Iterative cross-attention is performed along the epipolar direction (the horizontal direction after binocular correction) to narrow the search scope from global to the epipolar window [x﹣D]. max This significantly reduces computational complexity.

[0054] The initial first feature map and initial disparity map are iteratively refined through an iterative refinement module. In each iteration, cross-epipolar attention calculation and first feature map update are performed through the backbone network. The disparity values ​​corresponding to occluded and unoccluded pixels are differentially refined through residual prediction task heads, offset prediction task heads, and weight prediction task heads. After the iterative refinement is completed, in the uncertainty estimation module, the uncertainty estimation task head obtains the disparity uncertainty estimate value, which can represent the confidence of the refinement result corresponding to each reference pixel, based on the first feature map (i.e., the refined feature map) obtained in the last iteration. This estimate value participates in end-to-end training as part of the total loss function during the training phase, and is used to participate in the calculation of propagation weights in the subsequent depth propagation process during the inference phase.

[0055] Specifically, in the iterative refinement module, the iterative refinement process in each iteration round is as follows: Figure 2 As shown, for the first iteration, its first feature map "after the update of the previous iteration" and disparity map are the initial first feature map H. 0 and the initial disparity map D 0 The iteration process in the t-th iteration includes:

[0056] Step S111: Based on the second feature map and the first feature map updated in the previous iteration, obtain the cross epipolar attention distribution map corresponding to different candidate disparities in the current iteration.

[0057] For any reference pixel in the first feature map after the previous iteration: obtain the candidate pixels in the second feature map corresponding to the reference pixel under different candidate disparities, with each candidate disparity corresponding to one candidate pixel; obtain the query vector based on the feature vector of the reference pixel in the first feature map after the previous iteration through learnable linear projection.

[0058] For any candidate disparity: the key vector corresponding to the candidate pixel is obtained from the feature vector corresponding to the candidate pixel in the second feature map. The value vector corresponding to the candidate pixel can also be obtained from the feature vector corresponding to the candidate pixel in the second feature map. Among them, for candidate pixels whose horizontal or vertical coordinates are not integers, the feature vector of the candidate pixel can be determined by bilinear interpolation.

[0059] To enhance position awareness, this embodiment further injects relative position encoding into the key vector, encoding the disparity offset between candidate pixels and reference pixels as positional information. This allows the model to perceive the magnitude of the disparity offset between candidate pixels and reference pixels when calculating attention weights, making the attention weights explicitly dependent on the disparity offset and thus accelerating matching convergence. Specifically, relative position encoding is performed along the epipolar direction of the candidate disparity to obtain the relative position encoding result. Taking candidate disparity d as an example, the relative position encoding in the extreme direction in this embodiment can be expressed as:

[0060]

[0061]

[0062] Where d represents the candidate disparity, dmodel represents the Transformer embedding dimension, and its value is even. Every two dimensions are grouped together. 2j and 2j+1 are the two position encoding dimension indices in the j-th group. The even-numbered dimensions correspond to the sine operation result, and the odd-numbered dimensions correspond to the cosine operation result.

[0063] In some embodiments, the positional information can be converted into a scalar bias that depends on the relative offset without changing the key vector, and then directly added to the calculation of attention weights.

[0064] Based on the relative position encoding result and the feature vector of the candidate pixel corresponding to the candidate disparity in the second feature map, the key vector corresponding to the candidate pixel is obtained.

[0065] For example, the relative position encoding result is first linearly projected to obtain a first projection result; then, the feature vector corresponding to the candidate pixel in the second feature map is linearly projected to obtain a second projection result; the key vector corresponding to the candidate pixel is obtained based on the first and second projection results. For example, for a reference pixel in the first feature map, the query vector of the reference pixel and the key vector and value vector of the corresponding candidate pixel when the candidate disparity is d can be represented as:

[0066] Q=W Q H1(x, y); K d =W K H2(x﹣d,y)+W R PE(d); V d =W V H2(x﹣d,y)

[0067] Where Q represents the query vector corresponding to the reference pixel (x, y) in the current iteration; H1(x, y) represents the feature vector corresponding to the reference pixel in the first feature map after the update in the previous iteration; H2(x-d, y) represents the feature vector corresponding to the reference pixel in the candidate pixel (x-d, y) with candidate disparity d; PE(d) represents the relative position encoding result of the candidate disparity d; W Q W K W V and W R Both are learnable projection matrices; therefore, W R PE(d) is the first projection result when the candidate disparity is d; W K H2(x﹣d,y) is the second projection result when the candidate disparity is d; K d V represents the key vector of the candidate pixel corresponding to the candidate disparity d; d This represents the value vector of the candidate pixel corresponding to the candidate disparity d.

[0068] Then, based on the query vector of the reference pixel and the key vector corresponding to the candidate pixel, the cross-epipolar attention weight corresponding to the reference pixel under the candidate disparity is obtained; based on the cross-epipolar attention weights corresponding to all reference pixels under the candidate disparity, the cross-epipolar attention distribution map corresponding to the candidate disparity is obtained, and so on, so that each reference pixel corresponds to a cross-epipolar attention weight for each candidate disparity, and thus the cross-epipolar attention distribution map corresponding to each candidate disparity is obtained.

[0069] Taking candidate disparity d as an example, in the t-th iteration, the cross-epipolar attention weight A corresponding to the reference pixel (x, y) when the candidate disparity is d. t (x, y, d) can be represented as:

[0070]

[0071] Where Q represents the query vector corresponding to the reference pixel (x, y); K d This represents the key vector of the candidate pixel corresponding to the reference pixel when the candidate disparity is d; d k The vector dimension of the key vector.

[0072] The weights are then rearranged according to the magnitude of the corresponding candidate disparity to obtain the cross-epipolar attention distribution of the reference pixel in the disparity dimension. The obtained cross-epipolar attention weights are not only used for subsequent feature aggregation, but also to characterize the matching confidence. The peak position along the disparity dimension is the matching disparity, and the peak height reflects the matching confidence. When occlusion occurs, since the real matching point is not within the epipolar window, the response of the cross-epipolar attention weights is low in the entire disparity dimension. This embodiment realizes occlusion perception based on this characteristic.

[0073] Step S112: Obtain the aggregated feature map corresponding to the current iteration round based on the cross epipolar attention distribution map corresponding to different candidate disparities.

[0074] For any reference pixel, under any candidate disparity: obtain the candidate pixel corresponding to the reference pixel in the candidate disparity; obtain the value vector corresponding to the candidate pixel based on the feature vector corresponding to the candidate pixel in the second feature map; obtain the cross-epipolar attention weight corresponding to the reference pixel in the cross-epipolar attention distribution map corresponding to the candidate disparity; obtain the aggregated feature vector corresponding to the reference pixel based on the value vector of the candidate pixel corresponding to each candidate disparity and the cross-epipolar attention weight; obtain the aggregated feature map corresponding to the current iteration based on the aggregated feature vectors of all reference pixels.

[0075] Here, let V be the value vector of the reference pixel (x, y) corresponding to the candidate pixel when the candidate disparity is d. d The attention weight of the reference pixel corresponding to the cross epipolar line in the candidate disparity is A. t (x, y, d); then the aggregated feature vector corresponding to the reference pixel (x, y) can be expressed as:

[0076]

[0077] Among them, C t (x, y) represents the aggregated feature vector corresponding to the reference pixel (x, y) in the current t-th iteration.

[0078] Step S113: Based on the aggregated feature map, the first feature map updated in the previous iteration, and the disparity map, obtain the first feature map updated in the current iteration.

[0079] The aggregated feature map, along with the first feature map updated in the previous iteration and the disparity map, are input into the learnable feature update module within the CEO-Former backbone network for forward computation, thereby obtaining the first feature map updated in the current iteration. Specifically:

[0080] U t =LN(H t﹣1 +W c C t +W d Enc D (D t﹣1 ))

[0081] H t =LN(U t +FFN(U t ))

[0082] Among them, W c W d H represents the learnable projection matrix; t﹣1 D represents the first feature map after the update in the previous iteration (the t-1th iteration); t﹣1 This represents the disparity map updated in the previous iteration; Enc D LN represents disparity coding; LN represents layer normalization operation; FFN represents a two-layer feedforward network; H t U represents the first feature map after the current t-th iteration; t This represents the intermediate result of the learnable feature update module in the current t-th iteration.

[0083] Step S114: Obtain the occlusion mask corresponding to the current iteration round based on the cross epipolar attention distribution map corresponding to different candidate disparities.

[0084] This embodiment further obtains the occlusion state variables of each reference pixel through an occlusion inference mechanism. Specifically, it obtains the maximum value among all cross epipolar attention weights corresponding to the current reference pixel, and obtains the matching missing intensity of the reference pixel based on the maximum value. This is used to measure the degree of "no reliable matching pixel" in the first feature map and the second feature map. The smaller the maximum cross epipolar attention weight, the greater the corresponding matching missing intensity, indicating that there is no strong match within the epipolar window at this time, and the higher the probability that the reference pixel is in the occlusion area. For example, the difference between 1.0 and the maximum cross epipolar attention weight can be calculated, and the obtained difference can be used as the matching missing intensity corresponding to the reference pixel.

[0085] The occlusion state variable of the reference pixel is obtained based on the obtained missing match intensity to characterize whether the reference pixel is in an occluded state. Specifically, if the missing match intensity of the reference pixel is greater than or equal to a preset intensity threshold, the reference pixel is considered to be in an occluded region, and the occlusion state variable of the reference pixel is set to 1; otherwise, the occlusion state variable of the reference pixel is set to 0. For example, the preset intensity threshold is usually set to 0.5, and its specific value can be calibrated using a validation set during training.

[0086] Based on the occlusion state variables of all reference pixels, the occlusion mask corresponding to the current iteration is obtained. At this time, the occlusion state variable of each reference pixel can be directly used as the occlusion mask value corresponding to that reference pixel.

[0087] Step S115: Determine the refinement method of each reference pixel based on the occlusion mask corresponding to the current iteration round, so as to obtain the disparity map updated in the current iteration round.

[0088] In this embodiment, different iterative refinement strategies are used for occluded and unoccluded reference pixels. Specifically, the disparity correction value of unoccluded reference pixels is predicted through residual learning, and disparity information is aggregated from the adaptive neighborhood of adjacent unoccluded pixels through deformable attention to obtain the aggregated disparity value corresponding to the occluded reference pixels. This conforms to the geometric prior that the depth of the occluded region is related to the adjacent unoccluded region. Finally, the disparity map updated in the current iteration is obtained based on the corrected disparity value or aggregated disparity value of each reference pixel.

[0089] Specifically, based on the mask value corresponding to each reference pixel in the occlusion mask corresponding to the current iteration round, it is determined whether each reference pixel is occluded, so as to determine the refinement method of each reference pixel.

[0090] For any reference pixel: if the reference pixel is not occluded, residual refinement is performed on the reference pixel to obtain the corrected disparity value of the reference pixel; in this embodiment, the first feature map updated in the current iteration is input into the residual prediction head, and convolution operation or two-layer MLP (multilayer perceptron) mapping is performed on the feature vectors corresponding to each reference pixel to predict the disparity correction amount corresponding to each reference pixel. The disparity correction amount corresponding to each reference pixel and the disparity value corresponding to it in the disparity map updated in the previous iteration are added together to obtain the disparity value of the reference pixel after the current iteration.

[0091] If the reference pixel is occluded, support aggregation refinement is performed on the reference pixel to obtain the aggregated disparity value corresponding to the reference pixel. Specifically, all unoccluded reference pixels (i.e., non-occluded pixels) are obtained based on the occlusion mask in the current iteration round to obtain the non-occluded pixel set. Then, through deformable attention, multiple non-occluded pixels adjacent to each occluded reference pixel are selected from the non-occluded pixel set, and disparity aggregation is performed based on the updated disparity values ​​corresponding to these non-occluded pixels to obtain the disparity value (i.e., aggregated disparity value) corresponding to the occluded reference pixel.

[0092] Specifically, based on the aggregated feature map corresponding to the current iteration, the aggregated feature vector corresponding to the reference pixel is obtained; offset prediction and weight prediction are performed based on the aggregated feature vector corresponding to the reference pixel to obtain K offsets and K weight values; in this embodiment, two parallel lightweight sub-networks are used as the offset prediction task head and the weight prediction task head, respectively. The former is used to predict a preset number (i.e., K) of two-dimensional sampling offsets corresponding to the reference pixel, and the latter is used to output a preset number of weight logarithmic values. After Softmax operation or normalization operation with validity mask, a preset number of aggregated weight values ​​corresponding to the reference pixel are obtained, and the sum of the preset number of aggregated weight values ​​corresponding to a single reference pixel is 1; for example, K can be set to 8.

[0093] For example, for an occluded reference pixel p, its corresponding aggregated feature vector is C. p The offset prediction task head and weight prediction task head used are both two-layer multilayer perceptrons (MLPs). The aggregated feature vector of the reference pixel is input into these two task heads, and the outputs a preset number of offsets and weight values, i.e.:

[0094] {Δq h}=Φ offset (C p =W2·ReLU(W1·C p +b1)+b2

[0095] {a h}=softmax(Φ weight (C p ))

[0096] Where, △q h This represents the h-th offset; a h Φ represents the h-th weight value; offset Indicates the offset prediction task header; Φ weight W1 and W2 represent the learnable weight matrices of the offset prediction task head, and b1 and b2 represent the bias terms of the offset prediction task head.

[0097] It should be noted that during training, the eight positions in the 3×3 neighborhood of a reference pixel, excluding the center point, can be used as offsets for initialization. After training, during the actual inference process, the K offsets output by the offset prediction head can adaptively move within the maximum search radius.

[0098] Based on the coordinates of the reference pixel and each offset, K support points corresponding to the reference pixel are determined, and each support point corresponds to a weight value. In this embodiment, the coordinates of the reference pixel are added to each offset to obtain the corresponding support point, and so on. Subsequently, weighted fusion is performed based on the disparity value of each support point and the corresponding weight value.

[0099] It should be noted that if the obtained support point coordinates are not integer coordinates, the disparity value corresponding to the support point is calculated based on bilinear interpolation with mask constraints. Specifically, for any support point: obtain a preset number of first neighboring pixels corresponding to the support point. For example, integer neighbors within the four neighbors of the support point can be obtained, and these integer neighbors are used as the first neighboring pixels of the support point. Since the support point is determined from the set of unoccluded pixels, the support point itself is an unoccluded reference pixel (i.e., an unoccluded pixel). Then, based on the mask value corresponding to each first neighboring pixel and its disparity value in the disparity map updated in the previous iteration, the disparity value corresponding to the support point is obtained. Therefore, for the h-th support point corresponding to the current reference pixel, its support disparity can be expressed as:

[0100]

[0101] Where, q h This represents the h-th support point corresponding to the occluded reference pixel. N4(q) represents the support disparity corresponding to the support point in the current t-th iteration; h ) represents the four-neighborhood of the support point; r represents the r-th integer neighbor of the support point in the four-neighborhood; This represents the occlusion mask value corresponding to the r-th integer neighbor; through Occluded neighboring points are excluded, thus setting the validity weight of sampling points that fall outside the image range or have no valid non-occluded neighboring points to zero; b r (q h ) represents the bilinear interpolation coefficient of the r-th integer neighboring point to the support point; ε represents the disparity value of the r-th integer neighbor after the update in the current t-th iteration; s To prevent positive numbers with a denominator of zero.

[0102] Based on the disparity value and weight value of each support point corresponding to the reference pixel, the aggregated disparity value of the reference pixel is obtained. Then, for the occluded reference pixel p, its corresponding aggregated disparity value can be expressed as:

[0103]

[0104] in, This represents the aggregated disparity value corresponding to the occluded reference pixel p in the current t-th iteration; a h This represents the h-th weight value.

[0105] By analogy, the aggregated disparity value corresponding to each occluded reference pixel is obtained; based on the disparity value corresponding to the unoccluded reference pixel based on the disparity correction amount and the aggregated disparity value of the occluded reference pixel, the disparity map updated in the current iteration is obtained.

[0106] Step S116: Update the next iteration based on the second feature map and the first feature map and disparity map updated in the current iteration round until the preset iteration round is reached, and obtain the refined disparity map, refined feature map and refined occlusion mask.

[0107] If the current iteration is not the last iteration in the preset iteration, then continue to update the next iteration based on the second feature map and the first feature map and disparity map updated in the current iteration; if the current iteration is the last iteration, end the iterative refinement process, and use the first feature map and disparity map updated in the current iteration as the refined feature map and refined disparity map; and use the occlusion mask updated in the current iteration as the refined occlusion mask.

[0108] In the uncertainty estimation module, uncertainty estimation is performed based on the refined feature map through the uncertainty estimation task head to obtain the uncertainty estimation map.

[0109] In this embodiment, the uncertainty estimation task head is a small decoder consisting of two convolutional layers. For a reference pixel (x, y), its corresponding uncertainty estimate can be expressed as:

[0110] σ(x, y) = softplus(Conv) 1×1 (ReLU(Conv3×3(f(x,y))))+ε)

[0111] Where σ(x, y) represents the uncertainty estimate corresponding to the reference pixel (x, y); f(x, y) represents the refined feature vector corresponding to the reference pixel (x, y) in the refined feature map; Conv 1×1 This represents the convolution operation; softplus is the activation function used to ensure that the uncertain estimate is positive; ε=10-6 To prevent numerical instability.

[0112] This embodiment conveys the confidence level of the cross-polar occlusion perception iterative refinement network CEO-Former for the refinement result corresponding to each reference pixel through an uncertainty estimation map. The uncertainty estimation value is higher for reference pixels located at the occlusion boundary and in texture-poor regions, while the uncertainty estimation value is lower for reference pixels located in texture-rich regions with matching confidence.

[0113] Step S120: Input the refined disparity map, refined occlusion mask, uncertainty estimation map and reference view into the edge-aware depth completion network to guide the occluded area to perform depth completion based on the edge information in the reference view, and obtain a complete depth map.

[0114] In this embodiment, edge information from the reference view is introduced during the edge-aware depth completion stage to model depth completion as edge-guided anisotropic diffusion, ensuring that the completion does not cross object boundaries. The edge-aware depth completion network (or edge-aware depth propagation network EADP-Net) in this embodiment includes a data processing module and a complete depth map acquisition module. The data processing module processes the reference view and the refined disparity map, refined occlusion mask, and uncertainty estimation map output during the cross-epipolar occlusion perception iterative refinement stage. The processed data is then input into the complete depth map acquisition module for recursive propagation, ultimately obtaining the complete depth map.

[0115] It should be noted that since the refined disparity map, refined occlusion mask, and uncertainty estimation map are all obtained based on the first and second feature maps at 1 / 4 resolution, these data need to be restored to their original resolution before subsequent processing. That is, the pixels in this stage refer to the pixels in the original resolution coordinate system. For the refined disparity map, it is necessary to first upsample it to restore it to the original resolution, and then multiply the disparity values ​​by the corresponding factor according to the scale change. That is, after multiplying each disparity value in the refined disparity map by 4, it is converted into a refined depth map.

[0116] Specifically, in the data processing module, a refined depth map is obtained based on the refined disparity map. At this time, each pixel in the reference view corresponds to a refined depth value in the refined depth map.

[0117] This embodiment restores the refined disparity map from the last iteration to its original resolution, and then uses Z=fB / max(D) to achieve the same result. T , εd) is converted into a refined depth map, where ε d It is the lower limit protection parameter for parallax, satisfying ε d >0 is used to prevent the presence of zero or excessively small disparity values ​​in the refined depth map, which could lead to division by zero or abnormally large depth values.

[0118] Edge information is extracted from the reference view to obtain an edge response map. For example, edge information in the reference view is extracted through the edge extraction branch of an edge-aware depth completion network (a lightweight edge detection network, such as HED-lite) to obtain an edge response map. In this edge response map, the numerical range of the response value corresponding to each pixel is [0, 1]. The higher the response value, the stronger the edge signal at the corresponding position, the higher the confidence that it belongs to an edge, and the greater the probability that it is a strong edge.

[0119] The effective depth mask is obtained by refining the occlusion mask. For example, for the refined occlusion mask restored to the original resolution, the effective depth mask is obtained by subtracting the value 1 from the occlusion mask value corresponding to each pixel. Since the mask value corresponding to the occluded position in the refined occlusion mask is 1 and the mask value corresponding to the unoccluded position is 0, the effective depth obtained after subtraction has a mask value of 0 corresponding to the occluded position and a mask value of 1 corresponding to the unoccluded position. Therefore, it can be seen that the unoccluded pixels in the effective depth mask are the effective depth pixels.

[0120] In the complete depth map acquisition module, anisotropic recursive propagation is performed for multiple propagation steps based on the effective depth mask, refined occlusion mask, uncertainty estimation map, edge response map, and refined depth map to update the depth of each pixel in the reference view until the preset number of propagation steps is reached to obtain the complete depth map.

[0121] For example, the total number of propagation steps S is determined by an adaptive stopping criterion: during propagation, propagation can be stopped early based on a convergence threshold, for example, when the depth change of all occluded pixels is below the convergence threshold τ. conv =10 ﹣4 Early termination is possible. In practical applications, 16 to 32 steps are usually sufficient for convergence. In extreme occlusion scenarios (occlusion area > 30%), S can be automatically extended to 48 steps to ensure sufficient completion.

[0122] Taking the propagation process in step s as an example, in the propagation process in step s, the occluded pixels in the reference view are determined according to the refined occlusion mask, and a preset number of second neighboring pixels corresponding to the occluded pixels are obtained. In this embodiment, the pixels within the eight neighborhoods of the occluded pixel are called the second neighboring pixels of the occluded pixel. For any second neighboring pixel: the propagation weight corresponding to the second neighboring pixel is calculated according to the coordinates of the occluded pixel, the coordinates of the second neighboring pixels, the edge response map, and the uncertainty estimation map. In this embodiment, the propagation weight is jointly determined based on the edge intensity difference, uncertainty difference, and spatial distance between the occluded pixel and its second neighboring pixels. In the normal direction of the object boundary, the edge response change is usually large, and the propagation weight is small, thereby suppressing the depth from crossing the boundary. In the direction along the object surface or along the boundary tangent, the local features are more consistent, and the corresponding propagation weight is larger. Therefore, for the i-th occluded pixel, the propagation weight of its corresponding e-th second neighboring pixel can be expressed as:

[0123]

[0124]

[0125] Among them, w ie w' represents the propagation weight between the i-th occluded pixel and its corresponding e-th second neighbor pixel; ie E represents the unnormalized propagation weight between the i-th occluded pixel and its corresponding e-th second neighbor pixel; α, β, and γ are hyperparameters controlling the intensity of edge difference, uncertainty difference, and spatial distance influence, and their values ​​can be set by the user; i E represents the response value of the i-th occluded pixel in the edge response map; ie E represents the response value of the e-th second neighboring pixel corresponding to the i-th occluded pixel in the edge response map; i With E ie The absolute value of the difference between the two positions can be regarded as the first-order discrete difference of the edge response map along the adjacent direction. The larger the difference, the more likely the two positions are to pass through an edge response change region, and the smaller the corresponding propagation weight; σ i σ represents the value (i.e., the uncertainty estimate) of the i-th occluded pixel in the uncertainty estimation map; ie P represents the uncertainty estimate of the e-th second neighboring pixel corresponding to the i-th occluded pixel; i P represents the coordinates of the i-th occluded pixel in the reference view; ie ε represents the coordinates of the e-th second neighboring pixel corresponding to the i-th occluded pixel in the reference view; wTo prevent positive numbers with zero denominators; n represents the total number of pixels in the second neighboring region corresponding to the occluded pixel.

[0126] Then, based on the propagation weights corresponding to all second-neighbor pixels and the depth update value of the second-neighbor pixel after propagation in step s-1, the depth update value of the occluded pixel after propagation in the current step s is obtained; while the depth values ​​of the unoccluded pixels (i.e., the effective depth pixels) remain frozen, that is, only the occluded pixels participate in the depth propagation update, ensuring that the existing accurate depth is not affected by propagation. The depth update in the propagation process in step s can be expressed as:

[0127]

[0128] Among them, Z s (i) represents the depth update value of the i-th pixel after the current propagation step s; M valid (i) represents the mask value corresponding to the i-th pixel in the effective depth mask, M occ (i) represents the mask value corresponding to the i-th pixel in the refined occlusion mask at the original resolution; Z s-1 (i, e) represents the depth update value of the e-th second neighboring pixel corresponding to the i-th pixel after propagation in step s-1.

[0129] The recursive propagation process described above can be viewed as a discretized iterative solution of the anisotropic diffusion partial differential equation (PDE). Using the unobstructed effective depth as the anchor point, the reliable depth remains constant, and the diffusion term drives the obstructed region to converge towards the nearest reliable depth. The diffusion time τ corresponds to the total number of propagation steps S, and the step size Δτ is constrained by the CFL condition to ensure stability. The corresponding continuous PDE form is:

[0130]

[0131]

[0132] Where, ▽Z represents the first-order spatial gradient of the depth field in the x and y directions, approximated by the difference between adjacent pixels in the discrete image; ▽ represents the Hamiltonian operator; κ(E, σ) is the anisotropic diffusion coefficient, which allows different propagation intensities at different adjacent directions and spatial locations, determined by the edge strength and uncertainty gradient, and approaches 0 at strong edges to suppress cross-boundary propagation; ▽E represents the first-order spatial gradient of the edge response map in the x and y directions; ▽σ represents the first-order spatial gradient of the uncertainty estimation map in the x and y directions; div(κ(E, σ)·▽Z) represents the anisotropic diffusion term; λ anchor ·M valid ·(Z refined -Z) represents the source term; Z represents the latest depth map, Z refinedRepresents a refined depth map; M valid This represents the effective depth mask; ω1 and ω2 are preset weights.

[0133] It should be noted that during the training of the initial disparity estimation network, the transpolar occlusion perception iterative refinement network, and the edge perception depth completion network, the shared weights of the twin feature pyramid network and the 3D hourglass regularization network in the initial disparity estimation network can use pre-trained weight parameters; and the edge extraction branch in the edge perception depth completion network is initialized with pre-trained weights, and then the transpolar occlusion perception iterative refinement network (CEO-Former) and its multi-branch task head are concatenated with the edge perception depth completion network (EADP-Net) for end-to-end joint optimization with disparity supervision, final depth supervision, and various regularization terms.

[0134] In the pre-training of the initial disparity estimation network, the twin feature pyramid network with shared weights can be initialized using existing weights, but it needs to be combined with the 3D hourglass regularization network and the initial disparity obtained through expectation regression. Figure 1 This allows for optimization, enabling the extraction of multi-scale feature maps more suitable for binocular matching from dual-view images during the inference phase. Furthermore, after the initial disparity estimation network completes pre-training, its network parameters can be fixed before retraining other network structures. It can also continue to update with a smaller learning rate when jointly fine-tuning with the transpolar occlusion perception iterative refinement network and the edge perception depth completion network.

[0135] During training, the total loss function used includes the initial disparity supervision loss L. init shading loss L occ and / or occlusion continuity regularization term L occ_reg Iterative parallax supervision loss L disp Diversity regularization term L div Uncertainty loss L unc and deep supervision loss L depth The total loss function can then be expressed as:

[0136] L total =λ init L init +λ dispt L disp +λ occ L occ +λ occ_reg L occ_reg +λ div L div +λ unc L unc +λ depth L depth

[0137] Where, λ init , λ dispt , λ occ , λ occ_reg , λ div , λ unc , λ depth These are the weights corresponding to each loss function, and their values ​​can be adjusted according to actual needs.

[0138] Specifically, since the initial disparity map, the refined occlusion mask, the disparity map updated in each iteration of the iterative refinement process, and the refined disparity map are all generated based on the first feature map and the second feature map at 1 / 4 resolution, when calculating the corresponding loss function, it is necessary to first restore them to the original resolution through upsampling. For the initial disparity map, the disparity map in the iterative refinement process, and the refined disparity map, the disparity value also needs to be multiplied by 4, and then the corresponding loss function is calculated with the corresponding true value (i.e., true disparity / true occlusion mask).

[0139] Among them, the initial disparity supervision loss is used to supervise the error between the initial disparity map output by the initial disparity estimation network and the real disparity map; the occlusion loss is used to characterize the error between the refined occlusion mask and the real occlusion mask; and the depth supervision loss is used to supervise the error between the complete depth map output by the edge-aware depth completion network and the real depth map. For example, the mean square error (MSE) can be used to calculate the difference between the two images to obtain the corresponding loss function.

[0140] The occlusion continuity regularization term is used to constrain the spatial continuity of the occlusion probability between a reference pixel and its neighboring pixels. After obtaining the occlusion state variables (or occlusion mask values) of each reference pixel, to further smooth the occlusion mask and eliminate isolated noise, this embodiment applies occlusion continuity regularization inside the CEO-Former. It constrains the spatial continuity of the occlusion probability based on the differences between the occlusion state variable values ​​of each reference pixel and its neighboring pixels. This regularization term encourages the occlusion mask to be a piecewise constant in space, which aligns with the characteristic that occlusion regions are usually continuous blocks in real-world scenarios. Its weight λ in the total loss function... occ_reg It can be set to 0.01 to strike a balance between the clarity of the occlusion boundary and the smoothness of the mask; then the occlusion continuity regularization term can be expressed as:

[0141]

[0142] Among them, L occ_reg This represents the occlusion continuity regularization term; Ω represents the pixel domain, |Ω| represents the total number of pixels within the pixel domain, which is also the total number of reference pixels; p represents the reference pixel; q represents the pixel within the four-neighborhood (i.e., N4(p)) of the reference pixel p, which is the adjacent pixel of the reference pixel p; Oocc (p) represents the matching missing intensity of reference pixel p; O occ (q) represents the matching missing strength of adjacent pixel q.

[0143] It should be noted that the occlusion loss is used to supervise whether the occlusion judgment is correct. The true occlusion mask is preferably the occlusion or visibility label provided by the dataset. If there is no directly available occlusion label, it can be generated based on the consistency of the true disparity between the left and right views. The occlusion continuity regularization term is used to constrain the soft occlusion probability of adjacent pixels to maintain spatial continuity. If there is neither occlusion label in the training data nor can occlusion label be generated based on the true disparity between the left and right views, the occlusion loss can be deleted and only the occlusion continuity regularization term can be retained.

[0144] The iterative disparity supervision loss is used to supervise the error between the updated disparity value and the true disparity value in each iteration during multi-round iterative refinement. It can be expressed as:

[0145]

[0146] Where T represents the total number of iterations for refinement; t represents the t-th iteration for refinement; and L t This represents the error calculation result between the disparity map updated in the t-th iteration and the original resolution, multiplied by 4, and the true disparity map; ρ is the basic weight value. Since the larger the iteration t, the more accurate the resulting depth map, and the closer the difference between T and t is to 0, the corresponding weight value ρ... T﹣t The larger the value, the greater the reference weight is set for more accurate error calculation results.

[0147] It should be noted that in one iteration, for the occluded reference pixel, in the process of calculating the disparity value corresponding to its support point by bilinear interpolation based on mask constraints, the binary occlusion mask value needs to be replaced with a differentiable soft occlusion probability, that is, the binary occlusion mask value is replaced with the matching missing strength.

[0148] A diversity regularization term is used to constrain the spatial consistency of each support point during support aggregation refinement. In this example, to improve the spatial consistency of depth estimation in occluded regions, diversity regularization is applied among the K sampling points (i.e., support points). This regularization term imposes a larger penalty when support points are too close together, encouraging them to cover a wider neighborhood and preventing aggregation failure caused by all support points shrinking to the same location. Furthermore, the predicted offset is also constrained by the maximum offset range to prevent the offset from increasing unbounded. Its corresponding diversity weight λ in the total loss function... div It can be set to 0.005; then the diversity regularization term in this embodiment can be expressed as:

[0149]

[0150] Where K represents the total number of support points, or the total number of predicted offset / weight values; Δp h Δp represents the h-th offset; g τ represents the g-th offset; div It is the distance scale parameter in the diversity regularization term, also known as the temperature parameter, which satisfies τ div The value is >0 to avoid overly concentrated distribution of the generated support points. When the distance between two sampling points is much smaller than the distance scale parameter, the exponent term is close to 1, resulting in a large penalty; when the distance between the two points is greater than the distance scale parameter, the penalty decreases rapidly. Therefore, the distance scale parameter controls "how close two sampling points are considered overly concentrated," and its specific value is determined based on the feature map scale and the validation set, and is fixed as a training hyperparameter.

[0151] The uncertainty loss is used to constrain the accuracy of uncertainty estimation based on the refined disparity map and the true disparity map. In this embodiment, the uncertainty estimate is input into the edge-aware depth completion network to participate in the calculation of propagation weights, and also serves as input to the uncertainty loss (negative log-likelihood loss) in the total loss function, participating in the end-to-end training of the entire network architecture so that the network learns to output well-calibrated uncertainty estimation results. The uncertainty loss in this embodiment can be expressed as:

[0152] L unc =|D T -Z GT | / σ+log(σ)

[0153] Among them, D T This represents the disparity map after refining it to its original resolution and multiplying each disparity value by 4; Z GT σ represents the true disparity map; σ is the uncertainty estimation map; log is the logarithmic function.

[0154] This embodiment limits the attention computation to the candidate disparity range along the epipolar direction and replaces the global self-attention mechanism with a cross-epochal attention mechanism, reducing the computational complexity of attention from O(H²W²) to O(H·W·D²). maxThe computational complexity increases significantly with image resolution, resulting in a substantial reduction in computational cost while maintaining disparity refinement accuracy. By directly extracting the missing matching intensity from the cross-epipolar attention weight distribution to derive the occlusion state variable, joint optimization of occlusion detection and disparity refinement is achieved without requiring an additional independent occlusion prediction branch. A differentiated refinement strategy is employed: residual refinement for non-occluded pixels and deformable neighborhood support aggregation for occluded pixels. This strategy fully leverages the geometric prior of the stronger correlation between occluded pixel depth and spatially neighboring non-occluded pixels, effectively improving disparity refinement accuracy in occluded regions compared to a globally indiscriminate aggregation approach. By modulating the anisotropic propagation weights with pixel-wise uncertainty estimates, the depth propagation process can adaptively avoid high-uncertainty regions. This weight is also used in the total loss function calculation during training, forming a complete uncertainty propagation chain. Simultaneously, edge information in the reference view constrains the propagation direction, ensuring reliable depth recursively spreads along the same object surface to occluded regions without crossing object boundaries. This solves the problem of blurred depth values ​​at object boundaries in existing depth completion methods, maintaining the structural integrity of the depth map. End-to-end training of the three-stage network is performed using a multi-task joint loss function. The initial disparity supervision loss, iterative disparity supervision loss, occlusion continuity regularization term, support point diversity regularization term, uncertainty loss, and depth completion loss are mutually constrained, and each subtask is collaboratively optimized, improving the overall system's training stability and final depth estimation accuracy.

[0155] Please refer to Figure 3 Some embodiments provide an occlusion-aware depth estimation and edge-aware depth completion system, including:

[0156] Data acquisition module 200 is used to acquire the left and right views;

[0157] The initial disparity estimation module 210 is used to input the left view and the right view into the initial disparity estimation network to obtain the second feature map, the initial first feature map and the initial disparity map. The first feature map and the second feature map each correspond to a view. The view corresponding to the first feature map is used as the reference view and the pixel in the first feature map is used as the reference pixel.

[0158] In this embodiment, the initial disparity estimation module 210 includes a feature extraction module 211, an initial matching volume construction module 212, an initial matching volume regularization module 213, and an initial disparity map acquisition module 214. The feature extraction module 211 performs multi-scale feature extraction on the left and right views respectively. The initial matching volume construction module 212 constructs an initial matching volume based on the multi-scale feature maps corresponding to the left and right views. The initial matching volume regularization module 213 performs regularization processing on the initial matching volume to obtain a regularized matching volume. The initial disparity map acquisition module 214 obtains an initial disparity map based on the regularized matching volume.

[0159] The cross-epipolar occlusion perception module 220 is used to input the initial first feature map, second feature map, and initial disparity map into the cross-epipolar occlusion perception iterative refinement network to perform multiple rounds of iterative refinement on the initial first feature map and initial disparity map. In each iteration round: based on the second feature map and the first feature map updated in the previous iteration round, the cross-epipolar attention distribution map corresponding to different candidate disparities in the current iteration round is obtained; based on the cross-epipolar attention distribution map corresponding to different candidate disparities, the aggregated feature map corresponding to the current iteration round is obtained; based on the aggregated feature map and the first feature map updated in the previous iteration round... The algorithm generates a disparity map and obtains the first feature map updated in the current iteration. It then obtains the occlusion mask corresponding to the current iteration based on the cross-epipolar attention distribution map corresponding to different candidate disparities. Based on the occlusion mask, it determines the refinement method for each reference pixel to obtain the disparity map updated in the current iteration. The algorithm then updates the next iteration based on the second feature map, the first feature map updated in the current iteration, and the disparity map, until a preset iteration is reached, resulting in a refined disparity map, a refined feature map, and a refined occlusion mask. Finally, it performs uncertainty estimation based on the refined feature map to obtain an uncertainty estimation map.

[0160] In this embodiment, the transpolar occlusion sensing module 220 can be divided into an iterative refinement module 221 and an uncertainty estimation module 222. The iterative refinement module 221 is used to perform multiple rounds of iterative refinement on the initial first feature map and the initial disparity map to obtain a refined disparity map, a refined feature map, and a refined occlusion mask. After the iterative refinement is completed, the uncertainty estimation module 222 obtains an uncertainty estimation map based on the refined feature map to determine the confidence level of the final disparity value obtained by each reference pixel.

[0161] The edge-aware depth completion module 230 is used to input the refined disparity map, refined occlusion mask, uncertainty estimation map and reference view into the edge-aware depth completion network, so as to guide the occluded area to perform depth completion based on the edge information in the reference view, and obtain a complete depth map.

[0162] In this embodiment, the edge-aware depth completion module 230 can be divided into a data processing module 231 and a complete depth map acquisition module 232. In the data processing module 231, the refined disparity map, refined occlusion mask, and uncertainty estimation map output by the reference view and the transpolar occlusion perception iterative refinement module 220 are processed. The processed data is then input into the complete depth map acquisition module 232 for recursive propagation, and finally a complete depth map is obtained.

[0163] It should be noted that each system module in this embodiment corresponds to the method steps in the above-described occlusion perception depth estimation and edge perception depth completion method. The specific implementation methods have been described in the above embodiments and will not be repeated here.

[0164] In the above embodiments, implementation can be achieved, in whole or in part, by software, hardware, firmware, or any combination thereof. Furthermore, as those skilled in the art will understand, the principles herein can be reflected in a computer program product on a computer-readable storage medium pre-loaded with computer-readable program code. Any tangible, non-transitory computer-readable storage medium may be used, including magnetic storage devices (hard disks, floppy disks, etc.), optical storage devices (CDs, DVDs, Blu-ray discs, etc.), flash memory, and / or the like. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to form a machine, such that instructions executing on the computer or other programmable data processing apparatus can generate means for performing a specified function. These computer program instructions can also be stored in a computer-readable storage medium that can instruct the computer or other programmable data processing apparatus to operate in a particular manner, such that instructions stored in the computer-readable storage medium can form an article of manufacture, including means for implementing the specified function. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to perform a series of operational steps on the computer or other programmable apparatus to produce a computer-implemented process, such that instructions executing on the computer or other programmable apparatus can provide steps for implementing the specified function.

[0165] This document describes various exemplary embodiments with reference to them. However, those skilled in the art will recognize that changes and modifications can be made to the exemplary embodiments without departing from the scope of this document. For example, various operational steps and components for performing operational steps can be implemented in different ways depending on the specific application or considering any number of cost functions associated with the operation of the system (e.g., one or more steps can be deleted, modified, or combined with other steps).

[0166] While the principles herein have been illustrated in various embodiments, numerous modifications to the structures, arrangements, proportions, elements, materials, and components, particularly suited to specific environments and operational requirements, may be used without departing from the principles and scope of this disclosure. These modifications and other alterations or alterations will be included within the scope of this document. Those skilled in the art will recognize that many changes can be made to the details of the above embodiments without departing from the fundamental principles of the invention.

Claims

1. A method for occlusion-aware depth estimation and edge-aware depth completion, characterized in that, include: Get the left and right views; The left view and the right view are input into the initial disparity estimation network to obtain the second feature map, the initial first feature map and the initial disparity map. The first feature map and the second feature map each correspond to a view. The view corresponding to the first feature map is used as the reference view and the pixel in the first feature map is used as the reference pixel. The initial first feature map, the second feature map, and the initial disparity map are input into a cross-epipolar occlusion perception iterative refinement network. Multiple iterations of iterative refinement are performed on the initial first feature map and the initial disparity map. In each iteration: based on the second feature map and the first feature map updated in the previous iteration, a cross-epipolar attention distribution map corresponding to different candidate disparities is obtained; based on the cross-epipolar attention distribution map corresponding to different candidate disparities, an aggregated feature map corresponding to the current iteration is obtained; based on the aggregated feature map and the first feature map updated in the previous iteration, the network refines the network. The disparity map is used to obtain the first feature map updated in the current iteration round. An occlusion mask corresponding to the current iteration round is obtained based on the cross-epipolar attention distribution map corresponding to different candidate disparities. The refinement method for each reference pixel is determined based on the occlusion mask corresponding to the current iteration round to obtain the disparity map updated in the current iteration round. The next iteration round is updated based on the second feature map, the first feature map updated in the current iteration round, and the disparity map, until a preset iteration round is reached, resulting in a refined disparity map, a refined feature map, and a refined occlusion mask. Uncertainty estimation is performed based on the refined feature map to obtain an uncertainty estimation map. The refined disparity map, refined occlusion mask, uncertainty estimation map, and reference view are input into the edge-aware depth completion network to guide the occluded region for depth completion based on the edge information in the reference view, thereby obtaining a complete depth map.

2. The method as described in claim 1, characterized in that, The step of obtaining the cross-epipolar attention distribution map corresponding to different candidate disparities in the current iteration based on the second feature map and the first feature map updated in the previous iteration includes: For any reference pixel in the first feature map after the previous iteration: obtain the query vector based on the feature vector of the reference pixel in the first feature map after the previous iteration; obtain the candidate pixels at the corresponding positions of the reference pixel in the second feature map under different candidate disparities, with each candidate disparity corresponding to one candidate pixel; For any candidate disparity: obtain the key vector corresponding to the candidate pixel in the second feature map based on the feature vector corresponding to the candidate pixel of the candidate disparity; obtain the cross epipolar attention weight corresponding to the reference pixel under the candidate disparity based on the query vector of the reference pixel and the key vector corresponding to the candidate pixel. The cross-epipolar attention distribution map corresponding to the candidate disparity is obtained by using the cross-epipolar attention weights of all reference pixels under the candidate disparity.

3. The method as described in claim 2, characterized in that, The step of obtaining the key vector corresponding to the candidate pixel based on the feature vector corresponding to the candidate pixel in the second feature map according to the candidate disparity includes: The relative position encoding result is obtained by performing relative position encoding on the candidate disparity; Based on the relative position encoding result and the feature vector corresponding to the candidate pixel in the second feature map, the key vector corresponding to the candidate pixel is obtained; Specifically, the relative position encoding result is linearly projected to obtain a first projection result; the feature vector corresponding to the candidate pixel in the second feature map is linearly projected to obtain a second projection result; and the key vector corresponding to the candidate pixel is obtained based on the first projection result and the second projection result.

4. The method as described in claim 1, characterized in that, The step of obtaining the aggregated feature map corresponding to the current iteration round based on the cross-epipolar attention distribution map corresponding to different candidate disparities includes: For any reference pixel, under any candidate disparity: obtain the candidate pixel corresponding to the reference pixel in the candidate disparity; obtain the value vector corresponding to the candidate pixel according to the feature vector corresponding to the candidate pixel in the second feature map; obtain the cross-epipolar attention weight corresponding to the reference pixel in the cross-epipolar attention distribution map corresponding to the candidate disparity; Based on the value vector of the reference pixel in each candidate pixel corresponding to the candidate disparity and the cross epipolar attention weight, the aggregated feature vector corresponding to the reference pixel is obtained. The aggregated feature map corresponding to the current iteration is obtained based on the aggregated feature vector of all reference pixels.

5. The method as described in claim 1, characterized in that, The step of determining the refinement method for each reference pixel based on the occlusion mask corresponding to the current iteration round to obtain the disparity map updated in the current iteration round includes: Based on the mask value of each reference pixel in the occlusion mask corresponding to the current iteration round, determine whether each reference pixel is occluded, so as to determine the refinement method of each reference pixel. For any reference pixel: if the reference pixel is not occluded, perform residual refinement on the reference pixel to obtain the corrected disparity value of the reference pixel; if the reference pixel is occluded, perform support aggregation refinement on the reference pixel to obtain the aggregated disparity value of the reference pixel. The disparity map updated in the current iteration is obtained based on the corrected disparity value or aggregated disparity value of each reference pixel.

6. The method as described in claim 5, characterized in that, If the reference pixel is occluded, support aggregation refinement is performed on the reference pixel to obtain the aggregated disparity value corresponding to the reference pixel, including: Based on the aggregated feature map corresponding to the current iteration round, the aggregated feature vector corresponding to the reference pixel is obtained; Based on the aggregated feature vector corresponding to the reference pixel, offset prediction and weight prediction are performed to obtain K offsets and K weight values; Based on the coordinates of the reference pixel and each offset, determine the K support points corresponding to the reference pixel, and each support point corresponds to a weight value; For any support point: obtain the first neighboring pixels of the support point with a preset number of pixels, and obtain the disparity value of the support point based on the mask value of each first neighboring pixel and its disparity value in the disparity map after the previous iteration. The aggregated disparity value of the reference pixel is obtained based on the disparity value and weight value of each support point corresponding to the reference pixel.

7. The method as described in claim 1, characterized in that, The process of inputting the refined disparity map, refined occlusion mask, uncertainty estimation map, and reference view into an edge-aware depth completion network to guide depth completion of the occluded region based on edge information in the reference view, thereby obtaining a complete depth map, includes: A refined depth map is obtained based on the refined disparity map, where each pixel in the reference view corresponds to a refined depth value in the refined depth map. Edge information is extracted from the reference view to obtain an edge response map; The effective depth mask is obtained based on the refined occlusion mask; Anisotropic recursive propagation of multiple propagation steps is performed based on the effective depth mask, refined occlusion mask, uncertainty estimation map, edge response map, and refined depth map to update the depth of each pixel in the reference view until a preset number of propagation steps are reached to obtain a complete depth map. Specifically, during the propagation process in step s, the occluded pixels in the reference view are determined according to the refined occlusion mask, and a preset number of second neighboring pixels corresponding to the occluded pixels are obtained. For any second neighboring pixel: the propagation weight corresponding to the second neighboring pixel is calculated according to the coordinates of the occluded pixel, the coordinates of the second neighboring pixels, the edge response map, and the uncertainty estimation map. The depth update value of the occluded pixel after the current propagation in step s is obtained according to the propagation weights corresponding to all second neighboring pixels and the depth update value of the second neighboring pixels after propagation in step s-1.

8. The method as described in claim 1, characterized in that, The step of inputting the left view and the right view into the initial disparity estimation network to obtain the second feature map, the initial first feature map, and the initial disparity map includes: Multi-scale feature extraction is performed on the left view and the right view respectively to obtain the multi-scale feature map corresponding to the left view and the multi-scale feature map corresponding to the right view. Feature map fusion is performed on the multi-scale feature map corresponding to the left view and the multi-scale feature map corresponding to the right view to obtain a first feature map and a second feature map. An initial matching body is constructed based on the first feature map and the second feature map, and the initial matching body includes a similarity distribution map between the first feature map and the second feature map under different candidate disparities; The initial matching body is regularized to obtain the regularized matching body; An initial disparity map is obtained based on the regularized matching body, and each reference pixel in the first feature map corresponds to a disparity value in the initial disparity map.

9. The method as described in claim 6, characterized in that, During the training of the initial disparity estimation network, the transpolar occlusion perception iterative refinement network, and the edge perception depth completion network, the total loss function used includes the initial disparity supervision loss, iterative disparity supervision loss, diversity regularization term, uncertainty loss, depth supervision loss, occlusion loss, and / or occlusion continuity regularization term. The initial disparity supervision loss is used to monitor the error between the initial disparity map output by the initial disparity estimation network and the true disparity map; the occlusion loss is used to characterize the error between the refined occlusion mask and the true occlusion mask; the occlusion continuity regularization term is used to constrain the occlusion probability between the reference pixel and its neighboring pixels to maintain spatial continuity; the iterative disparity supervision loss is used to monitor the error between the disparity value updated in each iteration and the true disparity value during multi-round iterative refinement; the diversity regularization term is used to constrain the spatial consistency of each support point when performing support aggregation refinement; the uncertainty loss is used to constrain the accuracy of uncertainty estimation based on the refined disparity map and the true disparity map; the depth supervision loss is used to monitor the error between the complete depth map output by the edge-aware depth completion network and the true depth map.

10. An occlusion-aware depth estimation and edge-aware depth completion system, characterized in that, include: The data acquisition module is used to acquire the left and right views; An initial disparity estimation module is used to input the left view and the right view into an initial disparity estimation network to obtain a second feature map, an initial first feature map, and an initial disparity map. The first feature map and the second feature map each correspond to a view. The view corresponding to the first feature map is used as a reference view, and the pixels in the first feature map are used as reference pixels. A cross-polar occlusion perception module is used to input the initial first feature map, the second feature map, and the initial disparity map into a cross-polar occlusion perception iterative refinement network to perform multiple rounds of iterative refinement on the initial first feature map and the initial disparity map. In each iteration round: based on the second feature map and the first feature map updated in the previous iteration round, a cross-polar attention distribution map corresponding to different candidate disparities in the current iteration round is obtained; based on the cross-polar attention distribution map corresponding to different candidate disparities, an aggregated feature map corresponding to the current iteration round is obtained; based on the aggregated feature map and the first feature map updated in the previous iteration round... A first feature map is obtained by combining a feature map and a disparity map, and then updated to the first feature map for the current iteration. An occlusion mask for the current iteration is obtained based on the cross-epipolar attention distribution map corresponding to different candidate disparities. The refinement method for each reference pixel is determined based on the occlusion mask for the current iteration to obtain the disparity map updated for the current iteration. The next iteration is then updated based on the second feature map, the first feature map updated for the current iteration, and the disparity map, until a preset iteration is reached, resulting in a refined disparity map, a refined feature map, and a refined occlusion mask. Uncertainty estimation is performed based on the refined feature map to obtain an uncertainty estimation map. The edge-aware depth completion module is used to input the refined disparity map, refined occlusion mask, uncertainty estimation map, and reference view into the edge-aware depth completion network, so as to guide the occlusion area to perform depth completion based on the edge information in the reference view, and obtain a complete depth map.

Images