Depth estimation method based on multi-step iteration volume probability distribution diffusion model

Abstract

The invention discloses a depth pre-estimation method based on a multi-step iteration volume probability distribution diffusion model, and particularly relates to the technical field of depth pre-estimation. Three-dimensional volume characteristics and two-dimensional context information are obtained through a pre-training network, basic volume guidance is constructed according to the three-dimensional volume characteristics, and the depth pre-estimation of the volume probability distribution diffusion model is realized. Constructing confidence perception context guidance according to the two-dimensional context information, and in the volume probability distribution diffusion model, performing iterative processing on the input noise volume through the basic volume guidance and the confidence perception context guidance to generate a predicted volume; carrying out distribution conversion on a reverse process of generating a predicted volume from a noise volume through a Markov chain, finally weakening disturbance generated in a volume probability distribution diffusion model distribution conversion process through suppression represented by an intermediate state generated in the reverse process, and sampling to generate a depth estimation result. Progressive modeling is carried out on the complex depth volume by utilizing multi-step generation diffusion to obtain final accurate prediction, so that the efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of depth estimation, and in particular to a depth estimation method based on a multi-step iterative volume probability distribution diffusion model. Background Art

[0002] Depth estimation technology is at the core of many 3D vision applications, including autonomous driving, robotic navigation, augmented reality, and industrial manufacturing. Deep convolutional neural networks have recently achieved impressive results in multiple challenging real-world scenarios. However, state-of-the-art network architectures still have difficulty handling prediction ambiguities caused by textureless areas, repeated patterns, thin structures, occlusions, or reflections, which can lead to incorrect disparity predictions. In order to obtain accurate depth information, the most advanced methods tend to use 3D cost volumes to learn the distribution of correspondences on different depth hypothesis planes. Compared with these 2D contextual feature-based methods, 3D volume-based methods effectively exploit stereo geometric constraints through the matching process and achieve impressive results on different benchmark datasets. Therefore, how to achieve accurate depth based on 3D volumes has been the focus of recent work.

[0003] Existing volume-based research mainly focuses on using various complex architectures and loss functions to learn the volume correspondence distribution, treating the estimation of the 3D cost volume as a single-step approximation process. However, simply forcing the estimated distribution to approximate the complex target distribution will bring a huge burden to the learning process, and some complex and challenging cases such as occlusion and reflection cannot be handled thoroughly in one step. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is to propose a depth estimation method based on a multi-step iterative volume probability distribution diffusion model, so as to solve the problem that the depth estimation efficiency in the prior art is low and cannot thoroughly handle challenging cases.

[0005] The technical solution adopted by the present invention to solve the technical problem is a depth estimation method based on a multi-step iterative volume probability distribution diffusion model, which is characterized by comprising the steps of:

[0006] S1. Acquire three-dimensional volume features and two-dimensional context information through a pre-trained network, construct basic volume guidance according to the three-dimensional volume features, and construct confidence-aware context guidance according to the two-dimensional context information;

[0007] S2. In the volume probability distribution diffusion model, the input noise volume is iteratively processed to generate the prediction volume through the basic volume guidance and the confidence-aware context guidance, and the reverse process of the noise volume generating the prediction volume is transformed through the Markov chain;

[0008] S3. The disturbance generated in the distribution conversion process of the volume probability distribution diffusion model is weakened by suppressing the intermediate state representation generated in the reverse process, and the depth estimation result is generated by sampling.

[0009] Furthermore, the predicted volume generated by each iterative process is processed by an online filtering operation, and the processing result is transmitted to the next sampling step. The expression of the online filtering operation is as follows:

[0010] y′ t =Project Uni WTA D (y t ), Dim=D,

[0011] In the formula, y′ t The noise volume is filtered online. Uni For single-mode projection, WTA D It's a winner-takes-all operation, y t is the noise volume, Dim=D is the depth dimension.

[0012] Furthermore, the construction of the basis volume guides the conversion of the cost volume of the pre-trained network into a probabilistic form required by the conditional diffusion process through a softmax function.

[0013] Furthermore, the softmax function converts each pixel in the cost volume of the pre-trained network according to the depth dimension to generate a cost probability volume of the pre-trained network. The conversion process expression is as follows:

[0014]

[0015] In the formula, is the cost probability volume of the pre-trained network, h, w, m are the spatial width, height and depth, is the cost of the pre-trained network, For m th The cost volume of the plane in the depth dimension.

[0016] Furthermore, context features are obtained through the two-dimensional context information, context volume is obtained by projecting the context features to the three-dimensional context volume, and a filter volume is constructed by a confidence map obtained through confidence information on the dimension, and an optimized volume is obtained by an element-wise summation operation based on the context volume and the filter volume.

[0017] Furthermore, the confidence map is constructed by the highest probability values ​​of all depth planes in the depth dimension.

[0018] Furthermore, the highest probability value in the confidence map is used to obtain the query vector of the cross-attention operation through an inversion operation, and the query vector is used to measure the uncertainty of the depth volume matching. The acquisition process expression is as follows:

[0019]

[0020] In the formula, Q i is the query vector, C i is the confidence map, is the input depth volume.

[0021] Furthermore, the position points in the context feature generate a reference vector and a content vector through variable convolution, and the expression is as follows:

[0022]

[0023] In the formula, K i is the reference vector, V i is the content vector, p is the position point in the context feature, C′ is the total number of channels, W is the variability convolution weight, Δ (p) is the learnable offset, is the context feature, Chunk is the separation operation of the context feature on the channel dimension, and Dim=c is the channel dimension.

[0024] Furthermore, the context volume and the filter volume are summed up by elements to obtain the optimized volume, and the expression is as follows:

[0025]

[0026] In the formula, To optimize the volume, is the context volume, ⊙ is the element-wise multiplication symbol.

[0027] Compared with the prior art, the present invention has at least the following beneficial effects:

[0028] (1) A depth estimation method based on a multi-step iterative volume probability distribution diffusion model described in the present invention uses multi-step generative diffusion to progressively model complex depth volumes to obtain a final accurate prediction;

[0029] (2) The present invention has strong generalization ability and high portability. It adopts a plug-and-play network architecture design and can learn generalized features from data sets of multiple scenarios and transfer applications to generate high-quality prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flowchart of a depth estimation method based on a multi-step iterative volume probability distribution diffusion model;

[0031] Figure 2 The framework diagram of a depth estimation method based on a multi-step iterative volume probability distribution diffusion model. DETAILED DESCRIPTION

[0032] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to these embodiments.

[0033] Embodiment 1

[0034] To solve the problems of low efficiency of depth estimation and inability to thoroughly handle challenging cases, such as Figure 1 As shown, the present invention provides a depth estimation method based on a multi-step iterative volume probability distribution diffusion model, characterized in that it includes the steps of:

[0035] S1. Obtain 3D volume features and 2D context information through a pre-trained network, construct basic volume guidance based on the 3D volume features, and construct confidence-aware context guidance based on the 2D context information;

[0036] S2. In the volume probability distribution diffusion model, the input noise volume is iteratively processed to generate the prediction volume through the basic volume guidance and the confidence-aware context guidance, and the reverse process of the noise volume generating the prediction volume is transformed through the Markov chain;

[0037] S3. The disturbance generated in the distribution conversion process of the volume probability distribution diffusion model is weakened by suppressing the intermediate state representation generated in the reverse process, and the depth estimation result is generated by sampling.

[0038] First, the three-dimensional volume features are obtained through the pre-trained network, and the basic volume guidance is constructed according to the three-dimensional volume features. The cost volume of a pre-trained network is given first, and each pixel in the cost volume of the pre-trained network is converted into the probability form required by the conditional diffusion process along the depth dimension through the softmax function, which is the cost probability volume of the pre-trained network. As the basic geometric prior information, it is used for the prior knowledge or assumptions of the shape, structure and geometric properties of the scene or object. The conversion process expression is as follows:

[0039]

[0040] In the formula, is the cost probability volume of the pre-trained network, h, w, m are the spatial width, height and depth, is the cost of the pre-trained network, For m th The cost volume of the plane in the depth dimension.

[0041] Secondly, the two-dimensional context information is obtained through the pre-trained network, and the search is performed in the depth dimension to retrieve the highest probability value in all depth hypothesis planes. The query vector of the cross-attention operation is obtained by inverting the highest probability value retrieved in the confidence map, which is used to measure the uncertainty of the match in the input depth volume. The acquisition process expression is as follows:

[0042]

[0043] In the formula, Q i is the query vector, C i is the confidence map, WTA D It’s a winner-takes-all operation. is the input depth volume.

[0044] The reference vector and content vector are generated in the corresponding context features through variable convolution to achieve efficient set transformation modeling and receptive field adaptability. The position points in the context features are processed as follows:

[0045]

[0046] In the formula, K i is the reference vector, V i is the content vector, p is the position point in the context feature, C′ is the total number of channels, w is the variable convolution weight, Δ (p) is the learnable offset, is the context feature, Chunk is the separation operation of the context feature on the channel dimension, and Dim=c is the channel dimension.

[0047] The confidence-aware context features are obtained through the query vector, reference vector and content vector through a linear attention calculation method, and the expression is as follows:

[0048]

[0049] In the formula, is the confidence-aware context feature, φ q is the softmax operation for each row of the input matrix, φ k The softmax operation is performed on each column of the input matrix. Using this calculation method can reduce the computational cost in depth estimation.

[0050] According to the matching uncertainty of the depth volume, the relevant information of the context features is retrieved, and then the obtained depth volume is multiplied by the confidence map to obtain the filtered volume; in order to match each other in dimension, the confidence-aware context features are projected into the context volume, and then added to the filtered volume by restoring the image depth operation to obtain the context volume; the optimized volume is constructed by element-wise summing of the filtered volume and the context volume, and the expression is as follows:

[0051]

[0052] In the formula, To optimize the volume, is the context volume, C i is the confidence map, and ⊙ is the element-wise multiplication symbol. Confidence-aware context guidance is applied to each down-sampling block of the 3D diffusion model with different dimensional sizes. Through the confidence-aware context guidance of the refinement operation on the depth volume, the volume distribution of high-confidence regions is preserved, while the volume distribution of low-confidence regions is optimized in the multi-scale context.

[0053] Furthermore, in the volume probability distribution diffusion model, the sampling process iteratively processes the input noise volume through basic volume guidance and confidence-aware context guidance to generate a predicted volume, and performs distribution conversion on the reverse process of generating predicted volume from noise volume through Markov chain. The probability distribution conversion formula is as follows:

[0054]

[0055] In the formula, p θ is the Gaussian distribution condition, μ θ is the denoising prediction process of the diffusion network, is the multiplication of predefined coefficients, is the coefficient transformation, I is the unit matrix, and some parameters f θ is the diffusion model, α t is the predefined coefficient, t is the time step in the sampling process, is the standard normal distribution, x is the given guidance condition, including the basic volume guidance and the confidence-aware context guidance, y t is the noise volume.

[0056] The disturbance generated in the distribution conversion process of the volume probability distribution diffusion model is suppressed by the intermediate state representation generated in the reverse process. The predicted noise volume is filtered online at each iteration, and then the processed noise volume is sent to the next sampling step. The expression of the online filtering operation is as follows:

[0057] y′ t=Project Uni WTA D (y t ), Dim=D

[0058] In the formula, y′ t The noise volume is filtered online. Uni is a single-mode projection,

[0059] WTA D It is a winner-takes-all operation, y t is the noise volume, Dim = D is the depth dimension. Figure 2 As shown, through this method, the input image is sampled in the volume probability distribution diffusion model according to the basic volume guidance and confidence-aware context guidance and the depth estimation result is generated. The distribution diffusion model constructs an intermediate target distribution at each step of the Markov chain, thereby splitting the learning of complex empirical distribution into multi-step distribution transformation modeling. The converted multi-step distribution transformation modeling is suitable for volume distribution learning tasks. This method can transform complex cost volume prediction into progressive probability distribution modeling, thereby improving the efficiency of depth estimation, and has strong generalization ability and high transferability. It adopts a plug-and-play network architecture design, which can learn generalized features from data sets of multiple scenes and migrate applications to generate high-quality prediction results.

[0060] In summary, the present invention provides a depth estimation method based on a multi-step iterative volume probability distribution diffusion model, which obtains three-dimensional volume features and two-dimensional context information through a pre-trained network, constructs a basic volume guidance based on the three-dimensional volume features, and constructs a confidence-aware context guidance based on the two-dimensional context information. In the volume probability distribution diffusion model, the input noise volume is iteratively processed by the basic volume guidance and the confidence-aware context guidance to generate a predicted volume, and the reverse process of generating the predicted volume from the noise volume is distributedly transformed through the Markov chain. Finally, the disturbance generated in the distribution transformation process of the volume probability distribution diffusion model is weakened by suppressing the intermediate state representation generated in the reverse process, and the depth estimation result is generated by sampling. This method uses multi-step generation diffusion to progressively model complex depth volumes to obtain the final accurate prediction, and has strong generalization ability and high transferability. It adopts a plug-and-play network architecture design, and can learn generalized features from data sets of multiple scenes and migrate applications to generate high-quality prediction results.

[0061] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0062] In addition, in the present invention, descriptions such as "first", "second", "one", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0063] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0064] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. A depth estimation method based on a multi-step iterative volume probability distribution diffusion model. It is characterized in that Includes steps: S1. Acquire three-dimensional volume features and two-dimensional context information through a pre-trained network, construct basic volume guidance according to the three-dimensional volume features, and construct confidence-aware context guidance according to the two-dimensional context information; S2. In the volume probability distribution diffusion model, the input noise volume is iteratively processed to generate the prediction volume through the basic volume guidance and the confidence-aware context guidance, and the reverse process of the noise volume generating the prediction volume is transformed through the Markov chain; S3. The disturbance generated in the distribution conversion process of the volume probability distribution diffusion model is weakened by suppressing the intermediate state representation generated in the reverse process, and the depth estimation result is generated by sampling.

2. According to claim 1, a depth estimation method based on a multi-step iterative volume probability distribution diffusion model, It is characterized in that The predicted volume generated by each iterative process is processed by an online filtering operation, and the processing result is transmitted to the next sampling step. The online filtering operation expression is as follows: y' t =Project Uni WTA D (yt),Dim=D In the formula, y' t The noise volume is filtered online. Uni For single-mode projection, WTA D It is a winner-takes-all operation, y t is the noise volume, Dim=D is the depth dimension.

3. According to claim 1, a depth estimation method based on a multi-step iterative volume probability distribution diffusion model, It is characterized in that The constructed basis volume guides the conversion of the cost volume of the pre-trained network into the probabilistic form required by the conditional diffusion process through a softmax function.

4. The depth estimation method based on the multi-step iterative volume probability distribution diffusion model according to claim 3, It is characterized in that The softmax function converts each pixel in the cost volume of the pre-trained network according to the depth dimension to generate the cost probability volume of the pre-trained network. The conversion process expression is as follows: In the formula, is the cost probability volume of the pre-trained network, h, w, m are the spatial width, height and depth, is the cost of the pre-trained network, For m th The cost volume of the depth dimension plane.

5. The depth estimation method based on the multi-step iterative volume probability distribution diffusion model according to claim 1, It is characterized in that Context features are obtained through the two-dimensional context information, context volume is obtained by projecting the context features to the three-dimensional context volume, and a filter volume is constructed by a confidence map obtained through confidence information on the dimension, and an optimized volume is obtained by an element-wise summation operation based on the context volume and the filter volume.

6. A depth estimation method based on a multi-step iterative volume probability distribution diffusion model according to claim 5, It is characterized in that The confidence map is constructed by the highest probability values ​​of all depth planes in the depth dimension.

7. A depth estimation method based on a multi-step iterative volume probability distribution diffusion model according to claim 6, It is characterized in that The highest probability value in the confidence map is obtained by inverting the query vector of the cross-attention operation, and the query vector is used to measure the uncertainty of the depth volume matching. The acquisition process expression is as follows: In the formula, Q i is the query vector, C i is the confidence map, is the input depth volume.

8. The depth estimation method based on the multi-step iterative volume probability distribution diffusion model according to claim 5, It is characterized in that The position points in the context feature generate a reference vector and a content vector through variable convolution, and the expression is as follows: In the formula, K i is the reference vector, V i is the content vector, p is the position point in the context feature, C' is the total number of channels, W is the variability convolution weight, Δ (p) is the learnable offset, is the context feature, Chunk is the separation operation of the context feature on the channel dimension, and Dim=c is the channel dimension.

9. The depth estimation method based on the multi-step iterative volume probability distribution diffusion model according to claim 5, It is characterized in that The context volume and the filter volume are summed up to obtain the optimized volume, and the expression is as follows: In the formula, To optimize the volume, is the context volume, ⊙ is the element-wise multiplication symbol.

Images