Stereo matching method based on cycle cost aggregation

Abstract

The application provides a stereo matching method based on cyclic cost volume aggregation, which can gradually fuse global context and update matching details in the cost volume, thereby eliminating mismatches and ambiguities. In order to promote comprehensive cost aggregation, the application proposes a multi-peak search strategy, based on which several local cost volumes with high probability are indexed and summarized by GRU. In addition, the application also introduces a cascaded search range method, which uses a larger search range at the beginning of GRU iteration, and gradually reduces the search range with specific patterns. Based on the proposed cyclic cost aggregation method, the application constructs a new stereo matching network. A large number of experiments on the stereo matching dataset show that the method ranks first in the KITTI-12 and KITTI-15 stereo benchmark test rankings among all published methods.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision stereo matching, and more particularly relates to a stereo matching method based on recurrent cost aggregation. BACKGROUND

[0002] Stereo matching is a basic task in computer vision, with wide application prospects. In the past five years, deep learning-based methods have made significant progress in stereo matching and depth estimation. RAFT-Stereo uses a recurrent optimization framework to regress disparity residuals by iteratively retrieving local costs based on a GRU updater. This updater can save context information and previous states in the hidden layer, effectively combining global context with local matching details to gradually improve the accuracy of disparity estimation. However, existing recurrent iterative optimization methods repeatedly index from the initial cost volume, purely relying on the GRU updater to integrate global context into local matching details to infer global geometry, which can be inefficient. This is because the initial cost volume only encodes local matching information, which may contain mismatched and ambiguous ill-posed regions such as textureless / translucent regions and occluded regions. It is inefficient to optimize disparity directly from such coarse local cost volumes through the GRU updater. On the other hand, cost aggregation is widely used in traditional stereo matching models to regularize the cost volume to address this issue. SUMMARY

[0003] The present application proposes a stereo matching method based on recurrent cost aggregation, which combines the advantages of recurrent optimization and traditional aggregation methods to estimate high-precision disparity. Specifically, the present application aggregates the cost volume through a GRU updater to gradually integrate global context and matching details in the updated cost volume. Experiments show that using GRU to aggregate and improve the quality of the cost volume is more efficient than optimizing the disparity map from the initial cost volume to improve the final precision. To comprehensively aggregate the cost, the present application introduces a multi-peak finding strategy that indexes several local cost volumes with high potential and aggregates them through GRU. To obtain local cost volumes for updating and aggregation, the present application proposes a cascaded search method that uses a large search range at the beginning of recurrent optimization and gradually reduces the search range using specific patterns.

[0004] To achieve the above-mentioned purpose, the present application provides a stereo matching method based on recurrent cost aggregation, which comprises the following steps:

[0005] (1) Take two corrected images on the same horizontal line, called left and right images, to estimate the dense disparity between the left and right images; extract matching features of the left and right images, and context features of the left image;

[0006] (2) Use the matching features of the left image and the right image to construct a correlation cost volume as an initial cost volume C init Then, two layers of cost volume pyramids {C 1 ,C 2} are obtained by using average pooling with a kernel size of 1, 2 and an equal step length in the parallax dimension respectively, and the cost volume pyramid is referred to as a global cost volume C global .

[0007] (3) C init is converted into a probability volume by a softmax operation along the parallax dimension, and the first K parallaxes with the maximum probability are selected from the probability volume as initial parallaxes for local cost volume aggregation by using a multi-peak search;

[0008] (4) A multi-peak search is performed from the global cost volume according to the current parallax to obtain a local cost volume containing multiple peaks; the local cost volume with multiple peaks is encoded with the context features of the left image, and a local cost volume residual is updated iteratively using a GRU; in each iteration, when the multi-peak search is performed from the global cost volume according to the current parallax, the search range is changed according to the number of iterations;

[0009] (5) The local cost volume residual is added to the local cost volume to obtain a regularized local cost volume, and a parallax map is regressed from the regularized local cost volume, and the parallax map is up-sampled to the original resolution as the output.

[0010] In an embodiment of the present application, in the step (1), the matching features of the left image and the right image and the context features of the left image are extracted, and specifically comprising:

[0011] The feature extraction module includes two independent encoders, a matching feature encoder and a context feature encoder, both of which have the same structure and use the first two stages of a pre-trained ConvNext to extract features, and fuse features of two different scales through a U-Net type up-sampling module; wherein the matching feature encoder extracts features of the left image and the right image, denoted as f l ,f r ∈R C ×H×W , C = 256, H, W are 1 / 4 of the width and height of the original image respectively; the context feature encoder extracts only the context features f c ∈R C×H×W of the left image.

[0012] In an embodiment of the present application, in the step (2), the correlation cost volume is constructed using the matching features of the left image and the right image, and specifically comprising:

[0013] For the matching features f l ,f r ∈R C×H×WThe initial relevant cost body C is constructed using the following formula. init :

[0014] C init (d,x,y)= <f l (x,y),f r (xd,y)>

[0015] Where <·,·> denote the inner product, and d∈[0,D] max / 4],D max It is the maximum disparity assumption, C init The dimension is [D max / 4,H,W].

[0016] In one embodiment of the present invention, the multi-peak search in step (3) specifically includes:

[0017] The local cost volume obtained in the previous regularization Transform into probability body P i Its dimension is [N i [,H / 4,W / 4], where N i N is the number of sampling points in the i-th iteration. i =k×(2r i +1), where k is the number of peaks in the multi-peak search, r i It is the search range for the i-th iteration; from the probability volume P i Select the k disparities d with the highest probability j (j = 1, ..., k); for each d j With it as the center, r i Uniform sampling is performed on the radius to obtain candidate disparity Ω j The disparity candidate Ω in the current iteration i From all Ω j Combined into:

[0018] Ω j =[d j -r i ,...,d j ,...,d j +r i ],(j=1,...,k)

[0019] Ω i =[Ω 1 ,Ω 2 ,...,Ω k ]

[0020] According to Ω i From the global cost volume C global Searching for local cost volume C local,i .

[0021] In one embodiment of the present application, in the step (4), the search range is changed according to the number of iterations, specifically including:

[0022] The search range is divided into three levels of 25, 9 and 5, each level corresponds to a specific GRU update module, and in the first few iterations, a larger search range is used, and then the search range is gradually reduced, as follows:

[0023]

[0024] In one embodiment of the present application, in the step (4), the local cost volume residual is updated iteratively using GRU, specifically including:

[0025] The local cost volume C local,i of the current iteration is obtained by encoding the current iteration of the local cost volume C i-1 and the disparity map D local,i of the previous iteration through convolution and inputting into the GRU to estimate the local cost volume residual ΔC

[0026]

[0027] In one embodiment of the present application, in the step (5), the disparity map is regressed, specifically including:

[0028]

[0029] Where i is the number of iterations, Ω i is the disparity candidate of the current iteration, is the regularized local cost volume.

[0030] In one embodiment of the present application, in the step (3), when the number of iterations i = 1, is the global cost volume C global .

[0031] In one embodiment of the present application, in the step (5), the disparity map is upsampled to the original resolution, specifically including:

[0032] In each iteration, the up-sampling weight is generated from the hidden state h through convolution, and the low-resolution disparity map is up-sampled to the original resolution according to the up-sampling weight.

[0033] In one embodiment of the present application, for the sequence of disparity maps {d1,...,d N output by N iterations, given the disparity map label d gt , the L1 loss function is used for supervision, and the loss function is defined as:

[0034]

[0035] where γ = 0.8, ||*|| represents the absolute value of *, and N is the total number of iterations.

[0036] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects:

[0037] (1) The present application proposes a stereo matching method based on cyclic cost aggregation, which combines the advantages of cycle optimization and traditional aggregation methods to estimate high-precision disparity. Specifically, the present application aggregates the cost volume through the GRU updater to gradually fuse and update the global context and matching details in the cost volume;

[0038] (2) In order to comprehensively aggregate the cost, the present application introduces a multi-peak search strategy, which indexes several local cost volumes with high potential and aggregates them through GRU;

[0039] (3) In order to obtain local cost volumes for updating and aggregation, the present application proposes a cascaded search method, which uses a large search range at the beginning of cycle optimization and gradually reduces the search range using specific patterns.

[0040] (4) The disparity estimation accuracy of the method of the present application ranks first on the KITTI 2012 and KITTI 2015 rankings. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a schematic diagram of the stereo matching method based on cyclic cost aggregation of the present application;

[0042] Figure 2 is a schematic diagram of multi-peak search in an embodiment of the present application;

[0043] Figure 3 is a schematic diagram of cascaded search range in an embodiment of the present application;

[0044] Figure 4 is a comparison diagram of the qualitative results of the present method and other methods. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. The specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0046] The problem to be solved by the present application is the low accuracy of ill-conditioned regions in disparity estimation. Figure 1 The present application proposes a stereo matching method based on recurrent cost aggregation, which combines the advantages of recurrent optimization and traditional aggregation methods to estimate high-precision disparity. Specifically, the present application aggregates the cost volume through a GRU updater to gradually integrate and update the global context and matching details in the cost volume. Specifically, as shown in Figure 1 The present application provides a stereo matching method based on recurrent cost aggregation, which includes:

[0047] (1) Take two corrected images on the same horizontal line, called left image and right image, to estimate the dense disparity between the left image and the right image; extract the matching features of the left image and the right image, and the context features of the left image;

[0048] Specifically, the feature extraction module includes two independent encoders, a matching feature encoder and a context feature encoder, both of which have the same structure and use the first two stages of pre-trained ConvNext to extract features, and fuse features of two different scales through a U-Net type upsampling module; wherein the matching feature encoder extracts the features of the left image and the right image, denoted as f l ,f r ∈R C×H×W , C = 256, H, W are 1 / 4 of the width and height of the original image; the context feature encoder extracts only the context features f c ∈R C×H×W of the left image.

[0049] (2) Use the matching features of the left image and the right image to construct a correlation cost volume as the initial cost volume C init , and then obtain two layers of cost volume pyramid {C 1 ,C 2} by using average pooling with kernel size 1, 2 and equal step size in the disparity dimension respectively, and the cost volume pyramid is called global cost volume C global .

[0050] Specifically, for the matching features f l ,f r ∈R C×H×W of the left image and the right image, the initial correlation cost volume C init is constructed by the following formula:

[0051] C init (d,x,y) = <f l (x,y),f r (x-d,y)>

[0052] Where <·,·> denotes the inner product, d ∈ [0, D max / 4], D maxIt is the maximum disparity assumption, C init The dimension is [D max / 4,H,W].

[0053] (3) By performing a softmax operation along the disparity dimension, C init Convert to a probability volume, and use multi-peak search to select the top K disparities with the highest probabilities from the probability volume as the initial disparities for local cost volume aggregation;

[0054] For comprehensive cost aggregation, such as Figure 2 As shown, this invention introduces a multi-peak search strategy, which indexes several high-potential local cost bodies and aggregates them using GRU. Specifically, the multi-peak search retrieves the previously obtained regularized local cost bodies. Transform into probability body P i Its dimension is [N i [,H / 4,W / 4], where N i N is the number of sampling points in the i-th iteration. i =k×(2r i +1), where k is the number of peaks in the multi-peak search, r i This is the search range for the i-th iteration; when the iteration number i = 1, For the global cost body C global From probability body P i Select the k disparities d with the highest probability j (j = 1, ..., k); for each d j With it as the center, r i Uniform sampling is performed on the radius to obtain candidate disparity Ω j The disparity candidate Ω in the current iteration i From all Ω j Combined into:

[0055] Ω j =[d j -r i ,...,d j ,...,d j +r i ],(j=1,...,k)

[0056] Ω i =[Ω 1 ,Ω 2 ,...,Ω k ]

[0057] According to Ω i From the global cost volume C global Searching for local cost volume C local,i .

[0058] (4) Multi-peak search from the global cost volume according to the current disparity to obtain a local cost volume containing multiple peaks; the local cost volume with multiple peaks is encoded with the context features of the left image, and the local cost volume residual is updated iteratively using GRU; in each iteration, when the multi-peak search is performed from the global cost volume according to the current disparity, the search range is changed according to the iteration number;

[0059] In order to obtain the local cost volume for updating and aggregation, the application proposes a cascaded search method, as shown in the figure, which uses a large search range at the beginning of the loop optimization, and gradually reduces the search range using a specific pattern. Figure 3

[0060] Among them, the search range is divided into 25, 9, 5 three levels, each level corresponds to a specific GRU update module, in the first few iterations, a larger search range is used, and then the search range is gradually reduced, as follows:

[0061]

[0062] The GRU iterative update of the local cost volume residual specifically includes:

[0063] The local cost volume C local,i of the current iteration is input into the GRU after being encoded by convolution to estimate the local cost volume residual ΔC i-1 . local,i The local cost volume residual is added to the original cost volume to obtain the regularized local cost volume C

[0064]

[0065] (5) The local cost volume residual is added to the local cost volume to obtain the regularized local cost volume, and the disparity map is regressed from the regularized local cost volume. The disparity map is upsampled to the original resolution as output.

[0066] The regression of the disparity map specifically includes:

[0067]

[0068] Where i is the iteration number, Ω i is the disparity candidate of the current iteration, is the regularized local cost volume.

[0069] Where the disparity map is upsampled to the original resolution, specifically including:

[0070] In each iteration, the up-sampling weight is generated from the hidden state h by convolution, and the low-resolution disparity map is up-sampled to the original resolution according to the up-sampling weight. ​

[0071] Further, for the sequence of disparity maps {d1,...,dN} outputted by N iterations, given a disparity map label d N gt , supervised by L1 loss function, which is defined as:

[0072]

[0073] where γ = 0.8, ||*|| represents taking absolute value of *, and N is the total number of iterations.

[0074] Table 1 and Table 2 are quantitative comparisons of the method of the present application and other methods on KITTI 2015 and KITTI 2012 respectively. As can be seen from Table 1 and Table 2, the method of the present application is superior to other published methods in all indicators of KITTI 2012 and 2015. Figure 4 is a qualitative comparison of the method of the present application and other methods, and the method of the present application is superior to other methods in the accuracy of disparity estimation in multiple regions.

[0075] Table 1

[0076]

[0077] Table 2

[0078]

[0079] Those skilled in the art will readily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application, and any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.​

Claims

1. A stereo matching method based on cycle cost aggregation, characterized in that, The method comprises the following steps: (1) taking two corrected images on the same horizontal line as left and right images respectively, for estimating the dense disparity between the left and right images; extracting matching features of the left and right images, and context features of the left image; (2) Use the matching features of the left and right images to construct a correlation cost volume as the initial cost volume C init Then, two layers of cost volume pyramids {C 1 ,C 2} are obtained by using average pooling with kernel size 1, 2 and equal step size in the disparity dimension respectively, and the cost volume pyramid is called global cost volume C global ; (3) converting C into a probability volume by a softmax operation along the disparity dimension, and selecting the top K disparities with the largest probabilities from the probability volume as initial disparities for local cost volume aggregation using a multi-peak search; the multi-peak search specifically includes: init (3) converting C into a probability volume by a softmax operation along the disparity dimension, and selecting the top K disparities with the largest probabilities from the probability volume as initial disparities for local cost volume aggregation using a multi-peak search; the multi-peak search specifically includes: the regularized local cost volume obtained last time is converted into a probability volume P i with dimension [N i ,H / 4,W / 4], where N i is the number of sampling points of the i-th iteration, N i =k×(2r i +1), where k is the number of peaks of the multi-peak search, r i is the search range of the i-th iteration; k most probable disparities d i (j=1,...,k) are selected from the probability volume P j ; for each d j , a uniform sampling is performed with d i as the center and r j as the radius to obtain a disparity candidate Ω i ; the disparity candidates Ω j of the current iteration are combined into: Ω j = [d j -r i ,...,d j ,...,d j +r i ], (j = 1,...,k) Ω i = [Ω 1 , Ω 2 ,..., Ω k ] According to Ω i From the global cost volume C global Find the local cost volume C local,i ; (4) performing multi-peak search from the global cost volume according to the current disparity to obtain a local cost volume containing multiple peaks; encoding the local cost volume with multiple peaks and the context features of the left image, and updating the local cost volume residual using GRU iteration; in each iteration, when performing multi-peak search from the global cost volume according to the current disparity, the search range is changed according to the iteration number; the search range is changed according to the iteration number, specifically including: The search range is divided into 25, 9, 5 levels, each level corresponds to a specific GRU update module, in the first few iterations, a larger search range is used, and then the search range is gradually reduced, as follows: (5) adding the local cost volume residual to the local cost volume to obtain a regularized local cost volume, and regressing the disparity map from the regularized local cost volume, and upsampling the disparity map to the original resolution as output.

2. The cost volume aggregation based stereo matching method of claim 1, wherein, In step (1), the matching features of the left and right images, and the context features of the left image are extracted, specifically including: The feature extraction module includes two independent encoders, a matching feature encoder and a context feature encoder, both of which have the same structure and use the first two stages of a pre-trained ConvNext to extract features. The features of two different scales are fused through a U-Net type upsampling module; wherein the matching feature encoder extracts the features of the left image and the right image, denoted as f l ,f r ∈R C×H×W , C = 256, H, W are 1 / 4 of the width and height of the original image respectively; the context feature encoder extracts only the context features f c ∈R C ×H×W .

3. The cost volume aggregation based stereo matching method of claim 1 or 2, wherein, In step (2), the correlation cost volume is constructed using the matching features of the left and right images, specifically including: For matching features f of the left and right images l ,f r ∈R C×H×W The initial correlation cost volume C is constructed by init : C init (d,x,y) = <f l (x,y),f r (x-d,y)> where <·, ·> denotes the inner product, d ∈ [0, D max / 4], D max is the maximum disparity hypothesis, C init has dimension [D max / 4, H, W].

4. The cost volume aggregation based stereo matching method of claim 3, wherein, In step (4), the local cost volume residual is updated using GRU iteration, specifically including: the local cost volume C of the current iteration local,i the disparity map D resulting from the last iteration i-1 the local cost volume residual ΔC estimated by convolution and input into the GRU local,i the regularized local cost volume obtained by adding the local cost volume residual to the original cost volume 5. The cost volume aggregation based stereo matching method of claim 1 or 2, wherein, In step (5), the disparity map is regressed, specifically: where i is the iteration number, Ω i is the disparity candidate of the current iteration, is the regularized local cost volume.

6. The cost volume aggregation based stereo matching method of claim 1 or 2, wherein, In step (3), when the iteration number i = 1, is the global cost volume C global .

7. The cost volume aggregation based stereo matching method of claim 1, wherein, In step (5), the disparity map is upsampled to the original resolution, specifically including: In each iteration, the upsampling weight is generated from the hidden state h by convolution, and the low-resolution disparity map is upsampled to the original resolution according to the upsampling weight.

8. The stereo matching method based on cyclic cost aggregation according to claim 1 or 2, characterized in that: For the sequence of disparity maps {d1,..., dN} output for N iterations, given a disparity map label d N gt is supervised using the LI loss function defined as:​ wherein γ = 0.8, ||*|| represents taking the absolute value of *, and N is the total number of iterations.

Images