A Binocular Dense Disparity Estimation Method that Decouples Matching Cost Calculation and Cost Volume Optimization

Abstract

This invention belongs to the field of binocular dense disparity estimation technology in computer vision, and relates to a binocular dense disparity estimation method that decouples matching cost calculation and cost volume optimization, including matching cost volume calculation and cost volume optimization. The matching cost volume calculation stage is responsible for outputting an initial matching cost volume, which can be implemented by any network or traditional methods. The cost volume optimization stage consists of a guided cost volume optimization network designed in this invention. This network decouples the features used for matching and guided optimization, learns more targeted guided features for context weight calculation, and flexibly performs context aggregation on the input cost volume to achieve cost optimization. The advantages of this invention are: more flexible initial cost calculation, more efficient feature learning, and more effective context aggregation and inter-scale aggregation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision binocular dense disparity estimation technology, and relates to a binocular dense disparity estimation method that decouples matching cost calculation and cost volume optimization. Background Technology

[0002] Deep learning-based binocular dense disparity estimation techniques have made groundbreaking progress in recent years. These methods can be broadly categorized into two types: cost volume optimization-based methods and disparity optimization-based methods. Disparity optimization-based methods, such as Stereonet (Khamis, S., Fanello, SR, Rhemann, C., et al.: 'Stereonet: guided hierarchical refinement for real-time edge-aware depth prediction'. In ECCV 2018), first obtain a coarse disparity, then feed the disparity along with the image or features into an optimization network to obtain an optimized disparity map. These methods are fast but have lower accuracy. Cost volume optimization-based methods mainly involve feature extraction, cost volume construction, cost volume optimization, and disparity calculation. Typical methods include Adaptive Aggregation Network (H. Xu and J. Zhang, "Aanet: Adaptive aggregation network for efficient stereo matching," in CVPR 2020) and Group-wise Correlation Stereo Network (X. Guo, K. Yang, W. Yang, X. Wang, and H. Li, "Group-wise Correlation Stereo Network," in CVPR 2019). These methods typically perform feature extraction only once and couple the matching task with the cost volume optimization task, meaning matching and optimization use the same features. This approach, on the one hand, reduces the method's flexibility, making it impossible to optimize cost volumes obtained by any method; on the other hand, the coupling of the two processes prevents features from being learned more effectively, thus impacting the final disparity prediction performance. This is because the features required for matching should possess local discriminativity, while the features required for optimization need to possess local consistency; the two types of features have opposite properties. Summary of the Invention

[0003] To address the aforementioned issues, this invention proposes a method to decouple the matching and cost volume optimization processes. Based on this, a binocular dense disparity estimation framework is designed to decouple matching cost calculation and cost volume optimization. This framework comprises two separate steps: matching cost volume calculation and cost volume optimization. The matching cost volume calculation stage outputs an initial matching result: a 3D cost volume; this stage can be implemented using any network or traditional methods. The cost volume optimization stage consists of the guided cost volume optimization network designed in this invention. This network takes a guiding image and a coarse cost volume as input and outputs an optimized cost volume. By proposing a novel and independent cost optimization network, this invention decouples the features used for matching and guided optimization, enabling the learning of more targeted guiding features for context weight calculation and flexibly performing context aggregation on the input cost volume to achieve cost optimization.

[0004] The technical solution adopted in this invention is as follows:

[0005] A method for estimating dense disparity in binocular vision by decoupling matching cost calculation and cost volume optimization, comprising the following steps:

[0006] Step 1, Cost Calculation Process: This can be implemented using any network or traditional methods, characterized by the fact that the features used for matching and the features used for optimization are not shared; specifically including:

[0007] (1.1) The left and right images are input into a shared matching feature extraction network to extract the multi-scale features corresponding to the left and right images respectively, forming a matching feature pyramid;

[0008] (1.2) Construct a multi-scale cost volume using the matching feature pyramid to form a cost volume pyramid;

[0009] Step 2, Cost Optimization Process: Optimize the cost volume using independent guiding features; specifically including:

[0010] (2.1) Input the left image into the guided feature extraction network to calculate multi-scale guided features and construct a guided feature pyramid;

[0011] (2.2) Perform intra-scale aggregation of cost volumes based on the guiding features of the corresponding scale;

[0012] (2.3) Perform inter-scale aggregation of cost volumes based on the guiding features of the corresponding scale;

[0013] (2.4) Iterate through steps (2.2) and (2.3) to obtain the optimized cost body;

[0014] (2.5) Parallax calculation.

[0015] The beneficial effects of this invention are as follows: 1) The initial cost calculation method is more flexible, such as optimizing the cost volume obtained by traditional methods, thereby enhancing the generalization ability of the application; 2) Feature learning is more efficient, thereby improving the quality of the final disparity prediction; 3) The invented cost volume optimization network can perform more effective context aggregation and inter-scale aggregation. Its context aggregation can effectively solve the no-match problem caused by occlusion; its inter-scale aggregation can fuse the matching results of targets of different scales under different receptive field features, solving the mismatch problem caused by lack of texture. Attached Figure Description

[0016] Figure 1 This is a network structure diagram of the present invention. Detailed Implementation

[0017] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0018] This invention presents a binocular dense disparity estimation method that decouples matching cost calculation and cost volume optimization, based on two fundamental processes: a matching cost calculation process and a cost optimization process. Based on these two processes, a dense disparity estimation framework is proposed, such as... Figure 1 As shown. The matching cost calculation process provides a multi-scale cost volume pyramid, where each cost volume satisfies the shape "number of candidate points × height × width". It's worth noting that this cost volume pyramid can also be obtained from other networks or traditional methods. The cost optimization process utilizes guided features to obtain the weight of each point relative to its context, and aggregates the cost volume across scales and within scales based on these weights to remove noise from the cost, thus achieving cost optimization. The specific steps are as follows:

[0019] Step 1, Cost Calculation Process:

[0020] (1) The left and right images are input into a shared matching feature extraction network to extract the matching feature pyramids corresponding to the left and right images, respectively. The matching feature extraction network consists of a ResNet-like network and a Feature Pyramid Network (FPN). The outputs of the intermediate and final layers of the ResNet network are then processed. scale, scale, The scale feature maps are constructed into a form, input into the FPN, and the output is the feature pyramid FP. l and FP r Each feature map in the pyramid or The shape is C×(i·H)×(i·W). l and r represent left and right respectively, C represents the number of feature channels, and H and W represent the height and width of the input image respectively.

[0021] (2) Taking the maximum number of candidate matching points as D, the number of candidate points on the feature maps at the three scales in the feature pyramid are respectively Extract feature map and Constructing the cost body CV i The cost volume pyramid (CV) is constructed by matching feature maps at multiple scales within the feature pyramid. The construction process is shown in formula (1):

[0022]

[0023] Where (x,y) is the spatial location index, d is the candidate point index, and FP i (x, y) is a feature vector of dimension C, and ⊙ represents the Cartesian multiplication operation. CV i (x,y,d) represents the cost value at position (x,t,d) on the cost body, which is also a point on the left feature map. With the candidate point on the right feature map The similarity between them.

[0024] Step 2, Cost Optimization Process:

[0025] The process starts at the lowest scale. Within a scale, contextual information is aggregated to optimize the cost volume. Simultaneously, between scales, information is provided from lower scales to higher scales to optimize the cost volume at higher scales.

[0026] (1) Input the left image into the guided feature extraction network to calculate the guided feature pyramid. The guided feature extraction network has the same structure as the matching feature extraction network, but the weights are not shared. The guided feature pyramid is represented as GP, where each feature map in the pyramid is GP. i The shape is C×(i·H)×(i·W), where, C represents the number of feature channels, and H and W represent the height and width of the input image, respectively.

[0027] (2) Intra-scale aggregation: Take CV respectively i and GP i Using GP i Contextual relationships in CV i Aggregation optimization is performed. To improve computational efficiency, the cost body to be aggregated and the guiding features are divided into multiple windows, and the aggregation process is carried out within each window. The specific process of aggregation within each window is as follows:

[0028] (2.1) According to GP i Obtain the context relationship. Let the features within the w-th window be... Transform it into (w m ×w n )×C,w m and wn Let K be the height and width of the window, respectively; then the dictionary (K:key) and the query index (Q:query) can be obtained from formula (2):

[0029]

[0030] in For network weights, then the context is... Obtained from formula (3),

[0031]

[0032] Where softmax is the normalization function and T is the matrix transpose operation.

[0033] (2.2) Based on the context, calculate the cost volume within the w-th window. Perform an aggregate update to obtain the optimized cost volume within the window. As in formula (4):

[0034]

[0035] (3) Perform inter-scale aggregation of cost volumes based on the guiding features of the corresponding scale:

[0036] Retrieve from the w-th window and and The purpose of inter-scale aggregation is to update the high-scale cost volume using the low-scale cost volume, where... It is twice the size of the cost body pyramid. The cost body, It is twice the size of the guiding feature pyramid. The guiding features. Similar to the intra-scale aggregation process, the dictionary, query index, and inter-scale context relationship are first obtained according to formula (5):

[0037]

[0038] Here, GK and GQ represent the dictionary and query obtained from the guiding features, respectively. These represent the network parameters for generating the dictionary and for querying, respectively. This represents the inter-scale contextual relationships obtained from the guiding image.

[0039] Then, an aggregation operation is performed to obtain the updated high-scale cost volume, which is still used... This is represented as in formula (6).

[0040]

[0041] Among them, before the equal sign For the updated and optimized cost body, after the equals sign The cost body before the update.

[0042] (4) From Begin by iterating through steps (2) and (3). The final step of the iteration should be to execute step (2) and... The final optimized cost body is obtained

[0043] (5) Calculate the dense parallax according to formula (7):

[0044]

[0045] Where D max For the maximum possible parallax, P d Let d be the probability of a disparity of d.

Claims

1. A binocular dense disparity estimation method decoupling matching cost computation and cost volume optimization, characterized in that, The steps are as follows: Step 1, cost calculation process, specifically including: (1.1) the left image and the right image are input into the shared matching feature extraction network, and the matching feature pyramids corresponding to the left image and the right image are extracted respectively; The matching feature extraction network consists of a residual network-like network and a feature pyramid network (FPN); the outputs of the intermediate and last layers of the residual network are used to extract features. scale, scale, The scale feature maps are constructed into a form, input into the FPN, and the output is a feature pyramid. and Each feature map in the pyramid or The shape is ;in, , They represent left and right respectively. Represents the number of feature channels, These are the height and width of the input image, respectively; (1.2) a cost volume pyramid is constructed using the matching feature pyramids; The maximum number of candidate matching points is taken as , and the number of candidate points on the feature maps of three scales in the feature pyramid is respectively ; the feature maps and are taken to construct the cost volume ; the plurality of cost volumes constructed by matching the feature maps of multiple scales in the feature pyramid form a cost volume pyramid ; the construction process is as formula (1): (1) ； wherein, is a spatial position index, is a candidate point index, is a feature vector of dimension is a Cartesian multiplication operation; is a cost value at a position on the cost volume, i.e. a similarity between a point on the left feature map and a candidate point on the right feature map.​ Step 2, cost optimization process, specifically including: (2.1) the left image is input into the guided feature extraction network, and a guided feature pyramid is calculated; The guide feature extraction network and the matching feature extraction network have the same structure, and the weights are not shared; the guide feature gold tower is expressed as The shape of each feature map in the pyramid is ; (2.2) the scale intra-aggregation of the cost volume is performed according to the guided features of the corresponding scale; Take and , use the context relationship in to aggregate and optimize ; to improve the calculation efficiency, divide the to-be-aggregated cost body and the guide features into multiple windows, and perform the aggregation process in the windows; the aggregation in each window is as follows: (2.2.1) According to Obtain the context; let the first... The features within each window are Transform it into , and These are the height and width of the window, respectively; then the dictionary ( :key) and query index ( The query is obtained from formula (2): (2) ； wherein , are network weights; then the context relationship is obtained from equation (3), (3) ； wherein is a normalization function, is a matrix transpose operation; (2.2.2) According to the context relationship, the cost volume in the first window is aggregated and updated to obtain an optimized cost volume, as formula (4): ​​​ (4) ； (2.3) the scale inter-aggregation of the cost volume is performed according to the guided features of the corresponding scale; Take the first window and , and scale inter-aggregation to update the high scale cost volume with the low scale cost volume; wherein is the cost volume in the cost volume pyramid with scale 2 times , is the guiding feature in the guiding feature pyramid with scale 2 times . First, the dictionary and the query index and the context relationship are obtained according to formula (5): (5) ； wherein, respectively denote a dictionary and a query obtained from a guiding feature; respectively denote network parameters for generating the dictionary and the query; denotes an inter-scale contextual relationship obtained from a guiding image; Then the aggregation operation is performed to obtain the updated high-scale cost volume, still denoted as denoted as, as in equation (6), (6) ； wherein the equal sign before is the cost volume after the update is the cost volume before the update (2.4) steps (2.2) and (2.3) are iterated step by step to obtain the optimized cost volume; From Start, step (2.2) and step (2.3) are performed iteratively step by step, the last step of the iteration is to perform step (2) and , get the final optimized cost volume ; (2.5) disparity calculation; The dense disparity calculation is performed according to formula (7): (7) ； where is the maximum possible disparity, is the probability that the disparity is .

Images