A binocular stereo matching method based on superpixel segmentation

Abstract

The present invention discloses a binocular stereo matching method based on superpixel segmentation. The original left and right views are downsampled and segmented into SLIC superpixels, and the correlation matrix and adjacency matrix are fed into a superpixel-based graph neural network to obtain weights containing superpixel information. The downsampled left and right views and the correlation matrix are multiplied by a node matrix. The node matrix is ​​fed into a superpixel-based graph neural network to obtain superpixel feature weights. The original left and right views are fed into a convolutional neural network and multiplied with the superpixel feature weights. A four-dimensional cost volume is constructed using the left and right feature maps and then fed into a cost aggregation module to obtain a predicted disparity map. The present invention constructs an end-to-end deep learning framework for stereo matching based on superpixel segmentation, utilizing superpixel blocks to guide and constrain the stereo matching algorithm.

Description

Technical Field

[0001] The present invention relates to computer vision and image processing technology, and in particular to a binocular stereo matching method based on superpixel segmentation. Background Art

[0002] Research on stereo matching algorithms began in the 1970s. After decades of research, in 2002, Scharstein et al. summarized the general steps of stereo matching algorithms into four: matching cost calculation, cost aggregation, disparity calculation, and disparity optimization. They also established the Middlebury testbed for testing and quantitatively evaluating stereo matching algorithms. Based on the different approaches used in stereo matching research over the years, we categorize stereo matching algorithms into traditional stereo matching algorithms and deep learning stereo matching algorithms.

[0003] Traditional stereo matching algorithms can be categorized into three types based on their approach: global, local, and semi-global. Global stereo matching algorithms employ global constraints to address pixel mismatches caused by occlusion or repetitive textures, estimating disparity values ​​based on optimization theory. The global matching problem is often described as an energy minimization problem, where the energy function includes both a data term and a smoothness term. Common global algorithms include graph cuts, belief propagation, and dynamic programming. Local stereo matching algorithms employ local optimization theory to estimate pixel disparity, focusing on the calculation and accumulation of matching costs. Within the range of maximum disparity, the pixel with the lowest matching cost is identified as the target pixel, and local information is used to determine the pixel with the lowest matching cost, thereby calculating the disparity. This method has low computational complexity but is sensitive to noise. Semi-global stereo matching algorithms are an optimization of global matching algorithms. In 2005, H. Hirschmuller et al. proposed the semi-global matching algorithm (SGM), which reduces computational complexity and significantly improves matching accuracy compared to local matching algorithms.

[0004] With the recent advancement of deep learning in computer vision, Mayer et al. first proposed an end-to-end stereo matching network based on the optical flow estimation network FlowNet. This algorithm uses a correlation layer to measure the similarity of left and right image features and leverages this feature similarity for left and right view stereo matching. GCNet takes a different approach by directly concatenating left and right features, requiring 3D convolutions to aggregate the resulting 4D cost volume. In 2018, PSMNet further improved GCNet by using pyramid feature extraction and a stacked hourglass network with 25 3D convolutional layers, thereby improving stereo matching accuracy. While 3D convolutions can achieve state-of-the-art performance, their high computational cost and memory consumption make them prohibitively expensive to deploy in practice. The recent GA-Net work also noted the shortcomings of 3D convolutions and attempted to replace them with two guided aggregation layers; however, the final model still used 15 3D convolutions. To address this, Xu et al. proposed an intra-scale and inter-scale cost aggregation module to completely replace the cost volume in 3D convolutions, thereby improving matching accuracy.

[0005] However, these algorithms do not consider that pixels with locally similar colors in an image are likely to be located on the same plane. For example, pixels in a textureless area are likely to be located on a large plane. Summary of the Invention

[0006] Purpose of the invention: The purpose of the present invention is to address the deficiencies in the prior art, provide a binocular stereo matching method based on superpixel segmentation, construct an end-to-end stereo matching deep learning framework based on superpixel segmentation, and use superpixel blocks to guide and constrain the stereo matching algorithm.

[0007] Technical solution: A binocular stereo matching method based on superpixel segmentation of the present invention comprises the following steps:

[0008] Step S1: Downsample the original left and right views respectively, and then perform SLIC superpixel segmentation on the obtained downsampled left and right views respectively to obtain corresponding sets of multiple image sub-regions, as well as the corresponding correlation matrix Q and adjacency matrix A. Then, the correlation matrix Q and the adjacency matrix A are fed into the trained superpixel-based graph neural network to obtain the corresponding weights containing superpixel information; here, the correlation matrix is ​​the relationship between superpixel blocks and pixels; the adjacency matrix is ​​the relationship between superpixel blocks;

[0009] Step S2: Perform two 1×1 convolutions and one flatten process on the downsampled left and right views respectively, expand the multidimensional image output into one dimension, and multiply it with the normalized correlation matrix Q to obtain the preliminary node matrix H;

[0010] Step S3: Input the preliminary node matrix H obtained in step S2 into the trained two-layer superpixel-based graph neural network, learn the edge information between adjacent superpixel nodes through the superpixel-based graph neural network, and then decode and softmax the output of the superpixel-based graph neural network to obtain the superpixel feature weights;

[0011] Step S4: Input the original left and right views into the CNN convolutional neural network to obtain the corresponding left feature map and right feature map; then multiply them with the superpixel feature weights obtained in step S3, and then input them into the hybrid pooling module and the 1×1 convolution layer respectively;

[0012] Step S5: Use the left feature map and the right feature map to construct a four-dimensional cost volume, and then send it to the 3D cost aggregation module; finally, the output of the cost aggregation part is upsampled and disparity regression is performed to obtain the predicted disparity map.

[0013] Considering that most pixels of the same color are located on the same plane, the present invention proposes an end-to-end binocular stereo matching method based on superpixel segmentation, which includes two modules: a binocular stereo matching network and a superpixel-based graph neural network. The entire network adopts an end-to-end approach to train the superpixel segmentation network and the binocular stereo matching network respectively.

[0014] Furthermore, the hybrid pooling module includes a pyramid pooling module and a long strip pooling layer. The hybrid pooling module encodes global horizontal and vertical information and then uses these codes to balance its own weights for feature optimization. The spatial pyramid pooling module can expand the receptive field and obtain a fusion of global and local feature information. The feature dimension is reduced through a 1×1 convolutional layer, where the horizontal filter size is 1×3 and the vertical filter size is 3×1.

[0015] Furthermore, the specific method for constructing the cost volume is to concatenate the feature maps corresponding to each hyperparameter disparity value in the left and right feature maps, and use the feature maps of the spatial pyramid pooling module to form a four-dimensional matching cost volume. Using this four-dimensional matching cost volume, the original four-dimensional left and right feature maps can be concatenated in the new disparity dimension. In this invention, since the h and w of the cost volume are one-quarter of the h and w of the input image, the hyperparameter disparity value here is 48.

[0016] To obtain more environmental information and further optimize and denoise the matching cost values ​​in the cost volume, three stacked encoder-decoder structures are used for cost aggregation. Each encoder-decoder structure generates a disparity map, and the three encoder-decoder structures have three outputs and three losses. Each encoder-decoder structure consists of two layers of convolution and two layers of deconvolution, with skip connections added to enrich pixel information. At the same time, the output of each encoder-decoder structure is upsampled and disparity regressed to restore the disparity map to the same width and height as the input left and right views.

[0017] During the training process, the total loss is obtained by the weighted sum of the three loss values. The total loss function is as follows:

[0018]

[0019] Where N is the number of labeled pixels, y i is the true value, x i is the predicted value;

[0020] During testing, the final disparity map is obtained from the last of the three outputs.

[0021] Furthermore, when downsampling the left view and the right view in step S1, the original left view and the original right view are both downsampled to 1 / 4 of the original image.

[0022] Beneficial effects: In order to solve the difficult problems of weak texture areas, repeated textures, thin ropes / thin rods, occlusion and depth discontinuity in the field of binocular stereo matching research, the present invention proposes an end-to-end superpixel segmentation-based stereo matching deep learning framework, which uses superpixel blocks to guide and constrain the stereo matching algorithm to optimize stereo matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of the overall network structure of the present invention;

[0024] Figure 2 Schematic diagram of the encoder / decoder structure of the present invention;

[0025] Figure 3 This is a flowchart of the cost volume concat left and right feature maps in the present invention;

[0026] Figure 4 Schematic diagram of 3D cost aggregation in the present invention;

[0027] Figure 5 The original image of the scene to be processed in the embodiment;

[0028] Figure 6 This is a schematic diagram of an embodiment processed using the technical solution of the present invention. DETAILED DESCRIPTION

[0029] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0030] like Figure 1 As shown, a binocular stereo matching method based on superpixel segmentation of the present invention comprises the following steps:

[0031] Step S1: Downsample the original left and right views by one-quarter, then perform SLIC superpixel segmentation on the resulting downsampled left and right views to obtain corresponding sets of multiple image subregions, as well as the corresponding correlation matrix Q and adjacency matrix A. Then, the correlation matrix Q and adjacency matrix A are fed into the trained superpixel-based graph neural network to obtain the corresponding weights.

[0032] Step S2: Perform two 1×1 convolutions and one flatten process on the downsampled left and right views respectively, expand the multidimensional image output into one dimension, and multiply it with the normalized correlation matrix Q to obtain the preliminary node matrix H;

[0033] Step S3, as Figure 2 As shown, the preliminary node matrix H obtained in step S2 is input into the trained two-layer superpixel-based graph neural network, and then the superpixel-based graph neural network is output, decoded and softmax processed to obtain the superpixel feature weights;

[0034] Step S4: Input the original left view and right view into the CNN convolutional neural network to obtain the corresponding left feature map and right feature map; then multiply them with the superpixel feature weights obtained in step S3, and then input them into the hybrid pooling module and 1×1 convolution layer respectively; the hybrid pooling module includes a pyramid pooling module and a long strip pooling layer, which encodes the global horizontal and vertical information through the hybrid pooling module, and then uses these codes to balance its own weights for feature optimization

[0035] Step S5: Figure 3 As shown in , the left feature map and the right feature map are used to construct a four-dimensional cost volume, which is then fed into the 3D cost aggregation module. Finally, the output of the cost aggregation part is upsampled and regressed to obtain the predicted disparity map. The specific method for constructing the cost volume is as follows: concat the feature map corresponding to each disparity value in the left feature map and the right feature map, and use the feature map of the spatial pyramid pooling module to form a four-dimensional matching cost volume. Figure 4As shown, this embodiment adopts three stacked encoder-decoder structures during cost aggregation, and each encoder-decoder structure generates a disparity map; thus, the three hourglass structures will consist of three outputs and three losses; during training, the total loss is obtained by the weighted sum of the three loss values; during testing, the final disparity map is obtained from the last of the three outputs.

[0036] Figure 3 In the first branch, the feature map is first reduced in dimension using a 1×1 convolution before entering the two branches. In the first branch, there are three branches, and the first two branches are downsampled to half and one-third of the original size, respectively. The three branches are then convolved and upsampled, and the three outputs are added together before another convolution. In the second branch, two strip pooling layers are first performed, with pooling layer sizes of 1×3 and 3×1, respectively. This is followed by a 1D convolution and upsampling, and the two outputs are added together. Finally, the outputs of the two branches are concatinated together. This module's processing can better capture long-term dependencies, expand the receptive field, and further optimize areas with weak and repeated textures.

[0037] In order to overcome the problem that CNN networks and GCN networks are not directly compatible, the present invention designs a structural converter that allows features to propagate between image and graph spaces. First, the original left and right views are downsampled to 1 / 4 of the original size. Then, in order to quickly construct a suitable graph neural network structure containing pixel color information, the present invention uses simple linear iterative clustering (SLIC) to perform superpixel segmentation on the left and right views. Finally, by establishing the adjacency relationship between superpixels, the RGB image is converted into an undirected graph. In summary, an unweighted graph is constructed based on the superpixel segmentation of the RGB left and right views. The graph encoder of the present invention can project the association matrix Q from the image space to the graph space. Correspondingly, the graph decoder can assign the node features output by the graph network to pixels. Both the graph encoder and the graph decoder can be introduced into the network to integrate CNN and GCN. In addition, the adjacency mask records the adjacency relationship between adjacent superpixels, which will learn the weights of the edges together with the GCN.

[0038] This embodiment assumes that a graph G = (v, e) includes a node v and an edge e; in a superpixel-based graph neural network, v and e are encoded as a node matrix H and an adjacency matrix A. The i-th row edge of H represents the i-th node, and (i, j) of A represents the weight of the edge between the i-th node and the j-th node of the graph. To improve the adaptability of the graph neural network, the adjacency matrix Ak of ​​the k-th layer is learned based on the input of each layer after the output of each layer of the network. The weight of the edge is then compressed in the range of (0, 1) by the sigmoid function to prevent numerical instability. The adjacency mask A can cut off the association between non-adjacent nodes while maintaining the connectivity between adjacent nodes, thereby modeling the color space structure of the binocular image to generate superpixel-level features. Finally, the superpixel feature weight is obtained by softmax and multiplied into the hybrid pooling module.

[0039] like Figure 5 and Figure 6 As shown, after the binocular stereo image is processed using the technical solution of the present invention, the defects of structures such as weak texture areas, repeated textures, thin ropes / thin rods, occlusions and depth discontinuities can be overcome.

Claims

1. A binocular stereo matching method based on superpixel segmentation, characterized by: The following steps are involved: Step S1: Downsample the original left and right views respectively, and then perform SLIC superpixel segmentation on the obtained downsampled left and right views respectively to obtain corresponding sets of multiple image subregions, as well as the corresponding correlation matrix Q and adjacency matrix A. Then, the correlation matrix Q and adjacency matrix A are fed into the trained superpixel-based graph neural network to obtain the corresponding weights containing superpixel information; Step S2: Perform two 1×1 convolutions and one flatten process on the downsampled left and right views respectively, expand the multidimensional image output into one dimension, and multiply it with the normalized correlation matrix Q to obtain the preliminary node matrix H; Step S3: Input the preliminary node matrix H obtained in step S2 into the trained two-layer superpixel-based graph neural network, learn the edge information between adjacent superpixel nodes through the superpixel-based graph neural network, and then decode and softmax the output of the superpixel-based graph neural network to obtain the superpixel feature weights; Step S4: Input the original left and right views into the CNN convolutional neural network to obtain the corresponding left feature map and right feature map; then multiply them with the superpixel feature weights obtained in step S3, and then input them into the hybrid pooling module and the 1×1 convolution layer respectively; Step S5: Use the left feature map and the right feature map to construct a four-dimensional cost volume, and then send it to the 3D cost aggregation module; finally, the output of the cost aggregation part is upsampled and disparity regression is performed to obtain the predicted disparity map.

2. The binocular stereo matching method based on superpixel segmentation according to claim 1, characterized in that: The hybrid pooling module includes a pyramid pooling module and a strip-shaped pooling layer, and encodes global horizontal and vertical information through the hybrid pooling module, wherein the horizontal filter size is 1×3 and the vertical filter size is 3×1.

3. The binocular stereo matching method based on superpixel segmentation according to claim 1, wherein: The specific method of constructing the cost volume is: concat the feature maps corresponding to each hyperparameter disparity value in the left feature map and the right feature map, and use the feature maps of the spatial pyramid pooling module to form a four-dimensional matching cost volume.

4. The binocular stereo matching method based on superpixel segmentation according to claim 1, wherein: The cost aggregation is performed using three stacked encoder-decoder structures, each of which generates a disparity map. The three encoder-decoder structures have three outputs and three losses. Each encoder-decoder structure consists of two layers of convolution and two layers of deconvolution, with skip connections added. The output of each encoder-decoder structure undergoes upsampling and disparity regression to restore the disparity map to the same width and height as the input left and right views. During the training process, the total loss is obtained by the weighted sum of the three loss values. The total loss function loss(x,y) is: Where N is the number of labeled pixels, y i is the true value, x i is the predicted value; During testing, the final disparity map is obtained from the last of the three outputs.

5. The binocular stereo matching method based on superpixel segmentation according to claim 1, wherein: When downsampling the left view and the right view in step S1, both the original left view and the original right view are downsampled to 1 / 4 of the original image.

Images