A binocular stereo matching method

Abstract

The present invention discloses a binocular stereo matching method, which comprises the following steps: 1) a feature extraction network performs feature extraction on left and right images to be matched, and obtains feature maps of N resolutions of the left image and feature maps of N resolutions corresponding to the right image; 2) the feature maps of the same resolution corresponding to the left and right images are subjected to correlation operations to form a 4D matching cost volume; then a local cost aggregation operation is performed on each 4D matching cost volume through an intra-scale aggregation module to obtain a new matching cost volume with the same resolution as the original matching cost volume; 3) the N 4D new matching cost volumes obtained in step 2) are fused through an inter-scale aggregation module to obtain a final matching cost volume; 4) disparity maps corresponding to N different resolutions are obtained according to the final matching cost volume; then each obtained disparity map is upsampled and input into StereoDRNet to obtain a final predicted disparity map.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a binocular stereo matching method. Background Art

[0002] Binocular stereo vision is inspired by human vision. Without touching the target object, it imitates the human eye and uses two cameras to capture images from different angles. It obtains the three-dimensional information of the object based on the principle of parallax, and reconstructs the three-dimensional contour and position information of the object. It has the advantages of high efficiency, high precision, simple operation, low cost, and high degree of automation. It is widely used in precision measurement, target positioning, robot motion and environmental survey, mechanical grasping, intelligent driving, three-dimensional reconstruction, medical imaging, human-computer interaction and other fields.

[0003] The key technologies for binocular vision are: camera calibration, stereo correction, stereo matching, and 3D reconstruction. Stereo matching is one of the core steps of binocular vision technology and a research focus in the field of computer vision. Stereo matching is a process of finding the correspondence between spatial objects in the acquired left and right images, that is, finding the corresponding points of the binocular images, calculating the disparity, and estimating the depth based on the calculated disparity using the principle of similar triangles. Stereo matching is the most important part of establishing stereo vision. The disparity information obtained by stereo matching will directly affect the obtained 3D information.

[0004] Stereo matching is the key, research focus and difficulty of binocular vision technology. This subject has been studied for decades. Traditionally, the stereo matching process is considered to be a multi-stage optimization solution process, which is generally divided into four steps: matching cost calculation, matching cost aggregation, disparity calculation and disparity optimization (see Scharstein D, Szeliski RA taxonomy and evaluation of dense two-frame stereo correspondence algorithms [J]. International Journal of Computer Vision, 2002, 47 (1-3): 7-42).

[0005] In recent years, deep learning has demonstrated excellent performance in various fields such as image, speech, and natural language processing. Convolutional Neural Network (CNN) can effectively reduce the dimensionality of large data volumes to small data volumes while retaining image features for feature extraction. Reasonable use of convolutional neural networks can obtain high-precision and high-efficiency stereo matching algorithms. Large-scale binocular simulation data sets also provide possibilities for stereo matching algorithms based on deep learning.

[0006] Literature "N.Mayer,E.Ilg,P. P.Fischer, D.Cremers, A.Dosovitskiy and T.Brox.A Large Dataset to Train Convolutional Networks for Disparity,Optical Flow,and Scene Flow Estimation[J].2016IEEE Conference on Computer Vision and Pattern Recognition(CVPR),2016:4040-4048" For the first time, it was proposed to extend the concept of optical flow estimation to disparity and scene flow estimation through convolutional neural networks. The disparity estimation adopts the encoder-decoder architecture for end-to-end convolutional network training. The disparity accuracy obtained by the network is comparable to that of other algorithms at the time, but the running speed is improved by 1000 times, and a large-scale dataset suitable for training convolutional networks for stereo matching is created. Literature "Chang Jiaren, Chen Yongsheng.Pyramid stereo matching network[C].Proceedingsof the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018: 5410-5418" proposed to use spatial pyramid pooling (SPP) and dilated convolution in the matching cost calculation to expand the receptive field, and proposed a stacked hourglass 3D convolutional neural network combined with intermediate supervision to aggregate the matching cost volume. This network model achieved the best results on the KITTI evaluation website at the time. The literature "F. Zhang, V. Prisacariu, R. Yang and P. Torr. GA-Net: Guided Aggregation Net for End-to-end Stereo Matching [J]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019: 185-194" borrowed some traditional semi-local stereo matching algorithms (Semi-Global Matching, SGM for short) and filtering ideas, and proposed two network layers to replace the commonly used 3D convolution cost aggregation method to obtain local and global cost dependencies respectively.They are the semi-global guided aggregation layer (SGA) and the local guided aggregation layer (LGA). SGA is an approximate differentiable representation of SGM, and LGA draws on the cost filtering strategy in traditional methods to compensate for the detail loss of downsampling and upsampling. These two layers of the network achieve higher accuracy while reducing parameters.

[0007] In addition, the network structure of the end-to-end stereo matching algorithm is generally more complex. Usually, the feature extraction effect is better in ill-posed areas, such as occlusion, reflection, weak texture and no texture. A multi-scale and multi-layer feature extraction network is often selected, but this will result in a large number of redundant parameters, which will lead to the need for a huge cache during operation and high requirements for the device. Therefore, the algorithm has a large overhead for small devices. The literature "Wang Yufeng, Wang Hongwei, Yu Guang, Yang Mingquan, Yuan Yuwei, Quan Jicheng. Stereo matching algorithm based on three-dimensional convolutional neural network [J]. Acta Optica Sinica, 2019, 39(11): 227-234" constructs a sparse loss volume in the disparity dimension, which greatly shortens the running time while improving the accuracy. However, there are still large errors for irregular surfaces, shadow areas, or areas with extremely dark light and occluded areas. The document "Liu Jianguo, Feng Yunjian, Ji Guo, Yan Fuwu, Zhu Shizhuo. An improved stereo matching algorithm based on PSMNet [J]. Journal of South China University of Technology (Natural Science Edition), 2020, 48(01): 60-69+83" improves the Pyramid Stereo Matching Network (PSMNet) and proposes the SWNet stereo matching algorithm. In the feature extraction module, the Atrous Spatial Pyramid Pool (ASPP) structure is used, so that the final network model has fewer layers and parameters, takes up less memory and can quickly process matching tasks, and the matching achieves a higher accuracy. The paper "Li Tong, Ma Wei, Xu Shibiao, Zhang Xiaopeng. End-to-end deep network adapted to stereo matching tasks [J]. Computer Research and Development, 2020, 57(07): 1531-1538" takes PSMNet as the benchmark network, improves its feature extraction module and 3D convolution matching, i.e., cost aggregation module, and proposes a feature extraction network designed for stereo matching tasks. At the same time, in the disparity calculation module, a separate 3D convolution with a large receptive field is proposed to reduce the complexity of convolution and improve the accuracy of stereo matching. The paper "Cheng Mingyang, Gai Shaoyan, Da Feipeng. Research on Stereo Matching Network Based on Attention Mechanism [J]. Acta Optica Sinica, 2020, 40(14): 144-152" uses a convolutional neural network based on a residual network (Residual Network, abbreviated as ResNet) to calculate the feature maps of the left and right images, and captures rich contextual information by adding image attention mechanism modules and channel attention mechanism modules, thereby effectively enhancing the feature representation of stereo matching. At the same time, it improves the network loss function, adds semantic coding loss, and further standardizes network training, making the reconstruction results more accurate and improving the accuracy of stereo matching.The paper "Wang Yufeng, Wang Hongwei, Liu Yu, Yang Mingquan, Quan Jicheng. Progressive refinement of real-time stereo matching algorithm [J]. Acta Optica Sinica, 2020, 40(09): 99-109" proposes a lightweight model that can be used for stereo matching tasks. The main idea is to use a coarse-to-fine progressive method to predict dense disparity maps, and in the disparity refinement module, a Markov random field (MRF) is proposed to cluster multiple modes first, process them separately and then fuse them, and output the predicted disparity map residual, so as to process different areas in different situations. This improves the accuracy of the algorithm and improves the real-time performance of the algorithm. However, the existing feature extraction network is too complex, has too many parameters, and has low feature extraction efficiency. At present, the improvement of the running time of the end-to-end stereo matching algorithm is mainly in the matching cost calculation stage and the cost aggregation stage. In the matching cost calculation stage, the end-to-end stereo matching algorithm often relies on a good feature extraction network. However, in order to better extract features, the existing feature extraction network is complex and has many parameters. Although it can achieve better results, it makes the entire end-to-end network more complex and occupies huge memory, which increases the burden on the device and increases the time consumed to obtain the initial disparity map. Summary of the invention

[0008] In order to overcome the shortcomings of the existing feature extraction network being too complex and having too many parameters, the purpose of the present invention is to provide a binocular stereo matching method, which reduces the operating pressure of the equipment through the improved binocular stereo matching method and obtains an initial disparity map of high quality more quickly. The innovation of the present method is that an end-to-end deep network suitable for stereo matching is proposed, wherein the network structure is constructed with an end-to-end framework design, and after inputting the processed left and right images, wherein the left and right images mainly refer to the imaging using a parallel binocular vision system, such as a binocular camera, or the left and right eye images obtained by two parallel cameras, the disparity map can be directly obtained through the network, and the process of manual preprocessing and subsequent processing is reduced, so that the running time is reduced, and by reducing the error accumulation in the stereo matching process, at the same time, on the basis of AANet, a feature extraction network with fewer parameters and simpler is constructed to obtain better local features, and then through the intra-scale aggregation module and the inter-scale aggregation module, the network is efficiently obtained by using deformable convolution and traditional cross-scale aggregation methods, respectively. High-quality disparity maps.

[0009] The technical solution of the present invention is:

[0010] A binocular stereo matching method, the steps of which include:

[0011] 1) The feature extraction network extracts features from the left and right images to be matched, and obtains feature maps of N resolutions of the left image and feature maps of N resolutions corresponding to the right image; where N is an integer greater than 1;

[0012] 2) The feature maps of the same resolution corresponding to the left and right images are correlated to form a 4D matching cost volume, thereby obtaining N matching cost volumes, and the N matching cost volumes are locally aggregated through the in-scale aggregation module to obtain N new matching cost volumes with the same resolution as the original ones;

[0013] 3) The N new matching cost volumes obtained in step 2) are fused through the inter-scale aggregation module to obtain the final matching cost volumes required for N disparity calculations;

[0014] 4) According to the final matching cost volume, a soft argmin function is used to obtain the disparity maps corresponding to N different resolutions; then each disparity map is upsampled and input into StereoDRNet to obtain the final predicted disparity map.

[0015] Furthermore, the feature extraction network first downsamples the input image, and then uses the first residual network to process the input image to obtain a feature map M1 with a resolution of one-third of the input image; then the feature map M1 is input into the second residual network for processing to obtain a feature map M2 with a resolution of one-sixth of the input image; finally, the feature map M2 is input into the third residual network for processing to obtain a feature map M3 with a resolution of one-twelfth of the input image.

[0016] Furthermore, the intra-scale aggregation module is implemented by Get a new matching price roll; among them, K represents the matching cost after in-scale aggregation when the disparity is d at pixel p. 2 represents the total number of sampling points, w k represents the aggregation weight of the kth sampling point, p k is the fixed offset of pixel p, △p k is the learnable additional offset, m k is the position weight of the kth point, used to adjust the aggregation weight w k , function C is the matching cost calculation function.

[0017] Furthermore, the inter-scale aggregation module aggregates each matching cost volume with each other matching cost volume except itself. When the resolution of the aggregated matching cost volume is higher than the resolution of the current matching cost volume, the aggregated matching cost volume performs a downsampling operation. When the resolution of the aggregated matching cost volume is smaller than the resolution of the current matching cost volume, the aggregated matching cost volume performs an upsampling operation, and the final matching cost volume is obtained through the inter-scale aggregation module.

[0018] Further, through , Sum the sampling results to obtain the matching cost volume Among them, represents the matching cost volume of the r-th scale after in-scale aggregation; the function f r is divided into three cases. When r = s, it is an identity mapping function. That is, when the resolutions are the same, the matching cost volume of the r-th scale does not undergo any transformation. When r < s, that is, when the resolution of the matching cost volume of the r-th scale after in-scale aggregation is greater than the resolution of the matching cost volume of the current s-th scale to be aggregated, then downsample the matching cost volume of the r-th scale to make its resolution consistent with that of the matching cost volume of the s-th scale. If r > s, that is, when the resolution of the matching cost volume of the r-th scale after in-scale aggregation is less than the resolution of the matching cost volume of the current s-th scale to be aggregated, then upsample the matching cost volume of the r-th scale and then perform a 1x1 convolution. s = 1~S, and S has the same value as N.

[0019] Further, the network structures of the first residual network, the second residual network, and the third residual network are all first a 2-layer residual network with a convolution kernel of 1X1, then a 1-layer residual network with a first convolution kernel of 3X3 and a second convolution kernel of 1X1, and finally a 2-layer residual network with a convolution kernel of 1X1. The entire first residual network, second residual network, and third residual network are sequentially connected by these three parts.

[0020] The technical solution adopted by the present invention to solve its technical problems is the deep network - AEDNet.

[0021] 1 Network structure design

[0022] According to the four steps of the stereo matching algorithm: matching cost calculation, matching cost aggregation, disparity calculation and disparity optimization. The network structure design of the present invention firstly includes the matching cost calculation stage. Based on the differences between the stereo matching task and other computer vision tasks, an improved feature extraction network specially used for the stereo matching task is adopted to extract the input left and right images to obtain feature maps of three resolutions of 1 / 3, 1 / 6 and 1 / 12 of the original size of the image. After that, the left and right image features of the three resolutions are constructed into a multi-scale matching cost volume through relevant operations. In the cost aggregation stage, in order to replace the cumbersome 3D convolution operation, the intra-scale aggregation and inter-scale aggregation methods are adopted, which are composed of three intra-scale aggregation modules (Adaptive Cross-Scale Aggregation, referred to as ISA) and one inter-scale aggregation module (Adaptive Cross-Scale Aggregation, referred to as CSA). In the disparity calculation stage, the soft argmin function is used, that is, the fully differentiable flexible argmin function. The end-to-end stereo matching network GCNet document "A.Kendall,H Martirosyan,S Dasgupta,et al.End-to-End Learning of Geometry and Context for Deep StereoRegression[C].2017IEEE International Conference on Computer Vision(ICCV),Venice,2017:66-75" was first proposed. Through this operation, the matching cost volume can be formed into an initial disparity map. Finally, through upsampling and StereoDRNet, that is, a stereo-based 3D reconstruction network, from the document "Rohan Chabra,JulianStraub,Christopher Sweeney,Richard Newcombe and Henry Fuchs. Stereodrnet: Dilated residual stereonet[J]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019: 11786-11795", uses the refinement module in its network to fuse and optimize the three initial disparity maps to form the final disparity map of the input image. The designed network structure is as follows Figure 1 , the network data processing flow is as follows Figure 2 .

[0023] 2 Feature Extraction Module

[0024] A good matching cost calculation method is the basis of an accurate stereo matching algorithm, and its purpose is to measure the correlation between left and right pixels. It mainly uses similarity functions to measure costs under different disparities. In common end-to-end stereo matching algorithms, an approximate ResNet network structure is usually used in the matching cost stage. GC-Net uses 2D convolution to extract deep features, and then an 8-layer residual network is used to extract unary features after dimensionality reduction. PSMNet first uses basic residual block learning to extract unary features, and then uses dilated convolution to expand the receptive field for feature extraction. GA-Net uses stacked hourglass networks to extract features. In addition to solving the degradation problem of deep networks, the ResNet network itself is modified based on the VGG (Visual Geometry Group) network. ResNet and VGG are the champions of the ILSVRC (ImageNet Large Scale Visual Recognition Challenge), and their main task is image classification. Since image classification and stereo matching are two different image tasks, the main difference is that image classification tasks require the translation invariance of images, that is, the output classification results do not change no matter how much the input image is translated. In addition, the stacked hourglass network is mainly used for human pose estimation tasks, which require that it is not affected by perspective and position, that is, translation and rotation. The stereo matching task is sensitive to image translation, requiring that once the input image is translated, the image result should also change accordingly.

[0025] In addition, the stereo matching task does not completely require semantic level features. The image classification task requires highly abstract semantic level features to make a more accurate judgment on the image content type. The stacked hourglass network is to use multi-scale features to recognize postures. If the recognition result is only the features extracted by the convolution of the last layer, it is easy to cause information loss. In fact, human posture detection is an associative task. Adjacent body joints have a certain correlation, but different joints have different feature points, which shows that each body joint does not have the best recognition effect on the output of the same convolutional network layer. This requires the network to use the results of each layer of the network output at the same time. However, the stereo matching task is mainly to find the correspondence between the pixels in the left and right images, so as to find the same-name points in the left and right images and then calculate the disparity. Therefore, the stereo matching task should pay more attention to the local detail description ability of the features extracted by the feature extraction network in different spatial positions.

[0026] In summary, the present invention adopts an improved network SimpleResNet, namely SesNet, which is specially used for stereo matching tasks, for feature extraction. Since the purpose of the present invention is to reduce the number of parameters in the feature extraction stage and thus reduce the running time, three feature maps with different resolutions are obtained by continuously passing through three channels of SesNet, and after obtaining the three feature maps with different resolutions, the feature map pyramid network (Feature Pyramid Networks, FPN for short) is used. The algorithm flow of the feature extraction module is as follows: Figure 3 Among them, M2 and M3 represent the output layers after SesNet and dimensionality reduction operation, and P1, P2, and P3 represent the output layers after SesNet and FPN.

[0027] The main idea of ​​SimpleResNet design is to consider that the larger the receptive field, the lower the difference of the obtained features due to the overlap of the regions in the image, and propose to obtain features with a lower degree of abstraction by limiting the size of the convolution kernel. At the same time, the present invention reduces the corresponding convolution layer, and realizes that while reducing the amount of parameters, the extracted features have stronger local detail expression and difference by increasing the self-position information that the network can extract, that is, the detail information. The feature network improved according to SimpleResNet is called SesNet, and the specific network structure is shown in Table 1, where the specific network structure of the residual block in the table is represented by square brackets, such as the conv1_x layer, whose specific network structure is a residual network with a 2-layer convolution kernel of 1X1, and the output feature map channel is 32. The usage of batch normalization (Batch nomalization) and ReLU (Rectified Linear Units) is consistent with that of ResNet, and H and W represent the height and width of the input image respectively. The present invention inputs the processed left and right images into the network, firstly refers to ResNet and uses a 7x7 convolution kernel in the first layer of the entire feature extraction network to downsample the input image, the purpose is to retain the information of the original image as much as possible, and only executes in the layer closest to the input image, that is, the first layer, and then enters the first layer of SesNet network, that is, the conv1_x layer in Table 1 to obtain a feature map M1 with a resolution of one-third of the input image, and then passes M1 through the second layer of SesNet network, that is, the conv2_x layer in Table 1 to obtain a feature map M2 with a resolution of one-sixth of the input image, and finally passes M2 through the last layer of SesNet network to obtain a feature map M3 with a resolution of one-twelfth of the input image, and the obtained three feature maps are passed through the feature pyramid network as shown in FIG. Figure 3In order to reduce memory usage, M1 is not included in the pyramid, and only a 1x1 convolution layer is used to generate a high-resolution feature image. The feature pyramid first upsamples M3 by 2 times to obtain image features with the same spatial size as M2, and adds M2 element by element. The 1x1 convolution is to reduce the number of convolution kernels so that feature images of the same size can be better fused. For subsequent iterations, M3 is also attached with a 1x1 convolution layer to generate a low-resolution feature image. Finally, in order to reduce the aliasing effect of upsampling, a 3x3 convolution is attached to each feature image to form the final result of the feature extraction stage: P1, P2, and P3, which are left and right image features of three resolutions, respectively, for a total of 6 feature maps.

[0028] Table 1 SesNet network structure parameters

[0029]

[0030] 3. Cost aggregation module

[0031] The main purpose of cost aggregation is to make the calculated matching cost value more accurately reflect the relationship between adjacent pixels. Because the pixel-based matching cost value is easily affected by noise, the cost aggregation operation can establish the connection between adjacent pixels to obtain a new disparity space image. The end-to-end stereo matching algorithm usually obtains a 4D matching cost volume (height*width*disparity*feature size) by combining the left and right feature maps extracted in the previous step, and then uses 3D convolution operation to perform cost aggregation to obtain the final disparity map. However, since the obtained matching cost volume is a 4D tensor, it itself has a high parameter amount. In order to reduce the discrete cost of errors and make the calculated disparity smoother while maintaining a high accuracy, a larger adjacent pixel range is usually used as a constraint, that is, a larger convolution kernel is used. The 3D convolution calculation itself is highly complex, and expanding the 3D convolution kernel can only lead to an increase in the number of parameters. Therefore, the method of using 3D convolution for cost aggregation, although with high accuracy, has the disadvantages of high parameter amount and memory usage. Based on the idea of ​​reducing the number of network parameters and making the network compact and lightweight, the present invention adopts two modules in the cost aggregation stage, namely, the intra-scale aggregation module (Adaptive Intra-Scale Aggregation) and the cross-scale aggregation module (Adaptive Cross-Scale Aggregation). These two modules are used to replace the traditional 3D convolution based on the matching cost volume, so as to improve the running rate, reduce the running loss and keep the running result at a high accuracy. The present invention is consistent with the traditional end-to-end stereo matching algorithm in the cost aggregation stage. The matching cost volume is first obtained. The specific operation is to perform relevant operations on the left and right feature maps extracted in the feature extraction stage, thereby obtaining a 4D matching cost volume, and then perform cost aggregation operations on the matching cost volume. The specific operation is to first use the intra-scale aggregation module to make the cost of the matching cost volume of each scale more effective, and then use the inter-scale aggregation module to fuse the matching cost volume information of different scales. The present invention uses these two modules to replace the traditional 3D convolution based on matching cost volume, while improving the operating rate and reducing the operating loss, so that the operating results maintain a high accuracy.

[0032] 3.1 Intra-scale Aggregation Module

[0033] The in-scale aggregation module performs cost aggregation operations on the matching cost volume constructed by the features of the left and right images at the same resolution. It mainly performs local cost aggregation operations on the matching cost volume. The local cost aggregation in the traditional stereo matching method uses regular convolution of a fixed window for the matching cost volume, that is, the sampling points are distributed in a grid shape within a window of a certain size. The local cost aggregation formula in the traditional stereo matching method is (1).

[0034]

[0035] Where p and q represent two pixels respectively, and pixel q belongs to the adjacent pixel of p. It means that at pixel p, the disparity is d after cost aggregation, C(d,q) means that at pixel q, the disparity is d original, that is, the matching cost value without cost aggregation, and w(p,q) is the aggregation weight of pixel q for pixel p.

[0036] There are generally two cases of disparity discontinuity. One is the cost discreteness problem caused by the matching cost calculation, and the other is the disparity discontinuity caused by the difference in disparity values ​​between different objects or between objects and backgrounds. The local cost aggregation in the traditional stereo matching method treats these two cases indiscriminately. Although the convolutional neural network can automatically learn the weight w, it cannot adaptively perform sampling, which easily leads to the common problem of coarse edge (edge-fattening) of object edges and subtle structures. To address the disparity discontinuity of object edges and subtle structures, the present invention adopts an adaptive in-scale aggregation module (ISA). Similar to the local cost aggregation in the traditional stereo matching algorithm, this module adopts a feature representation based on sparse points, and draws on deformable convolution for adaptive sampling to reduce the impact of foreground and background on cost aggregation, so as to achieve more efficient cost aggregation. The cost aggregation formula of the in-scale aggregation module is (2). In the cost aggregation stage, the present invention first adopts an adaptive scale aggregation module (ISA), which is similar to the local cost aggregation in the traditional stereo matching algorithm. The obtained matching cost volumes of three different scales are respectively input into the ISA module. This module adopts a feature representation based on sparse points, from which deformable convolution is used. Through continuous training, adaptive sampling is performed. Each original cost value is aggregated by the surrounding 9 adaptive sampling points, so as to obtain three new matching cost volumes with the same scale as the previous ones. The new matching cost value is calculated by the cost aggregation formula (2). This can reduce the problem of abnormal matching cost values ​​caused by mismatching and reduce the influence of foreground and background on cost aggregation, so as to achieve more efficient cost aggregation.

[0037]

[0038] in K represents the matching cost after in-scale aggregation when the disparity is d at pixel p. 2 represents the total number of sampling points. The convolution kernel used in the present invention is a 3x3 deformable convolution, so K = 3, w k represents the aggregation weight of the kth point, p k is a fixed offset of pixel p, and △p k is the learnable additional offset, m kis the position weight of the kth point, which is specifically reflected in the distance relationship between the sampling point and the pixel point, and is mainly used to adjust the aggregation weight w k , function C is the matching cost calculation function.

[0039] 3.2 Inter-scale Aggregation Module

[0040] The inter-scale aggregation module is used to aggregate features between matching cost volumes of different scales. For textureless or weakly textured areas, such as the sky, glass, and walls, the coarse scale obtained by downsampling can better extract discriminative features, but for some detail features, higher resolution feature representation is required, so multi-scale aggregation is also a common aggregation method, and the use of this module in the present invention makes up for the missing global information in the matching cost calculation to a certain extent. The present invention draws on the traditional cross-scale aggregation method, and inputs the three matching cost volumes of different scales obtained by the intra-scale aggregation module into the inter-scale aggregation module. The three matching cost volumes are respectively aggregated with the other two matching cost volumes except themselves. When the resolution of the aggregated matching cost volume is higher than the resolution of the current matching cost volume, the aggregated matching cost volume is downsampled. When the resolution of the aggregated matching cost volume is smaller than the resolution of the current matching cost volume, the aggregated matching cost volume is upsampled. The specific operation is to divide the six matching cost volumes of the left and right images after intra-scale aggregation into two parts according to the left and right images, and obtain the final matching cost volumes of three different resolutions for the left and right images through the inter-scale aggregation module. The aggregation formula of the inter-scale aggregation module is as shown in formula (3) and formula (4).

[0041]

[0042]

[0043] in represents the matching cost volume of the s-th scale after inter-scale aggregation, a total of S scale matching cost volumes, S is 3 in the present invention, and represents the matching cost volume after in-scale aggregation at the rth scale, f r It is a general function representation of multiple scale aggregation, that is, the aggregation method is represented by the function f r OK. Function f rThere are three cases. When r = s, it is an identity mapping function. That is, when the resolutions are the same, the matching cost volume at the r-th scale is not transformed. When r < s, the matching cost volume at the r-th scale needs to be downsampled to make its resolution consistent with that of the s-th scale. The specific operation is to use a 3x3 convolutional network with a stride of 2 for s - r steps. At the same time, when r > s, the matching cost volume at the r-th scale needs to be upsampled by bilinear interpolation and then followed by a 1x1 convolution to enhance the learning ability of the network and ensure cost smoothness.

[0044] The beneficial effects of the present invention are as follows:

[0045] The present invention proposes an end-to-end deep network for stereo matching, AEDNet, which is improved based on AANet. In the feature extraction stage, SesNet is used to reduce the number of parameters of the overall network. Among them, simple convolutions are used to extract local information for the stereo matching task, and feature information at different scales is fused through FPN. Three-resolution matching cost volumes are constructed, and through the intra-scale aggregation module and inter-scale aggregation module without 3D convolutions. Compared with the basic network, the network of the present invention has fewer parameters. Although the error rate of the matching points is relatively high, compared with other stereo matching networks, the network of the present invention can efficiently achieve stereo matching while ensuring accuracy, and in the case of a poor hardware environment, the network of the present invention will have obvious advantages and can have higher operating efficiency.

[0046] The present invention first evaluates the model by using different settings. Among them, FPN represents the Feature Pyramid Networks. ResNet+FPN represents the feature extraction network in the basic network AANet. The structure of ResNet is similar to ResNet-50, and deformable convolutions with a convolution kernel of 3 are used for feature extraction. SesNet+FPN refers to the network AEDNet proposed by the present invention. SimpleResNet represents SimpleResNet, that is, the feature extraction network dedicated to the stereo matching task. SesNet represents the improved SimpleResNet in the feature extraction stage of the present invention. The comparison of each setting network on the SceneFlow test set is shown in Table 2. Among them, the running time refers to the time used to process binocular images with a resolution of 576x960. The end-point-error (EPE) refers to the average disparity error between the predicted disparity value and the true value in pixels. FLOPs (FLoating point Operations), that is, the number of floating-point operations, is usually used to measure the complexity of the network model. MACC (Multiply-ACCumulate operation), also known as MAdd, that is, the multiply-accumulate operation, is usually used to measure the computational amount of the model.

[0047] Table 2. Evaluating models using different settings

[0048]

[0049] As shown in Table 2, the SesNet model has improved accuracy and running time compared with the SimpleResNet model. The experimental results show that SesNet has certain advantages. The network AEDNet proposed in this invention has reduced the number of parameters by nearly 25% compared with the reference network AANet. The number of parameters has increased from 4x10 6 down to 3x10 6 This makes it possible to transplant the stereo matching network model to small computing devices, such as mobile platforms. While the matching rate is reduced to a certain extent, there is a relative reduction in running time, but the reduction is not large. There are probably two reasons for this. First, the improvement of the present invention based on AANet is mainly in the feature extraction stage. The end-to-end network has an advantage in running time compared to other networks, and the impact of improving a certain part is relatively small. Second, the present invention uses simple convolution operations to replace complex convolutions, but the corresponding network structure becomes complex, and at the same time leads to a decrease in the accuracy of the proposed network. The present invention compares the network AEDNet with the basic network AANet by calculating the disparity of the Sceneflow dataset. The network of the present invention performs better than AANet in some areas of the Sceneflow dataset, including reflective objects. The pipes made of reflective materials maintain continuity in the disparity map estimated by the network of the present invention, and objects with repeated textures, such as rectangular wooden blocks, have a certain gap with the real disparity map, but the network of the present invention is better than AANet in processing repeated textures. In the case of mutual occlusion of objects, the network of the present invention can estimate the disparity of the cutting tool behind the wooden frame, while the AANet network does not fully display the disparity of the cutting tool, and the parallax between the handle and the blade is greatly separated, and the leaves of the green plants are mismatched in the edge area of ​​the green plants due to mutual occlusion.

[0050] The present invention uses the best model obtained by training on the Sceneflow data set to perform fine-tuning training using the training sets of the KITTI2012 data set and the KITTI2015 data set, and uses the corresponding test sets to verify network performance, wherein the KITTI2012 data set has 194 groups of binocular training image pairs and 195 groups of binocular test image pairs, respectively, and the KITTI2015 data set has 200 groups of binocular training image pairs and 200 groups of binocular test image pairs, during the training process, the original image is randomly cropped to 336x960 as input, each training is 1000 epochs, the batch size is set to 1, the learning rate starts from 0.0001, and after the model is trained for 400 epochs, the learning rate is reduced by half every 200 epochs, and after the training is completed for 900 epochs, the learning rate is further reduced by half.

[0051] The fine-tuned trained network model is evaluated according to the evaluation criteria of the KITTI evaluation website and compared with other excellent algorithms. The evaluation results of each algorithm on the KITTI2012 data set and the KITTI2015 data set are shown in Tables 3 and 4. The data in the table are from the KITTI evaluation website or publicly published papers with a certain degree of credibility. The benchmark used is that when the endpoint error of the matching point in the image is less than 3px, the pixel is correctly estimated. In addition, to ensure better comparison of network performance. The AEDNet proposed in the present invention and its reference network AANet are trained and run under the same experimental conditions.

[0052] Table 3 Evaluation results of each network on the KITTI2012 dataset

[0053] Network Name Out-Noc / % Out-All / % Avg-Noc / px Avg-All / px Running time / s AcfNet 1.17 1.54 0.5 0.5 0.48 GwcNet 1.32 1.80 0.5 0.5 0.32 GA-Net 1.36 1.70 0.5 0.5 0.36 SGNet 1.38 1.85 0.5 0.5 0.6 PSMNet 1.49 1.89 0.5 0.6 0.41 SegStereo 1.68 2.03 0.5 0.6 0.6 GC-Net 1.77 2.06 0.6 0.7 0.9 AANet 2.54 3.18 0.6 0.7 0.09 DispNetC 4.11 4.65 0.9 1.0 0.06 Flow2Stereo 4.58 5.11 1.0 1.1 0.05 DispSegNet 4.68 5.66 0.9 1.0 0.9 ikB 4.68 6.13 1.0 1.4 7.92 The network of the present invention 3.40 4.11 0.7 0.8 0.08

[0054] Note: Out represents the proportion of mismatched points in the disparity map, Avg represents the average error of matching points in the disparity map, Noc represents the non-occluded area, and All represents the entire area.

[0055] Table 4 Evaluation results of each network on the KITTI2015 dataset

[0056] Network Name D1-bg / % D1-fg / % D1-all / % Running time / s GANet 1.48 3.46 1.81 1.8 AcfNet 1.51 3.80 1.89 0.48 EdgeStereo 1.84 3.30 2.08 0.32 SegStereo 1.88 4.07 2.25 0.6 GC-Net 2.21 6.16 2.87 0.9 AANet 2.54 4.58 2.88 0.09 DWARF 3.20 3.94 3.33 0.14 DispNetC 4.32 4.41 4.34 0.06 MADnet 3.75 9.20 4.66 0.02 DeepCostAggr 5.34 11.35 6.34 0.03 The network of the present invention 2.93 5.11 3.29 0.08

[0057] Note: D1 represents the proportion of mismatched points in the disparity map, bg represents the background area, fg represents the foreground area, and all represents the entire area.

[0058] It can be seen from Tables 3 and 4 that since the network of the present invention reduces the number of network parameters by limiting the size of the convolution kernel and using ordinary convolution in the feature matching stage, the running time is reduced to a certain extent. Although the accuracy is not as good as the basic network, compared with other high-precision stereo matching networks, the network of the present invention has certain advantages in running time. Compared with other networks that consider network operation efficiency and focus on real-time performance, the accuracy of the network of the present invention is slightly higher. Experiments show that the network proposed in the present invention can efficiently realize stereo matching while ensuring accuracy.

[0059] Among them, AcfNet, GwcNet, PSMNet, SegStereo, and GC-Net in Table 3 are all end-to-end stereo matching algorithms. At the same time, these networks all use 3D convolution in the cost aggregation stage. Compared with the network of the present invention, the fastest processing time of these network models in terms of running time is 0.32s, while the processing time of the network of the present invention can be controlled within 0.1s, which verifies that the intra-scale aggregation module and the inter-scale aggregation module improve the real-time performance after reducing a large number of redundant parameters, but at the same time also sacrifices the accuracy of the network. DispNetC and Flow2Stereo combine optical flow estimation and stereo matching, and finally directly use 2D convolution regression to obtain the disparity map, resulting in poor effect of the final disparity map. In addition, the models of Flow2Stereo and DispSegNet are obtained through unsupervised training. Therefore, compared with the supervised network proposed in the present invention, they can extract fewer effective features, which affects the final result of the network.

[0060] In Table 4, DWARF and MADnet are lightweight, especially MADnet aims to improve and optimize with low-resolution disparity, but this approach ignores the local detail information of the original input image and loses accuracy to a certain extent. The present invention fuses feature information at different scales through feature pyramid and cross-scale aggregation to provide more effective information for disparity calculation. Finally, DeepCostAggr is a stereo matching algorithm that combines deep learning with traditional matching algorithms. Compared with end-to-end stereo matching algorithms, it has a large error accumulation, and the design of its network matching cost calculation affects the final disparity map effect.

[0061] The network AEDNet proposed by the present invention and the basic network AANet are compared through the disparity estimation results on the KITTI2012 test set and the KITTI2015 test set. In the KITTI2012 test set, the network of the present invention can more accurately calculate the disparity value of vehicles, especially car windows, and the disparity information of slender objects, such as tree branches, is more correct. In the KITTI2015 test set, the network of the present invention has better performance in object edges and weak texture areas, such as circular road signs, tree trunks, telephone poles, grass, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Diagram of the adaptive end-to-end deep network structure for stereo matching.

[0063] Figure 2 It is the network data processing flow.

[0064] Figure 3 This is the algorithm flow of the feature extraction module. DETAILED DESCRIPTION

[0065] The present invention proposes an end-to-end network suitable for stereo matching. First, the dataset and training process used in the network training method of the present invention are introduced. Then, each model is evaluated by different settings, and the advantages of the proposed network in terms of time and video memory usage are confirmed by the performance on the Sceneflow test set. Finally, the algorithm performance is fine-tuned and verified on the KITTI2012 and KITTI2015 datasets.

[0066] 1. Dataset and training process

[0067] The present invention first uses the Sceneflow dataset to train the network. The datasets commonly used in previous stereo matching algorithms, such as KITTI and Middlebury, have fewer training images. In order to improve the performance of stereo matching networks, especially end-to-end stereo matching networks, a large amount of data is required for training. In 2016, CVPR (Computer Vision and Pattern Recognition) proposed to build a large-scale synthetic dataset, namely the Sceneflow dataset. The Sceneflow dataset has about 39,000 pairs of training images, and the resolution of each pair of images is 576x960 pixels. It is called a synthetic dataset because this dataset is composed of three-dimensional scenes and is rendered from these three-dimensional scenes. The Sceneflow dataset contains relevant completely dense and accurate label information such as parallax, optical flow and scene flow. The Sceneflow dataset is divided into three subsets, namely FlyingThing3D, Monkaa and Driving. Among them, the FlyThing3D content is the background composed of random geometric structures and the foreground of dynamic objects obtained from ShapeNet. Monkaa is a large and diverse dataset obtained by modeling scenes in cartoons. In terms of image generation, Driving is similar to Monkaa, but in terms of image content, Driving mainly provides data on driving scenes. Such a huge dataset enables deep networks to be fully trained.

[0068] The KITTI dataset is currently the largest computer vision algorithm evaluation dataset in the world for autonomous driving scenarios. The KITTI dataset contains real binocular image data and LIDAR point cloud data collected from 61 scenes such as urban areas, rural areas, and highways. Among them, the KITTI 2012 dataset and the KITTI 2015 dataset are commonly used datasets for binocular stereo matching. The resolution of each pair of images is 384x1248 pixels. The KITTI 2012 dataset is the first static stereo matching dataset for outdoor scenes. It consists of 194 training image pairs with depth labels and 195 test image pairs without depth labels. It is saved in lossless png format, and the collected depth map is semi-dense, covering about one-third of the image. The subsequent KITTI2015 dataset was expanded to consist of 200 training scenes and 200 test scenes, with 4 color images in each scene, also saved in lossless png format. The collection method of this dataset is similar to that of KITTI 2012. Unlike the previous dataset, this dataset is collected in dynamic scenes, mainly moving targets such as cars.

[0069] The hardware configuration of the experimental environment of the present invention is Inter(R)CoreTM i7-6700 CPU@3.40GHz x 8, a single NVIDIA RTX 2080GPU is used for network training, the memory is 15.6GiB, the hard disk space size is 1.0TB, the Ubuntu20.04 system is used, the GNOME version is 3.36.3, and the window system is X11. The software configuration uses the general parallel computing architecture version of cuda10.0, and uses the corresponding GPU acceleration library cudnn. In addition, an Anaconda virtual environment is built, and its important environment dependency packages are shown in Table 5.

[0070] Table 5 Important dependency package versions

[0071]

[0072]

[0073] Adam is used as the optimizer in network training. Its method parameters are β1=0.9, β2=0.999. All training sets of the Sceneflow dataset, totaling 35,454 image pairs, are used for training, and evaluated on its standard test set, totaling 4,370 image pairs. The original image is randomly cropped to 288×576 as input, and the input image is normalized. Random color enhancement and vertical flipping are used, and the maximum disparity is set to 192 pixels. The model of the present invention is trained for 64 epochs, the batch size is set to 4, and the learning rate starts from 0.001. To prevent the model from failing to converge, the learning rate is reduced by half every 10 epochs after the model is trained for 20 epochs.

[0074] The specific embodiments and drawings of the present invention disclosed above are intended to help understand the content and essence of the present invention, and are not intended to limit the present invention. Any modifications, substitutions and improvements made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A binocular stereo matching method, the steps of which include: 1) The feature extraction network extracts features from the left and right images to be matched, and obtains feature maps of N resolutions for the left image and feature maps of N resolutions for the right image; Wherein N is an integer greater than 1; 2) The feature maps of the same resolution corresponding to the left and right images are correlated to form a 4D matching cost volume; then the in-scale aggregation module is used to perform a local cost aggregation operation on each 4D matching cost volume to obtain a new matching cost volume with the same resolution as the original matching cost volume; 3) The N new matching cost volumes obtained in step 2) are merged through the inter-scale aggregation module to obtain the final matching cost volume; wherein, the inter-scale aggregation module aggregates each matching cost volume with each other matching cost volume except itself, and when the resolution of the aggregated matching cost volume is higher than the resolution of the current matching cost volume, the aggregated matching cost volume is downsampled; when the resolution of the aggregated matching cost volume is smaller than the resolution of the current matching cost volume, the aggregated matching cost volume is upsampled, and then , sum the sampling results to get the matching cost volume in, represents the matching cost volume of the rth scale after in-scale aggregation; function f r There are three cases: when r=s is an identity mapping function, that is, when the resolution is the same, the matching cost volume of the rth scale does not undergo any transformation; when r<s, that is, when the resolution of the matching cost volume of the rth scale after in-scale aggregation is greater than the resolution of the matching cost volume of the sth scale to be aggregated, the matching cost volume of the rth scale is downsampled to make it consistent with the resolution of the matching cost volume of the sth scale; if r>s, that is, when the resolution of the matching cost volume of the rth scale after in-scale aggregation is less than the resolution of the matching cost volume of the sth scale to be aggregated, the matching cost volume of the rth scale is upsampled and then followed by a 1x1 convolution, s=1~S, S is the same as N; 4) According to the final matching cost volume, the disparity maps corresponding to N different resolutions are obtained; then the obtained disparity maps are upsampled and input into StereoDRNet to obtain the final predicted disparity map.

2. The method according to claim 1, characterized in that The feature extraction network first downsamples the input image, and then uses the first residual network to process the input image to obtain a feature map M1 with a resolution of one-third of the input image; then the feature map M1 is input into the second residual network for processing to obtain a feature map M2 with a resolution of one-sixth of the input image; finally, the feature map M2 is input into the third residual network for processing to obtain a feature map M3 with a resolution of one-twelfth of the input image.

3. The method according to claim 1 or 2, characterized in that The intra-scale aggregation module is implemented by Get a new matching price roll; among them, K represents the matching cost after in-scale aggregation when the disparity is d at pixel p. 2 represents the total number of sampling points, w k represents the aggregation weight of the kth sampling point, p k is the fixed offset of pixel p, Δp k is the learnable additional offset, m k is the position weight of the kth point, used to adjust the aggregation weight w k , function C is the matching cost calculation function.

4. The method according to claim 2, characterized in that The first residual network, the second residual network, and the third residual network are all first passed through a residual network with a 2-layer convolution kernel of 1X1, then passed through a residual network with a first-layer convolution kernel of 3X3 and a second-layer convolution kernel of 1X1, and finally connected in sequence with a 2-layer convolution kernel of 1X1.

Images