Binocular stereo vision matching network construction method and binocular stereo vision matching method

Abstract

The application discloses a binocular stereo vision matching network construction method and a binocular stereo vision matching method, and belongs to the field of binocular stereo vision matching. The method comprises the following steps: collecting left images, right images and corresponding disparity maps to construct a data set, training and testing a stereo matching network to be trained, and obtaining a binocular stereo vision matching network. The stereo matching network comprises a disparity edge prediction branch for performing disparity edge prediction on images, and a disparity estimation backbone network comprising a cost cube construction subnetwork for constructing a cost cube, a cost aggregation module for regularizing the cost cube to obtain an aggregated cost volume, an aggregated cost optimization module for suppressing edge regions of the aggregated cost volume according to a disparity edge prediction map of a reference image, and a disparity regression module for performing upsampling on the optimized aggregated cost volume and then performing regression to obtain a disparity map. The application can accurately realize matching of edge and occlusion regions, and improve matching precision and reliability.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of binocular stereo vision matching, and more particularly relates to a binocular stereo vision matching network construction method and a binocular stereo vision matching method. BACKGROUND

[0002] Stereo vision plays an important role in computer vision. In practical applications, binocular stereo vision technology has become the current focus and hotspot. The essence of binocular stereo vision is to obtain two-dimensional plane images, i.e. left and right images, of an object in the same environment through two cameras, and to extract spatial three-dimensional information from the two-dimensional plane image information to realize the understanding of the shape, size and position of the object, so it is widely used in robot navigation, non-contact measurement, medical image processing, three-dimensional reconstruction and other fields.

[0003] The main purpose of stereo matching is to find a one-to-one corresponding point between the two images taken by the binocular camera, and to match it to obtain the disparity of the corresponding point. In the matching process, the left image is usually taken as the reference image, and the right image is taken as the image to be matched to search and match the corresponding relationship.

[0004] Traditional stereo matching using feature extraction algorithms can usually only identify simple shape and texture features, and cannot well handle complex scenes and changes in lighting, etc., resulting in poor performance in practical applications. With the development of deep learning and the support of big data, the performance of stereo matching has been greatly improved. However, in complex and variable actual scenes, the current stereo vision technology often has difficulty in accurately matching the edges and occluded areas, thus there are deficiencies in precision and reliability. SUMMARY

[0005] In view of the defects and improvement needs of the prior art, the present application provides a binocular stereo vision matching network construction method and a binocular stereo vision matching method, which aims to introduce disparity edge information to guide binocular stereo vision matching, so that the matching of edges and occluded areas can be accurately realized in complex and variable actual scenes, and the matching precision and reliability are improved.

[0006] To achieve the above-mentioned purpose, according to one aspect of the present application, a binocular stereo vision matching network construction method is provided, comprising:

[0007] Collecting left images, right images and corresponding disparity maps to construct a data set, after establishing a stereo matching network to be trained, using the data set to train and test the stereo matching network to be trained, obtaining a binocular stereo vision matching network for estimating the disparity map between the left image and the right image;

[0008] The stereo matching network comprises a disparity edge prediction branch and a disparity estimation backbone network.

[0009] The parallax edge prediction branch is configured to perform parallax edge prediction on the image to obtain a corresponding parallax edge prediction map;

[0010] The parallax estimation backbone network comprises:

[0011] The cost cube construction subnetwork is configured to construct a cost cube between a left image and a right image to be matched; in the left image and the right image to be matched, one is a reference image and the other is a to-be-matched image;

[0012] The cost aggregation module is configured to regularize the cost cube to obtain an aggregated cost volume;

[0013] The aggregated cost optimization module is configured to identify an edge region of the aggregated cost volume according to the parallax edge prediction map of the reference image predicted by the parallax edge prediction branch and suppress the edge region to obtain an optimized aggregated cost volume;

[0014] The parallax regression module is configured to perform regression on the optimized aggregated cost volume after up-sampling the optimized aggregated cost volume to be consistent with the size of the reference image to obtain a parallax map.

[0015] Further, the aggregated cost optimization module suppresses the edge region of the aggregated cost volume according to the following formula:

[0016] C new (x, y) = C(x, y) x (2 - P(x, y))

[0017] wherein C new is the optimized aggregated cost volume, C is the aggregated cost volume, P is the parallax edge prediction map of the reference image, and (x, y) represents a pixel coordinate.

[0018] Further, the cost cube construction subnetwork comprises:

[0019] The left image low-scale feature extraction module is configured to perform original-scale feature extraction on the left image to obtain low-scale features of the left image;

[0020] The left image high-scale feature extraction module is configured to perform a spatial pyramid pooling operation on the low-scale features of the left image to obtain high-scale features of the left image;

[0021] The left image feature fusion module is configured to perform weighted fusion on the low-scale features and the high-scale features of the left image to obtain fused features of the left image; during the fusion, the low-scale features have a higher weight in the edge region, and the high-scale features have a higher weight in the non-edge region, and the edge region and the non-edge region are determined according to the parallax edge prediction map of the left image predicted by the parallax edge prediction branch;

[0022] The right image low-scale feature extraction module is configured to perform original-scale feature extraction on the right image to obtain low-scale features of the right image;

[0023] a right image high scale feature extraction module configured to perform a spatial pyramid pooling operation on low scale features of the right image to obtain high scale features of the right image;

[0024] a right image feature fusion module configured to fuse the low scale features and the high scale features of the right image to obtain fused features of the right image, wherein the low scale features have a higher weight in an edge region and the high scale features have a higher weight in a non-edge region, and the edge region and the non-edge region are determined according to a disparity edge prediction map of the right image predicted by the disparity edge prediction branch;

[0025] and a cost volume construction module configured to concatenate the fused features of the left image and the fused features of the right image at each disparity level according to a predetermined maximum disparity value to obtain a cost cube between the left image and the right image.

[0026] Further, the left image feature fusion module fuses the low scale features and the high scale features of the left image according to the following formula:

[0027]

[0028] The right image feature fusion module fuses the low scale features and the high scale features of the right image according to the following formula:

[0029]

[0030] wherein F new _ l and F new _ r represent the fused features of the left image and the right image respectively, F low _ l and F low _ r represent the low scale features of the left image and the right image respectively, F high _ l and F high _ r represent the high scale features of the left image and the right image respectively, P l and P r represent the disparity edge prediction maps of the left image and the right image respectively, and (x, y) represents a pixel coordinate.

[0031] Further, the disparity edge prediction branch comprises:

[0032] a 2D convolution module configured to extract shallow features of an input image;

[0033] a multi-scale feature extraction module configured to perform spatial pyramid pooling operations of different scales on the shallow features to obtain deep features of multiple scales;

[0034] an upsampling module, configured to upsample the deep features at each scale to be consistent with the size of the input image;

[0035] a feature fusion module, configured to perform convolution operation on the features output by the upsampling module after concatenation to obtain edge features in a single channel;

[0036] and an edge regression module, configured to regress the pixel values in the edge features to the range of [0, 1] to obtain a disparity edge prediction map.

[0037] Further, the stereo matching network to be trained is trained and tested by using a dataset, including:

[0038] performing edge detection on the disparity maps in the dataset to obtain disparity edge maps, and constructing an edge detection sample set together with the corresponding left images and right images;

[0039] training and testing the disparity edge prediction branch in the stereo matching network by using the edge detection sample set;

[0040] freezing the parameters of the disparity edge prediction branch in the stereo matching network, training and testing the stereo matching network by using the dataset, and obtaining the binocular stereo vision matching network.

[0041] Further, when training the disparity edge prediction branch in the stereo matching network by using the edge detection sample set, a weighted cross-entropy loss function is adopted.

[0042] Further, when training the stereo matching network by freezing the parameters of the disparity edge prediction branch in the stereo matching network and using the dataset, a smooth L1 loss function based on disparity is adopted.

[0043] According to another aspect of the present application, a binocular stereo vision matching method is provided, including:

[0044] inputting the left image and the right image to be matched into the binocular stereo vision matching network, and outputting a disparity map between the left image and the right image by the binocular stereo vision matching network;

[0045] converting the disparity map between the left image and the right image into a depth map to complete the matching;

[0046] wherein the binocular stereo vision matching network is established by the binocular stereo vision matching network construction method provided by the present application.

[0047] According to still another aspect of the present application, a computer readable storage medium is provided, comprising a stored computer program which, when executed by a processor, controls the device in which the computer readable storage medium is located to perform the above-mentioned binocular stereo matching network construction method according to the present application, and / or the above-mentioned binocular stereo matching method according to the present application.

[0048] In general, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0049] (1) The binocular stereo matching network established by the present application introduces a disparity edge prediction branch on the basis of the disparity estimation backbone network, which is used to predict edge information, and guides the cost aggregation stage of the disparity estimation backbone network based on the predicted edge information to weaken the cost volume at the edge, so as to reduce the false matching calculation caused by occlusion, improve the matching difficulty of the edge and occlusion area, and accurately realize the matching of the edge and occlusion area in a complex and variable environment, effectively improving the precision and reliability of binocular stereo matching.

[0050] (2) In the preferred scheme of the present application, the present application further guides the feature extraction stage of the disparity estimation backbone network based on the edge information predicted by the disparity edge prediction branch. Specifically, the low-scale and high-scale features are fused based on the edge information to assign weights, so that the low-scale features have a higher weight in the edge area, and the high-scale features have a higher weight in the non-edge area, thereby strengthening the low-scale features at the edge and the high-scale features at the non-edge, fully utilizing the rich edge information carried by the low-scale features, and further improving the matching precision at the edge.

[0051] (3) In the preferred scheme of the present application, when training the stereo matching network, the disparity edge prediction branch therein is first trained, and then the training of the disparity estimation backbone network is completed based on the trained disparity edge prediction branch, so that the training process of the disparity estimation backbone network can be guided by accurate edge information, and the training efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 Structure schematic diagram of the binocular stereo matching network provided by the embodiment of the present application;

[0053] Figure 2 Schematic diagram of the disparity edge prediction branch provided by the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0055] In the present application, the terms "first", "second", etc. (if any) in the present application and the accompanying drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0056] In order to improve the accuracy and reliability of binocular stereo vision matching, the present application provides a binocular stereo vision matching network construction method and a binocular stereo vision matching method, and the overall idea is to improve the matching network structure to introduce edge information to guide binocular stereo vision matching, so that the matching of edge and occlusion area can be accurately realized in complex and variable environment, thereby improving the accuracy and reliability of binocular stereo vision matching.

[0057] Based on the above idea, the present application further analyzes the specific process of binocular stereo vision matching and finds that on the basis of disparity edge prediction, the edge information can be used to guide the feature extraction and cost aggregation process of binocular images, which can effectively improve the matching accuracy and reliability. Experimental results further show that the effect of using edge information to guide cost aggregation is more significant. Based on this, in practical application, based on the specific accuracy and reliability requirements, the feature extraction process can be selectively improved, or the cost aggregation process can be selectively improved, or both processes can be selectively improved.

[0058] The following is an embodiment.

[0059] Embodiment 1

[0060] A binocular stereo vision matching network construction method comprises:

[0061] After collecting left images, right images and corresponding disparity maps to construct a data set, establishing a stereo matching network to be trained, using the data set to train and test the stereo matching network to be trained, and obtaining a binocular stereo vision matching network for estimating the disparity map between the left image and the right image.

[0062] Optionally, in the present embodiment, the data for constructing the data set is derived from SceneFlow and KITTIStereo. In order to facilitate network input, the images in the data set are all cropped to 256x512 size in the present embodiment.

[0063] The stereo matching network constructed in the present embodiment is as follows Figure 1As shown, the embodiment introduces a disparity edge prediction branch on the basis of a traditional network, and adds corresponding modules in the feature extraction part and the cost aggregation part to introduce edge information predicted by the disparity edge prediction branch. Accordingly, in the embodiment, the stereo matching network comprises:

[0064] a disparity edge prediction branch and a disparity estimation backbone network.

[0065] In the stereo matching network, the disparity edge prediction branch is configured to perform disparity edge prediction on the image to obtain a corresponding disparity edge prediction map. In the embodiment, the structure of the disparity edge prediction branch is as shown in Figure 2 , and comprises:

[0066] a 2D convolution module configured to perform multiple convolutions on the input image to extract shallow features of the input image;

[0067] a multi-scale feature extraction module configured to perform a spatial pyramid pooling operation of four different scales on the shallow features to obtain deep features of four scales; the spatial pyramid pooling (SPP) can be connected to a convolution layer to allow the network to input an arbitrary picture and generate a fixed-size output;

[0068] an up-sampling module configured to up-sample the deep features at each scale to a size consistent with the input image;

[0069] a feature fusion module configured to perform convolution on the features output by the up-sampling module after concatenation to obtain an edge feature in a single channel;

[0070] and an edge regression module configured to regress pixel values in the edge feature to a range of [0, 1] through a sigmoid regression function to obtain a disparity edge prediction map.

[0071] As shown in Figure 1 , in the embodiment, the disparity estimation backbone network comprises:

[0072] a cost cube construction sub-network configured to construct a cost cube between a left image and a right image to be matched;

[0073] In the left image and the right image to be matched, one is a reference image and the other is a to-be-matched image; in the embodiment, the left image is specifically taken as the reference image and the right image is specifically taken as the to-be-matched image;

[0074] As shown in Figure 1 , the cost cube construction sub-network specifically comprises:

[0075] a left image low-scale feature extraction module configured to perform original-scale feature extraction on the left image to obtain low-scale features of the left image;

[0076] The left image high-scale feature extraction module is configured to perform a spatial pyramid pooling operation on the low-scale features of the left image to obtain high-scale features of the left image.

[0077] The left image feature fusion module is configured to perform weighted fusion on the low-scale features and the high-scale features of the left image to obtain fused features of the left image. During the fusion, the low-scale features have a higher weight in the edge region, and the high-scale features have a higher weight in the non-edge region. The edge region and the non-edge region are determined according to a disparity edge prediction map of the left image that is predicted by the disparity edge prediction branch.

[0078] The right image low-scale feature extraction module is configured to perform original-scale feature extraction on the right image to obtain low-scale features of the right image.

[0079] The right image high-scale feature extraction module is configured to perform a spatial pyramid pooling operation on the low-scale features of the right image to obtain high-scale features of the right image.

[0080] The right image feature fusion module is configured to perform weighted fusion on the low-scale features and the high-scale features of the right image to obtain fused features of the right image. During the fusion, the low-scale features have a higher weight in the edge region, and the high-scale features have a higher weight in the non-edge region. The edge region and the non-edge region are determined according to a disparity edge prediction map of the right image that is predicted by the disparity edge prediction branch.

[0081] The cost volume construction module is configured to connect the fused features of the left image and the fused features of the right image at each disparity level according to a predetermined maximum disparity value to obtain a cost cube between the left image and the right image. The cost cube is a 4D spatial cost volume with a size of “image width × image height × disparity × feature dimension”.

[0082] In binocular stereo matching, the matching error at the edge is often large. Traditional matching methods often directly construct a cost cube using high-scale features after extracting the high-scale features of the image. However, in the process of extracting the high-scale features, part of the detail information at the edge is lost, which affects the final matching accuracy. In this embodiment, the low-scale features and the high-scale features are fused, and the fusion weight is assigned based on the prediction result of the disparity edge prediction branch, so that the low-scale features have a higher weight in the edge region, and the high-scale features have a higher weight in the non-edge region. In this way, the low-scale features at the edge and the high-scale features at the non-edge are strengthened. Since the low-scale features can better preserve the information at the edge, this embodiment can fully utilize the edge information carried by the low-scale features and effectively improve the matching accuracy at the edge.

[0083] Optionally, in the embodiment, the left image low scale feature extraction module, the left image high scale feature extraction module, the right image low scale feature extraction module and the right image high scale feature extraction module have the same structure as the PSMNet model.

[0084] The left image low scale feature extraction module and the right image low scale feature extraction module are specifically convolutional neural networks sharing weights, the first three layers of the convolutional neural networks are standard convolutional layers, the convolution kernel size is 3x3, the channel number is 32, and the step length is 2, so that the image size is down-sampled to half of the original input. Then, four residual layers are arranged, the first residual layer has three residual blocks, the second residual layer has 16 residual blocks, and the original image is down-sampled to half of the original, the third residual layer has three residual blocks, and a dilated convolution with a dilation rate of 2 is selected, and the fourth residual layer has three residual blocks, and a dilated convolution with a dilation rate of 4 is selected. The convolution kernel of the four residual layers is 3x3. After the above convolution operation, low scale image features are extracted.

[0085] The left image high scale feature extraction module and the right image high scale feature extraction module are specifically spatial pyramid pooling modules, which include four fixed size average pooling layers with sizes of 64x64, 32x32, 16x16 and 8x8 respectively. Then, a 1x1 convolution is used to reduce the feature dimension. Then, the low-dimensional features are up-sampled to the original feature map size by bilinear interpolation. The features of the four scales and the features output by the second layer and the fourth layer in the above residual layer are fused to obtain high scale feature maps.

[0086] Based on the extracted low scale feature maps and high scale feature maps, in the embodiment, the left image feature fusion module fuses the low scale features and the high scale features of the left image according to the following formula:

[0087]

[0088] The right image feature fusion module fuses the low scale features and the high scale features of the right image according to the following formula:

[0089]

[0090] wherein, F new _ l and F new _ r respectively represent the fusion features of the left image and the right image, F low _ l and F low _ r respectively represent the low scale features of the left image and the right image, F high _l and F high _ r respectively represent the high scale features of the left image and the right image, Pl and P r denote the left and right disparity edge prediction maps respectively, (x, y) denotes the pixel coordinate; x denotes multiplication, and denotes concatenation.

[0091] Based on the above fusion formula, the embodiment can effectively enhance the low-scale features at the edges and the high-scale features at the non-edges based on the prediction result of the disparity edge prediction branch.

[0092] As shown in FIG. 1, in the embodiment, the disparity estimation backbone network further includes, after the cost volume construction subnetwork: Figure 1

[0093] a cost aggregation module, configured to regularize the cost volume to obtain an aggregated cost volume;

[0094] an aggregated cost optimization module, configured to identify an edge region of the aggregated cost volume according to a disparity edge prediction map of a reference image predicted by the disparity edge prediction branch and suppress the edge region to obtain an optimized aggregated cost volume;

[0095] and a disparity regression module, configured to regress the optimized aggregated cost volume after up-sampling the optimized aggregated cost volume to the size of the reference image to obtain a disparity map.

[0096] In the embodiment, the aggregated cost optimization module suppresses the edge region of the aggregated cost volume based on the prediction result of the disparity edge prediction branch, which can weaken the cost volume at the edges, reduce the false matching calculation caused by occlusion, and improve the matching difficulty of the edges and the occlusion regions.

[0097] Optionally, in the embodiment, the aggregated cost optimization module suppresses the edge region of the aggregated cost volume according to the following formula:

[0098] C new (x, y) = C(x, y) x (2 - P(x, y))

[0099] wherein C new is the optimized aggregated cost volume, C is the aggregated cost volume, P is the disparity edge prediction map of the reference image, and (x, y) denotes the pixel coordinate; based on the above formula, the embodiment suppresses the edge region of the cost volume while enhancing the non-edge region.

[0100] ​Optionally, in this embodiment, the cost aggregation module and the disparity regression module are similar to the related structures in the PSMNet, specifically, the cost aggregation module adopts a stacked hourglass 3D convolution structure, which includes three hourglass networks; the disparity regression module performs bilinear interpolation on the output of each hourglass network respectively to obtain a disparity map with the same size as the reference image, i.e., WxHxD, where W and H represent the width and height of the image, and D represents the number of disparities; then, for each possible disparity d of each pixel, the matching cost C of the pixel is obtained according to the above-mentioned aggregated cost volume d , and the probability is calculated using the softmax operation σ(·). The final disparity prediction value is calculated by weighted summation according to the probability , and the formula is:

[0101]

[0102] where D max represents the preset maximum disparity.

[0103] After the data set is constructed and the stereo matching network to be trained is constructed, when training, first, the disparity edge prediction branch in the stereo matching network is trained, and the training manner is as follows:

[0104] An edge detection algorithm such as canny or hed is used to perform edge detection on the disparity map in the data set to obtain a disparity edge map, and the corresponding left image and right image form an edge detection sample set;

[0105] The disparity edge prediction branch in the stereo matching network is trained and tested using the edge detection sample set.

[0106] When training the disparity edge prediction branch, a weighted cross-entropy loss function is used. The weight setting formula is as follows:

[0107]

[0108] where weight pixel is the weight corresponding to each pixel point, num pos is the total number of pixels that are regarded as edge regions, and here the pixels with a pixel value greater than 0.5 are regarded as edge region pixels, num neg is the total number of pixels that are regarded as non-edge regions, i.e., the number of pixels with a pixel value less than 0.5. The pixel values of the points come from the edge regression estimated disparity edge prediction map.

[0109] After the training of the disparity edge prediction branch is completed, the disparity prediction backbone network is further trained according to the embodiment, and the specific manner is as follows:

[0110] The parameters of the disparity edge prediction branch in the frozen stereo matching network are frozen, the stereo matching network is trained and tested by using a data set, and a binocular stereo vision matching network is obtained; when training the disparity prediction backbone network, a smooth L1 loss function based on disparity is used, and the formula is as follows:

[0111]

[0112]

[0113] Wherein, N represents the number of pixels, d represents the true value of the disparity, is the predicted value of the network output.

[0114] In the embodiment, the cost aggregation module uses an hourglass stack structure, each hourglass structure calculates a loss function, and the total loss is a weighted average of the three, according to the PSMNet benchmark model, the weight parameters of the loss functions of the three hourglass structures are set as W dips-loss1 =0.5, W dips-loss2 =0.7 and W dips-loss3 =1 in turn. Only the disparity map obtained by the aggregated cost of the last hourglass structure through disparity regression is taken as the final result.

[0115] The parameter settings of the above training process are as follows:

[0116] The improved model is realized based on PyTorch, and the Adam optimizer with β1=0.9 and β2=0.999 is used to train the model. For the KITTIStereo data set, the batch size of each round of training is set to 2, and the maximum disparity is set to 192.

[0117] First, the edge branch and the disparity backbone are trained for 40 rounds in the SceneFlow data set, the learning rate of the edge branch is 1E-7, the disparity backbone is pre-trained for the first 15 rounds, the learning rate is 0.001, and the learning rate is 0.0005 for the last 25 rounds. After the model training is completed, fine-tune on the KITTIStereo data set, divide the training set according to the ratio of 5:1, respectively for training and testing, a total of 300 rounds of training, the initial learning rate is 0.001, and the learning rate is 0.0001 for the last 200 rounds.

[0118] Embodiment 2:

[0119] A binocular stereo vision matching network construction method. The embodiment is similar to the above-mentioned embodiment 1, the difference is that in the embodiment, in the disparity estimation backbone network, the cost cube construction subnetwork is consistent with the PMSNet model, that is, the cost cube is constructed directly according to the high-scale features extracted by the spatial pyramid pooling operation, without left image feature fusion module and right image feature fusion module.

[0120] The remaining structures in the network can refer to the description of the above-mentioned embodiment 1; the construction of the data set, and the specific manner of model training, can also refer to the description of the above-mentioned embodiment 1, which will not be repeated here.

[0121] Embodiment 3:

[0122] A binocular stereo vision matching method comprises:

[0123] inputting the left image and the right image to be matched into a binocular stereo vision matching network, and outputting a disparity map between the left image and the right image by the binocular stereo vision matching network;

[0124] converting the disparity map between the left image and the right image into a depth map, and completing the matching;

[0125] The binocular stereo vision matching network is established by the binocular stereo vision matching network construction method provided in the above-mentioned embodiments 1 or 2.

[0126] Embodiment 4:

[0127] According to another aspect of the present application, a computer readable storage medium is provided, comprising a stored computer program, which, when executed by a processor, controls the device where the computer readable storage medium is located to execute the binocular stereo vision matching network construction method provided in the above-mentioned embodiments 1 or 2, and / or the binocular stereo vision matching method provided in the above-mentioned embodiment 3.

[0128] Those skilled in the art can easily understand that the above-mentioned is only the preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for constructing a binocular stereo matching network, characterized in that, The application relates to a stereo matching network and a stereo matching method. The stereo matching network comprises a disparity edge prediction branch and a disparity estimation backbone network. The disparity edge prediction branch is used for disparity edge prediction of an image to obtain a corresponding disparity edge prediction image. The disparity estimation backbone network comprises: a cost cube construction subnetwork used for constructing a cost cube between a left image and a right image to be matched; one of the left image and the right image to be matched is a reference image, and the other is a to-be-matched image; a cost aggregation module used for regularizing the cost cube to obtain an aggregated cost body; an aggregated cost optimization module used for identifying an edge region of the aggregated cost body according to a disparity edge prediction image of the reference image predicted by the disparity edge prediction branch and performing suppression on the edge region to obtain an optimized aggregated cost body; and a disparity regression module used for performing regression on the optimized aggregated cost body after up-sampling the optimized aggregated cost body to be consistent with the size of the reference image to obtain a disparity image. The cost cube construction subnetwork comprises: a left image low-scale feature extraction module used for performing original-scale feature extraction on the left image to obtain low-scale features of the left image; a left image high-scale feature extraction module used for performing spatial pyramid pooling operation on the low-scale features of the left image to obtain high-scale features of the left image; a left image feature fusion module used for performing weighted fusion on the low-scale features and the high-scale features of the left image to obtain fusion features of the left image; during the fusion, the low-scale features have higher weights in the edge region, and the high-scale features have higher weights in the non-edge region, and the edge region and the non-edge region are determined according to a disparity edge prediction image of the left image predicted by the disparity edge prediction branch; a right image low-scale feature extraction module used for performing original-scale feature extraction on the right image to obtain low-scale features of the right image; a right image high-scale feature extraction module used for performing spatial pyramid pooling operation on the low-scale features of the right image to obtain high-scale features of the right image; a right image feature fusion module used for performing weighted fusion on the low-scale features and the high-scale features of the right image to obtain fusion features of the right image; during the fusion, the low-scale features have higher weights in the edge region, and the high-scale features have higher weights in the non-edge region, and the edge region and the non-edge region are determined according to a disparity edge prediction image of the right image predicted by the disparity edge prediction branch; and a cost body construction module used for connecting the fusion features of the left image and the fusion features of the right image at each disparity level according to a predetermined maximum disparity value to obtain the cost cube between the left image and the right image. The aggregated cost optimization module suppresses the edge region of the aggregated cost body according to the following formula: 2.The method of claim 1, wherein, The left image feature fusion module performs weighted fusion on the low-scale features and the high-scale features of the left image according to the following formula: wherein, is the optimized aggregation cost, is the aggregation cost, is a disparity edge prediction map of the reference image, denotes a pixel coordinate. 3.The method of claim 1, wherein, The right image feature fusion module performs weighted fusion on the low-scale features and the high-scale features of the right image according to the following formula: ​ wherein, and F and F' denote the fused features of the left and right images, respectively, and L and L' denote the low scale features of the left and right images, respectively, and H and H' denote the high scale features of the left and right images, respectively, and D and D' denote the disparity edge prediction maps of the left and right images, respectively, denotes a pixel coordinate; denotes a multiplication, denotes a concatenation.

4. The binocular stereo matching network construction method of any one of claims 1-3, characterized in that, The disparity edge prediction branch comprises: a 2D convolution module for extracting shallow features of an input image; a multi-scale feature extraction module for performing spatial pyramid pooling operations of different scales on the shallow features to obtain deep features of multiple scales; an up-sampling module for up-sampling the deep features at each scale to be consistent with the size of the input image; a feature fusion module for performing convolution operations on the features output by the up-sampling module after concatenation to obtain an edge feature of a single channel; and an edge regression module for regressing pixel values in the edge feature to the range of [0, 1] to obtain a disparity edge prediction map.

5. The method of claim 1-3, wherein, The training and testing of the stereo matching network to be trained using the data set comprises: performing edge detection on the disparity maps in the data set to obtain disparity edge maps, and constructing an edge detection sample set together with the corresponding left images and right images; training and testing the disparity edge prediction branch in the stereo matching network using the edge detection sample set; freezing the parameters of the disparity edge prediction branch in the stereo matching network, training and testing the stereo matching network using the data set, and obtaining a binocular stereo vision matching network. 6.The method of constructing a binocular stereo matching network according to claim 5, wherein, When training the disparity edge prediction branch in the stereo matching network using the edge detection sample set, a weighted cross-entropy loss function is used. 7.The method of claim 5, wherein, When training the stereo matching network using the data set with the parameters of the disparity edge prediction branch in the stereo matching network frozen, a smooth L1 loss function based on disparity is used.

8. A binocular stereo matching method, characterized by, The method comprises: inputting left and right images to be matched into a binocular stereo vision matching network, and outputting a disparity map between the left and right images from the binocular stereo vision matching network; converting the disparity map between the left and right images into a depth map to complete matching; wherein the binocular stereo vision matching network is established by the binocular stereo vision matching network construction method of any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer program stored in the computer readable storage medium is executed by a processor to control a device where the computer readable storage medium is located to perform the binocular stereo vision matching network construction method of any one of claims 1-7 and / or the binocular stereo vision matching method of claim 8. The computer program stored in the computer readable storage medium is executed by a processor to control a device where the computer readable storage medium is located to perform the binocular stereo vision matching network construction method of any one of claims 1-7 and / or the binocular stereo vision matching method of claim 8.

Images