Real-time stereo matching method based on multi-scale and multi-class cost volumes

Abstract

The present invention proposes a real-time stereo matching method based on multi-scale and multi-type cost volumes, including obtaining left and right RGB images; inputting the left and right RGB images into an end-to-end disparity prediction model; wherein the end-to-end disparity prediction model includes a feature extraction module, a cost volume construction module, a cost aggregation module, a disparity prediction module and a disparity refinement module; and outputting a disparity prediction result. The method proposed by the present invention can accurately restore the disparity map result of the original image size.

Description

Technical Field

[0001] The invention belongs to the field of stereo matching and binocular depth estimation. Background Art

[0002] With the continuous progress of society and the rapid development of artificial intelligence technology, autonomous intelligent robots have received widespread attention. Mobile robots, as a type of autonomous intelligent robot, have many applications, such as warehousing and logistics, autonomous driving, service robots, etc. In the past, robots generally worked according to pre-set trajectories, which worked well in known environments, but were often difficult to handle in unknown environments. How robots perceive new environments has become a very important issue.

[0003] Vision plays an extremely important role in humans' acquisition of information in environmental scenes, accounting for more than 75% of the total information acquisition. Through vision, humans can know the three-dimensional relevant information of the environment they are in, so they can judge the shape, size and position of all objects in the environment. Therefore, it is an indispensable link to apply visual sensors to robots, that is, machine vision. The goal of machine vision is to simulate human eyes through cameras to recognize and explore the world. The most important and basic step is to obtain information about the external world. Visual sensors mainly include monocular cameras, binocular cameras and RGB-D cameras. Monocular cameras cannot restore the scale information of the surrounding environment from an image alone. RGB-D cameras are easily affected by the lighting environment in outdoor environments, while binocular cameras can obtain accurate scale information through mathematical methods, so that they can overcome the problem that RGB-D cameras are easily affected by light when obtaining scale information, and the scale inconsistency caused by monocular cameras.

[0004] However, at this stage, there are still many problems to be solved in the field of binocular vision. For example, the research on real-time performance is still insufficient, especially how to achieve sufficient accuracy under real-time conditions. A large number of algorithms have high accuracy but sacrifice speed. Although StereoNet is fast enough, its accuracy is lower than that of later algorithms, which makes the algorithm truly applicable in the required scenarios. At the same time, the utilization of multi-scale features and information in multi-scale and multi-type cost volumes is not effective and sufficient enough, and the relationship between speed and accuracy cannot be well balanced. Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] To this end, the purpose of the present invention is to propose a real-time stereo matching method based on multi-scale and multi-type cost volumes, which is used to accurately restore the disparity map result of the original image size.

[0007] To achieve the above-mentioned purpose, the first embodiment of the present invention proposes a real-time stereo matching method based on multi-scale and multi-type cost volumes, comprising:

[0008] Get left and right RGB images;

[0009] Inputting the left and right RGB images into an end-to-end disparity prediction model; wherein the end-to-end disparity prediction model includes a feature extraction module, a cost volume construction module, a cost aggregation module, a disparity prediction module and a disparity refinement module;

[0010] Output disparity prediction results.

[0011] In addition, the real-time stereo matching method based on multi-scale and multi-type cost volumes according to the above embodiment of the present invention may also have the following additional technical features:

[0012] Further, in one embodiment of the present invention, inputting the left and right RGB images into an end-to-end disparity prediction model comprises:

[0013] Acquire feature maps of three scales from the left and right RGB images through the feature extraction module;

[0014] The cost volume construction module fuses the feature maps of the left and right RGB images at the three scales to obtain the cost volumes of the three scales;

[0015] Aggregating the minimum-scale cost volume using 3D convolution through the cost aggregation module;

[0016] Regressing the aggregated results through the disparity prediction module to obtain a coarse disparity map;

[0017] The disparity refinement module uses bilinear upsampling and convolution on the coarse disparity map to enlarge and optimize the disparity value.

[0018] Furthermore, in one embodiment of the present invention, the acquiring feature maps of three scales from the left and right RGB images by the feature extraction module includes:

[0019] The input RGB image of size H×W×3 is convolved with a 7×7 convolution with a stride of 3, where the BatchNorm and ReLU activation functions of the 7×7 convolution are scaled down to one-third of the size and the number of channels is adjusted to 32, obtaining a first-layer feature map of one-third scale, with a size of H / 3×W / 3×128;

[0020] The first layer feature map is passed through four bottleneck structures, where only the first one has a step size of 2, an input channel number of 128, and an output channel number of 256, and the other three have a step size of 1, an input channel number and an output channel number of 256, and the number of channels inside all bottlenecks is 64, to obtain a second layer feature map with a size of H / 6×W / 6×256;

[0021] The second layer feature map is passed through 6 bottleneck structures, wherein only the first of the 6 bottleneck structures has a step size of 2, an input channel number of 256, and an output channel number of 512, and the other 5 have a step size of 1, an input channel number and an output channel number of 512, and the number of channels inside all bottlenecks is 128. The convolution in the 6 bottleneck structures is a deformable convolution, and the third layer feature map is obtained, with a size of H / 12×W / 12×512;

[0022] After obtaining the feature maps of three scales from bottom to top, it also includes:

[0023] The third layer feature map is subjected to 1×1 convolution to adjust the number of channels from 512 to 128, and then subjected to 3×3 convolution, BatchNorm and ReLU activation functions to form the first layer output of the feature pyramid network; the second layer feature map is subjected to 1×1 convolution to adjust the number of channels from 256 to 128, and then directly added to the first layer output after 2 times the nearest neighbor upsampling, and then subjected to 3×3 convolution, BatchNorm and ReLU activation functions to form the second layer output;

[0024] After the third layer feature map is subjected to 1×1 convolution to keep the number of channels at 128, it is directly added to the second layer output after 2 times nearest neighbor upsampling, and then subjected to 3×3 convolution, BatchNorm and ReLU activation function to form the third layer output; at this time, the output of the feature pyramid network is still three feature maps of the same scale, i.e. 1 / 3, 1 / 6, and 1 / 12, but the number of channels is adjusted to 128.

[0025] Further, in one embodiment of the present invention, the cost volume construction module fuses the feature maps of the left and right RGB images at the three scales to obtain the cost volumes of the three scales, including:

[0026] The four-dimensional cost volume is constructed by the difference method for the feature maps with the lowest resolution at three scales, and the three-dimensional cost volume is obtained by the inner product method using the correlation method for the feature maps with the other two pairs of resolutions.

[0027] Further, in one embodiment of the present invention, the aggregating the minimum-scale cost volume using 3D convolution by the cost aggregation module includes:

[0028] Input the lowest resolution 4D cost volume into multiple 3D convolution blocks to filter the cost volume;

[0029] The result of filtering the cost volume is input into a separate convolutional layer without BatchNorm and activation function, with a channel number of 1, that is, a one-dimensional output is produced for each candidate disparity at each pixel.

[0030] Further, in one embodiment of the present invention, regressing the aggregated result through the disparity prediction module to obtain a coarse disparity map includes:

[0031] The soft argmin method is selected to regress the coarse disparity map from the aggregated result.

[0032] Further, in one embodiment of the present invention, the using bilinear upsampling and convolution of the coarse disparity map by the disparity refinement module to amplify and optimize the disparity value includes:

[0033] The disparity map is refined hierarchically using three upsampling passes. The first upsampling pass has 36 input channels, including the original RGB image reduced to H / 6×W / 6×3, the coarse disparity map enlarged to H / 6×W / 6, and the cost volume of H / 6×W / 6×32. The second upsampling pass has 68 input channels, including the original RGB image reduced to H / 3×W / 3×3, the coarse disparity map enlarged to H / 6×W / 6×1, and the cost volume of H / 3×W / 3×64. The third upsampling pass combines the original RGB image and the disparity map, with 4 input channels.

[0034] Each time the input is first passed through a 3×3 convolution layer to adjust the number of output channels to 32, and then passed through 6 residual blocks, also using 3×3 convolution, BatchNorm and Leaky ReLU activation functions, the number of channels remains 32, and the step size is 1.

[0035] Furthermore, in one embodiment of the present invention, the input end-to-end disparity prediction model is trained using a smooth L1 loss function.

[0036] To achieve the above-mentioned purpose, the second aspect of the present invention proposes a computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it implements the real-time stereo matching method based on multi-scale and multi-type cost volumes as described above.

[0037] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, a real-time stereo matching method based on multi-scale and multi-type cost volumes as described above is implemented.

[0038] The embodiment of the present invention proposes a real-time stereo matching method based on multi-scale and multi-class cost volumes, and proposes a real-time, multi-scale cost volume-fused end-to-end disparity prediction deep architecture StereoFCNet, referred to as FC-Net. This network is divided into five modules: feature extraction module, cost volume construction module, cost aggregation module, disparity prediction module and disparity refinement module, as shown in the schematic diagram Figure 4 As shown in the figure, the feature extraction module obtains feature maps of 1 / 3, 1 / 6, and 1 / 12 sizes from the left and right RGB images through a shared weighted twin network; the cost volume construction module fuses the feature maps of the left and right images at three scales to obtain cost volumes of 1 / 3, 1 / 6, and 1 / 12 sizes; the cost aggregation module aggregates and reduces the dimension of the 1 / 12 cost volume using 3D convolution; the disparity prediction module regresses the initial coarse disparity map from the aggregated result; the disparity refinement module upsamples the disparity map three times, and splices the RGB image, disparity map, and cost volume of the corresponding size as input each time it upsamples, learns the disparity residual through the residual block, and directly adds it to the disparity map to continuously correct the disparity map, and finally outputs the disparity map result restored to the original size. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0040] Figure 1 A schematic flow chart of a real-time stereo matching method based on multi-scale and multi-type cost volumes provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of the structure of an end-to-end disparity prediction model provided by an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of a feature extraction module provided by an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of a Bottleneck layer structure provided by an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of a deformable convolution provided by an embodiment of the present invention;

[0045] Figure 6 A schematic diagram of the structure of a feature extractor provided by an embodiment of the present invention;

[0046] Figure 7 A schematic diagram of a dilated convolution provided by an embodiment of the present invention;

[0047] Figure 8 A schematic diagram of a residual block structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0049] The following describes a real-time stereo matching method based on multi-scale and multi-type cost volumes according to an embodiment of the present invention with reference to the accompanying drawings.

[0050] Figure 1 A schematic flow chart of a real-time stereo matching method based on multi-scale and multi-type cost volumes provided in an embodiment of the present invention.

[0051] like Figure 1 As shown, the real-time stereo matching method based on multi-scale and multi-class cost volumes includes the following steps:

[0052] S101: Obtain left and right RGB images;

[0053] S102: inputting the left and right RGB images into an end-to-end disparity prediction model; wherein the end-to-end disparity prediction model includes a feature extraction module, a cost volume construction module, a cost aggregation module, a disparity prediction module and a disparity refinement module;

[0054] like Figure 2 The figure shows the structure of the input end-to-end disparity prediction model (FC-Net). Figure 2 The operation at each disparity refinement input is: bilinearly upsampling the disparity to the output size, adjusting the original RGB image to the same size, and splicing it with the cost volume of the same scale as a guide, and inputting them together into the refinement network to restore high-resolution details. In this way, the present invention not only uses color input as a guide to refine the coarse disparity map, but also uses the cost volume at high resolution containing global correlation information as a guide to improve the optimization effect.

[0055] Furthermore, in one embodiment of the present invention, the left and right RGB images are input into an end-to-end disparity prediction model, including:

[0056] The feature extraction module obtains feature maps of three scales from the left and right RGB images;

[0057] The cost volume construction module fuses the feature maps of the left and right RGB images at three scales to obtain cost volumes of three scales.

[0058] The minimum-scale cost volume is aggregated using 3D convolution through the cost aggregation module;

[0059] The coarse disparity map is regressed from the aggregated results through the disparity prediction module;

[0060] The disparity refinement module uses bilinear upsampling and convolution on the coarse disparity map to enlarge and refine the disparity value.

[0061] The single-scale cost volume lacks global context information, which easily causes mismatching in areas with weak textures. Based on the idea of ​​FPN (Feature Pyramid Network) and residual network, this paper proposes a multi-scale feature extraction module, which uses FPN to fuse sufficient underlying semantic information and uses deformable convolution technology to adaptively adjust the size and shape of the receptive field to provide rich global context information for pathological areas. The structure is shown in the figure below: Figure 3 shown.

[0062] The feature pyramid network first uses a feature extraction network to extract feature maps of multiple scales from the original image, namely Figure 3 The bottom-up process on the left side of the figure, and then the top-down process on the right side, form a complete feature pyramid network. Taking the extracted feature map containing three scales as an example, the top-down process is as follows: the third-layer output of the feature extraction network, that is, the output of the smallest scale, is subjected to 1×1 convolution to form the first-layer output of the feature pyramid network; the second-layer output of the feature extraction network is directly added together after 1×1 convolution and the first-layer output obtained in the previous step after 2 times the nearest upsampling to form the second-layer output; the third-layer output is the same as the second layer. The output of the feature pyramid network is still a feature map of three scales.

[0063] The advantages of the feature pyramid network structure can be mainly divided into three points: (1) It utilizes multi-scale features, which is better than the extractor that only uses the lowest resolution features; (2) The number of parameters is much smaller than directly extracting features from original images of different resolutions, and the memory usage is also much smaller; (3) Adding a top-down process is better than using only three-scale feature extractors, which can integrate sufficient underlying semantic information.

[0064] The bottleneck structure is a core structure proposed in the residual network ResNet. Figure 4 As shown, it is a special residual structure.

[0065] First, use 1×1 convolution, BatchNorm, and ReLU activation functions to reduce the number of channels, and then use 3×3 convolution, BatchNorm, and ReLU activation functions to process the data as needed. For example, setting the bottleneck structure step size to 2 means changing the step size parameter of the 3×3 convolution to 2, so that the input scale is reduced by half. Finally, the number of channels is restored through 1×1 convolution and BatchNorm, and the output is directly added to the original input and activated through the ReLU function to get the final output.

[0066] The bottleneck structure is called a bottleneck structure because it is narrow in the middle and thick on both sides, and it looks like a bottleneck, but the number of channels is not fixed and cannot be changed. The advantage of the bottleneck structure is that the 1×1 convolution can easily change the dimension, so as to achieve flexible network design and reduce the number of parameters, and deepen the network, making it relatively easier to train, and the effect is equivalent to using a 3×3 convolution with the same number of input channels.

[0067] The schematic diagram of deformable convolution is as follows Figure 5 Deformable convolution means that the convolution kernel adds an additional learnable offset to each element, so that the convolution kernel can be expanded to a large range during training. Traditional convolution kernels are usually fixed in size. For example, the convolution kernel of a traditional 3×3 convolution is Figure 5 The 9 blocks in the middle have poor adaptability to unknown changes and weak generalization ability. Deformable convolution can be close to the actual shape of the object, maintain tracking coverage, and adaptively change the range and shape. From a mathematical point of view, the formula of traditional convolution is:

[0068]

[0069] The formula for deformable convolution is:

[0070]

[0071] The idea of ​​deformable convolution is: at the beginning, just like traditional convolution, the feature map is extracted based on the input image using the convolution kernel. Then the obtained feature map is used as input, and the offset is learned through a convolution layer. The number of channels of the offset is 2N, including the offset in both the horizontal and vertical directions. During training, the convolution kernel used to generate the output feature and the convolution kernel used to generate the offset are learned synchronously. Since the corresponding position after the offset may not be an integer point, it is necessary to use bilinear interpolation to calculate the weighted sum of the four integer points in the neighborhood to obtain the value of the position, and finally learn through back propagation.

[0072] The feature extractor structure of the multi-scale feature extraction module of the present invention is as follows: Figure 6As shown. H represents height, W represents width, D represents disparity range, and F represents the number of channels. The feature extractor ResNet-40 is modified based on the first three stages of ResNet-50. The bottleneck residual structure deepens the network, reduces the number of parameters, improves accuracy, and solves the problems of network degradation and gradient disappearance. At the same time, the convolutions in the 6 bottleneck structures of 1 / 12 scale are replaced with deformable convolutions, thereby adaptively changing the shape and size of the receptive field to be closer to the shape of the object, improving the effect of feature extraction.

[0073] Furthermore, in one embodiment of the present invention, feature maps of three scales are obtained from the left and right RGB images by a feature extraction module, including:

[0074] The input RGB image of size H×W×3 is convolved with a 7×7 convolution with a stride of 3, where the BatchNorm and ReLU activation functions of the 7×7 convolution are reduced to one-third of the size and the number of channels is adjusted to 32, obtaining the first-layer feature map of one-third scale, with a size of H / 3×W / 3×128;

[0075] The first layer feature map passes through 4 bottleneck structures, among which only the first one has a step size of 2, 128 input channels, and 256 output channels, and the other three have a step size of 1, 256 input channels, and 64 channels inside all bottlenecks to obtain the second layer feature map with a size of H / 6×W / 6×256;

[0076] The second-layer feature map passes through 6 bottleneck structures, among which only the first one has a step size of 2, an input channel number of 256, and an output channel number of 512, and the other 5 have a step size of 1, an input channel number and an output channel number of 512, and the number of channels inside all bottlenecks is 128. The convolution in the 6 bottleneck structures is a deformable convolution, and the third-layer feature map is obtained, with a size of H / 12×W / 12×512;

[0077] After obtaining the feature maps of three scales from bottom to top, it also includes:

[0078] The third layer feature map is subjected to 1×1 convolution to adjust the number of channels from 512 to 128, and then subjected to 3×3 convolution, BatchNorm and ReLU activation functions to form the first layer output of the feature pyramid network; the second layer feature map is subjected to 1×1 convolution to adjust the number of channels from 256 to 128, and then directly added to the first layer output after 2 times the nearest neighbor upsampling, and then subjected to 3×3 convolution, BatchNorm and ReLU activation functions to form the second layer output;

[0079] After the third layer feature map is subjected to 1×1 convolution to keep the number of channels at 128, it is directly added to the second layer output after 2 times nearest neighbor upsampling, and then subjected to 3×3 convolution, BatchNorm and ReLU activation function to form the third layer output; at this time, the output of the feature pyramid network is still three feature maps of the same scale, i.e. 1 / 3, 1 / 6, and 1 / 12, but the number of channels is adjusted to 128.

[0080] Furthermore, in one embodiment of the present invention, the cost volume construction module is used to fuse the feature maps of the left and right RGB images at three scales to obtain cost volumes of three scales, including:

[0081] The four-dimensional cost volume is constructed by the difference method for the feature maps with the lowest resolution at three scales, and the three-dimensional cost volume is obtained by the inner product method using the correlation method for the feature maps with the other two pairs of resolutions.

[0082] Specifically, considering that the low-resolution four-dimensional cost volume contains sufficient information, is small in size, and occupies a small amount of computation, the four-dimensional cost volume at high resolution has a large number of parameters, and the speed of three-dimensional convolution is slow. Although the construction of a three-dimensional cost volume will lose some information, it occupies a small amount of computation and is fast, and the three-dimensional cost volume is sufficient to restore the parallax with considerable accuracy. Therefore, the present invention constructs a four-dimensional cost volume for the lowest resolution feature map by the difference method, and its size is H / 12×W / 12×D / 12×F. The other two pairs of high-resolution feature maps use the correlation method to perform inner products to obtain a three-dimensional cost volume, respectively, with sizes of H / 6×W / 6×D / 6 and H / 3×W / 3×D / 3. Among them, the maximum parallax D is set to 192, and the number of feature channels F is 32×4, thereby constructing cost volumes of three scales.

[0083] Further, in one embodiment of the present invention, the minimum-scale cost volume is aggregated using 3D convolution by a cost aggregation module, including:

[0084] Input the lowest resolution 4D cost volume into multiple 3D convolution blocks to filter the cost volume;

[0085] The result of filtering the cost volume is input into a separate convolutional layer without BatchNorm and activation function, with a channel number of 1, that is, a one-dimensional output is produced for each candidate disparity at each pixel.

[0086] Specifically, the present invention learns the correct judgment method through multiple 3D convolutions and sets nonlinear activation functions. The input is a four-dimensional cost volume with the lowest resolution, the size of which is H / 12×W / 12×D / 12×F. First, four 3D convolution blocks are used to filter the cost volume. The convolution kernel size is 3×3×3 and contains BatchNorm and Leaky ReLU activation functions. The first 3D convolution reduces the number of channels from 128 to 64, the second 3D convolution reduces the number of channels from 64 to 32, and the third and fourth 3D convolutions keep the number of channels at 32. Finally, a separate 3×3×3 convolution layer without BatchNorm and activation function is used, and the number of output channels is 1, that is, a one-dimensional output is generated for each candidate disparity at each pixel, and the output size obtained is H / 12×W / 12×D / 12×1. In this way, the contextual information of the spatial dimension and the disparity dimension is aggregated, and the output obtained after the feature dimension is eliminated is H / 12×W / 12×D / 12.

[0087] Further, in one embodiment of the present invention, regressing the aggregated result through the disparity prediction module to obtain a coarse disparity map includes:

[0088] The soft argmin method is selected to regress the coarse disparity map from the aggregated result.

[0089] Specifically, after using soft argmin on the cost volume of size H / 12×W / 12×D / 12, an initial coarse disparity prediction map of size H / 12×W / 12 is obtained.

[0090] Further, in one embodiment of the present invention, the disparity refinement module uses bilinear upsampling and convolution on the coarse disparity map to amplify and optimize the disparity value, including:

[0091] The disparity map is refined hierarchically using three upsampling passes. The first upsampling pass has 36 input channels, including the original RGB image reduced to H / 6×W / 6×3, the coarse disparity map enlarged to H / 6×W / 6, and the cost volume of H / 6×W / 6×32. The second upsampling pass has 68 input channels, including the original RGB image reduced to H / 3×W / 3×3, the coarse disparity map enlarged to H / 6×W / 6×1, and the cost volume of H / 3×W / 3×64. The third upsampling pass combines the original RGB image and the disparity map, with 4 input channels.

[0092] Each time the input is first passed through a 3×3 convolution layer to adjust the number of output channels to 32, and then passed through 6 residual blocks, also using 3×3 convolution, BatchNorm and Leaky ReLU activation functions, the number of channels remains 32, and the step size is 1.

[0093] Specifically, dilated convolution is also called dilated convolution. Figure 7 As shown in the figure, the convolution kernel of traditional convolution is compact N×N in size, while the convolution kernel of dilated convolution has a gap between every two weights, or fills with 0 weights. The size of the gap is called the dilation factor. Traditional convolution can be considered a special case of dilated convolution, that is, dilated convolution with a dilation factor of 1, while Figure 7 The dilation factor shown is 2. If you want the input and output sizes to remain the same when the convolution stride is 1, you need to add a number of zeros around the input that is equal to the dilation factor, so when setting the dilated convolution in use, the number of zeros will be equal to the dilation factor.

[0094] The residual block structure, called Residual Block or Basic Block, is the basic structural block proposed in the residual network ResNet. Figure 8 As shown in the figure, the input first passes through 3×3 convolution, BatchNorm and ReLU function activation, then passes through 3×3 convolution and BatchNorm to obtain the residual, and then the original input and the output are skipped and added together, and finally the sum of the two is activated by the ReLU function. When the step size of the convolution is not 1, the size of the input will change. Finally, when adding directly to the original input, it is necessary to ensure that the dimensions of the two tensors are consistent, that is, the original input needs to be downsampled to the same size. The idea of ​​using skip connections can effectively reduce the problem of gradient vanishing and model degradation, so the residual structure can greatly deepen the network depth.

[0095] Furthermore, in one embodiment of the present invention, the input end-to-end disparity prediction model is trained using a smooth L1 loss function.

[0096] The present invention is trained in a fully supervised manner, and the loss function uses a smoothed L1 loss. L1 loss is also called mean absolute error, which measures the absolute value of the difference between the predicted value and the true value. It is robust to outliers and does not cause gradient explosion problems, but its derivative at point 0 is discontinuous, making the solution possibly unstable. L2 loss is also called mean square error, which measures the square of the difference between the predicted value and the true value. It is sensitive to outliers, but the solution is more stable. The smoothed L1 loss combines the advantages of L1 loss and L2 loss, and smoothes the L1 loss near point 0. Its formula is:

[0097]

[0098] The total loss function is:

[0099]

[0100] Reference Figure 2The loss function part of the 4 resolutions, namely 1 / 12, 1 / 6, 1 / 3, 1, corresponding to the disparity map of k = 0, 1, 2, 3, constitutes the total loss function. is the predicted disparity at pixel i at the kth resolution, is the true value at the same pixel, w k is the weight of the output disparity map for each resolution, which is set to 1 / 3, 2 / 3, 1, and 1 from low to high resolution. The predicted disparity map is always bilinearly upsampled to match the resolution of the true value.

[0101] S103: Output the disparity prediction result.

[0102] The embodiment of the present invention proposes a real-time stereo matching method based on multi-scale and multi-class cost volumes, and proposes a real-time, multi-scale cost volume-fused end-to-end disparity prediction deep architecture StereoFCNet, referred to as FC-Net. This network is divided into five modules: feature extraction module, cost volume construction module, cost aggregation module, disparity prediction module and disparity refinement module, as shown in the schematic diagram Figure 4 As shown. The feature extraction module obtains feature maps of 1 / 3, 1 / 6, and 1 / 12 sizes from the left and right RGB images respectively through a twin network with shared weights; the cost volume construction module fuses the feature maps of the left and right images at three scales to obtain cost volumes of 1 / 3, 1 / 6, and 1 / 12 sizes; the cost aggregation module aggregates and reduces the dimension of the cost volume of 1 / 12 using 3D convolution; the disparity prediction module regresses the initial coarse disparity map from the aggregated result; the disparity refinement module upsamples the disparity map three times, and splices the RGB image, disparity map, and cost volume of corresponding sizes as input each time the upsampling occurs, learns the disparity residual through the residual block, and directly adds it to the disparity map to continuously correct the disparity map, and finally outputs the disparity map result restored to the original image size. Compared with the prior art, the advantages of the present invention are:

[0103] 1) Compared with StereoNet, StereoNet only extracts features at the lowest resolution (1 / 8 or 1 / 16). This single-scale feature lacks global context information, and only has rough information at low resolution, without detailed features. The present invention implements a multi-scale feature extraction network based on the idea of ​​feature pyramid network, and utilizes the feature information of these different scales (1 / 8, 1 / 4 and 1 / 2), obtains multi-level information from coarse to detailed, and fuses the context information together; the deformable convolution technology is used to adaptively change the size and shape of the receptive field, which is more in line with the shape of the object in the actual image, and can also extract object information of different sizes, fully integrate the underlying semantic information, and improve the accuracy and effect of feature extraction.

[0104] 2) Single-scale, single-type cost volumes are difficult to balance the relationship between the amount of information and the amount of parameters. Single-scale, low-resolution cost volumes lack information, and single-scale, high-resolution 4D cost volumes have too many parameters when applying 3D convolution, and the amount of information of a single 3D cost volume is insufficient. Therefore, the present invention proposes to construct multi-scale and multi-type cost volumes to obtain multi-scale similarity relationships, construct 4D cost volumes at the lowest resolution, and splice 3D cost volumes at high resolution, balancing accuracy and speed. Under the premise of maintaining real-time performance, it improves considerable accuracy and optimizes the processing effect of stereo matching in inherent pathological areas.

[0105] 3) The hierarchical refinement network that only uses RGB images as color guidance in StereoNet is easily disturbed by colors and has low robustness. The present invention proposes a refinement network structure that splices multi-scale cost volumes, disparity maps and RGB images. Through a residual block based on dilated convolution, the receptive field is expanded and image details are restored in combination with rich contextual correlation information, thereby improving accuracy and anti-interference ability.

[0106] To achieve the above-mentioned purpose, a second aspect of the present invention proposes a computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it implements the real-time stereo matching method based on multi-scale and multi-type cost volumes as described above.

[0107] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the real-time stereo matching method based on multi-scale and multi-class cost volumes as described above is implemented.

[0108] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0109] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0110] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A real-time stereo matching method based on multi-scale and multi-class cost volumes, characterized in that: The following steps are involved: Get left and right RGB images; Inputting the left and right RGB images into an end-to-end disparity prediction model; wherein the end-to-end disparity prediction model includes a feature extraction module, a cost volume construction module, a cost aggregation module, a disparity prediction module and a disparity refinement module; Output disparity prediction result; The inputting the left and right RGB images into an end-to-end disparity prediction model comprises: Acquire feature maps of three scales from the left and right RGB images through the feature extraction module; The cost volume construction module fuses the feature maps of the left and right RGB images at the three scales to obtain the cost volumes of the three scales; Aggregating the minimum-scale cost volume using 3D convolution through the cost aggregation module, wherein the four-dimensional cost volume of the lowest resolution is input into multiple 3D convolution blocks to filter the cost volume, and the result obtained by filtering the cost volume is input into a separate convolution layer without BatchNorm and activation function, and the number of channels is 1, that is, each candidate disparity generates a one-dimensional output at each pixel, and a coarse disparity map is regressed from the aggregated result through the disparity prediction module, and the coarse disparity map is amplified and optimized by using bilinear upsampling and convolution through the disparity refinement module; The cost volume construction module fuses the feature maps of the left and right RGB images at the three scales to obtain the cost volumes of the three scales, including: The four-dimensional cost volume is constructed by the difference method for the feature maps with the lowest resolution at three scales, and the three-dimensional cost volume is obtained by the inner product method using the correlation method for the feature maps with the other two pairs of resolutions.

2. The method according to claim 1, characterized in that The step of obtaining feature maps of three scales from the left and right RGB images by the feature extraction module includes: The input RGB image of size H×W×3 is convolved with a 7×7 convolution with a stride of 3, where the BatchNorm and ReLU activation functions of the 7×7 convolution are scaled down to one-third of the size and the number of channels is adjusted to 32, obtaining a first-layer feature map of one-third scale, with a size of H / 3×W / 3×128; The first layer feature map is passed through four bottleneck structures, where only the first one has a step size of 2, an input channel number of 128, and an output channel number of 256, and the other three have a step size of 1, an input channel number and an output channel number of 256, and the number of channels inside all bottlenecks is 64, to obtain a second layer feature map with a size of H / 6×W / 6×256; The second layer feature map is passed through 6 bottleneck structures, wherein only the first of the 6 bottleneck structures has a step size of 2, an input channel number of 256, and an output channel number of 512, and the other 5 have a step size of 1, an input channel number and an output channel number of 512, and the number of channels inside all bottlenecks is 128. The convolution in the 6 bottleneck structures is a deformable convolution, and the third layer feature map is obtained, with a size of H / 12×W / 12×512; After obtaining the feature maps of three scales from bottom to top, it also includes: The third layer feature map is subjected to 1×1 convolution to adjust the number of channels from 512 to 128, and then subjected to 3×3 convolution, BatchNorm and ReLU activation functions to form the first layer output of the feature pyramid network; the second layer feature map is subjected to 1×1 convolution to adjust the number of channels from 256 to 128, and then directly added to the first layer output after 2 times the nearest neighbor upsampling, and then subjected to 3×3 convolution, BatchNorm and ReLU activation functions to form the second layer output; After the third layer feature map is subjected to 1×1 convolution to keep the number of channels at 128, it is directly added to the second layer output after 2 times nearest neighbor upsampling, and then subjected to 3×3 convolution, BatchNorm and ReLU activation function to form the third layer output; at this time, the output of the feature pyramid network is still three feature maps of the same scale, i.e. 1 / 3, 1 / 6, and 1 / 12, but the number of channels is adjusted to 128.

3. The method according to claim 1, characterized in that The step of regressing the aggregated result through the disparity prediction module to obtain a coarse disparity map includes: The soft argmin method is selected to regress the coarse disparity map from the aggregated result.

4. The method according to claim 1, characterized in that: The method of using bilinear upsampling and convolution on the coarse disparity map by the disparity refinement module to amplify and optimize the disparity value includes: The disparity map is refined hierarchically using three upsampling passes. The first upsampling pass has 36 input channels, including the original RGB image reduced to H / 6×W / 6×3, the coarse disparity map enlarged to H / 6×W / 6, and the cost volume of H / 6×W / 6×32. The second upsampling pass has 68 input channels, including the original RGB image reduced to H / 3×W / 3×3, the coarse disparity map enlarged to H / 6×W / 6×1, and the cost volume of H / 3×W / 3×64. The third upsampling pass combines the original RGB image and the disparity map, with 4 input channels. Each time the input is first passed through a 3×3 convolution layer to adjust the number of output channels to 32, and then passed through 6 residual blocks, also using 3×3 convolution, BatchNorm and Leaky ReLU activation functions, the number of channels remains 32, and the step size is 1.

5. The method according to claim 1, characterized in that The input end-to-end disparity prediction model is trained using a smooth L1 loss function.

6. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the real-time stereo matching method based on multi-scale and multi-class cost volumes as described in any one of claims 1 to 5 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the real-time stereo matching method based on multi-scale and multi-type cost volumes as described in any one of claims 1 to 5 is implemented.

Images