Binocular stereo matching method for non-polar correction

Abstract

The invention relates to the technical field of binocular vision, in particular to a binocular stereo matching method for non-epipolar correction, which comprises the following steps: collecting binocular images by using a binocular camera, the binocular images comprising a first image and a second image; respectively inputting the first image and the second image into a multi-scale feature extraction network to obtain a first multi-scale feature map corresponding to the first image and a second multi-scale feature map corresponding to the second image; constructing a correlation-based three-dimensional cost volume based on the first multi-scale feature map and the second multi-scale feature map; performing two-dimensional cost aggregation on the three-dimensional cost volume to obtain an aggregation feature map; sequentially performing multi-scale feature extraction and multi-scale fusion on the aggregated feature map by adopting a multi-scale convolution attention module to obtain a multi-dimensional feature map; and performing parallax regression and optimization processing on the multi-dimensional feature map to obtain a parallax estimation map. The method is at least beneficial to improving the accuracy and calculation efficiency of stereo matching.

Description

Technical Field

[0001] The invention belongs to the field of binocular vision technology, and in particular relates to a binocular stereo matching method with non-epipolar line correction. Background Art

[0002] Stereo matching technology has been widely used in fields such as autonomous driving and augmented reality. The current mainstream method of binocular stereo matching usually relies on epipolar correction technology, that is, the images obtained by the left and right cameras are subjected to epipolar correction before stereo matching.

[0003] However, this method of stereo matching after epipolar correction has some inherent limitations, as follows: epipolar correction relies on accurate camera intrinsics and extrinsics. If there are errors in the calibration process, the corrected image may not strictly meet the epipolar geometry constraints, which will lead to matching errors, namely calibration errors; epipolar correction involves the perspective transformation of the image, which may introduce geometric distortion in the image, especially in the edge area of ​​the image, thus affecting the matching accuracy, namely perspective transformation distortion; when the baseline of the binocular camera is large, epipolar correction may cause the perspective transformation of the image to be too drastic, causing serious image distortion, thereby reducing the matching quality, namely large baseline effect; in dynamic scenes, the movement of objects may cause the matching points to shift in the corrected image, which will increase the difficulty of matching and even lead to matching failure; although epipolar correction can reduce the search range during matching, it itself requires camera parameter calculation and image transformation, which will increase the additional computational burden. Therefore, it is necessary to provide a non-episodic correction stereo matching method to solve the above problems. Summary of the invention

[0004] In view of this, the present invention aims to provide a binocular stereo matching method with non-episodic correction, which is at least beneficial to improving the accuracy and computational efficiency of stereo matching.

[0005] To achieve the above object, the technical solution created by the present invention is implemented as follows: The invention provides a binocular stereo matching method with non-epipolar correction, comprising: using a binocular camera to collect binocular images, the binocular images comprising a first image and a second image; inputting the first image and the second image into a multi-scale feature extraction network respectively to obtain a first multi-scale feature map corresponding to the first image and a second multi-scale feature map corresponding to the second image; constructing a three-dimensional cost volume based on correlation based on the first multi-scale feature map and the second multi-scale feature map; performing two-dimensional cost aggregation on the three-dimensional cost volume to obtain an aggregated feature map; using a multi-scale convolutional attention module to perform multi-scale feature extraction and multi-scale fusion on the aggregated feature map in sequence to obtain a multi-dimensional feature map; performing disparity regression and optimization processing on the multi-dimensional feature map to obtain a disparity estimation map.

[0006] In some embodiments, before the first image and the second image are respectively input into the multi-scale feature extraction network, the method further includes: using a preprocessing module to perform denoising and edge enhancement processing on the first image and the second image.

[0007] In some embodiments, the multi-scale feature extraction network includes a MobileNetV2 network.

[0008] In some embodiments, the step of obtaining a first multi-scale feature map and a second multi-scale feature map includes: using a multi-scale feature extraction network to perform dimensionality reduction and then dimensionality increase operations on the first image and the second image respectively, and the dimensionality reduction and then dimensionality increase operations include: gradually reducing the resolution of the first image or the second image to 1 / 4, 1 / 8 and 1 / 16 of the original image, and then gradually expanding it to 1 / 4 of the original image.

[0009] In some embodiments, constructing the correlation-based three-dimensional cost volume includes: generating a three-dimensional cost volume by acquiring feature similarities between the first multi-scale feature map and the second multi-scale feature map at different disparities.

[0010] In some embodiments, performing two-dimensional cost aggregation on the three-dimensional cost volume to obtain an aggregated feature map includes: performing two-dimensional cost aggregation using a reverse residual block.

[0011] In some embodiments, using a reverse residual block for two-dimensional cost aggregation includes: performing a 1×1 convolution operation, a Relu6 activation function activation, a depth 3×3 convolution operation, a Relu6 activation function activation, a 1×1 convolution operation, and a Linear linear activation function activation on the three-dimensional cost volume in sequence to obtain processed features, and performing an aggregation and splicing operation on the processed features and the three-dimensional cost volume in the channel dimension to obtain an aggregated feature map.

[0012] In some embodiments, multi-scale feature extraction adopts a deep convolution approach, using multiple convolution kernels of different sizes for multi-scale feature extraction, and the multiple convolution kernels of different sizes include: 1×1 convolution kernel, 7×1 convolution kernel, 1×7 convolution kernel, 11×1 convolution kernel, 1×11 convolution kernel, 21×1 convolution kernel and 1×21 convolution kernel.

[0013] In some embodiments, multi-scale fusion includes fusing feature maps of different scales obtained by multi-scale feature extraction to obtain a multi-dimensional feature map, and the fusion is completed by splicing or by weighted summation.

[0014] In some embodiments, performing disparity regression and optimization processing on the multi-dimensional feature map includes: performing Softmax normalization processing on the multi-dimensional feature map to obtain a disparity probability distribution for each pixel, and obtaining a disparity estimation map based on the disparity probability distribution with the highest probability.

[0015] Compared with the prior art, the invention can achieve the following beneficial effects: the binocular stereo matching method with non-episode correction provided by the embodiment of the invention solves the problems of calibration error, perspective transformation distortion, large baseline effect and computational complexity caused by epipolar correction in the traditional binocular stereo matching process, and the invention introduces a multi-scale convolutional attention mechanism and adopts a two-dimensional cost body for cost calculation, thereby realizing efficient, stable and fast binocular stereo matching, and greatly improving the accuracy and computational efficiency of stereo matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings: Figure 1 A schematic diagram of the overall framework of the binocular stereo matching method with non-epipolar line correction described in an embodiment of the present invention; Figure 2 A schematic diagram of extracting features from a first image and a second image according to an embodiment of the present invention; Figure 3 A schematic diagram of the structure of the reverse residual block described in the embodiment of the present invention; Figure 4 A schematic diagram of the multi-scale convolutional attention module described in the embodiment of the present invention sequentially performing multi-scale feature extraction and multi-scale fusion on the aggregated feature map. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0018] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0019] In the description of the invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the invention, unless otherwise specified, the meaning of "multiple" is two or more.

[0020] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0021] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0022] refer to Figure 1 The present invention provides a binocular stereo matching method with non-epipolar correction, comprising: using a binocular camera to collect binocular images, the binocular images comprising a first image and a second image; inputting the first image and the second image into a multi-scale feature extraction network respectively to obtain a first multi-scale feature map corresponding to the first image and a second multi-scale feature map corresponding to the second image; constructing a three-dimensional cost volume based on correlation based on the first multi-scale feature map and the second multi-scale feature map; performing two-dimensional cost aggregation on the three-dimensional cost volume to obtain an aggregated feature map; using a multi-scale convolutional attention module to perform multi-scale feature extraction and multi-scale fusion on the aggregated feature map in sequence to obtain a multi-dimensional feature map; performing disparity regression and optimization processing on the multi-dimensional feature map to obtain a disparity estimation map.

[0023] In some embodiments, before the first image and the second image are respectively input into the multi-scale feature extraction network, the method further includes: using a preprocessing module to perform denoising and edge enhancement processing on the first image and the second image.

[0024] In some embodiments, the multi-scale feature extraction network includes a MobileNetV2 network.

[0025] In some embodiments, the step of obtaining a first multi-scale feature map and a second multi-scale feature map includes: using a multi-scale feature extraction network to perform dimensionality reduction and then dimensionality increase operations on the first image and the second image respectively, and the dimensionality reduction and then dimensionality increase operations include: gradually reducing the resolution of the first image or the second image to 1 / 4, 1 / 8 and 1 / 16 of the original image, and then using an upsampling operation to gradually expand it to 1 / 4 of the original image.

[0026] Specifically, the MobileNetV2 network can be used to extract multi-scale feature maps from the first image and the second image respectively. In order to effectively process objects and texture information of different scales, the MobileNetV2 network processes the input image through a series of convolutional layers to gradually reduce the original resolution. Figure 2 The resolution of the input image can be reduced to 1 / 4, 1 / 8 and 1 / 16 of the original image. The feature map of each scale contains image information at different levels. Among them, the feature map with 1 / 4 resolution retains relatively rough global information and is suitable for capturing the contours and backgrounds of larger objects. The feature maps with smaller resolutions of 1 / 8 and 1 / 16 focus on local detail information, such as the edges, textures and discontinuity areas of objects. This multi-scale feature extraction mechanism ensures that the network can still capture rich image information when processing scenes with complex backgrounds or different depths, which helps to improve the accuracy of subsequent stereo matching and disparity estimation.

[0027] In some embodiments, constructing the correlation-based three-dimensional cost volume includes: generating a three-dimensional cost volume by acquiring feature similarities between the first multi-scale feature map and the second multi-scale feature map at different disparities.

[0028] In some embodiments, performing two-dimensional cost aggregation on a three-dimensional cost volume to obtain an aggregated feature map includes: performing two-dimensional cost aggregation using an inverse residual block, which can achieve enhancement on the channel dimension of the three-dimensional cost volume, thereby effectively capturing more useful feature information.

[0029] In some embodiments, using a reverse residual block for two-dimensional cost aggregation includes: performing a 1×1 convolution operation, a Relu6 activation function activation, a depth 3×3 convolution operation, a Relu6 activation function activation, a 1×1 convolution operation, and a Linear linear activation function activation on the three-dimensional cost volume in sequence to obtain processed features, and performing an aggregation and splicing operation on the processed features and the three-dimensional cost volume in the channel dimension to obtain an aggregated feature map.

[0030] Specifically, refer to Figure 3The reverse residual block first expands the channel dimension through 1×1 convolution and activates it with Relu6 activation function, then uses deep 3×3 convolution and Relu6 activation function to extract spatial features, and finally restores the number of channels to the original dimension through 1×1 convolution and Linear activation function. Finally, the initial input features (3D cost volume) and the features that have undergone a series of processing are aggregated and spliced ​​in the channel dimension, thereby improving the feature expression ability while ensuring computational efficiency.

[0031] In some embodiments, multi-scale feature extraction adopts a deep convolution method, using multiple convolution kernels of different sizes for multi-scale feature extraction, and multiple convolution kernels of different sizes include: 1×1 convolution kernel, 7×1 convolution kernel, 1×7 convolution kernel, 11×1 convolution kernel, 1×11 convolution kernel, 21×1 convolution kernel and 1×21 convolution kernel. It should be noted that these convolution kernels are used to capture features of different scales in the image, especially strip-like structural features, such as object edges and textures. For example, a 7×1 convolution kernel and a 1×7 convolution kernel are used to extract horizontal strip features and vertical strip features in the image, respectively. These convolution kernels are more suitable for capturing the slender structures of objects, such as edges, discontinuous areas, and shape contours of objects than traditional square convolution kernels. This information is very important for disparity estimation because disparity estimation relies on accurate object contour matching. The reason for introducing convolution operations of different scales is that by introducing convolution kernels of different sizes, local and global information of the image can be captured at different scales at the same time. Larger convolution kernels can capture larger contextual information, while smaller convolution kernels focus on local details. This multi-scale operation ensures the comprehensive extraction of image features.

[0032] In some embodiments, multi-scale fusion includes fusing feature maps of different scales obtained by multi-scale feature extraction to obtain a multi-dimensional feature map, and the fusion is completed by splicing or by weighted summation.

[0033] Specifically, for the use of a multi-scale convolutional attention module to perform multi-scale feature extraction and multi-scale fusion on the aggregated feature map in turn to obtain a multi-dimensional feature map, in some examples, refer to Figure 4, after the aggregated feature map is subjected to 1×1 convolution, it is then subjected to 7×1 convolution and 1×7 convolution in sequence, and then to 11×1 convolution and 1×11 convolution in sequence, and then to 21×1 convolution and 1×21 convolution in sequence. Finally, the three features obtained and the original features (aggregated feature map) are spliced ​​together. The multi-scale feature map is spliced ​​along the channel dimension, that is, the feature maps from different convolution kernels are combined to form a multi-dimensional feature map. This splicing method can retain the features extracted by each convolution kernel and merge them into a more expressive feature map. After splicing, 1×1 convolution can be used for channel mixing. The role of channel mixing is to reweight and integrate features of different scales, so that it can focus on important feature information and ignore redundant or irrelevant parts. This step helps to improve the understanding and utilization of features of different scales by the multi-scale convolution attention module, thereby enhancing the accuracy of cost aggregation.

[0034] In some embodiments, performing disparity regression and optimization processing on the multi-dimensional feature map includes: performing Softmax normalization processing on the multi-dimensional feature map to obtain the disparity probability distribution of each pixel, and obtaining a disparity estimation map based on the disparity probability distribution with the highest probability. Specifically, the disparity with the highest probability in the disparity probability distribution of each pixel can be used as the disparity of the pixel, thereby obtaining a complete disparity estimation map. In this way, the disparity information can be effectively captured, ensuring that the final estimation result has a high accuracy.

[0035] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0036] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A binocular stereo matching method with non-episodic correction, characterized in that: include: Using a binocular camera to collect a binocular image, wherein the binocular image includes a first image and a second image; Inputting the first image and the second image into a multi-scale feature extraction network respectively to obtain a first multi-scale feature map corresponding to the first image and a second multi-scale feature map corresponding to the second image; Constructing a correlation-based three-dimensional cost volume based on the first multi-scale feature map and the second multi-scale feature map; Performing two-dimensional cost aggregation on the three-dimensional cost volume to obtain an aggregated feature map; Using a multi-scale convolutional attention module to sequentially perform multi-scale feature extraction and multi-scale fusion on the aggregated feature map to obtain a multi-dimensional feature map; The multi-dimensional feature map is subjected to disparity regression and optimization processing to obtain a disparity estimation map.

2. The binocular stereo matching method with non-epipolar correction according to claim 1, characterized in that: Before the first image and the second image are respectively input into the multi-scale feature extraction network, the method further includes: using a preprocessing module to perform denoising and edge enhancement processing on the first image and the second image.

3. The binocular stereo matching method with non-epipolar correction according to claim 1, characterized in that: The multi-scale feature extraction network includes a MobileNetV2 network.

4. The binocular stereo matching method with non-episodic correction according to claim 1 or 3, characterized in that: The step of obtaining the first multi-scale feature map and the second multi-scale feature map includes: using a multi-scale feature extraction network to perform dimensionality reduction and then dimensionality increase operations on the first image and the second image respectively, and the dimensionality reduction and then dimensionality increase operations include: gradually reducing the resolution of the first image or the second image to 1 / 4, 1 / 8 and 1 / 16 of the original image, and then gradually expanding it to 1 / 4 of the original image.

5. The binocular stereo matching method with non-epipolar correction according to claim 1, characterized in that: Constructing the correlation-based three-dimensional cost volume includes: generating the three-dimensional cost volume by acquiring feature similarities between the first multi-scale feature map and the second multi-scale feature map at different disparities.

6. The binocular stereo matching method with non-episodic correction according to claim 1, characterized in that: Performing two-dimensional cost aggregation on the three-dimensional cost volume to obtain an aggregated feature map includes: performing two-dimensional cost aggregation using a reverse residual block.

7. The binocular stereo matching method with non-epipolar correction according to claim 6, characterized in that: Using the reverse residual block for two-dimensional cost aggregation includes: performing a 1×1 convolution operation, a Relu6 activation function activation, a depth 3×3 convolution operation, a Relu6 activation function activation, a 1×1 convolution operation, and a Linear linear activation function activation on the three-dimensional cost volume in sequence to obtain processed features, and performing an aggregation and splicing operation on the processed features and the three-dimensional cost volume in the channel dimension to obtain the aggregated feature map.

8. The binocular stereo matching method with non-episodic correction according to claim 1, characterized in that: The multi-scale feature extraction adopts a deep convolution method and uses multiple convolution kernels of different sizes for multi-scale feature extraction. The multiple convolution kernels of different sizes include: 1×1 convolution kernel, 7×1 convolution kernel, 1×7 convolution kernel, 11×1 convolution kernel, 1×11 convolution kernel, 21×1 convolution kernel and 1×21 convolution kernel.

9. The binocular stereo matching method with non-episodic correction according to claim 8, characterized in that: The multi-scale fusion includes fusing feature maps of different scales obtained by the multi-scale feature extraction to obtain the multi-dimensional feature map, and the fusion is completed by splicing or by weighted summation.

10. The binocular stereo matching method with non-episodic correction according to claim 1 or 9, characterized in that: The disparity regression and optimization processing of the multi-dimensional feature map includes: performing Softmax normalization processing on the multi-dimensional feature map to obtain a disparity probability distribution of each pixel, and obtaining the disparity estimation map based on the disparity probability distribution with the maximum probability.

Images