Stereo matching method and system based on context geometric cube and distortion parallax optimization

Abstract

The invention discloses a stereo matching method based on context geometric features and distortion parallax optimization, which comprises the following steps of: firstly, shooting a calibrated left picture and a calibrated right picture through a binocular camera, extracting multi-scale context features of the two pictures, and calculating the similarity between feature points according to the feature points of the left feature graph and the right feature graph under different parallaxes; the matching cost is the matching cost, so that the constructed geometric cube contains rich geometric matching information. Meanwhile, context features are copied and expanded into context feature cubes with the same size according to parallax, and rich context information is contained. The geometric cube contains matching conditions of feature points of left and right views with different parallax errors, cost aggregation is guided through context features of the left view, and context feature information, feature point cost matching information under different parallax errors and geometric information of adjacent feature points are preliminarily fused to improve cost matching reliability. And generating a geometric attention weight cube. And filtering the contextual feature cube.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically, to a stereo matching method and system based on contextual geometric cube and distortion parallax optimization. Background Art

[0002] Stereo matching is a basic technology in computer vision, which is used to calculate the disparity (or depth) between corresponding pixels in the rectified images of the same scene taken from two perspectives. This process is based on the principle of stereo vision and simulates the way the human eye obtains depth information. Depth estimation plays a huge role in the real world and is a hot topic in the development of AI today, such as autonomous driving, 3D reconstruction, robotics, and object detection. Although a large number of researchers have made their work public, it is still a major difficulty and challenge to achieve high efficiency, high accuracy, and low computational complexity when processing image edge areas, repetitive structures, textureless and transparent objects.

[0003] There are four existing binocular vision stereo matching methods. The first method is the traditional binocular vision stereo matching of image pixels, which can be divided into four steps: matching cost calculation, cost aggregation, disparity calculation, and disparity optimization. The matching cost calculation calculates the matching cost on the corresponding scan line of the right image for each pixel (or feature point) in the left image; cost aggregation means that the matching of a pixel needs to be constrained by the matching of other surrounding pixels, and the matching of a single pixel cannot be considered in isolation, so as to improve the reliability of cost matching; disparity calculation and optimization calculate the best disparity based on the matching cost and a series of post-processing optimization disparities to obtain more robust disparity predictions; the second method is stereo matching based on CNN cost cube aggregation, which can also be roughly divided into four steps: feature extraction, cost cube construction, cost aggregation, disparity regression and optimization. Its technology can be roughly divided into two categories: 2D and 3D architectures. In the cost cube construction stage, both extract features from the left and right view images to construct the cost cube. In the 2D architecture, a correlation layer (Correlation layer), which is used to calculate the similarity of corresponding pixels in the left and right images; specifically, the 2D architecture constructs the cost cube by calculating the correlation between the feature maps of the left and right images at different disparities; while the 3D architecture concatenates or calculates the feature difference, and this operation is performed over the entire disparity range; this means that the 3D architecture considers more spatial information to process the cost volume. In the cost aggregation stage, the cost volume in the 2D architecture is usually processed by a simple 2D convolution, which can retain some spatial information and perform detailed feature extraction. The 3D architecture processes the cost volume through more complex three-dimensional convolutions, which enables the model to optimize in three dimensions of space and better capture geometric information; the third method is stereo matching based on iterative optimization, which is inspired by the success of iterative refinement in optical flow estimation, especially the RAFT architecture, which bypasses explicit cost aggregation and iteratively updates the disparity of the high-resolution cost cube using a multi-resolution ConvGRU with cross connections; the fourth method is stereo matching based on the Vision Transformer (VIT) architecture. Transformers were originally developed for natural language processing and have shown excellent performance in capturing long-range dependencies and global context information. The key components of stereo matching based on visual transformations include self-attention, cross-attention, and position encoding schemes that provide spatial cues. These methods move away from the traditional cost-volume construction and instead formulate stereo matching as a sequence-to-sequence problem, using attention mechanisms to establish correspondences between pixels in the left and right images.

[0004] However, the above methods all have some drawbacks that cannot be ignored:

[0005] First, the traditional stereo matching method has poor adaptability to complex scenes. For complex scenes (such as occlusion, uneven texture, lighting changes, etc.), the matching effect of this method is often not ideal, especially when there are fewer textures or large lighting changes, this method may have matching errors.

[0006] Second, the stereo matching method based on CNN cost cube aggregation can obtain smaller parameters and good calculation speed based on 2D architecture, but due to the limitation of two-dimensional convolution, the information extracted is limited, resulting in low model accuracy. Based on 3D architecture, it can process spatial and geometric information at the same time, which is more effective in understanding and processing depth information, but it consumes huge memory space;

[0007] Third, the stereo matching method based on iterative optimization iteratively updates the disparity generated by the model, and cannot directly generate the final disparity map end-to-end. It needs to be calculated repeatedly. Each iteration involves matching, cost calculation, optimization and update of the entire image, resulting in high computational complexity and huge consumption of memory and computing time.

[0008] Fourth, the stereo matching method based on the VIT architecture relies on the self-attention mechanism to deal with the long-range dependencies of images, which results in very high computational overhead when processing high-resolution images. It usually requires a large amount of training data and a lot of training time to fully exert its performance, especially in visual tasks such as stereo matching. The ViT model needs to learn the deep feature relationships and geometric information between images. Summary of the invention

[0009] In view of the above defects or improvement needs of the prior art, the present invention provides a stereo matching method and system based on contextual geometric features and distortion parallax optimization, which utilizes rich geometric information and contextual information to design an adaptive fusion context information to guide 3D cost aggregation, and at the same time designs a lightweight 3D regularization network to aggregate the cost volume, thereby reducing the number of parameters and calculation time, and uses accurate cost volume attention weights to filter and splice feature volumes to generate the final feature geometric cost cube. In addition, a disparity optimization module is designed according to the initial disparity obtained by the model to construct a 3D distortion cost volume, which makes the disparity search range more refined, and finds more accurate residuals in the unconstrained residual search space, thereby sacrificing less speed in exchange for higher accuracy. Its purpose is to solve the problem that the existing stereo matching methods based on traditional methods have poor adaptability to complex scenes and are not very suitable for complex scenes. The matching effect of this method is often not ideal, especially when there is less texture or large changes in lighting, the method may have technical problems such as matching errors; the existing CNN-based cost cube aggregation stereo matching method based on 2D architecture can obtain a smaller number of parameters and a good calculation speed, but the model accuracy is not high, and the 3D architecture consumes a huge amount of memory space. The existing stereo matching method based on iterative optimization has a high computational complexity and consumes a huge amount of memory and computing time. The existing stereo matching method based on VIT architecture has a very large computational overhead when processing high-resolution images, and usually requires a large amount of training data and a lot of training time to fully exert its performance, especially in visual tasks such as stereo matching, the ViT model needs to learn the deep feature relationship and geometric information between images.

[0010] To achieve the above object, according to one aspect of the present invention, a stereo matching method based on contextual geometric features and distortion parallax optimization is provided, comprising the following steps:

[0011] (1) Obtain the left and right view images to be matched that have been calibrated by the binocular camera, and preprocess the left and right view images to be matched to obtain preprocessed left and right view images.

[0012] (2) The preprocessed left and right view images obtained in step (1) are input into a pre-trained stereo matching model to obtain a final disparity map as a stereo matching result.

[0013] Preferably, the preprocessing process in step (1) is specifically as follows: first, obtain the left and right view images taken by the binocular camera and their image data, including the left and right view image channel number C, the height H of the left and right view images, and the width W of the left and right view images; then, use the Sobel function in OpenCV to perform edge detection processing on the obtained left and right view images to obtain the edges in the left and right view images; then, perform convolution operations on the left and right view images in the horizontal and vertical directions and normalize them respectively to obtain the left and right view gradient intensity maps in the horizontal and vertical directions respectively; finally, according to the obtained left and right view gradient intensity maps, the irrelevant edge parts in the left and right view images are cropped respectively to make them meet the input image size required by the stereo matching model of the present invention, so as to obtain the preprocessed left and right view images.

[0014] The stereo matching model consists of four parts: a lightweight multi-scale feature extraction network, a 3D regularization network, a distortion parallax optimization module, and a 3D depth-separable convolution module.

[0015] Preferably, the lightweight multi-scale feature extraction network includes three layers:

[0016] The first layer is a multi-scale deep feature extraction layer, whose input is the left and right view images, both of which are four-dimensional tensors of shape (B, C, H, W). The multi-scale deep feature extraction layer uses the pre-trained lightweight feature extraction network MobileNet-V3 in the timm library to perform deep feature extraction on the input left view image and right view image respectively, and changes the number of its input channels to 48, 64, 160, and 192, and outputs 4 four-dimensional tensors corresponding to the left view image and the right view image respectively, whose feature scales are equal to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the feature scales of the left and right view images, and their shapes are (B, 48, H / 4, W / 4), (B, 96, H / 8, W / 8), (B, 160, H / 16, W / 16), and (B, 192, H / 32, W / 32), respectively, where B represents the batch size;

[0017] The second layer is the multi-scale shallow feature extraction layer, whose input is also the left-view image and the right-view image, both of which are four-dimensional tensors of shape (B, C, H, W). The multi-scale shallow feature extraction layer inputs the four-dimensional tensors corresponding to the left-view image and the right-view image into the ordinary convolution layer respectively, and uses a 3×3 convolution kernel, a step size of 2 pixels and a padding of 1 pixel to change the number of input channels of the left-view image and the right-view image to 48, 64, 160, and 192, and outputs 4 four-dimensional tensors corresponding to the left and right view images respectively, whose feature scales are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the feature scales of the left and right view images, and their shapes are (B, 48, H / 4, W / 4), (B, 96, H / 8, W / 8), (B, 160, H / 16, W / 16), and (B, 192, H / 32, W / 32).

[0018] The third layer is the multi-scale fusion layer. Its input is the four-dimensional tensor corresponding to the deep features of the left and right view images obtained in the first layer and the four-dimensional tensor corresponding to the shallow features of the left and right view images obtained in the second layer. The multi-scale fusion layer first concatenates the two four-dimensional tensors of corresponding scales according to the channel dimension to obtain shapes of (B,92,H / 4,W / 4), (B,192,H / 8,W / 8), (B,320,H / 16,W / 16), (B,384, Then, we use ordinary convolution with a kernel of 3×3, a stride of 1, and a padding of 1 to reduce the number of channels of the four concatenated four-dimensional tensors of different scales by 1 / 2, and output multi-scale four-dimensional tensors with shapes of (B,48,H / 4,W / 4), (B,96,H / 8,W / 8), (B,160,H / 16,W / 16), and (B,192,H / 32,W / 32).

[0019] Preferably, the 3D regularization network includes an hourglass network module and a contextual geometry attention module.

[0020] The hourglass network module consists of two parts: downsampling and upsampling;

[0021] The downsampling part includes:

[0022] The first layer decoder takes as input a five-dimensional tensor of shape (B, C, D / 4, H / 4, W / 4). It first feeds the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3, converts the number of channels of the five-dimensional tensor into 2C, and outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8). Then, the five-dimensional tensor is successively fed into a second 3D convolution kernel of dimension 3×3×3 and a BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8), where D represents the disparity of the left and right viewing angles.

[0023] The second layer decoder, whose input is the five-dimensional tensor of the shape (B, 2C, D / 8, H / 8, W / 8) output by the first layer decoder, first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3, converts the number of channels of the five-dimensional tensor into 4C, and outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); then, the five-dimensional tensor is successively input into the second 3D convolution kernel of dimension 3×3×3 and the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16).

[0024] The third-layer decoder takes as input the five-dimensional tensor of the shape (B, 4C, D / 16, H / 16, W / 16) output by the second-layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3, converts the number of channels of the five-dimensional tensor into 6C, and outputs a five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32). Then, the five-dimensional tensor is successively input into the second 3D convolution kernel of dimension 3×3×3 and the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32).

[0025] The upsampling part includes:

[0026] The first-layer encoder takes as input the five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32) output by the downsampled third-layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4, converts the number of channels of the five-dimensional tensor into 4C, and outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); then, the five-dimensional tensor is successively input into the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16).

[0027] The second-layer encoder takes as input the five-dimensional tensor of the first-layer decoder output shape (B, 4C, D / 16, H / 16, W / 16). It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4, converts the number of channels of the five-dimensional tensor into 2C, and outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8). Then, the five-dimensional tensor is successively input into the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8).

[0028] The third-layer encoder takes as input the five-dimensional tensor of the second-layer decoder output shape (B, 2C, D / 8, H / 8, W / 8). It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4, converts the number of channels of the five-dimensional tensor into C, and outputs a five-dimensional tensor of shape (B, C, D / 4, H / 4, W / 4). Then, the five-dimensional tensor is successively input into the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, C, D / 4, H / 4, W / 4).

[0029] Preferably, the contextual geometric attention module includes two inputs, one of which is the left view feature map F c , which is a four-dimensional tensor of shape (B, C, H, W) that contains rich context feature information; the other input is the geometric cube F G , a five-dimensional tensor of shape (B, C, D, H, W), which contains rich geometric information;

[0030] The contextual geometric attention module first expands the left view feature map into a five-dimensional tensor of shape (B, C, D, H, W) according to the disparity dimension, namely the context feature cube F′ c , and the geometric cube F G and context feature cube F′ c Splicing is performed along the channel dimension to obtain a five-dimensional tensor with a shape of (B, 2C, D, H, W), namely a spliced ​​cube; then, the five-dimensional tensor is input into a 3D convolution kernel with a dimension of 1×5×5, and the number of channels of the five-dimensional tensor is reduced to C, thereby outputting a five-dimensional tensor with a shape of (B, C, D, H, W) (which is used to preliminarily fuse context information with geometric information); subsequently, the five-dimensional tensor is activated using the σ (Sigmoid) function (used to determine the area that should be emphasized or suppressed during the fusion process), thereby outputting a five-dimensional tensor with a shape of (B, C, D, H, W), namely a spliced ​​fusion cube A CF , the formula is as follows:

[0031] A CF =σ(conv3d(concat{F′ c ,F G}))

[0032] Among them, concat means that the input tensor is concatenated along the channel dimension, and conv3d means 3D convolution.

[0033] Then, by splicing the fused cube A CF ∈R B×C×D×H×W and the context feature cube F′ cTo multiply element by element, we get the initial context set attention, which is a five-dimensional tensor with a shape of (B, C, D, H, W). Then, we concatenate the five-dimensional tensor with the geometric cube along the channel dimension and output a five-dimensional tensor with a shape of (B, 2C, D, H, W). Finally, we input the five-dimensional tensor into a 3D convolution kernel with a dimension of 1×5×5, and reduce the number of channels of the five-dimensional tensor to C, thereby outputting a five-dimensional tensor (context geometric attention cube G) with a shape of (B, C, D, H, W), which fully integrates the context feature information and geometric information to obtain the context geometric attention cube G containing rich information. The formula is as follows:

[0034] G = conv3d(concat{FG,A CF ⊙F′ c})

[0035] where ⊙ represents element-wise multiplication.

[0036] Preferably, the distortion parallax optimization module includes:

[0037] The first part is the warped disparity transformation layer, whose input is the right view feature map and the initial disparity map. The right view feature map is a four-dimensional tensor with a shape of (B, C, H, W), and the initial disparity map is a four-dimensional tensor with a shape of (B, 1, H, W). The warped disparity transformation layer first creates grid coordinates (xx, yy) according to the number of channels C, height H, and width W of the input right view feature map to represent the coordinates of each pixel in the right view feature map; then, the grid horizontal coordinate xx is distorted and adjusted according to the initial disparity map to obtain the updated grid horizontal coordinate, and the grid horizontal coordinate is adjusted to the standardized range [-1, 1], and the standardized adjusted grid horizontal coordinate is converted to the PyTorch grid format to obtain the distorted grid coordinate (warped xx, warped yy); then, use the nn.functional.grid_sample function and use the distorted grid coordinates after the initial disparity adjustment to perform bilinear interpolation sampling on the right view feature map to obtain a distorted image; finally, create a mask mask with all 1s of the same size as the right view feature map, use the nn.functional.grid_sample function and use the distorted grid coordinates after the initial disparity adjustment to perform bilinear interpolation sampling on the mask mask to obtain the value of the mask at the target position of the distorted grid coordinates to determine the valid pixels in the right view feature map, and retain the valid pixels, thereby obtaining a four-dimensional tensor (distorted right view feature map) with a shape of (B, C, H, W) after parallax distortion.

[0038] The second part is the warped cost cube construction layer, whose input is the warped right view feature map, left view feature map and maximum disparity value maxdisp obtained by the warped disparity transformation layer. The warped cost cube construction layer first creates a zero tensor volume of the same data type as the left view feature map according to the shape of the left view feature map to store the correlation volume; then, the disparity value range [-maxdisp, maxdisp] is determined according to the maximum disparity value maxdisp, all integer disparities in the disparity value range are traversed, and the cosine similarity is used to calculate the correlation between the left view feature map and the warped right view feature map at each integer disparity; finally. The correlations obtained under all integer disparities are stored in the zero tensor volume to obtain the warped cost volume, which is a five-dimensional tensor with a shape of (B, C, D, H, W).

[0039] The third part is the warped disparity optimization layer, whose input is the left view feature map, the initial disparity map, and the warped cost volume obtained in the second layer. The warped disparity optimization layer first subtracts the warped right view feature map from the left view feature map to obtain the reconstruction error, which is a four-dimensional tensor with a shape of (B, C, H, W); then, the reconstruction error, the left view feature map, the initial disparity map, and the warped cost volume are concatenated along the channel dimension to obtain a concatenated volume, which is a four-dimensional tensor with a shape of (B, C ′ , H, W); then, the spliced ​​body and the initial disparity are input into a sub-precision module consisting of 8 convolutional layers for disparity optimization to extract the residual disparity value disp, that is, the disparity dimension value.

[0040] Preferably, for the sub-precision module, its specific structure is:

[0041] The first to fourth layers are four sequentially connected convolutional layers with the same structure. Each convolutional layer contains a sequence of convolution and activation functions. The input of the first layer is a concatenation, which has a shape of (B, C ′ , H, W), the first layer first inputs the four-dimensional tensor into a 3×3 convolution kernel, reduces the input channel of the four-dimensional tensor to C, and outputs a four-dimensional tensor with a shape of (B, C, H, W). Then, a nonlinear transformation is performed through the activation function Mish to output a four-dimensional tensor with a shape of (B, C, H, W). The input of the second layer is the output of the first layer, and so on. Finally, the fourth layer outputs a four-dimensional tensor with a shape of (B, C, H, W).

[0042] The fifth to seventh layers are three sequentially connected residual blocks with the same structure. Their input is the output of the fourth layer, a four-dimensional tensor with a shape of (B, C, H, W). The fifth layer first inputs the four-dimensional tensor into three convolution kernels with a dimension of 3x3, and connects the obtained output residuals, reducing the number of input channels of the four-dimensional tensor to 96, 64, and 32 respectively, and outputs a four-dimensional tensor with a shape of (B, 32, H, W). The input of the sixth layer is the output of the fifth layer, and so on. Finally, the seventh layer outputs a four-dimensional tensor with a shape of (B, 32, H, W).

[0043] The eighth layer is a separate convolutional layer, whose input is the four-dimensional tensor of shape (B, 32, H, W) output by the seventh layer. The eighth layer inputs the four-dimensional tensor into a convolution kernel of dimension 3x3, reduces the number of channels of the four-dimensional tensor to 1, and outputs a four-dimensional tensor of shape (B, 1, H, W), and extracts the residual disparity value disp, i.e., the disparity dimension value, from the four-dimensional tensor.

[0044] The input of the 3D depth separable convolution module is a five-dimensional tensor of shape (B, C, D, H, W). The 3D depth separable convolution module first inputs the five-dimensional tensor into a 3D point convolution kernel of dimension 1×1×1, and increases the input channel of the five-dimensional tensor to tC (where t is the expansion factor); then, the output result is input into the BN+ReLU layer to output a five-dimensional tensor of shape (B, tC, D, H, W); then, the five-dimensional tensor is successively input into a 3D depth convolution kernel of dimension 3×3×3 and the BN+ReLU layer to output a five-dimensional tensor of shape (B, tC, D, H, W); finally, the five-dimensional tensor is input into a 3D point convolution kernel of dimension 1×1×1 to reduce the output channel to C of arbitrary size. ′ , and input the output result into the BN+ReLU layer, and finally output a shape (B, C ′ , D, H, W) five-dimensional tensor.

[0045] Preferably, the stereo matching model is trained by the following steps:

[0046] (2-1) Obtain a stereo image pair dataset, and preprocess the stereo image pair dataset to obtain a preprocessed stereo image pair dataset, wherein the stereo image pair dataset includes a training set, a validation set, and a test set. The training set also includes the disparity of each stereo image pair.

[0047] (2-2) For each stereoscopic image pair in the training set of the stereoscopic image pair data set preprocessed in step (2-1), the left and right view images in the stereoscopic image pair are respectively input into a lightweight multi-scale feature extraction network for multi-scale feature extraction, so as to obtain four four-dimensional tensors corresponding to the stereoscopic image pair, whose scales are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the size of the left and right view images, i.e., left and right view feature maps of scales 1 / 4, 1 / 8, 1 / 16, and 1 / 32, respectively.

[0048] (2-3) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 4 scale left and right viewing angle feature maps f corresponding to the stereo image pair obtained in step (2-2) are l and f r ∈R B ×C×H / 4×W / 4 Construct a 1 / 4 scale 5D cost cube, that is Figure 1 The geometric cube in

[0049] (2-4) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the geometric cube corresponding to the stereo image pair obtained in step (2-3) is input into the downsampling part of the hourglass network module in the 3D regularization network to obtain an initial geometric cube of 1 / 8, 1 / 16, and 1 / 32 scales.

[0050] (2-5) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 32 scale initial geometric cube corresponding to the stereo image pair obtained in step (2-4) is input into the first layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 16 scale geometric cube, and the 1 / 16 scale initial geometric cube obtained by downsampling in step (2-4) is spliced ​​according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 16 scale geometric cube corresponding to the stereo image pair.

[0051] (2-6) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 16 scale geometric cube corresponding to the stereo image pair obtained in step (2-5) and the 1 / 16 scale left view feature map obtained in step (2-2) are input into the context geometry attention module in the 3D regularized network to obtain a 1 / 16 scale context geometry attention weight cube corresponding to the stereo image pair, which ultimately integrates the context feature information and geometric information.

[0052] (2-7) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 16 scale context geometric attention weight cube corresponding to the stereo image pair obtained in (2-6) is input into the second layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 8 scale geometric cube, and it is concatenated with the 1 / 8 scale initial geometric cube obtained by downsampling in step (2-4) according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 8 scale geometric cube corresponding to the stereo image pair.

[0053] (2-8) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale geometric cube corresponding to the stereo image pair obtained in step (2-7) and the feature map of the 1 / 8 scale left view obtained in step (2-2) are input into the context geometry attention module in the 3D regularized network to obtain a 1 / 8 scale context geometry attention weight cube corresponding to the stereo image pair, which ultimately integrates the context feature information and geometric information.

[0054] (2-9) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale context geometric attention weight cube corresponding to the stereo image pair obtained in step (2-8) is input into the third layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 4 scale geometric cube, and it is concatenated with the 1 / 4 scale geometric cube of step (2-3) according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 4 scale geometric cube corresponding to the stereo image pair.

[0055] (2-10) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale geometric cube corresponding to the stereo image pair obtained in step (2-9) and the feature map of the 1 / 4 scale left view obtained in step (2-2) are input into the context geometry attention module in the 3D regularization network to obtain the final 1 / 4 scale context geometry attention weight cube corresponding to the stereo image pair.

[0056] (2-11) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), all weights in the 1 / 4 scale context geometric attention weight cube corresponding to the stereo image pair obtained in step (2-10) are sorted by the disparity value in its disparity dimension, and all weights corresponding to the top 12 disparities are selected as the final context geometric attention weight cube A corresponding to the stereo image pair.cg ∈R C ×D′×H / 4×W / 4 .

[0057] (2-12) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 4 scale left view feature map f corresponding to the stereo image pair obtained in step (2-2) is l ∈R B×C×H / 4×W / 4 , also called the context feature map, is indexed according to the disparity value in step (2-3), the context feature map is increased by the disparity dimension, and the context feature map is copied along the disparity dimension to obtain the context feature cube V corresponding to the stereo image pair of the same shape as the context geometric attention weight cube c ∈R B×C×D′×H / 4×W / 4 .

[0058] (2-13) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the context geometric attention weight cube A corresponding to the stereo image pair obtained in step (2-11) is cg The context feature cube V corresponding to the stereo image pair obtained in the filtering step (2-12) c (i.e., the two are multiplied element by element) to obtain the final context geometric cube V corresponding to the stereo image pair cgv ∈R C×D′×H / 4×W / 4 .

[0059] (2-14) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the context geometry cube obtained in step (2-13) is input into the first two encoder layers and the first two decoder layers of the hourglass network module in the 3D regularization network. cgv Cost aggregation is performed (the context geometry cube is aggregated along the disparity dimension, the channel dimension, and the feature dimension) to obtain a target cube corresponding to the stereo image pair.

[0060] (2-15) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), a softmax function is used to weight all disparity candidate values ​​(i.e., 12 disparities in step (2-11)) of the target cube corresponding to the stereo image pair obtained in step (2-15) at each pixel to obtain the disparity value at the pixel corresponding to the stereo image pair. The disparity values ​​of the target cube at all pixels are combined to obtain an initial disparity map corresponding to the stereo image pair with a disparity dimension of 1.

[0061] (2-16) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the initial disparity map corresponding to the stereo image pair obtained in step (2-15) and the 1 / 4 scale right view feature map obtained in step (2-2) are input into the distorted disparity transformation layer of the disparity optimization module to obtain the 1 / 4 scale distorted right view feature map f corresponding to the stereo image pair. rw,4 .

[0062] (2-17) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 4 scale left view feature map obtained in step (2-2) and the 1 / 4 scale distorted right view feature map f corresponding to the stereo image pair obtained in step (2-16) are converted into rw,4 A bilinear interpolation upsampling operation is performed to obtain a left view feature map and a distorted right view feature map of the corresponding scale of the stereo image pair, and the left view feature map and the distorted right view feature map are input into the distortion cost cube construction layer of the distortion parallax optimization module to obtain a distorted cost volume V corresponding to the stereo image pair. w .

[0063] (2-18) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the left perspective feature map and the distorted right perspective feature map corresponding to the stereo image pair obtained in step (2-17) are subtracted to obtain the reconstruction error corresponding to the stereo image pair.

[0064] (2-19) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the initial disparity map corresponding to the stereo image pair obtained in step (2-14), the left view feature map corresponding to the stereo image pair obtained in step (2-17), and the distorted cost volume V w , and the reconstruction error corresponding to the stereoscopic image pair obtained in step (2-18) is input into the warped parallax layer of the warped parallax optimization module to obtain the fine-grained residual parallax value corresponding to the stereoscopic image pair, and the fine-grained residual parallax value is added to the initial parallax to obtain the refined parallax value corresponding to the stereoscopic image pair.

[0065] (2-20) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 3D convolution operations in steps (2-4) to (2-10) and step (2-14) are modified to 3D depth-separable convolutions, and the above steps (2-4) to (2-19) are repeated to obtain the predicted disparity value of the model corresponding to the stereo image pair;

[0066] The advantage of this step is that, compared with conventional 3D convolution, 3D separable convolution not only ensures that most of the information output by the convolution is preserved, but also greatly reduces the number of parameters and calculation time, greatly improving the efficiency of model training and testing.

[0067] (2-21) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), all disparity values ​​of the context geometry attention weight cube obtained in step (2-10) are weighted using the softmax function to obtain the disparity value of the pixel corresponding to the stereo image pair. The disparity values ​​of the context geometry attention weight cube at all pixels are combined to obtain the context feature geometry attention weight disparity map d corresponding to the stereo image pair with a disparity dimension of 1. att .

[0068] (2-22) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), a loss function is constructed using the predicted disparity value of the model corresponding to the stereo image pair obtained in step (2-20).

[0069] (2-23) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the loss function corresponding to the stereo image pair obtained in step (2-22) is used, and the stereo matching model is iteratively trained using the back propagation method until the stereo matching model converges, thereby obtaining a preliminary trained stereo matching model corresponding to the stereo image pair.

[0070] (2-24) For each stereo image pair in the verification set of the stereo image pair data set preprocessed in step (2-1), the stereo matching model preliminarily trained in step (3-23) is verified until the obtained disparity accuracy reaches the optimal level, thereby obtaining the trained stereo matching model corresponding to the stereo image pair.

[0071] Preferably, the calculation formula of the geometric cube in step (2-3) is as follows:

[0072]

[0073] Where x represents the horizontal coordinate of the pixel on the left or right view feature map, y represents the vertical coordinate of the pixel on the left or right view feature map, d represents the disparity value between the left view feature map and the right view feature map, <,> represents the dot product of the feature vector of the pixel on the left or right view feature map, which is used to measure the similarity between the pixel of the left view image and the corresponding pixel of the right view image, ‖ ‖2 represents the L2 norm of the feature vector of the pixel on the left or right view feature map, and f l(x, y, d) represents the feature vector at the pixel position of the left image feature map with disparity d and coordinates (x, y), f r (x―d, y, d) represents the feature vector at the pixel position of the right image feature map with a parallax of d and a parallax offset of (x―d, y).

[0074] The 1 / 4 scale distorted right view feature map f in step (2-17) rw,4 The calculation formula is as follows:

[0075] f rw,4 =warping(f r,4 ,d0)

[0076] where f r,4 is a 1 / 4 scale right perspective feature map, d0 is the initial disparity value of the pixel in the initial disparity map, and the warping function is the warping disparity operation of the warped disparity transformation layer.

[0077] The distortion in step (2-18) is V w The expression is as follows:

[0078]

[0079] where f l and f rw They are the left view feature map and the distorted right view feature map of the corresponding scale of the stereo image pair, and △d represents the range of fine-grained residual search.

[0080] The loss function formula in step (2-23) is as follows:

[0081]

[0082] where d att and d gt It is the disparity directly obtained by the geometric attention weight of the context feature and the real disparity map provided in the training set. att represents the coefficient of the predicted attention weight disparity, λ represents the coefficient of the final predicted disparity, and λ att The value range of and λ is from 0 to 1, and the sum of the two is 1. represents the smoothed L1 loss.

[0083] According to another aspect of the present invention, a stereo matching system based on contextual geometric features and distortion parallax optimization is provided, comprising:

[0084] The first module is used to obtain the left and right perspective images to be matched that have been calibrated by the binocular camera, and preprocess the left and right perspective images to be matched to obtain preprocessed left and right perspective images.

[0085] The second module is used to input the preprocessed left and right view images obtained by the first module into a pre-trained stereo matching model to obtain a final disparity map as a stereo matching result.

[0086] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0087] 1. Since the present invention adopts step (2-1) and step (2-2), it uses image preprocessing and lightweight multi-scale feature extraction to extract useful features, so that the model can better capture the global information and contextual information in the image when calculating the disparity, thereby improving the matching accuracy in complex scenes.

[0088] 2. Since the present invention adopts steps (2-4) to (2-10) and step (2-14), it uses 3D depth separable convolution. Compared with 2D architecture and 3D architecture, 3D depth separable convolution can not only process spatial and geometric information at the same time, but also explicitly encode the geometric structure in the stereo image, so that the accuracy of the model is greatly improved. At the same time, the number of parameters and calculation time are greatly reduced. The efficiency of model training and testing is greatly improved.

[0089] 3. Since the present invention adopts steps (2-1) to (2-24), it can directly optimize from the original image to the final disparity map through an end-to-end training framework, without explicitly designing feature matching or cost calculation processes to cause memory consumption and increase in computing speed. At the same time, the disparity is optimized by the warped disparity optimization module to obtain a refined disparity.

[0090] 4. Since the present invention adopts steps (2-3) to (2-22), it uses the hourglass network module and the contextual geometric attention module in the 3D regularization network to combine the contextual feature relationship and geometric information between learning images, and performs effective feature fusion and matching in a variety of complex scenes. It can obtain better results with less training data, and the generalization of the model is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 is a schematic diagram of a stereo matching method based on contextual geometric features and distortion parallax optimization of the present invention;

[0092] Figure 2 It is an overall structural block diagram of the stereo matching method based on contextual geometric features and distortion parallax optimization of the present invention;

[0093] Figure 3 Schematic diagram of the contextual geometric attention module used in the present invention;

[0094] Figure 4 is a schematic diagram of the structure of the distortion parallax optimization module used in the present invention;

[0095] Figure 5 It is a schematic diagram of the 3D separable convolution structure designed based on the MobileNet-V2 inverted residual structure of the present invention;

[0096] Figure 6 It is a comparison chart of the stereo matching results of the algorithm of the present application and the algorithm in the related art in the scenceflow public data set;

[0097] Figure 7 It is a comparison chart of the stereo matching results of the algorithm of the present application and the algorithm in the related art in the KITTI autonomous driving public data set;

[0098] Figure 8 It is a comparison chart of the stereo matching generalization results of the algorithm of the present application and the algorithm in the related art in any data set of the present invention. DETAILED DESCRIPTION

[0099] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0100] The basic idea of ​​the present invention is that, on the one hand, the present invention obtains accurate disparity. First, the calibrated left and right pictures are taken by a binocular camera, and the multi-scale context features of the two pictures are extracted. The similarity between the feature points under different disparities corresponding to the left and right feature maps is calculated, that is, the matching cost, so as to construct a geometric cube containing rich geometric matching information. At the same time, the context features are copied and expanded into context feature cubes of the same size according to the disparity, which contain rich context information. The geometric cube contains the matching conditions of the feature points of the left and right views with different disparities. The cost aggregation is guided by the context features of the left view, and the context feature information, the cost matching information of the feature points under different disparities, and the geometric information of the adjacent feature points are initially fused to improve the reliability of the cost matching, generate a geometric attention weight cube, and then filter the context feature cube to the final context geometric cube, so that the context information and the geometric information are fully integrated. Finally, the initial disparity is obtained through disparity regression, and then the distortion optimization constructs a 4D distortion cost volume with the obtained initial disparity. The initial disparity is optimized with image features, initial disparity map and reconstruction error, and the initial disparity search range is narrowed to a fine-grained range, which greatly reduces the difficulty of the network to find the correct residual in the unconstrained residual search space, greatly improving the accuracy of the disparity and the generalization ability of the model. On the other hand, under the premise of obtaining accurate errors, the present invention should consider the speed issue as much as possible. First, the feature extraction uses a lightweight structure. Secondly, the key to the speed lies in the 3D convolution in the model. Therefore, the 3D separable convolution designed according to the inverted residual structure of MobileNet-V2 reduces the model parameters and speeds up the calculation speed while reducing the accuracy as much as possible.

[0101] like Figure 1 As shown, the present invention provides a stereo matching method based on contextual geometric features and distortion parallax optimization, comprising the following steps:

[0102] (1) Obtain the left and right view images to be matched that have been calibrated by the binocular camera, and preprocess the left and right view images to be matched to obtain preprocessed left and right view images.

[0103] Specifically, a binocular camera simultaneously captures a target, thereby obtaining two perspective images, the left perspective image is usually used as the reference image, and the right perspective image is the stereo image to be matched, which is an image captured at different perspectives of the same scene at the same time. By obtaining the parallax between the pixels in the left and right perspective images, the depth information of the target can be obtained based on the parallax.

[0104] The preprocessing process in this step is specifically as follows: first, obtain the left and right view images taken by the binocular camera and their image data, including the number of channels C, height H and width W; then, use the Sobel function in OpenCV to perform edge detection processing on the obtained left and right view images to obtain the edges in the left and right view images (i.e., the places where the pixel values ​​in the image change greatly, which usually correspond to the boundaries or contours of the objects); then, perform convolution operations on the left and right view images in the horizontal and vertical directions and normalize them respectively to obtain the left and right view gradient intensity maps in the horizontal and vertical directions respectively (the left and right view gradient intensity maps can better understand the local features in the left and right view images and identify the edges, textures and other areas); finally, according to the obtained left and right view gradient intensity maps, the irrelevant edge parts of the left and right view images are cropped respectively to meet the input image size required by the stereo matching model of the present invention (the stereo matching model input will perform multi-scale operations, and the input image size needs to be divisible by 32) to obtain the preprocessed left and right view images.

[0105] The advantage of this step is that it unifies the size of the input model images, which facilitates subsequent training.

[0106] (2) The preprocessed left and right view images obtained in step (1) are input into a pre-trained stereo matching model to obtain a final disparity map as a stereo matching result.

[0107] like Figure 2 As shown, the stereo matching model of the present invention includes four parts: a lightweight multi-scale feature extraction network, a 3D regularization network, a distortion parallax optimization module, and a 3D depth separable convolution module, which are connected in sequence.

[0108] Specifically, the lightweight multi-scale feature extraction network consists of three layers:

[0109] The first layer is a multi-scale deep feature extraction layer. Its input is the left and right view images, both of which are four-dimensional tensors of shape (B, C, H, W). The multi-scale deep feature extraction layer uses the pre-trained lightweight feature extraction network MobileNet-V3 in the timm (PyTorch Image Models) library to perform deep feature extraction on the input left view image and right view image respectively, and converts the number of input channels to 48, 64, 160, and 192, and outputs 4 four-dimensional tensors corresponding to the left view image and the right view image respectively. The feature scale is equal to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the feature scale of the left and right view images, and the shapes are (B, 48, H / 4, W / 4), (B, 96, H / 8, W / 8), (B, 160, H / 16, W / 16), and (B, 192, H / 32, W / 32), respectively. B represents the batch size (Batch Size), that is, the number of input images fed into the multi-scale feature extraction network at one time. C represents the number of left and right view images (when C is 3, the input image is usually an RGB image with 3 channels, representing red, green and blue channels respectively). H represents the height of the left and right view images. W represents the width of the left and right view images.

[0110] The second layer is the multi-scale shallow feature extraction layer, whose input is also the left-view image and the right-view image, both of which are four-dimensional tensors of shape (B, C, H, W). The multi-scale shallow feature extraction layer inputs the four-dimensional tensors corresponding to the left-view image and the right-view image into the ordinary convolution layer respectively, and uses a 3×3 convolution kernel, a step size of 2 pixels and a padding of 1 pixel to change the number of input channels of the left-view image and the right-view image to 48, 64, 160, and 192, and outputs 4 four-dimensional tensors corresponding to the left and right view images respectively, whose feature scales are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the feature scales of the left and right view images, and their shapes are (B, 48, H / 4, W / 4), (B, 96, H / 8, W / 8), (B, 160, H / 16, W / 16), and (B, 192, H / 32, W / 32).

[0111] The third layer is the multi-scale fusion layer. Its input is the four-dimensional tensor corresponding to the deep features of the left and right view images obtained in the first layer and the four-dimensional tensor corresponding to the shallow features of the left and right view images obtained in the second layer. The multi-scale fusion layer first concatenates the two four-dimensional tensors of corresponding scales according to the channel dimension to obtain shapes of (B,92,H / 4,W / 4), (B,192,H / 8,W / 8), (B,320,H / 16,W / 16), (B,384, Then, we use ordinary convolution with a kernel of 3×3, a stride of 1, and a padding of 1 to reduce the number of channels of the four concatenated four-dimensional tensors of different scales by 1 / 2, and output multi-scale four-dimensional tensors with shapes of (B,48,H / 4,W / 4), (B,96,H / 8,W / 8), (B,160,H / 16,W / 16), and (B,192,H / 32,W / 32).

[0112] The 3D regularization network includes an hourglass network module and a contextual geometry attention module.

[0113] The hourglass network module consists of two parts: downsampling and upsampling. The downsampling part includes 3 decoders, and the upsampling part includes 3 encoders.

[0114] The specific structure of the downsampling part is:

[0115] The first layer decoder takes as input a five-dimensional tensor (geometric cube) of shape (B, C, D / 4, H / 4, W / 4). It first feeds the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3 (with a stride of 2 and padding of 1), converts the number of channels of the five-dimensional tensor into 2C, and outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8). Then, the five-dimensional tensor is successively fed into a second 3D convolution kernel of dimension 3×3×3 (with a stride of 1 and padding of 1) and a BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8), where D represents the disparity of the left and right viewing angles.

[0116] The second layer decoder, whose input is the five-dimensional tensor of the shape (B, 2C, D / 8, H / 8, W / 8) output by the first layer decoder, first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3 (with a stride of 2 and padding of 1), converts the number of channels of the five-dimensional tensor into 4C, and outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); then, the five-dimensional tensor is successively input into the second 3D convolution kernel of dimension 3×3×3 (with a stride of 1 and padding of 1) and the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16).

[0117] The third layer decoder, whose input is the five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16) output by the second layer decoder, first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3 (with a stride of 2 and padding of 1), converts the number of channels of the five-dimensional tensor into 6C, and outputs a five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32); then, the five-dimensional tensor is successively input into the second 3D convolution kernel of dimension 3×3×3 (with a stride of 1 and padding of 1) and the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32).

[0118] The specific structure of the upsampling part is:

[0119] The first-layer encoder takes as input the five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32) output by the downsampled third-layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4 (with a step size of 2 and padding of 1), converts the number of channels of the five-dimensional tensor into 4C, and outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); then, the five-dimensional tensor is successively input into the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16).

[0120] The second-layer encoder takes as input the five-dimensional tensor of the first-layer decoder output shape (B, 4C, D / 16, H / 16, W / 16). It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4 (with a stride of 2 and padding of 1), converts the number of channels of the five-dimensional tensor into 2C, and outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8). Then, the five-dimensional tensor is successively input into the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8).

[0121] The third-layer encoder takes as input the five-dimensional tensor of the second-layer decoder output shape (B, 2C, D / 8, H / 8, W / 8). It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4 (with a step size of 2 and a padding of 1), converts the number of channels of the five-dimensional tensor into C, and outputs a five-dimensional tensor of shape (B, C, D / 4, H / 4, W / 4). Then, the five-dimensional tensor is successively input into the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, C, D / 4, H / 4, W / 4).

[0122] The contextual geometry attention module is as follows Figure 3As shown, it has two inputs, one of which is the left view feature map F c , which is a four-dimensional tensor of shape (B, C, H, W) containing rich context feature information; the other input is the geometric cube F G , which is a five-dimensional tensor of shape (B, C, D, H, W) and contains rich geometric information. The contextual geometric attention module first expands the left view feature map into a five-dimensional tensor of shape (B, C, D, H, W) according to the disparity dimension, namely the context feature cube F′ c , and the geometric cube F G and context feature cube F′ c Splicing is performed along the channel dimension to obtain a five-dimensional tensor with a shape of (B, 2C, D, H, W), namely a spliced ​​cube; then, the five-dimensional tensor is input into a 3D convolution kernel with a dimension of 1×5×5, and the number of channels of the five-dimensional tensor is reduced to C, thereby outputting a five-dimensional tensor with a shape of (B, C, D, H, W) (which is used to preliminarily fuse context information with geometric information); subsequently, the five-dimensional tensor is activated using the σ (Sigmoid) function (used to determine the area that should be emphasized or suppressed during the fusion process), thereby outputting a five-dimensional tensor with a shape of (B, C, D, H, W), namely a spliced ​​fusion cube A CF , the formula is as follows:

[0123] A CF =σ(conv3d(concat{F′ c ,F G}))

[0124] Among them, concat means that the input tensor is concatenated along the channel dimension, and conv3d means 3D convolution.

[0125] Then, by splicing the fused cube A CF ∈R B×C×D×H×W and the context feature cube F′ c To multiply element by element, we get the initial context set attention, which is a five-dimensional tensor with a shape of (B, C, D, H, W). Then, we concatenate the five-dimensional tensor with the geometric cube along the channel dimension and output a five-dimensional tensor with a shape of (B, 2C, D, H, W). Finally, we input the five-dimensional tensor into a 3D convolution kernel with a dimension of 1×5×5, and reduce the number of channels of the five-dimensional tensor to C, thereby outputting a five-dimensional tensor (context geometric attention cube G) with a shape of (B, C, D, H, W), which fully integrates the context feature information and geometric information to obtain the context geometric attention cube G containing rich information. The formula is as follows:

[0126] G = conv3d(concat{F G,A CF ⊙F′ c})

[0127] where ⊙ represents element-wise multiplication.

[0128] The warped parallax optimization module consists of three parts, such as Figure 4 As shown, specifically:

[0129] The first part is the warped disparity transformation layer, whose input is the right view feature map and the initial disparity map. The right view feature map is a four-dimensional tensor with a shape of (B, C, H, W), and the initial disparity map is a four-dimensional tensor with a shape of (B, 1, H, W). The warped disparity transformation layer first creates grid coordinates (xx, yy) according to the number of channels C, height H, and width W of the input right view feature map to represent the coordinates of each pixel in the right view feature map; then, the grid horizontal coordinate xx is distorted and adjusted according to the initial disparity map to obtain the updated grid horizontal coordinate, and the grid horizontal coordinate is adjusted to the standardized range [-1, 1], and the standardized adjusted grid horizontal coordinate is converted to the PyTorch grid format to obtain the distorted grid coordinate (warped xx, warped yy); then, use the nn.functional.grid_sample function and use the distorted grid coordinates after the initial disparity adjustment to perform bilinear interpolation sampling on the right view feature map to obtain a distorted image; finally, create a mask mask with all 1s of the same size as the right view feature map, use the nn.functional.grid_sample function and use the distorted grid coordinates after the initial disparity adjustment to perform bilinear interpolation sampling on the mask mask to obtain the value of the mask at the target position of the distorted grid coordinates to determine the valid pixels in the right view feature map, and retain the valid pixels, thereby obtaining a four-dimensional tensor (distorted right view feature map) with a shape of (B, C, H, W) after parallax distortion.

[0130] The second part is the warped cost cube construction layer, whose input is the warped right view feature map obtained by the warped disparity transformation layer (which is a four-dimensional tensor with a shape of (B, C, H, W)), the left view feature map (which is a four-dimensional tensor with a shape of (B, C, H, W)) and the maximum disparity value maxdisp (that is, the set disparity threshold). The warped cost cube construction layer first creates a full-zero tensor volume of the same data type as the left view feature map according to the shape of the left view feature map (including the number of channels C, height H and width W) to store the correlation volume; then, according to the maximum disparity value maxdisp, the disparity value range [-maxdisp, maxdisp] is determined, all integer disparities in the disparity value range are traversed, and the cosine similarity is used to calculate the correlation between the left view feature map and the warped right view feature map at each integer disparity; finally. The correlations obtained under all integer disparities are stored in the full-zero tensor volume to obtain the warped cost volume, which is a five-dimensional tensor with a shape of (B, C, D, H, W).

[0131] The third part is the warped disparity optimization layer, whose input is the left view feature map, the initial disparity map (both of which are four-dimensional tensors with a shape of (B, C, H, W)), and the warped cost volume obtained in the second layer. The warped disparity optimization layer first subtracts the warped right view feature map from the left view feature map to obtain a reconstruction error, which is a four-dimensional tensor with a shape of (B, C, H, W); then, the reconstruction error, the left view feature map, the initial disparity map, and the warped cost volume are concatenated along the channel dimension to obtain a concatenated volume, which is a four-dimensional tensor with a shape of (B, C ′ , H, W); then, the spliced ​​body and the initial disparity are input into a sub-precision module consisting of 8 convolutional layers for disparity optimization to extract the residual disparity value disp, that is, the disparity dimension value.

[0132] For the sub-precision module, its specific structure is:

[0133] The first to fourth layers are four sequentially connected convolutional layers with the same structure. Each convolutional layer contains a sequence of convolution and activation functions. The input of the first layer is a concatenation, which has a shape of (B, C ′ , H, W), the first layer first inputs the four-dimensional tensor into a 3×3 convolution kernel, reduces the input channel of the four-dimensional tensor to C, and outputs a four-dimensional tensor with a shape of (B, C, H, W). Then, a nonlinear transformation is performed through the activation function Mish to output a four-dimensional tensor with a shape of (B, C, H, W). The input of the second layer is the output of the first layer, and so on. Finally, the fourth layer outputs a four-dimensional tensor with a shape of (B, C, H, W).

[0134] The fifth to seventh layers are three sequentially connected residual blocks with the same structure. Their input is the output of the fourth layer, a four-dimensional tensor with a shape of (B, C, H, W). The fifth layer first inputs the four-dimensional tensor into three convolution kernels with a dimension of 3x3, and connects the obtained output residuals, reducing the number of input channels of the four-dimensional tensor to 96, 64, and 32 respectively, and outputs a four-dimensional tensor with a shape of (B, 32, H, W). The input of the sixth layer is the output of the fifth layer, and so on. Finally, the seventh layer outputs a four-dimensional tensor with a shape of (B, 32, H, W).

[0135] The eighth layer is a separate convolutional layer, whose input is the four-dimensional tensor of shape (B, 32, H, W) output by the seventh layer. The eighth layer inputs the four-dimensional tensor into a convolution kernel of dimension 3x3, reduces the number of channels of the four-dimensional tensor to 1, and outputs a four-dimensional tensor of shape (B, 1, H, W), and extracts the residual disparity value disp, i.e., the disparity dimension value, from the four-dimensional tensor.

[0136] 3D depth-wise separable convolutional module such as Figure 5 As shown in the figure, MobileNet-V2 first uses 1×1 convolution to achieve dimensionality increase, then uses 3×3DW convolution, and finally uses 1×1 convolution to achieve dimensionality reduction to achieve a structure with small ends and large middle. Inspired by the MobileNet-V2 network structure, deep convolution and point-by-point convolution are also applied in the channel dimension of 3D convolution to obtain a 3D separable convolution module, whose structure is as follows:

[0137] The input of the 3D depth separable convolution module is a five-dimensional tensor of shape (B, C, D, H, W). The 3D depth separable convolution module first inputs the five-dimensional tensor into a 3D point convolution kernel of dimension 1×1×1, and increases the input channel of the five-dimensional tensor to tC (where t is the expansion factor, t=2 in the present invention); then, the output result is input into the BN+ReLU layer, thereby outputting a five-dimensional tensor of shape (B, tC, D, H, W); then, the five-dimensional tensor is successively input into a 3D depth convolution kernel of dimension 3×3×3 and the BN+ReLU layer, thereby outputting a five-dimensional tensor of shape (B, tC, D, H, W); finally, the five-dimensional tensor is input into a 3D point convolution kernel of dimension 1×1×1, and the output channel is reduced to C of arbitrary size. ′ , and input the output result into the BN+ReLU layer, and finally output a shape (B, C ′ , D, H, W) five-dimensional tensor.

[0138] The stereo matching model of the present invention is obtained by training through the following steps:

[0139] (2-1) A stereo image pair (left view image and right view image) dataset is obtained, and the stereo image pair dataset is preprocessed to obtain a preprocessed stereo image pair dataset, wherein the stereo image pair dataset includes a training set, a validation set, and a test set. The training set also includes the disparity of each stereo image pair.

[0140] In other words, the training set in the present invention is used to adjust the trainable weights and bias parameters in the stereo matching model, the validation set is used to adjust the learning rate and other hyperparameters of the stereo matching model, and the test set does not participate in the training of the model, but is used to test the effect of the finally trained stereo matching model.

[0141] Specifically, the preprocessing process in this step is exactly the same as the preprocessing process in the above step (1), and will not be repeated here.

[0142] (2-2) For each stereoscopic image pair in the training set of the stereoscopic image pair data set preprocessed in step (2-1), the left and right view images in the stereoscopic image pair are respectively input into a lightweight multi-scale feature extraction network for multi-scale feature extraction, so as to obtain four four-dimensional tensors corresponding to the stereoscopic image pair, whose scales are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the size of the left and right view images, i.e., left and right view feature maps of scales 1 / 4, 1 / 8, 1 / 16, and 1 / 32, respectively.

[0143] The advantage of this step is that multi-scale features can not only capture the global information of the image, but also help the model understand the local details of the image. By extracting features at multiple scales, the model's ability to represent the image in multiple levels and dimensions can be enhanced, while facilitating information transfer between different scales. By fusing features at different scales, the model can extract complementary information from each scale.

[0144] (2-3) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 4 scale left and right viewing angle feature maps f corresponding to the stereo image pair obtained in step (2-2) are l and f r ∈R B ×C×H / 4×W / 4 Construct a 1 / 4 scale 5D cost cube, that is Figure 1 The geometric cube in

[0145] Specifically, the 5D cost cube is constructed by calculating the similarity of the corresponding matching points of the left and right view feature maps under different parallaxes through cosine similarity, that is, Figure 1 Geometric cube in .

[0146] The calculation formula for the geometric cube is as follows:

[0147]

[0148] Where x represents the horizontal coordinate of the pixel on the left or right view feature map, y represents the vertical coordinate of the pixel on the left or right view feature map, d represents the disparity value between the left view feature map and the right view feature map, <,> represents the dot product of the feature vector of the pixel on the left or right view feature map, which is used to measure the similarity between the pixel of the left view image and the corresponding pixel of the right view image, ‖ ‖2 represents the L2 norm of the feature vector of the pixel on the left or right view feature map, and f l (x, y, d) represents the feature vector at the pixel position of the left image feature map with disparity d and coordinates (x, y), f r (x―d, y, d) represents the feature vector at the pixel position of the right image feature map with a parallax of d and a parallax offset of (x―d, y).

[0149] The advantage of this step is that it only uses the left and right feature maps of 1 / 4 the size of the left and right perspective images to construct the cost cube, which not only retains most of the feature information, but also greatly reduces the computational complexity of the model.

[0150] (2-4) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the geometric cube corresponding to the stereo image pair obtained in step (2-3) is input into the downsampling part of the hourglass network module in the 3D regularization network to obtain an initial geometric cube of 1 / 8, 1 / 16, and 1 / 32 scales.

[0151] (2-5) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 32 scale initial geometric cube corresponding to the stereo image pair obtained in step (2-4) is input into the first layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 16 scale geometric cube, and the 1 / 16 scale initial geometric cube obtained by downsampling in step (2-4) is spliced ​​according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 16 scale geometric cube corresponding to the stereo image pair.

[0152] (2-6) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 16 scale geometric cube corresponding to the stereo image pair obtained in step (2-5) and the 1 / 16 scale left view feature map obtained in step (2-2) are input into the context geometry attention module in the 3D regularized network to obtain a 1 / 16 scale context geometry attention weight cube corresponding to the stereo image pair, which ultimately integrates the context feature information and geometric information.

[0153] (2-7) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 16 scale context geometric attention weight cube corresponding to the stereo image pair obtained in (2-6) is input into the second layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 8 scale geometric cube, and it is concatenated with the 1 / 8 scale initial geometric cube obtained by downsampling in step (2-4) according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 8 scale geometric cube corresponding to the stereo image pair.

[0154] (2-8) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale geometric cube corresponding to the stereo image pair obtained in step (2-7) and the feature map of the 1 / 8 scale left view obtained in step (2-2) are input into the context geometry attention module in the 3D regularized network to obtain a 1 / 8 scale context geometry attention weight cube corresponding to the stereo image pair, which ultimately integrates the context feature information and geometric information.

[0155] (2-9) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale context geometric attention weight cube corresponding to the stereo image pair obtained in step (2-8) is input into the third layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 4 scale geometric cube, and it is concatenated with the 1 / 4 scale geometric cube of step (2-3) according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 4 scale geometric cube corresponding to the stereo image pair.

[0156] (2-10) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale geometric cube corresponding to the stereo image pair obtained in step (2-9) and the feature map of the 1 / 4 scale left view obtained in step (2-2) are input into the context geometry attention module in the 3D regularization network to obtain the final 1 / 4 scale context geometry attention weight cube corresponding to the stereo image pair.

[0157] The advantage of the above steps (2-4) to (2-10) is that, in the process of cost aggregation through the 3D regularization network, not only the rich contextual information between adjacent feature points of the left and right view feature maps is taken into account, but also the rich geometric information between the corresponding feature points of the left and right view feature maps under adjacent parallax is taken into account.

[0158] (2-11) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), all weights in the 1 / 4 scale context geometric attention weight cube corresponding to the stereo image pair obtained in step (2-10) are sorted by importance according to the disparity value in its disparity dimension (that is, the weights are sorted in descending order along the disparity direction), and all weights corresponding to the top 12 disparities are selected as the final context geometric attention weight cube A corresponding to the stereo image pair. cg ∈R C×D′×H / 4×W / 4 .

[0159] The advantage of this step is that, on the one hand, the ultimate goal is to obtain the best matching disparity among these candidate disparities, so the disparity with a smaller weight will not affect the result too much. On the other hand, it can greatly reduce the size of the context geometry attention weight cube, thereby reducing the amount of model training and greatly speeding up the calculation.

[0160] (2-12) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 4 scale left view feature map f corresponding to the stereo image pair obtained in step (2-2) is l ∈R B×C×H / 4×W / 4 , also called the context feature map, is indexed according to the disparity value in step (2-3), the context feature map is increased by the disparity dimension, and the context feature map is copied along the disparity dimension to obtain the context feature cube V corresponding to the stereo image pair of the same shape as the context geometric attention weight cube c ∈R B×C×D′×H / 4×W / 4 .

[0161] (2-13) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the context geometric attention weight cube A corresponding to the stereo image pair obtained in step (2-11) is cg The context feature cube V corresponding to the stereo image pair obtained in the filtering step (2-12) c (i.e., the two are multiplied element by element) to obtain the final context geometric cube V corresponding to the stereo image pair cgv ∈R C×D′×H / 4×W / 4 .

[0162] The specific formula is as follows:

[0163] V cgv =A cg ⊙V c

[0164] The advantage of this step is that the context feature cube is known according to the context geometric attention weight to obtain the cost volume that integrates rich context features and geometric features, so that the model can better capture the global information and context information in the image when calculating the disparity, thereby improving the matching accuracy in complex scenes.

[0165] (2-14) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the context geometry cube obtained in step (2-13) is input into the first two encoder layers and the first two decoder layers of the hourglass network module in the 3D regularization network. cgv Cost aggregation is performed (the context geometry cube is aggregated along the disparity dimension, the channel dimension, and the feature dimension) to obtain a target cube corresponding to the stereo image pair.

[0166] (2-15) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), a softmax function is used to weight all disparity candidate values ​​(i.e., 12 disparities in step (2-11)) of the target cube corresponding to the stereo image pair obtained in step (2-15) at each pixel to obtain the disparity value at the pixel corresponding to the stereo image pair. The disparity values ​​of the target cube at all pixels are combined to obtain an initial disparity map corresponding to the stereo image pair with a disparity dimension of 1.

[0167] The advantages of steps (2-15) and (2-16) are that cost aggregation combines multi-dimensional matching costs to optimize the disparity estimation of each pixel to improve matching accuracy, remove noise, enhance robustness, improve computational efficiency, enhance boundary preservation, and adapt to complex geometric structures.

[0168] (2-16) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the initial disparity map corresponding to the stereo image pair obtained in step (2-15) and the 1 / 4 scale right view feature map obtained in step (2-2) are input into the distorted disparity transformation layer of the disparity optimization module to obtain the 1 / 4 scale distorted right view feature map f corresponding to the stereo image pair. rw,4 .

[0169] Specifically, the 1 / 4 scale distorted right view feature map f rw,4 The calculation formula is as follows:

[0170] f rw,4 =warping(f r,4 ,d0)

[0171] where f r,4is a 1 / 4 scale right perspective feature map, d0 is the initial disparity value of the pixel in the initial disparity map, and the warping function is the warping disparity operation of the warped disparity transformation layer.

[0172] (2-17) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 4 scale left view feature map obtained in step (2-2) and the 1 / 4 scale distorted right view feature map f corresponding to the stereo image pair obtained in step (2-16) are converted into rw,4 A bilinear interpolation upsampling operation is performed to obtain a left view feature map and a distorted right view feature map of the corresponding scale of the stereo image pair, and the left view feature map and the distorted right view feature map are input into the distortion cost cube construction layer of the distortion parallax optimization module to obtain a distorted cost volume V corresponding to the stereo image pair. w .

[0173] Specifically, the distortion body V w The expression is as follows:

[0174]

[0175] where f l and f rw They are the left view feature map and the distorted right view feature map of the corresponding scale of the stereo image pair, and △d represents the range of fine-grained residual search.

[0176] The advantage of this step is that the optimal disparity value is generally distributed within a certain range around the initial disparity, so a fine-grained residual search range △d is defined, that is, the residual search range is narrowed from the initial disparity search range (0, d) to the fine-grained search range (d-△d, d+△d), which reduces the difficulty of the network to find the correct residual disparity in the unconstrained residual disparity search space and greatly speeds up the calculation time.

[0177] (2-18) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the left view feature map and the distorted right view feature map corresponding to the stereo image pair obtained in step (2-17) are subtracted to obtain the reconstruction error corresponding to the stereo image pair (which is used to indicate the incorrect area of ​​the initial disparity).

[0178] Specifically, the expression of reconstruction error is as follows

[0179] Error = f l (x,y,d)―f rw (x―d,y,d)

[0180] (2-19) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the initial disparity map corresponding to the stereo image pair obtained in step (2-15), the left view feature map corresponding to the stereo image pair obtained in step (2-17), and the distorted cost volume V w , and the reconstruction error corresponding to the stereo image pair obtained in step (2-18) is input into the distortion disparity optimization layer of the distortion disparity optimization module to obtain the fine-grained residual disparity value corresponding to the stereo image pair, and the fine-grained residual disparity value is added to the initial disparity to obtain the refined disparity value corresponding to the stereo image pair.

[0181] The advantage of step (2-16) to step (2-19) is that, in theory, the distorted right view feature map should perfectly match the left view feature map. Due to the existence of errors, the present invention needs to further optimize the disparity. The present invention obtains a 5D distorted cost volume based on the cosine similarity between the left view feature and the distorted right view feature at each disparity level, guiding the network to learn the best residual disparity with the maximum similarity.

[0182] (2-20) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), if Figure 2 As shown, the 3D convolution operations in steps (2-4) to (2-10) and step (2-14) are modified into 3D depth-separable convolutions, and the above steps (2-4) to (2-19) are repeated to obtain the predicted disparity value of the model corresponding to the stereo image pair;

[0183] The advantage of this step is that, compared with conventional 3D convolution, 3D separable convolution not only ensures that most of the information output by the convolution is preserved, but also greatly reduces the number of parameters and calculation time, greatly improving the efficiency of model training and testing.

[0184] (2-21) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), all disparity values ​​of the context geometry attention weight cube at each pixel obtained by (2-10) are weighted using the softmax function to obtain the disparity value at the pixel corresponding to the stereo image pair, and the disparity values ​​of the context geometry attention weight cube at all pixels are combined to obtain the context feature geometry attention weight disparity map d corresponding to the stereo image pair with a disparity dimension of 1. att .

[0185] The advantage of this step is that the predicted disparity of the model has a large error in the initial training. The refined training of steps (2-11) to (2-20) will cause a relatively large waste of video memory and computing time. This step first trains part of the model of steps (2-2) to (2-10), which greatly reduces the consumption of video memory and improves computing efficiency.

[0186] (2-22) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), a loss function is constructed using the predicted disparity value of the model corresponding to the stereo image pair obtained in step (2-20).

[0187] Specifically, the loss function formula is as follows:

[0188]

[0189] where d att and d gt It is the disparity directly obtained by the geometric attention weight of the context feature and the real disparity map provided in the training set. att represents the coefficient of the predicted attention weight disparity, λ represents the coefficient of the final predicted disparity, and λ att The value range of and λ is 0 to 1, and the sum of the two is 1. In the present invention, λ att The values ​​of and λ are 0.3 and 0.7, represents the smoothed L1 loss.

[0190] The advantage of this step is that the smoothed L1 loss is robust and has low sensitivity to outliers, ensuring the stability and efficiency of the training process.

[0191] (2-23) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the loss function corresponding to the stereo image pair obtained in step (2-22) is used, and the stereo matching model is iteratively trained using the back propagation method until the stereo matching model converges, thereby obtaining a preliminary trained stereo matching model corresponding to the stereo image pair.

[0192] (2-24) For each stereo image pair in the verification set of the stereo image pair data set preprocessed in step (2-1), the stereo matching model preliminarily trained in step (3-23) is verified until the obtained disparity accuracy reaches the optimal level, thereby obtaining the trained stereo matching model corresponding to the stereo image pair.

[0193] The advantages of steps (2-23) and (2-24) are that they perform iterative training through back propagation and use the validation set to help gradually optimize the model parameters, thereby improving the performance of the final model. As the training progresses, the model can gradually improve the matching accuracy by adjusting and optimizing its feature representation through each iteration.

[0194] Experimental Results

[0195] The present invention is developed using PyTorch and executed on the RTX 3090 GPU. For pre-training on SceneFlow, the input image is normalized using standard deviation statistics and ImageNet mean, and then image enhancement is performed with a probability of 0.2. Enhancement includes color adjustment, brightness enhancement, and contrast modification to simulate different exposure conditions. In addition, random vertical and rotational shifts are applied to the right image, which is then randomly cropped to a resolution of 512×384. The present invention uses the Adam

[37] optimizer with β1=0.9, β2=0.999. The feature geometry attention network is trained for 20 epochs using the scene flow dataset, and then the entire complete network is trained for 20 epochs. The coefficients of the two outputs are set to λ att =0.5, λ=1, the initial learning rate is set to 0.001, and decays by a factor of 2 after epochs 10, 13, 16, and 18.

[0196] In order to illustrate the improvement of the accuracy and speed of the method of the present invention for disparity estimation, tests were conducted on multiple data sets. Taking the SceneFlow data set as an example, the test results obtained by the present invention were compared with the currently commonly used methods. For various state-of-the-art methods such as AANet, GWCNet, and LEAstereo, the present invention improved the computation time (Runtime) and accuracy (EPE) by more than 30% and 50% respectively while keeping the parameter amount (Params) small, achieving the best performance in terms of both speed and accuracy. For real-time and fast methods such as BGNet and CoEx, although the method of the present invention does not have much advantage over them in terms of speed (Runtime) and parameter amount (Params), the performance of the present invention is significantly better than other real-time methods in terms of EPE evaluation index. Compared with Fast-ACVNet, the accuracy performance of the present invention is significantly improved when the computation time (Runtime) is equivalent, and the EPE index is reduced to 0.54. At the same time, compared with HitNet, although the accuracy of the present invention is almost the same as that of HitNet, the computation time of the present invention is reduced by 26%. The experimental results on the scene flow dataset show that the method of the present invention has a good balance between efficiency and performance, and has good results in both accuracy and speed. Figure 6The evaluation results are shown in Table 1. The present invention is also trained and tested on the KITTI dataset, and the comparison with the most advanced method is shown in Table 1. Figure 7 At the same time, under the premise of ensuring the accuracy and speed of the model, the model trained with only the SceneFlow dataset was tested on randomly selected stereo image pairs in the Middlebury dataset and the ETH3D dataset, and the results were compared with those of the advanced model Fast-ACVNet. The comparison chart is shown in the figure below. Figure 8 shown.

[0197]

[0198] Table 1

[0199] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A stereo matching method based on contextual geometric features and distortion parallax optimization, characterized in that: The steps include: (1) Obtain the left and right view images to be matched that have been calibrated by the binocular camera, and preprocess the left and right view images to be matched to obtain preprocessed left and right view images. (2) The preprocessed left and right view images obtained in step (1) are input into a pre-trained stereo matching model to obtain a final disparity map as a stereo matching result.

2. The stereo matching method based on contextual geometric features and distortion parallax optimization according to claim 1, characterized in that: The preprocessing process in step (1) is specifically as follows: first, obtain the left and right view images taken by the binocular camera and their image data, including the left and right view image channel number C, the height H of the left and right view images, and the width W of the left and right view images; then, use the Sobel function in OpenCV to perform edge detection processing on the obtained left and right view images to obtain the edges in the left and right view images; then, perform convolution operations on the left and right view images in the horizontal and vertical directions and normalize them respectively to obtain the left and right view gradient intensity maps in the horizontal and vertical directions respectively; finally, according to the obtained left and right view gradient intensity maps, the irrelevant edge parts in the left and right view images are cropped respectively to make them meet the input image size required by the stereo matching model of the present invention, so as to obtain the preprocessed left and right view images. The stereo matching model consists of four parts: a lightweight multi-scale feature extraction network, a 3D regularization network, a distortion parallax optimization module, and a 3D depth-separable convolution module.

3. The stereo matching method based on contextual geometric features and distortion parallax optimization according to claim 1 or 2, characterized in that: The lightweight multi-scale feature extraction network consists of three layers: The first layer is a multi-scale deep feature extraction layer, whose input is the left and right view images, both of which are four-dimensional tensors of shape (B, C, H, W). The multi-scale deep feature extraction layer uses the pre-trained lightweight feature extraction network MobileNet-V3 in the timm library to perform deep feature extraction on the input left view image and right view image respectively, and changes the number of its input channels to 48, 64, 160, and 192, and outputs 4 four-dimensional tensors corresponding to the left view image and the right view image respectively, whose feature scales are equal to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the feature scales of the left and right view images, and their shapes are (B, 48, H / 4, W / 4), (B, 96, H / 8, W / 8), (B, 160, H / 16, W / 16), and (B, 192, H / 32, W / 32), respectively, where B represents the batch size; The second layer is the multi-scale shallow feature extraction layer, whose input is also the left-view image and the right-view image, both of which are four-dimensional tensors of shape (B, C, H, W). The multi-scale shallow feature extraction layer inputs the four-dimensional tensors corresponding to the left-view image and the right-view image into the ordinary convolution layer respectively, and uses a 3×3 convolution kernel, a step size of 2 pixels and a padding of 1 pixel to change the number of input channels of the left-view image and the right-view image to 48, 64, 160, and 192, and outputs 4 four-dimensional tensors corresponding to the left and right view images respectively, whose feature scales are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the feature scales of the left and right view images, and their shapes are (B, 48, H / 4, W / 4), (B, 96, H / 8, W / 8), (B, 160, H / 16, W / 16), and (B, 192, H / 32, W / 32). The third layer is the multi-scale fusion layer. Its input is the four-dimensional tensor corresponding to the deep features of the left and right view images obtained in the first layer and the four-dimensional tensor corresponding to the shallow features of the left and right view images obtained in the second layer. The multi-scale fusion layer first concatenates the two four-dimensional tensors of corresponding scales according to the channel dimension to obtain shapes of (B,92,H / 4,W / 4), (B,192,H / 8,W / 8), (B,320,H / 16,W / 16), (B,384, Then, we use ordinary convolution with a kernel of 3×3, a stride of 1, and a padding of 1 to reduce the number of channels of the four concatenated four-dimensional tensors of different scales by 1 / 2, and output multi-scale four-dimensional tensors with shapes of (B,48,H / 4,W / 4), (B,96,H / 8,W / 8), (B,160,H / 16,W / 16), and (B,192,H / 32,W / 32).

4. The stereo matching method based on contextual geometric features and distortion parallax optimization according to any one of claims 1 to 3, characterized in that: The 3D regularization network includes an hourglass network module and a contextual geometry attention module. The hourglass network module consists of two parts: downsampling and upsampling; The downsampling part includes: The first layer decoder takes as input a five-dimensional tensor of shape (B, C, D / 4, H / 4, W / 4). It first feeds the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3, converts the number of channels of the five-dimensional tensor into 2C, and outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8). Then, the five-dimensional tensor is successively fed into a second 3D convolution kernel of dimension 3×3×3 and a BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8), where D represents the disparity of the left and right viewing angles. The second layer decoder, whose input is the five-dimensional tensor of the shape (B, 2C, D / 8, H / 8, W / 8) output by the first layer decoder, first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3, converts the number of channels of the five-dimensional tensor into 4C, and outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); then, the five-dimensional tensor is successively input into the second 3D convolution kernel of dimension 3×3×3 and the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16). The third-layer decoder takes as input the five-dimensional tensor of the shape (B, 4C, D / 16, H / 16, W / 16) output by the second-layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3, converts the number of channels of the five-dimensional tensor into 6C, and outputs a five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32). Then, the five-dimensional tensor is successively input into the second 3D convolution kernel of dimension 3×3×3 and the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32). The upsampling part includes: The first-layer encoder takes as input the five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32) output by the downsampled third-layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4, converts the number of channels of the five-dimensional tensor into 4C, and outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); then, the five-dimensional tensor is successively input into the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16). The second-layer encoder takes as input the five-dimensional tensor of the first-layer decoder output shape (B, 4C, D / 16, H / 16, W / 16). It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4, converts the number of channels of the five-dimensional tensor into 2C, and outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8). Then, the five-dimensional tensor is successively input into the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8). The third-layer encoder takes as input the five-dimensional tensor of the second-layer decoder output shape (B, 2C, D / 8, H / 8, W / 8). It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4, converts the number of channels of the five-dimensional tensor into C, and outputs a five-dimensional tensor of shape (B, C, D / 4, H / 4, W / 4). Then, the five-dimensional tensor is successively input into the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, C, D / 4, H / 4, W / 4).

5. The stereo matching method based on contextual geometric features and distortion parallax optimization according to claim 4, characterized in that: The contextual geometric attention module includes two inputs, one of which is the left view feature map F c , which is a four-dimensional tensor of shape (B, C, H, W) that contains rich context feature information; the other input is the geometric cube F G , a five-dimensional tensor of shape (B, C, D, H, W), which contains rich geometric information; The contextual geometric attention module first expands the left view feature map into a five-dimensional tensor of shape (B, C, D, H, W) according to the disparity dimension, namely the context feature cube F c ′ , and the geometric cube F G and context feature cube F c ′ Splicing is performed along the channel dimension to obtain a five-dimensional tensor with a shape of (B, 2C, D, H, W), namely a spliced ​​cube; then, the five-dimensional tensor is input into a 3D convolution kernel with a dimension of 1×5×5, and the number of channels of the five-dimensional tensor is reduced to C, thereby outputting a five-dimensional tensor with a shape of (B, C, D, H, W) (which is used to preliminarily fuse context information with geometric information); subsequently, the five-dimensional tensor is activated using the σ (Sigmoid) function (used to determine the area that should be emphasized or suppressed during the fusion process), thereby outputting a five-dimensional tensor with a shape of (B, C, D, H, W), namely a spliced ​​fusion cube A CF , the formula is as follows: TO CF =σ(conv3d(concat{F′ c ,F G })) Among them, concat means that the input tensor is concatenated along the channel dimension, and conv3d means 3D convolution. Then, by splicing the fused cube A CF ∈R B×C×D×H×W With the context feature cube F c ′ To multiply element by element, we get the initial context set attention, which is a five-dimensional tensor with a shape of (B, C, D, H, W). Then, we concatenate the five-dimensional tensor with the geometric cube along the channel dimension and output a five-dimensional tensor with a shape of (B, 2C, D, H, W). Finally, we input the five-dimensional tensor into a 3D convolution kernel with a dimension of 1×5×5, and reduce the number of channels of the five-dimensional tensor to C, thereby outputting a five-dimensional tensor (context geometric attention cube G) with a shape of (B, C, D, H, W), which fully integrates the context feature information and geometric information to obtain the context geometric attention cube G containing rich information. The formula is as follows: G=Conv3d(concat{F G ,A CF ⊙F′ c }) where ⊙ represents element-wise multiplication.

6. The stereo matching method based on contextual geometric features and distortion parallax optimization according to claim 5, characterized in that: The warped parallax optimization module includes: The first part is the warped disparity transformation layer, whose input is the right view feature map and the initial disparity map. The right view feature map is a four-dimensional tensor with a shape of (B, C, H, W), and the initial disparity map is a four-dimensional tensor with a shape of (B, 1, H, W). The warped disparity transformation layer first creates grid coordinates (xx, yy) according to the number of channels C, height H, and width W of the input right view feature map to represent the coordinates of each pixel in the right view feature map; then, the grid horizontal coordinate xx is distorted and adjusted according to the initial disparity map to obtain the updated grid horizontal coordinate, and the grid horizontal coordinate is adjusted to the standardized range [-1, 1], and the standardized adjusted grid horizontal coordinate is converted to the PyTorch grid format to obtain the distorted grid coordinate (warped xx, warped yy); then, use the nn.functional.grid_sample function and use the distorted grid coordinates after the initial disparity adjustment to perform bilinear interpolation sampling on the right view feature map to obtain a distorted image; finally, create a mask mask with all 1s of the same size as the right view feature map, use the nn.functional.grid_sample function and use the distorted grid coordinates after the initial disparity adjustment to perform bilinear interpolation sampling on the mask mask to obtain the value of the mask at the target position of the distorted grid coordinates to determine the valid pixels in the right view feature map, and retain the valid pixels, thereby obtaining a four-dimensional tensor (distorted right view feature map) with a shape of (B, C, H, W) after parallax distortion. The second part is the warped cost cube construction layer, whose input is the warped right view feature map, left view feature map and maximum disparity value maxdisp obtained by the warped disparity transformation layer. The warped cost cube construction layer first creates a zero tensor volume of the same data type as the left view feature map according to the shape of the left view feature map to store the correlation volume; then, the disparity value range [-maxdisp, maxdisp] is determined according to the maximum disparity value maxdisp, all integer disparities in the disparity value range are traversed, and the cosine similarity is used to calculate the correlation between the left view feature map and the warped right view feature map at each integer disparity; finally. The correlations obtained under all integer disparities are stored in the zero tensor volume to obtain the warped cost volume, which is a five-dimensional tensor with a shape of (B, C, D, H, W). The third part is the warped disparity optimization layer, whose input is the left view feature map, the initial disparity map, and the warped cost volume obtained in the second layer. The warped disparity optimization layer first subtracts the warped right view feature map from the left view feature map to obtain the reconstruction error, which is a four-dimensional tensor with a shape of (B, C, H, W); then, the reconstruction error, the left view feature map, the initial disparity map, and the warped cost volume are concatenated along the channel dimension to obtain a concatenated volume, which is a four-dimensional tensor with a shape of (B, C ′ , H, W); then, the spliced ​​body and the initial disparity are input into a sub-precision module consisting of 8 convolutional layers for disparity optimization to extract the residual disparity value disp, that is, the disparity dimension value.

7. The stereo matching method based on contextual geometric features and distortion parallax optimization according to claim 6, characterized in that: For the sub-precision module, its specific structure is: The first to fourth layers are four sequentially connected convolutional layers with the same structure. Each convolutional layer contains a sequence of convolution and activation functions. The input of the first layer is a concatenation, which has a shape of (B, C ′ , H, W), the first layer first inputs the four-dimensional tensor into a 3×3 convolution kernel, reduces the input channel of the four-dimensional tensor to C, and outputs a four-dimensional tensor with a shape of (B, C, H, W). Then, a nonlinear transformation is performed through the activation function Mish to output a four-dimensional tensor with a shape of (B, C, H, W). The input of the second layer is the output of the first layer, and so on. Finally, the fourth layer outputs a four-dimensional tensor with a shape of (B, C, H, W). The fifth to seventh layers are three sequentially connected residual blocks with the same structure. Their input is the output of the fourth layer, a four-dimensional tensor with a shape of (B, C, H, W). The fifth layer first inputs the four-dimensional tensor into three convolution kernels with a dimension of 3x3, and connects the obtained output residuals, reducing the number of input channels of the four-dimensional tensor to 96, 64, and 32 respectively, and outputs a four-dimensional tensor with a shape of (B, 32, H, W). The input of the sixth layer is the output of the fifth layer, and so on. Finally, the seventh layer outputs a four-dimensional tensor with a shape of (B, 32, H, W). The eighth layer is a separate convolutional layer, whose input is the four-dimensional tensor of shape (B, 32, H, W) output by the seventh layer. The eighth layer inputs the four-dimensional tensor into a convolution kernel of dimension 3x3, reduces the number of channels of the four-dimensional tensor to 1, and outputs a four-dimensional tensor of shape (B, 1, H, W), and extracts the residual disparity value disp, i.e., the disparity dimension value, from the four-dimensional tensor. The input of the 3D depth separable convolution module is a five-dimensional tensor of shape (B, C, D, H, W). The 3D depth separable convolution module first inputs the five-dimensional tensor into a 3D point convolution kernel of dimension 1×1×1, and increases the input channel of the five-dimensional tensor to tC (where t is the expansion factor); Then, the output result is input into the BN+ReLU layer to output a five-dimensional tensor of shape (B, tC, D, H, W); then, the five-dimensional tensor is successively input into a 3D deep convolution kernel of dimension 3×3×3 and the BN+ReLU layer to output a five-dimensional tensor of shape (B, tC, D, H, W); finally, the five-dimensional tensor is input into a 3D point convolution kernel of dimension 1×1×1 to reduce the output channel to C of arbitrary size. ′ , and input the output result into the BN+ReLU layer, and finally output a shape (B, C ′ , D, H, W) five-dimensional tensor.

8. The stereo matching method based on contextual geometric features and distortion parallax optimization according to claim 7, characterized in that: The stereo matching model is trained through the following steps: (2-1) Obtain a stereo image pair dataset, and preprocess the stereo image pair dataset to obtain a preprocessed stereo image pair dataset, wherein the stereo image pair dataset includes a training set, a validation set, and a test set. The training set also includes the disparity of each stereo image pair. (2-2) For each stereoscopic image pair in the training set of the stereoscopic image pair data set preprocessed in step (2-1), the left and right view images in the stereoscopic image pair are respectively input into a lightweight multi-scale feature extraction network for multi-scale feature extraction, so as to obtain four four-dimensional tensors corresponding to the stereoscopic image pair, whose scales are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the size of the left and right view images, i.e., left and right view feature maps of scales 1 / 4, 1 / 8, 1 / 16, and 1 / 32, respectively. (2-3) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 4 scale left and right viewing angle feature maps f corresponding to the stereo image pair obtained in step (2-2) are l and f r ∈R B×C×H / 4×W / 4 Construct a 1 / 4 scale 5D cost cube, which is the geometric cube in Figure 1; (2-4) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the geometric cube corresponding to the stereo image pair obtained in step (2-3) is input into the downsampling part of the hourglass network module in the 3D regularization network to obtain an initial geometric cube of 1 / 8, 1 / 16, and 1 / 32 scales. (2-5) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 32 scale initial geometric cube corresponding to the stereo image pair obtained in step (2-4) is input into the first layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 16 scale geometric cube, and the 1 / 16 scale initial geometric cube obtained by downsampling in step (2-4) is spliced ​​according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 16 scale geometric cube corresponding to the stereo image pair. (2-6) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 16 scale geometric cube corresponding to the stereo image pair obtained in step (2-5) and the 1 / 16 scale left view feature map obtained in step (2-2) are input into the context geometry attention module in the 3D regularized network to obtain a 1 / 16 scale context geometry attention weight cube corresponding to the stereo image pair, which ultimately integrates the context feature information and geometric information. (2-7) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 16 scale context geometric attention weight cube corresponding to the stereo image pair obtained in (2-6) is input into the second layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 8 scale geometric cube, and it is concatenated with the 1 / 8 scale initial geometric cube obtained by downsampling in step (2-4) according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 8 scale geometric cube corresponding to the stereo image pair. (2-8) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale geometric cube corresponding to the stereo image pair obtained in step (2-7) and the feature map of the 1 / 8 scale left view obtained in step (2-2) are input into the context geometry attention module in the 3D regularized network to obtain a 1 / 8 scale context geometry attention weight cube corresponding to the stereo image pair, which ultimately integrates the context feature information and geometric information. (2-9) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale context geometric attention weight cube corresponding to the stereo image pair obtained in step (2-8) is input into the third layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 4 scale geometric cube, and it is concatenated with the 1 / 4 scale geometric cube of step (2-3) according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 4 scale geometric cube corresponding to the stereo image pair. (2-10) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale geometric cube corresponding to the stereo image pair obtained in step (2-9) and the feature map of the 1 / 4 scale left view obtained in step (2-2) are input into the context geometry attention module in the 3D regularization network to obtain the final 1 / 4 scale context geometry attention weight cube corresponding to the stereo image pair. (2-11) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), all weights in the 1 / 4 scale context geometric attention weight cube corresponding to the stereo image pair obtained in step (2-10) are sorted by the disparity value in its disparity dimension, and all weights corresponding to the top 12 disparities are selected as the final context geometric attention weight cube A corresponding to the stereo image pair. cg ∈R C ×D′×H / 4×W / 4 . (2-12) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 4 scale left view feature map f corresponding to the stereo image pair obtained in step (2-2) is l ∈R B×C×H / 4×W / 4 , also called the context feature map, is indexed according to the disparity value in step (2-3), the context feature map is increased by the disparity dimension, and the context feature map is copied along the disparity dimension to obtain the context feature cube V corresponding to the stereo image pair of the same shape as the context geometric attention weight cube c ∈R B×C×D′×H / 4×W / 4 . (2-13) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the context geometric attention weight cube A corresponding to the stereo image pair obtained in step (2-11) is cg The context feature cube V corresponding to the stereo image pair obtained in the filtering step (2-12) c (i.e., the two are multiplied element by element) to obtain the final context geometric cube V corresponding to the stereo image pair cgv ∈R C×D′×H / 4×W / 4 . (2-14) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the context geometry cube obtained in step (2-13) is input into the first two encoder layers and the first two decoder layers of the hourglass network module in the 3D regularization network. cgv Cost aggregation is performed (the context geometry cube is aggregated along the disparity dimension, the channel dimension, and the feature dimension) to obtain a target cube corresponding to the stereo image pair. (2-15) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), a softmax function is used to weight all disparity candidate values ​​(i.e., 12 disparities in step (2-11)) of the target cube corresponding to the stereo image pair obtained in step (2-15) at each pixel to obtain the disparity value at the pixel corresponding to the stereo image pair. The disparity values ​​of the target cube at all pixels are combined to obtain an initial disparity map corresponding to the stereo image pair with a disparity dimension of 1. (2-16) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the initial disparity map corresponding to the stereo image pair obtained in step (2-15) and the 1 / 4 scale right view feature map obtained in step (2-2) are input into the distorted disparity transformation layer of the disparity optimization module to obtain the 1 / 4 scale distorted right view feature map f corresponding to the stereo image pair. rw,4 . (2-17) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 4 scale left view feature map obtained in step (2-2) and the 1 / 4 scale distorted right view feature map f corresponding to the stereo image pair obtained in step (2-16) are converted into rw,4 A bilinear interpolation upsampling operation is performed to obtain a left view feature map and a distorted right view feature map of the corresponding scale of the stereo image pair, and the left view feature map and the distorted right view feature map are input into the distortion cost cube construction layer of the distortion parallax optimization module to obtain a distorted cost volume V corresponding to the stereo image pair. w . (2-18) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the left perspective feature map and the distorted right perspective feature map corresponding to the stereo image pair obtained in step (2-17) are subtracted to obtain the reconstruction error corresponding to the stereo image pair. (2-19) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the initial disparity map corresponding to the stereo image pair obtained in step (2-14), the left view feature map corresponding to the stereo image pair obtained in step (2-17), and the distorted cost volume V w , and the reconstruction error corresponding to the stereoscopic image pair obtained in step (2-18) is input into the warped parallax layer of the warped parallax optimization module to obtain the fine-grained residual parallax value corresponding to the stereoscopic image pair, and the fine-grained residual parallax value is added to the initial parallax to obtain the refined parallax value corresponding to the stereoscopic image pair. (2-20) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 3D convolution operations in steps (2-4) to (2-10) and step (2-14) are modified to 3D depth-separable convolutions, and the above steps (2-4) to (2-19) are repeated to obtain the predicted disparity value of the model corresponding to the stereo image pair; The advantage of this step is that, compared with conventional 3D convolution, 3D separable convolution not only ensures that most of the information output by the convolution is preserved, but also greatly reduces the number of parameters and calculation time, greatly improving the efficiency of model training and testing. (2-21) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), all disparity values ​​of the context geometry attention weight cube obtained in step (2-10) are weighted using the softmax function to obtain the disparity value of the pixel corresponding to the stereo image pair. The disparity values ​​of the context geometry attention weight cube at all pixels are combined to obtain the context feature geometry attention weight disparity map d corresponding to the stereo image pair with a disparity dimension of 1. att . (2-22) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), a loss function is constructed using the predicted disparity value of the model corresponding to the stereo image pair obtained in step (2-20). (2-23) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the loss function corresponding to the stereo image pair obtained in step (2-22) is used, and the stereo matching model is iteratively trained using the back propagation method until the stereo matching model converges, thereby obtaining a preliminary trained stereo matching model corresponding to the stereo image pair. (2-24) For each stereo image pair in the verification set of the stereo image pair data set preprocessed in step (2-1), the stereo matching model preliminarily trained in step (3-23) is verified until the obtained disparity accuracy reaches the optimal level, thereby obtaining the trained stereo matching model corresponding to the stereo image pair.

9. The stereo matching method based on contextual geometric features and distortion parallax optimization according to claim 8, characterized in that: The calculation formula of the geometric cube in step (2-3) is as follows: Where x represents the horizontal coordinate of the pixel on the left or right view feature map, y represents the vertical coordinate of the pixel on the left or right view feature map, d represents the disparity value between the left view feature map and the right view feature map, <,> represents the dot product of the feature vector of the pixel on the left or right view feature map, which is used to measure the similarity between the pixel of the left view image and the corresponding pixel of the right view image, ‖‖2 represents the L2 norm of the feature vector of the pixel on the left or right view feature map, and f l (x, y, d) represents the feature vector at the pixel position of the left image feature map with disparity d and coordinates (x, y), f r (xd,y,d) represents the feature vector at the pixel position of the right image feature map with a parallax of d and a parallax offset of (xd,y). The 1 / 4 scale distorted right view feature map f in step (2-17) rw,4 The calculation formula is as follows: f rw,4 =warping(f r,4 ,d0) where f r,4 is a 1 / 4 scale right perspective feature map, d0 is the initial disparity value of the pixel in the initial disparity map, and the warping function is the warping disparity operation of the warped disparity transformation layer. The distortion in step (2-18) is V w The expression is as follows: where f l and f rw They are the left view feature map and the distorted right view feature map of the corresponding scale of the stereo image pair, and △d represents the range of fine-grained residual search. The loss function formula in step (2-23) is as follows: where d att and d gt It is the disparity directly obtained by the geometric attention weight of the context feature and the real disparity map provided in the training set. att represents the coefficient of the predicted attention weight disparity, λ represents the coefficient of the final predicted disparity, and λ att The value range of and λ is from 0 to 1, and the sum of the two is 1. represents the smoothed L1 loss.

10. A stereo matching system based on contextual geometric features and distortion parallax optimization, characterized in that: include: The first module is used to obtain the left and right perspective images to be matched that have been calibrated by the binocular camera, and preprocess the left and right perspective images to be matched to obtain preprocessed left and right perspective images. The second module is used to input the preprocessed left and right view images obtained by the first module into a pre-trained stereo matching model to obtain a final disparity map as a stereo matching result.

Images