Stereo matching method and system based on contextual geometric cube and warped parallax optimization

Abstract

The present invention discloses a stereo matching method based on contextual geometric features and distortion parallax optimization. First, two calibrated left and right pictures are captured by a binocular camera, and multi-scale context features of the two pictures are extracted. The similarity between feature points under different parallaxes corresponding to the left and right feature maps is calculated, that is, the matching cost, thereby constructing a geometric cube containing rich geometric matching information. At the same time, the context features are copied and expanded into context feature cubes of the same size according to the parallax, which contain rich context information. The geometric cube contains the matching status of feature points of the left and right views with different parallaxes. The cost aggregation is guided by the context features of the left view, and the context feature information, the cost matching information of feature points under different parallaxes, and the geometric information of adjacent feature points are preliminarily fused to improve the reliability of cost matching and generate a geometric attention weight cube. The context feature cube is used to filter the context feature cube.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically, to a stereo matching method and system based on contextual geometric cube and distortion parallax optimization. Background Art

[0002] Stereo matching is a fundamental technique in computer vision. It calculates the disparity (or depth) between corresponding pixels in rectified images of the same scene captured from two perspectives. This process, based on the principles of stereo vision, mimics the way the human eye acquires depth information. Depth estimation plays a significant role in real-world applications and is a hot topic in current AI developments, such as autonomous driving, 3D reconstruction, robotics, and object detection. Despite the extensive research published, achieving high efficiency, accuracy, and computational complexity when processing image edges, repetitive structures, textureless objects, and transparent objects remains a significant challenge.

[0003] There are roughly four existing binocular vision stereo matching methods. The first method is the traditional binocular vision stereo matching of image pixels, which can be divided into four steps: matching cost calculation, cost aggregation, disparity calculation, and disparity optimization. The matching cost calculation calculates the matching cost for each pixel (or feature point) in the left image on the corresponding scan line of the right image; cost aggregation means that the matching of a pixel needs to be constrained by the matching of other surrounding pixels, and the matching of a single pixel cannot be considered in isolation, thereby improving the reliability of cost matching; disparity calculation and optimization calculate the best disparity based on the matching cost and a series of post-processing optimization disparities to obtain more robust disparity predictions; the second is stereo matching based on CNN cost cube aggregation, which can also be roughly divided into four steps: feature extraction, cost cube construction, cost aggregation, disparity regression and optimization. Its technologies can be roughly divided into two categories: 2D and 3D architectures. In the cost cube construction stage, both extract features from the left and right view images to construct the cost cube. In the 2D architecture, a correlation layer (Correlation Layer) is usually applied to the features extracted from the left and right images. The 2D architecture constructs the cost volume by calculating the correlation between the feature maps of the left and right images at different disparities. The 3D architecture concatenates or calculates feature differences across the entire disparity range, which means that the 3D architecture considers more spatial information when processing the cost volume. During the cost aggregation stage, the cost volume in the 2D architecture is typically processed using simple two-dimensional convolutions, which preserves some spatial information and allows for detailed feature extraction. The 3D architecture processes the cost volume through more complex three-dimensional convolutions, which enables the model to optimize in three dimensions of space and better capture geometric information. The third method is stereo matching based on iterative optimization, which is inspired by the success of iterative refinement in optical flow estimation, especially the RAFT architecture. By bypassing explicit cost aggregation, the high-resolution cost cube is iteratively updated with disparity using a multi-resolution ConvGRU with cross connections. The fourth method is stereo matching based on the Vision Transformer (VIT) architecture. Transformers were originally developed for natural language processing and have shown excellent performance in capturing long-range dependencies and global context information. The key components of visual transformation-based stereo matching include self-attention, cross-attention, and position encoding schemes that provide spatial cues. These methods move away from the traditional cost-volume construction and instead formulate stereo matching as a sequence-to-sequence problem, using attention mechanisms to establish correspondences between pixels in the left and right images.

[0004] However, the above methods all have some drawbacks that cannot be ignored:

[0005] First, traditional stereo matching methods have poor adaptability to complex scenes. For complex scenes (such as occlusion, uneven texture, lighting changes, etc.), the matching effect of this method is often unsatisfactory, especially when there is less texture or large lighting changes, this method may result in matching errors.

[0006] Second, the CNN-based cost cube aggregation stereo matching method, based on a 2D architecture, can achieve a smaller number of parameters and a good computational speed. However, due to the limitations of two-dimensional convolution, the information extracted is limited, resulting in low model accuracy. A 3D architecture can simultaneously process spatial and geometric information and is more effective in understanding and processing depth information, but consumes a huge amount of memory space.

[0007] Third, the stereo matching method based on iterative optimization iteratively updates the disparity generated by the model and cannot directly generate the final disparity map end-to-end. It requires multiple repeated calculations. Each iteration involves matching the entire image, cost calculation, optimization update, and other operations, resulting in high computational complexity and consuming huge amounts of memory and computing time.

[0008] Fourth, the stereo matching method based on the VIT architecture relies on the self-attention mechanism to handle the long-range dependencies of images, which results in very high computational overhead when processing high-resolution images. It usually requires a large amount of training data and a lot of training time to fully realize its performance. Especially in visual tasks such as stereo matching, the ViT model needs to learn the deep feature relationships and geometric information between images. Summary of the Invention

[0009] In response to the above defects or improvement needs of the prior art, the present invention provides a stereo matching method and system based on contextual geometric features and distortion disparity optimization, which utilizes rich geometric information and contextual information to design an adaptive fusion context information to guide 3D cost aggregation, and at the same time designs a lightweight 3D regularization network to aggregate cost volume, thereby reducing the number of parameters and calculation time, and accurately uses the cost volume attention weight to filter the spliced ​​feature volume to generate the final feature geometric cost cube. In addition, a disparity optimization module is designed according to the initial disparity obtained by the model to construct a 3D distortion cost volume, which makes the disparity search range more refined, and finds more accurate residuals in the unconstrained residual search space, thereby sacrificing less speed in exchange for higher accuracy. Its purpose is to solve the problem that the existing stereo matching methods based on traditional methods have poor adaptability to complex scenes, and are not very adaptable to complex scenes. The matching effect of this method is often not ideal, especially when there is less texture or large changes in lighting, and the method may have technical problems such as matching errors; the existing CNN-based cost cube aggregation stereo matching method is based on a 2D architecture, which can obtain a smaller number of parameters and a good calculation speed, but the model accuracy is not high; the 3D architecture consumes a huge amount of memory space; the existing stereo matching method based on iterative optimization has a high computational complexity and consumes a huge amount of memory and computing time; the existing stereo matching method based on the VIT architecture has a very high computational overhead when processing high-resolution images, and usually requires a large amount of training data and a lot of training time to fully exert its performance, especially in visual tasks such as stereo matching, the ViT model needs to learn the deep feature relationships and geometric information between images.

[0010] To achieve the above object, according to one aspect of the present invention, a stereo matching method based on contextual geometric features and distortion parallax optimization is provided, comprising the following steps:

[0011] (1) Obtain the left and right perspective images to be matched that have been calibrated by the binocular camera, and preprocess the left and right perspective images to be matched to obtain preprocessed left and right perspective images.

[0012] (2) The preprocessed left and right view images obtained in step (1) are input into a pre-trained stereo matching model to obtain the final disparity map as the stereo matching result.

[0013] Preferably, the preprocessing process in step (1) is specifically as follows: first, obtaining the left and right perspective images and image data thereof taken by the binocular camera, including the left and right perspective image channel number C, the height H of the left and right perspective images, and the width W of the left and right perspective images; then, using the Sobel function in OpenCV to perform edge detection processing on the obtained left and right perspective images to obtain the edges in the left and right perspective images; then, performing convolution operations on the left and right perspective images in the horizontal and vertical directions and normalizing them respectively to obtain left and right perspective gradient intensity maps in the horizontal and vertical directions respectively; finally, according to the obtained left and right perspective gradient intensity maps, the irrelevant edge parts of the left and right perspective images are cropped respectively to make them meet the input image size required by the stereo matching model of the present invention, so as to obtain the preprocessed left and right perspective images.

[0014] The stereo matching model consists of four sequentially connected parts: a lightweight multi-scale feature extraction network, a 3D regularization network, a warped parallax optimization module, and a 3D depth-separable convolution module.

[0015] Preferably, the lightweight multi-scale feature extraction network includes three layers:

[0016] The first layer is a multi-scale deep feature extraction layer. Its input is the left and right view images, both of which are four-dimensional tensors of shape (B, C, H, W). The multi-scale deep feature extraction layer uses the pre-trained lightweight feature extraction network MobileNet-V3 in the timm library to perform deep feature extraction on the input left view image and right view image respectively, and changes the number of input channels to 48, 64, 160, and 192. It outputs four four-dimensional tensors corresponding to the left view image and the right view image respectively, whose feature scales are equal to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the feature scales of the left and right view images, and their shapes are (B, 48, H / 4, W / 4), (B, 96, H / 8, W / 8), (B, 160, H / 16, W / 16), and (B, 192, H / 32, W / 32), respectively, where B represents the batch size;

[0017] The second layer is a multi-scale shallow feature extraction layer, whose input is also the left view image and the right view image, both of which are four-dimensional tensors of shape (B, C, H, W). The multi-scale shallow feature extraction layer inputs the four-dimensional tensors corresponding to the left view image and the right view image into the ordinary convolution layer respectively, and uses a 3×3 convolution kernel, a step size of 2 pixels and a padding of 1 pixel to change the number of input channels of the left view image and the right view image to 48, 64, 160 and 192 respectively, and outputs four four-dimensional tensors corresponding to the left and right view images respectively, whose feature scales are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the feature scales of the left and right view images, and their shapes are (B, 48, H / 4, W / 4), (B, 96, H / 8, W / 8), (B, 160, H / 16, W / 16) and (B, 192, H / 32, W / 32).

[0018] The third layer is the multi-scale fusion layer, whose input is the four-dimensional tensor corresponding to the deep features of the left and right view images obtained in the first layer and the four-dimensional tensor corresponding to the shallow features of the left and right view images obtained in the second layer. The multi-scale fusion layer first concatenates the two four-dimensional tensors of corresponding scales according to the channel dimension to obtain the shapes of (B,92,H / 4,W / 4), (B,192,H / 8,W / 8), (B,320,H / 16,W / 16), (B,384, The concatenated 4D tensors of four different scales (H / 32, W / 32) are constructed. Then, ordinary convolution with a 3×3 kernel, a stride of 1, and a padding of 1 is used to reduce the number of channels of the four concatenated 4D tensors of different scales by 1 / 2, and the output multi-scale 4D tensors are (B, 48, H / 4, W / 4), (B, 96, H / 8, W / 8), (B, 160, H / 16, W / 16), and (B, 192, H / 32, W / 32).

[0019] Preferably, the 3D regularization network includes an hourglass network module and a contextual geometry attention module.

[0020] The hourglass network module consists of two parts: downsampling and upsampling;

[0021] The downsampling part includes:

[0022] The first layer decoder takes as input a five-dimensional tensor of shape (B, C, D / 4, H / 4, W / 4). It first feeds the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3, converts the number of channels of the five-dimensional tensor into 2C, and outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8). The five-dimensional tensor is then fed into a second 3D convolution kernel of dimension 3×3×3 and a BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8), where D represents the disparity between the left and right view images.

[0023] The second-layer decoder takes as input the five-dimensional tensor of the shape (B, 2C, D / 8, H / 8, W / 8) output by the first-layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3, converts the number of channels of the five-dimensional tensor into 4C, and outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); then, the five-dimensional tensor is input into the second 3D convolution kernel of dimension 3×3×3 and the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16).

[0024] The third-layer decoder takes as input the five-dimensional tensor of the shape (B, 4C, D / 16, H / 16, W / 16) output by the second-layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3, converts the number of channels of the five-dimensional tensor into 6C, and outputs a five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32); then, the five-dimensional tensor is input into the second 3D convolution kernel of dimension 3×3×3 and the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32).

[0025] The upsampling part includes:

[0026] The first-layer encoder takes as input the five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32) output by the downsampled third-layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4, converts the number of channels of the five-dimensional tensor into 4C, and outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); then, the five-dimensional tensor is input into the BN+ReLU layer successively, and finally outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16).

[0027] The second-layer encoder takes as input the five-dimensional tensor of the first-layer decoder output shape (B, 4C, D / 16, H / 16, W / 16). It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4, converts the number of channels of the five-dimensional tensor into 2C, and outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8); then, the five-dimensional tensor is input into the BN+ReLU layer successively, and finally outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8).

[0028] The third-layer encoder takes as input the five-dimensional tensor output by the second-layer decoder with a shape of (B, 2C, D / 8, H / 8, W / 8). It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4, converts the number of channels of the five-dimensional tensor into C, and outputs a five-dimensional tensor with a shape of (B, C, D / 4, H / 4, W / 4); then, the five-dimensional tensor is input into the BN+ReLU layer in sequence, and finally outputs a five-dimensional tensor with a shape of (B, C, D / 4, H / 4, W / 4).

[0029] Preferably, the contextual geometric attention module includes two inputs, one of which is the left view feature map F c , which is a four-dimensional tensor of shape (B, C, H, W) that contains rich context feature information; the other input is the geometric cube F G , a five-dimensional tensor of shape (B, C, D, H, W), which contains rich geometric information;

[0030] The contextual geometric attention module first expands the left view feature map into a five-dimensional tensor of shape (B, C, D, H, W) according to the disparity dimension, namely the context feature cube F′ c , and the geometric cube F G and context feature cube F′ c Splicing is performed along the channel dimension to obtain a five-dimensional tensor with a shape of (B, 2C, D, H, W), that is, a splicing cube; then, the five-dimensional tensor is input into a 3D convolution kernel with a dimension of 1×5×5, and the number of channels of the five-dimensional tensor is reduced to C, thereby outputting a five-dimensional tensor with a shape of (B, C, D, H, W) (which is used to preliminarily fuse contextual information with geometric information); subsequently, the five-dimensional tensor is activated using the σ (Sigmoid) function (used to determine the area that should be emphasized or suppressed during the fusion process), thereby outputting a five-dimensional tensor with a shape of (B, C, D, H, W), that is, a splicing fusion cube A CF , the formula is as follows:

[0031] A CF =σ(conv3d(concat{F′ c ,F G}))

[0032] Among them, concat means that the input tensor is spliced ​​along the channel dimension, and conv3d means 3D convolution.

[0033] Then, by splicing the fused cube A CF ∈R B×C×D×H×W and context feature cube F′ cTo multiply element-by-element, we get the initial contextual set attention, which is a five-dimensional tensor of shape (B, C, D, H, W). Subsequently, this five-dimensional tensor is concatenated with the geometric cube along the channel dimension to output a five-dimensional tensor of shape (B, 2C, D, H, W). Finally, this five-dimensional tensor is input into a 3D convolution kernel of dimension 1×5×5, which reduces the number of channels of the five-dimensional tensor to C, thereby outputting a five-dimensional tensor (contextual geometric attention cube G) of shape (B, C, D, H, W). It fully integrates contextual feature information and geometric information to obtain a contextual geometric attention cube G containing rich information. Its formula is as follows:

[0034] G=conv3d(concat{FG,A CF ⊙F′ c})

[0035] where ⊙ represents element-wise multiplication.

[0036] Preferably, the distortion parallax optimization module includes:

[0037] The first part is the warped disparity transformation layer, whose input is the right view feature map and the initial disparity map. The right view feature map is a four-dimensional tensor with a shape of (B, C, H, W), and the initial disparity map is a four-dimensional tensor with a shape of (B, 1, H, W). The warped disparity transformation layer first creates grid coordinates (xx, yy) according to the number of channels C, height H, and width W of the input right view feature map to represent the coordinates of each pixel in the right view feature map; then, the grid horizontal coordinate xx is distorted and adjusted according to the initial disparity map to obtain the updated grid horizontal coordinate, which is adjusted to the standardized range [-1, 1] and converted to the PyTorch grid format to obtain the warped grid coordinates after initial disparity adjustment (warped xx, warped yy); then, use the nn.functional.grid_sample function and use the distorted grid coordinates after initial disparity adjustment to perform bilinear interpolation sampling on the right view feature map to obtain a distorted image; finally, create a mask mask of the same size as the right view feature map with all 1s, use the nn.functional.grid_sample function and use the distorted grid coordinates after initial disparity adjustment to perform bilinear interpolation sampling on the mask mask to obtain the value of the mask at the target position of the distorted grid coordinates to determine the valid pixels in the right view feature map, and retain the valid pixels, thereby obtaining a four-dimensional tensor (distorted right view feature map) with a shape of (B, C, H, W) after parallax distortion.

[0038] The second part is the warped cost cube construction layer, whose input is the warped right-view feature map, left-view feature map, and maximum disparity value maxdisp obtained by the warped disparity transformation layer. The warped cost cube construction layer first creates an all-zero tensor volume of the same data type as the left-view feature map based on its shape to store the correlation volume. Then, based on the maximum disparity value maxdisp, it determines the disparity value range [-maxdisp, maxdisp], traverses all integer disparities in the disparity value range, and uses cosine similarity to calculate the correlation between the left-view feature map and the warped right-view feature map at each integer disparity. Finally, the obtained correlations at all integer disparities are stored in the all-zero tensor volume to obtain the warped cost volume, which is a five-dimensional tensor with a shape of (B, C, D, H, W).

[0039] The third part is the warped disparity optimization layer, whose input is the left view feature map, the initial disparity map, and the warped cost volume obtained in the second layer. The warped disparity optimization layer first subtracts the warped right view feature map from the left view feature map to obtain the reconstruction error, which is a four-dimensional tensor with a shape of (B, C, H, W); then, the reconstruction error, the left view feature map, the initial disparity map, and the warped cost volume are spliced ​​along the channel dimension to obtain a spliced ​​volume, which is a four-dimensional tensor with a shape of (B, C ′ , H, W) four-dimensional tensor; then, the spliced ​​body and the initial disparity are input into a sub-precision module consisting of 8 convolutional layers for disparity optimization to extract the residual disparity value disp, that is, the disparity dimension value.

[0040] Preferably, for the sub-precision module, its specific structure is:

[0041] The first to fourth layers are four sequentially connected convolutional layers with the same structure. Each convolutional layer contains a sequence of convolution and activation functions. The input of the first layer is a concatenation, which has a shape of (B, C ′ , H, W), the first layer first inputs the four-dimensional tensor into a 3×3 convolution kernel, reduces the input channel of the four-dimensional tensor to C, and outputs a four-dimensional tensor with a shape of (B, C, H, W). Then, a nonlinear transformation is performed through the activation function Mish to output a four-dimensional tensor with a shape of (B, C, H, W). The input of the second layer is the output of the first layer, ..., and so on. Finally, the fourth layer outputs a four-dimensional tensor with a shape of (B, C, H, W).

[0042] The fifth to seventh layers are three sequentially connected residual blocks with the same structure. Their input is the output four-dimensional tensor of the fourth layer with a shape of (B, C, H, W). The fifth layer first inputs the four-dimensional tensor into three 3x3 convolution kernels, and connects the obtained output residuals, reducing the number of input channels of the four-dimensional tensor to 96, 64, and 32 respectively, and outputs a four-dimensional tensor with a shape of (B, 32, H, W). The input of the sixth layer is the output of the fifth layer, and so on. Finally, the seventh layer outputs a four-dimensional tensor with a shape of (B, 32, H, W).

[0043] The eighth layer is a separate convolutional layer. Its input is the four-dimensional tensor of shape (B, 32, H, W) output by the seventh layer. The eighth layer inputs the four-dimensional tensor into a 3x3 convolution kernel, reduces the number of channels of the four-dimensional tensor to 1, and outputs a four-dimensional tensor of shape (B, 1, H, W). The residual disparity value disp, that is, the disparity dimension value, is extracted from the four-dimensional tensor.

[0044] The input of the 3D depth-wise separable convolution module is a five-dimensional tensor of shape (B, C, D, H, W). The 3D depth-wise separable convolution module first inputs the five-dimensional tensor into a 3D point convolution kernel of dimension 1×1×1, and increases the input channel of the five-dimensional tensor to tC (where t is the expansion factor); then, the output result is input into the BN+ReLU layer to output a five-dimensional tensor of shape (B, tC, D, H, W); then, the five-dimensional tensor is successively input into a 3D depth-wise convolution kernel of dimension 3×3×3 and a BN+ReLU layer to output a five-dimensional tensor of shape (B, tC, D, H, W); finally, the five-dimensional tensor is input into a 3D point convolution kernel of dimension 1×1×1 to reduce the output channel to C of arbitrary size. ′ And input the output result into the BN+ReLU layer, and finally output a shape (B, C ′ , D, H, W) five-dimensional tensor.

[0045] Preferably, the stereo matching model is trained by the following steps:

[0046] (2-1) Obtain a stereo image pair dataset, and preprocess the stereo image pair dataset to obtain a preprocessed stereo image pair dataset, wherein the stereo image pair dataset includes a training set, a validation set, and a test set. The training set also includes the disparity of each stereo image pair.

[0047] (2-2) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the left and right perspective pictures in the stereo image pair are respectively input into the lightweight multi-scale feature extraction network for multi-scale feature extraction to obtain four four-dimensional tensors corresponding to the stereo image pair, whose scales are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the size of the left and right perspective pictures, i.e., left and right perspective feature maps of scales 1 / 4, 1 / 8, 1 / 16, and 1 / 32, respectively.

[0048] (2-3) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 4 scale left and right view feature maps f corresponding to the stereo image pair obtained in step (2-2) are l and f r ∈R B ×C×H / 4×W / 4 Construct a 1 / 4 scale 5D cost cube, that is Figure 1 The geometric cube in

[0049] (2-4) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the geometric cube corresponding to the stereo image pair obtained in step (2-3) is input into the downsampling part of the hourglass network module in the 3D regularization network to obtain initial geometric cubes of 1 / 8, 1 / 16, and 1 / 32 scales.

[0050] (2-5) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 32 scale initial geometric cube corresponding to the stereo image pair obtained in step (2-4) is input into the first layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 16 scale geometric cube, and the 1 / 16 scale initial geometric cube obtained by downsampling in step (2-4) is spliced ​​according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 16 scale geometric cube corresponding to the stereo image pair.

[0051] (2-6) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 16 scale geometric cube corresponding to the stereo image pair obtained in step (2-5) and the 1 / 16 scale left view feature map obtained in step (2-2) are input into the contextual geometric attention module in the 3D regularization network to obtain a 1 / 16 scale contextual geometric attention weight cube corresponding to the stereo image pair, which ultimately integrates the contextual feature information and geometric information.

[0052] (2-7) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 16 scale context geometric attention weight cube corresponding to the stereo image pair obtained in (2-6) is input into the second layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 8 scale geometric cube, and it is spliced ​​with the 1 / 8 scale initial geometric cube obtained by downsampling in step (2-4) according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 8 scale geometric cube corresponding to the stereo image pair.

[0053] (2-8) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale geometric cube corresponding to the stereo image pair obtained in step (2-7) and the 1 / 8 scale feature map of the left view obtained in step (2-2) are input into the contextual geometric attention module in the 3D regularization network to obtain a 1 / 8 scale contextual geometric attention weight cube corresponding to the stereo image pair, which ultimately integrates the contextual feature information and geometric information.

[0054] (2-9) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale context geometric attention weight cube corresponding to the stereo image pair obtained in step (2-8) is input into the third layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 4 scale geometric cube, and it is spliced ​​with the 1 / 4 scale geometric cube of step (2-3) according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 4 scale geometric cube corresponding to the stereo image pair.

[0055] (2-10) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale geometric cube corresponding to the stereo image pair obtained in step (2-9) and the feature map of the 1 / 4 scale left view obtained in step (2-2) are input into the context geometry attention module in the 3D regularization network to obtain the final 1 / 4 scale context geometry attention weight cube corresponding to the stereo image pair.

[0056] (2-11) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), all weights in the 1 / 4 scale context geometric attention weight cube corresponding to the stereo image pair obtained in step (2-10) are sorted by the importance of the disparity value in its disparity dimension, and all weights corresponding to the top 12 disparities are selected as the final context geometric attention weight cube A corresponding to the stereo image pair.cg ∈R C ×D′×H / 4×W / 4 .

[0057] (2-12) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 4 scale left view feature map f corresponding to the stereo image pair obtained in step (2-2) is l ∈R B×C×H / 4×W / 4 , also known as the context feature map, is indexed according to the disparity value in step (2-3), the context feature map is increased by the disparity dimension, and the context feature map is copied along the disparity dimension to obtain the context feature cube V corresponding to the stereo image pair with the same shape as the context geometric attention weight cube c ∈R B×C×D′×H / 4×W / 4 .

[0058] (2-13) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the context geometric attention weight cube A corresponding to the stereo image pair obtained in step (2-11) is cg The context feature cube V corresponding to the stereo image pair obtained in the filtering step (2-12) c (ie, the two are multiplied element by element) to obtain the final context geometric cube V corresponding to the stereo image pair cgv ∈R C×D′×H / 4×W / 4 .

[0059] (2-14) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the context geometry cube obtained in step (2-13) is input into the first two encoder layers and the first two decoder layers of the hourglass network module in the 3D regularization network. cgv Cost aggregation is performed (the context geometric cube is aggregated along the disparity dimension, the channel dimension, and the feature dimension) to obtain the target cube corresponding to the stereo image pair.

[0060] (2-15) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), use a softmax function to weight all disparity candidate values ​​at each pixel of the target cube corresponding to the stereo image pair obtained in step (2-15) (i.e., the 12 disparities in step (2-11)) to obtain the disparity value at the pixel corresponding to the stereo image pair. The disparity values ​​of the target cube at all pixels are combined to obtain an initial disparity map corresponding to the stereo image pair with a disparity dimension of 1.

[0061] (2-16) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the initial disparity map corresponding to the stereo image pair obtained in step (2-15) and the 1 / 4 scale right view feature map obtained in step (2-2) are input into the warped disparity transformation layer of the disparity optimization module to obtain the 1 / 4 scale warped right view feature map f corresponding to the stereo image pair. rw,4 .

[0062] (2-17) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 4 scale left view feature map obtained in step (2-2) and the 1 / 4 scale distorted right view feature map f corresponding to the stereo image pair obtained in step (2-16) are converted into rw,4 Perform bilinear interpolation upsampling to obtain the left view feature map and the distorted right view feature map of the corresponding scale of the stereo image pair, and input the left view feature map and the distorted right view feature map into the distortion cost cube construction layer of the distortion parallax optimization module to obtain the distortion cost volume V corresponding to the stereo image pair. w .

[0063] (2-18) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the left perspective feature map and the distorted right perspective feature map corresponding to the stereo image pair obtained in step (2-17) are subtracted to obtain the reconstruction error corresponding to the stereo image pair.

[0064] (2-19) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the initial disparity map corresponding to the stereo image pair obtained in step (2-14), the left view feature map corresponding to the stereo image pair obtained in step (2-17), and the distorted cost volume V w , and the reconstruction error corresponding to the stereo image pair obtained in step (2-18) is input into the warped parallax layer of the warped parallax optimization module to obtain the fine-grained residual parallax value corresponding to the stereo image pair, and the fine-grained residual parallax value is added to the initial parallax to obtain the refined parallax value corresponding to the stereo image pair.

[0065] (2-20) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), modify the 3D convolution operations in steps (2-4) to (2-10) and step (2-14) to 3D depth-separable convolution, and repeat the above steps (2-4) to (2-19) to obtain the predicted disparity value of the model corresponding to the stereo image pair;

[0066] The advantage of this step is that, compared with conventional 3D convolution, 3D separable convolution not only ensures that most of the information output by the convolution is preserved, but also greatly reduces the number of parameters and calculation time, greatly improving the efficiency of model training and testing.

[0067] (2-21) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), all disparity values ​​of the context geometry attention weight cube obtained in step (2-10) are weighted using the softmax function to obtain the disparity value of the pixel corresponding to the stereo image pair. The disparity values ​​of the context geometry attention weight cube at all pixels are combined to obtain the context feature geometry attention weight disparity map d corresponding to the stereo image pair with a disparity dimension of 1. att .

[0068] (2-22) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), a loss function is constructed using the predicted disparity value of the model corresponding to the stereo image pair obtained in step (2-20).

[0069] (2-23) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the loss function corresponding to the stereo image pair obtained in step (2-22) is used, and the stereo matching model is iteratively trained using the back propagation method until the stereo matching model converges, thereby obtaining a preliminary trained stereo matching model corresponding to the stereo image pair.

[0070] (2-24) For each stereo image pair in the verification set of the stereo image pair dataset preprocessed in step (2-1), the stereo matching model preliminarily trained in step (3-23) is verified until the obtained disparity accuracy reaches the optimal level, thereby obtaining the trained stereo matching model corresponding to the stereo image pair.

[0071] Preferably, the calculation formula of the geometric cube in step (2-3) is as follows:

[0072]

[0073] Where x represents the horizontal coordinate of the pixel on the left or right view feature map, y represents the vertical coordinate of the pixel on the left or right view feature map, d represents the disparity value between the left view feature map and the right view feature map, <,> represents the dot product of the feature vector of the pixel on the left or right view feature map, which is used to measure the similarity between the pixel of the left view image and the corresponding pixel of the right view image, ‖ ‖2 represents the L2 norm of the feature vector of the pixel on the left or right view feature map, and f l(x, y, d) represents the feature vector at the pixel position of the left image feature map with disparity d and coordinates (x, y), f r (x―d,y,d) represents the feature vector at the pixel position of the right image feature map with disparity d and parallax offset coordinates (x―d,y).

[0074] The 1 / 4 scale distorted right view feature map f in step (2-17) rw,4 The calculation formula is as follows:

[0075] f rw,4 =warping(f r,4 ,d0)

[0076] where f r,4 is the right perspective feature map at 1 / 4 scale, d0 is the initial disparity value of the pixel point in the initial disparity map, and the warpping function is the warping disparity operation of the warped disparity transformation layer.

[0077] The distortion in step (2-18) is the original body V w The expression is as follows:

[0078]

[0079] where f l and f rw are the left view feature map and the distorted right view feature map of the corresponding scale of the stereo image pair, and △d represents the range of fine-grained residual search.

[0080] The loss function formula in step (2-23) is as follows:

[0081]

[0082] where d att and d gt It is the disparity directly obtained from the geometric attention weight of the context feature and the real disparity map provided in the training set. att represents the coefficient of the predicted attention weight disparity, λ represents the coefficient of the final predicted disparity, λ att The values ​​of and λ range from 0 to 1, and their sum is 1. represents the smoothed L1 loss.

[0083] According to another aspect of the present invention, a stereo matching system based on contextual geometric features and warped parallax optimization is provided, comprising:

[0084] The first module is used to obtain the left and right perspective images to be matched that have been calibrated by the binocular camera, and preprocess the left and right perspective images to be matched to obtain preprocessed left and right perspective images.

[0085] The second module is used to input the preprocessed left and right view images obtained by the first module into a pre-trained stereo matching model to obtain a final disparity map as a stereo matching result.

[0086] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0087] 1. Since the present invention adopts steps (2-1) and (2-2), it uses image preprocessing and lightweight multi-scale feature extraction to extract useful features, so that the model can better capture the global information and contextual information in the image when calculating disparity, thereby improving the matching accuracy in complex scenes.

[0088] 2. Because the present invention employs steps (2-4) to (2-10) and step (2-14), which utilize 3D depthwise separable convolution, compared to 2D and 3D architectures, 3D depthwise separable convolution can not only simultaneously process spatial and geometric information, but also explicitly encode the geometric structure in stereo images, thereby significantly improving the accuracy of the model. At the same time, the number of parameters and computation time are significantly reduced, greatly improving the efficiency of model training and testing.

[0089] 3. Because the present invention employs steps (2-1) to (2-24), it can optimize directly from the original image to the final disparity map through an end-to-end training framework, without explicitly designing feature matching or cost calculation processes that would increase memory consumption and computational speed. Furthermore, the warped disparity optimization module optimizes the disparity to obtain refined disparity.

[0090] 4. Since the present invention adopts steps (2-3) to (2-22), it uses the hourglass network module and the contextual geometric attention module in the 3D regularization network to combine the contextual feature relationship and geometric information between learning images, and performs effective feature fusion and matching in a variety of complex scenes. It can obtain better results with less training data, and the generalization of the model is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 is a schematic diagram of a stereo matching method based on contextual geometric features and distortion parallax optimization according to the present invention;

[0092] Figure 2 This is a block diagram of the overall structure of the stereo matching method based on contextual geometric features and distortion parallax optimization of the present invention;

[0093] Figure 3 Schematic diagram of the contextual geometric attention module used in the present invention;

[0094] Figure 4 Schematic diagram of the structure of the distortion parallax optimization module used in the present invention;

[0095] Figure 5 This is a schematic diagram of the 3D separable convolution structure designed based on the MobileNet-V2 inverted residual structure of the present invention;

[0096] Figure 6 This is a comparison chart of the stereo matching results of the algorithm of the present invention and the algorithm in the related art in the ScenceFlow public dataset;

[0097] Figure 7 This is a comparison chart of the stereo matching results of the algorithm of the present invention and the algorithm in the related art in the KITTI autonomous driving public dataset;

[0098] Figure 8 This is a comparison chart of the stereo matching generalization results of the algorithm of the present application and the algorithm in the related art in any data set of the present invention. DETAILED DESCRIPTION

[0099] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0100] The basic idea of ​​the present invention is that, on the one hand, the present invention obtains accurate disparity. First, the calibrated left and right pictures are taken by a binocular camera, and the multi-scale context features of the two pictures are extracted. The similarity between the feature points under different disparities corresponding to the left and right feature maps is calculated, that is, the matching cost, so as to construct a geometric cube containing rich geometric matching information. At the same time, the context features are copied and expanded into context feature cubes of the same size according to the disparity, which contain rich context information. The geometric cube contains the matching status of the feature points of the left and right views with different disparities. The cost aggregation is guided by the context features of the left view, and the context feature information, the cost matching information of the feature points under different disparities, and the geometric information of the adjacent feature points are preliminarily fused to improve the reliability of the cost matching, generate a geometric attention weight cube, and then filter the context feature cube to the final context geometric cube, so that the context information and geometric information are fully integrated. Finally, the initial disparity is obtained through disparity regression, and then the warping optimization is used to construct a 4D warped cost volume with the obtained initial disparity. The initial disparity is optimized with the image features, the initial disparity map, and the reconstruction error, and the initial disparity search range is narrowed to a fine-grained range, which greatly reduces the difficulty of the network in finding the correct residual in the unconstrained residual search space, greatly improving the accuracy of the disparity and the generalization ability of the model. On the other hand, under the premise of obtaining accurate errors, the present invention takes speed into consideration as much as possible. First, a lightweight structure is used for feature extraction. Secondly, the key to speed is the 3D convolution in the model. Therefore, the 3D separable convolution designed according to the inverted residual structure of MobileNet-V2 reduces the model parameters while minimizing the accuracy, thereby speeding up the calculation speed.

[0101] like Figure 1 As shown, the present invention provides a stereo matching method based on contextual geometric features and distortion parallax optimization, comprising the following steps:

[0102] (1) Obtain the left and right perspective images to be matched that have been calibrated by the binocular camera, and preprocess the left and right perspective images to be matched to obtain preprocessed left and right perspective images.

[0103] Specifically, a binocular camera simultaneously captures a target, producing two images from the left and right perspectives. The left perspective image is typically used as the baseline image, while the right perspective image is the stereo image to be matched. These images are captured from different perspectives of the same scene at the same time. By obtaining the disparity between the pixels in the left and right perspective images, the depth information of the target can be derived from the disparity.

[0104] The preprocessing process in this step is specifically as follows: first, obtain the left and right view images and their image data taken by the binocular camera, including the number of channels C, height H, and width W; then, use the Sobel function in OpenCV to perform edge detection processing on the obtained left and right view images to obtain edges in the left and right view images (i.e., places where the pixel values ​​in the image change greatly, which usually correspond to the boundaries or contours of the object); then, convolve the left and right view images in the horizontal and vertical directions and normalize them respectively to obtain left and right view gradient intensity maps in the horizontal and vertical directions respectively (the left and right view gradient intensity maps can better understand the local features in the left and right view images and identify edges, textures, etc.); finally, according to the obtained left and right view gradient intensity maps, the irrelevant edge parts of the left and right view images are cropped respectively to meet the input image size required by the stereo matching model of the present invention (the stereo matching model input will perform multi-scale operations, and the input image size must be divisible by 32) to obtain preprocessed left and right view images.

[0105] The advantage of this step is that it unifies the size of the input model images, which facilitates subsequent training.

[0106] (2) The preprocessed left and right view images obtained in step (1) are input into a pre-trained stereo matching model to obtain the final disparity map as the stereo matching result.

[0107] like Figure 2 As shown, the stereo matching model of the present invention includes four parts connected in sequence: a lightweight multi-scale feature extraction network, a 3D regularization network, a distortion parallax optimization module, and a 3D depth-separable convolution module.

[0108] Specifically, the lightweight multi-scale feature extraction network consists of three layers:

[0109] The first layer is a multi-scale deep feature extraction layer. Its input is the left and right view images, both of which are four-dimensional tensors of shape (B, C, H, W). The multi-scale deep feature extraction layer uses the pre-trained lightweight feature extraction network MobileNet-V3 in the timm (PyTorch Image Models) library to perform deep feature extraction on the input left view image and right view image respectively, and changes the number of its input channels to 48, 64, 160, and 192. It outputs four four-dimensional tensors corresponding to the left view image and the right view image respectively, whose feature scales are equal to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the feature scales of the left and right view images, and their shapes are (B, 48, H / 4, W / 4), (B, 96, H / 8, W / 8), (B, 160, H / 16, W / 16), and (B, 192, H / 32, W / 32). B represents the batch size (Batch Size, which is the number of input images fed into the multi-scale feature extraction network at once. C represents the number of left and right view images (when C is 3, the input image is typically an RGB image with three channels, representing red, green, and blue). H represents the height of the left and right view images. W represents the width of the left and right view images.

[0110] The second layer is a multi-scale shallow feature extraction layer, whose input is also the left view image and the right view image, both of which are four-dimensional tensors of shape (B, C, H, W). The multi-scale shallow feature extraction layer inputs the four-dimensional tensors corresponding to the left view image and the right view image into the ordinary convolution layer respectively, and uses a 3×3 convolution kernel, a step size of 2 pixels and a padding of 1 pixel to change the number of input channels of the left view image and the right view image to 48, 64, 160 and 192 respectively, and outputs four four-dimensional tensors corresponding to the left and right view images respectively, whose feature scales are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the feature scales of the left and right view images, and their shapes are (B, 48, H / 4, W / 4), (B, 96, H / 8, W / 8), (B, 160, H / 16, W / 16) and (B, 192, H / 32, W / 32).

[0111] The third layer is the multi-scale fusion layer, whose input is the four-dimensional tensor corresponding to the deep features of the left and right view images obtained in the first layer and the four-dimensional tensor corresponding to the shallow features of the left and right view images obtained in the second layer. The multi-scale fusion layer first concatenates the two four-dimensional tensors of corresponding scales according to the channel dimension to obtain the shapes of (B,92,H / 4,W / 4), (B,192,H / 8,W / 8), (B,320,H / 16,W / 16), (B,384, The concatenated 4D tensors of four different scales (H / 32, W / 32) are constructed. Then, ordinary convolution with a 3×3 kernel, a stride of 1, and a padding of 1 is used to reduce the number of channels of the four concatenated 4D tensors of different scales by 1 / 2, and the output multi-scale 4D tensors are (B, 48, H / 4, W / 4), (B, 96, H / 8, W / 8), (B, 160, H / 16, W / 16), and (B, 192, H / 32, W / 32).

[0112] The 3D regularization network includes an hourglass network module and a contextual geometric attention module.

[0113] The hourglass network module consists of two parts: downsampling and upsampling. The downsampling part includes 3 decoders, and the upsampling part includes 3 encoders.

[0114] The specific structure of the downsampling part is:

[0115] The first layer decoder takes as input a five-dimensional tensor (geometric cube) of shape (B, C, D / 4, H / 4, W / 4). It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3 (with a stride of 2 and padding of 1), converts the number of channels of the five-dimensional tensor into 2C, and outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8); then, the five-dimensional tensor is input into a second 3D convolution kernel of dimension 3×3×3 (with a stride of 1 and padding of 1) and a BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8), where D represents the disparity of the left and right view images.

[0116] The second-layer decoder takes as input the five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8) output by the first-layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3 (with a stride of 2 and padding of 1), converts the number of channels of the five-dimensional tensor into 4C, and outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); then, the five-dimensional tensor is input into the second 3D convolution kernel of dimension 3×3×3 (with a stride of 1 and padding of 1) and the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16).

[0117] The third layer decoder takes as input the five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16) output by the second layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3 (with a stride of 2 and padding of 1), converts the number of channels of the five-dimensional tensor into 6C, and outputs a five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32). Then, the five-dimensional tensor is successively input into the second 3D convolution kernel of dimension 3×3×3 (with a stride of 1 and padding of 1) and the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32).

[0118] The specific structure of the upsampling part is:

[0119] The first-layer encoder takes as input a five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32) output by the downsampled third-layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4 (with a stride of 2 and a padding of 1), converts the number of channels of the five-dimensional tensor into 4C, and outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); then, the five-dimensional tensor is input into the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16).

[0120] The second-layer encoder takes as input the five-dimensional tensor output by the first-layer decoder with a shape of (B, 4C, D / 16, H / 16, W / 16). It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4 (with a stride of 2 and a padding of 1), converts the number of channels of the five-dimensional tensor into 2C, and outputs a five-dimensional tensor with a shape of (B, 2C, D / 8, H / 8, W / 8). Then, the five-dimensional tensor is input into the BN+ReLU layer in sequence, and finally outputs a five-dimensional tensor with a shape of (B, 2C, D / 8, H / 8, W / 8).

[0121] The third-layer encoder takes as input the five-dimensional tensor output by the second-layer decoder with a shape of (B, 2C, D / 8, H / 8, W / 8). It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4 (with a stride of 2 and a padding of 1), converts the number of channels of the five-dimensional tensor into C, and outputs a five-dimensional tensor with a shape of (B, C, D / 4, H / 4, W / 4); then, the five-dimensional tensor is input into the BN+ReLU layer in sequence, and finally outputs a five-dimensional tensor with a shape of (B, C, D / 4, H / 4, W / 4).

[0122] Contextual geometry attention module such as Figure 3As shown, it has two inputs, one of which is the left view feature map F c , which is a four-dimensional tensor of shape (B, C, H, W) containing rich context feature information; the other input is the geometric cube F G , which is a five-dimensional tensor of shape (B, C, D, H, W) and contains rich geometric information. The contextual geometric attention module first expands the left view feature map into a five-dimensional tensor of shape (B, C, D, H, W) according to the disparity dimension, namely the context feature cube F′ c , and the geometric cube F G and context feature cube F′ c Splicing is performed along the channel dimension to obtain a five-dimensional tensor with a shape of (B, 2C, D, H, W), that is, a splicing cube; then, the five-dimensional tensor is input into a 3D convolution kernel with a dimension of 1×5×5, and the number of channels of the five-dimensional tensor is reduced to C, thereby outputting a five-dimensional tensor with a shape of (B, C, D, H, W) (which is used to preliminarily fuse contextual information with geometric information); subsequently, the five-dimensional tensor is activated using the σ (Sigmoid) function (used to determine the area that should be emphasized or suppressed during the fusion process), thereby outputting a five-dimensional tensor with a shape of (B, C, D, H, W), that is, a splicing fusion cube A CF , the formula is as follows:

[0123] A CF =σ(conv3d(concat{F′ c ,F G}))

[0124] Among them, concat means that the input tensor is spliced ​​along the channel dimension, and conv3d means 3D convolution.

[0125] Then, by splicing and fusing cube A CF ∈R B×C×D×H×W and context feature cube F′ c To multiply element-by-element, we get the initial contextual set attention, which is a five-dimensional tensor of shape (B, C, D, H, W). Subsequently, this five-dimensional tensor is concatenated with the geometric cube along the channel dimension to output a five-dimensional tensor of shape (B, 2C, D, H, W). Finally, this five-dimensional tensor is input into a 3D convolution kernel of dimension 1×5×5, which reduces the number of channels of the five-dimensional tensor to C, thereby outputting a five-dimensional tensor (contextual geometric attention cube G) of shape (B, C, D, H, W). It fully integrates contextual feature information and geometric information to obtain a contextual geometric attention cube G containing rich information. Its formula is as follows:

[0126] G=conv3d(concat{F G,A CF ⊙F′ c})

[0127] where ⊙ represents element-wise multiplication.

[0128] The warped parallax optimization module consists of three parts, such as Figure 4 As shown, specifically:

[0129] The first part is the warped disparity transformation layer, whose input is the right view feature map and the initial disparity map. The right view feature map is a four-dimensional tensor with a shape of (B, C, H, W), and the initial disparity map is a four-dimensional tensor with a shape of (B, 1, H, W). The warped disparity transformation layer first creates grid coordinates (xx, yy) according to the number of channels C, height H, and width W of the input right view feature map to represent the coordinates of each pixel in the right view feature map; then, the grid horizontal coordinate xx is distorted and adjusted according to the initial disparity map to obtain the updated grid horizontal coordinate, which is adjusted to the standardized range [-1, 1] and converted to the PyTorch grid format to obtain the warped grid coordinates after initial disparity adjustment (warped xx, warped yy); then, use the nn.functional.grid_sample function and use the distorted grid coordinates after initial disparity adjustment to perform bilinear interpolation sampling on the right view feature map to obtain a distorted image; finally, create a mask mask of the same size as the right view feature map with all 1s, use the nn.functional.grid_sample function and use the distorted grid coordinates after initial disparity adjustment to perform bilinear interpolation sampling on the mask mask to obtain the value of the mask at the target position of the distorted grid coordinates to determine the valid pixels in the right view feature map, and retain the valid pixels, thereby obtaining a four-dimensional tensor (distorted right view feature map) with a shape of (B, C, H, W) after parallax distortion.

[0130] The second part is the warped cost cube construction layer, whose input is the warped right view feature map obtained by the warped disparity transformation layer (which is a four-dimensional tensor with a shape of (B, C, H, W)), the left view feature map (which is a four-dimensional tensor with a shape of (B, C, H, W)), and the maximum disparity value maxdisp (i.e., the set disparity threshold). The warped cost cube construction layer first creates an all-zero tensor volume of the same data type as the left view feature map based on the shape of the left view feature map (including the number of channels C, height H, and width W) to store the correlation volume; then, the disparity value range [-maxdisp, maxdisp] is determined based on the maximum disparity value maxdisp, all integer disparities in the disparity value range are traversed, and the cosine similarity is used to calculate the correlation between the left view feature map and the warped right view feature map at each integer disparity; finally, the obtained correlations at all integer disparities are stored in the all-zero tensor volume to obtain the warped cost volume, which is a five-dimensional tensor with a shape of (B, C, D, H, W).

[0131] The third part is the warped disparity optimization layer, whose input is the left view feature map, the initial disparity map (both of which are four-dimensional tensors with a shape of (B, C, H, W)), and the warped cost volume obtained in the second layer. The warped disparity optimization layer first subtracts the warped right view feature map from the left view feature map to obtain the reconstruction error, which is a four-dimensional tensor with a shape of (B, C, H, W); then, the reconstruction error, the left view feature map, the initial disparity map, and the warped cost volume are spliced ​​along the channel dimension to obtain a spliced ​​volume, which is a four-dimensional tensor with a shape of (B, C ′ , H, W) four-dimensional tensor; then, the spliced ​​body and the initial disparity are input into a sub-precision module consisting of 8 convolutional layers for disparity optimization to extract the residual disparity value disp, that is, the disparity dimension value.

[0132] For the sub-precision module, its specific structure is:

[0133] The first to fourth layers are four sequentially connected convolutional layers with the same structure. Each convolutional layer contains a sequence of convolution and activation functions. The input of the first layer is a concatenation, which has a shape of (B, C ′ , H, W), the first layer first inputs the four-dimensional tensor into a 3×3 convolution kernel, reduces the input channel of the four-dimensional tensor to C, and outputs a four-dimensional tensor with a shape of (B, C, H, W). Then, a nonlinear transformation is performed through the activation function Mish to output a four-dimensional tensor with a shape of (B, C, H, W). The input of the second layer is the output of the first layer, ..., and so on. Finally, the fourth layer outputs a four-dimensional tensor with a shape of (B, C, H, W).

[0134] The fifth to seventh layers are three sequentially connected residual blocks with the same structure. Their input is the output four-dimensional tensor of the fourth layer with a shape of (B, C, H, W). The fifth layer first inputs the four-dimensional tensor into three 3x3 convolution kernels, and connects the obtained output residuals, reducing the number of input channels of the four-dimensional tensor to 96, 64, and 32 respectively, and outputs a four-dimensional tensor with a shape of (B, 32, H, W). The input of the sixth layer is the output of the fifth layer, and so on. Finally, the seventh layer outputs a four-dimensional tensor with a shape of (B, 32, H, W).

[0135] The eighth layer is a separate convolutional layer. Its input is the four-dimensional tensor of shape (B, 32, H, W) output by the seventh layer. The eighth layer inputs the four-dimensional tensor into a 3x3 convolution kernel, reduces the number of channels of the four-dimensional tensor to 1, and outputs a four-dimensional tensor of shape (B, 1, H, W). The residual disparity value disp, that is, the disparity dimension value, is extracted from the four-dimensional tensor.

[0136] 3D depth-wise separable convolutional modules such as Figure 5 As shown in the figure, MobileNet-V2 first uses 1×1 convolution to increase the dimension, then uses 3×3 DW convolution, and finally uses 1×1 convolution to reduce the dimension to achieve a structure with small ends and a large center. Inspired by the network structure of MobileNet-V2, depthwise convolution and pointwise convolution are also applied in the channel dimension of 3D convolution to obtain a 3D separable convolution module, whose structure is as follows:

[0137] The input of the 3D depth separable convolution module is a five-dimensional tensor of shape (B, C, D, H, W). The 3D depth separable convolution module first inputs the five-dimensional tensor into a 3D point convolution kernel of dimension 1×1×1, and increases the input channel of the five-dimensional tensor to tC (where t is the expansion factor, t=2 in the present invention); then, the output result is input into the BN+ReLU layer, thereby outputting a five-dimensional tensor of shape (B, tC, D, H, W); then, the five-dimensional tensor is successively input into a 3D depth convolution kernel of dimension 3×3×3 and a BN+ReLU layer, thereby outputting a five-dimensional tensor of shape (B, tC, D, H, W); finally, the five-dimensional tensor is input into a 3D point convolution kernel of dimension 1×1×1 to reduce the output channel to C of any size. ′ And input the output result into the BN+ReLU layer, and finally output a shape (B, C ′ , D, H, W) five-dimensional tensor.

[0138] The stereo matching model of the present invention is obtained by training through the following steps:

[0139] (2-1) A stereo image pair dataset (left-view image and right-view image) is obtained, and the stereo image pair dataset is preprocessed to obtain a preprocessed stereo image pair dataset, wherein the stereo image pair dataset includes a training set, a validation set, and a test set. The training set also includes the disparity of each stereo image pair.

[0140] In other words, the training set in the present invention is used to adjust the trainable weights and bias parameters in the stereo matching model, the validation set is used to adjust the hyperparameters such as the learning rate of the stereo matching model, and the test set does not participate in the training of the model and is used to test the effect of the finally trained stereo matching model.

[0141] Specifically, the preprocessing process in this step is exactly the same as the preprocessing process in the above step (1), and will not be repeated here.

[0142] (2-2) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the left and right perspective pictures in the stereo image pair are respectively input into the lightweight multi-scale feature extraction network for multi-scale feature extraction to obtain four four-dimensional tensors corresponding to the stereo image pair, whose scales are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the size of the left and right perspective pictures, i.e., left and right perspective feature maps of scales 1 / 4, 1 / 8, 1 / 16, and 1 / 32, respectively.

[0143] The advantage of this step is that multi-scale features can not only capture the global information of the image, but also help the model understand the local details of the image. By extracting features at multiple scales, the model can enhance the multi-level and multi-dimensional representation of the image, while facilitating information transfer between different scales. By fusing features at different scales, the model can extract complementary information from each scale.

[0144] (2-3) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 4 scale left and right view feature maps f corresponding to the stereo image pair obtained in step (2-2) are l and f r ∈R B ×C×H / 4×W / 4 Construct a 1 / 4 scale 5D cost cube, that is Figure 1 The geometric cube in

[0145] Specifically, the 5D cost cube is constructed by calculating the similarity of the corresponding matching points of the left and right view feature maps under different parallaxes through cosine similarity, that is, Figure 1 Geometric cube in .

[0146] The calculation formula of the geometric cube is as follows:

[0147]

[0148] Where x represents the horizontal coordinate of the pixel on the left or right view feature map, y represents the vertical coordinate of the pixel on the left or right view feature map, d represents the disparity value between the left view feature map and the right view feature map, <,> represents the dot product of the feature vector of the pixel on the left or right view feature map, which is used to measure the similarity between the pixel of the left view image and the corresponding pixel of the right view image, ‖ ‖2 represents the L2 norm of the feature vector of the pixel on the left or right view feature map, and f l (x, y, d) represents the feature vector at the pixel position of the left image feature map with disparity d and coordinates (x, y), f r (x―d,y,d) represents the feature vector at the pixel position of the right image feature map with disparity d and parallax offset coordinates (x―d,y).

[0149] The advantage of this step is that it only uses the left and right feature maps of 1 / 4 the size of the left and right perspective images to construct the cost cube, which not only retains most of the feature information but also greatly reduces the computational complexity of the model.

[0150] (2-4) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the geometric cube corresponding to the stereo image pair obtained in step (2-3) is input into the downsampling part of the hourglass network module in the 3D regularization network to obtain initial geometric cubes of 1 / 8, 1 / 16, and 1 / 32 scales.

[0151] (2-5) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 32 scale initial geometric cube corresponding to the stereo image pair obtained in step (2-4) is input into the first layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 16 scale geometric cube, and the 1 / 16 scale initial geometric cube obtained by downsampling in step (2-4) is spliced ​​according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 16 scale geometric cube corresponding to the stereo image pair.

[0152] (2-6) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 16 scale geometric cube corresponding to the stereo image pair obtained in step (2-5) and the 1 / 16 scale left view feature map obtained in step (2-2) are input into the contextual geometric attention module in the 3D regularization network to obtain a 1 / 16 scale contextual geometric attention weight cube corresponding to the stereo image pair, which ultimately integrates the contextual feature information and geometric information.

[0153] (2-7) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 16 scale context geometric attention weight cube corresponding to the stereo image pair obtained in (2-6) is input into the second layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 8 scale geometric cube, and it is spliced ​​with the 1 / 8 scale initial geometric cube obtained by downsampling in step (2-4) according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 8 scale geometric cube corresponding to the stereo image pair.

[0154] (2-8) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale geometric cube corresponding to the stereo image pair obtained in step (2-7) and the 1 / 8 scale feature map of the left view obtained in step (2-2) are input into the contextual geometric attention module in the 3D regularization network to obtain a 1 / 8 scale contextual geometric attention weight cube corresponding to the stereo image pair, which ultimately integrates the contextual feature information and geometric information.

[0155] (2-9) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale context geometric attention weight cube corresponding to the stereo image pair obtained in step (2-8) is input into the third layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 4 scale geometric cube, and it is spliced ​​with the 1 / 4 scale geometric cube of step (2-3) according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 4 scale geometric cube corresponding to the stereo image pair.

[0156] (2-10) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale geometric cube corresponding to the stereo image pair obtained in step (2-9) and the feature map of the 1 / 4 scale left view obtained in step (2-2) are input into the context geometry attention module in the 3D regularization network to obtain the final 1 / 4 scale context geometry attention weight cube corresponding to the stereo image pair.

[0157] The advantage of the above steps (2-4) to (2-10) is that, in the process of cost aggregation through the 3D regularization network, not only the rich contextual information between adjacent feature points of the left and right view feature maps is taken into account, but also the rich geometric information between the corresponding feature points of the left and right view feature maps under adjacent parallax is taken into account.

[0158] (2-11) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), all weights in the 1 / 4 scale context geometric attention weight cube corresponding to the stereo image pair obtained in step (2-10) are sorted by the importance of the disparity value in its disparity dimension (that is, the weights are sorted in descending order along the disparity direction), and all weights corresponding to the top 12 disparities are selected as the final context geometric attention weight cube A corresponding to the stereo image pair. cg ∈R C×D′×H / 4×W / 4 .

[0159] The advantage of this step is that, on the one hand, the ultimate goal is to find the best matching disparity among these candidate disparities, so disparities with smaller weights will not significantly affect the results. On the other hand, it can greatly reduce the size of the contextual geometric attention weight cube, thereby reducing the amount of model training and greatly speeding up the calculation.

[0160] (2-12) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 4 scale left view feature map f corresponding to the stereo image pair obtained in step (2-2) is l ∈R B×C×H / 4×W / 4 , also known as the context feature map, is indexed according to the disparity value in step (2-3), the context feature map is increased by the disparity dimension, and the context feature map is copied along the disparity dimension to obtain the context feature cube V corresponding to the stereo image pair with the same shape as the context geometric attention weight cube c ∈R B×C×D′×H / 4×W / 4 .

[0161] (2-13) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the context geometric attention weight cube A corresponding to the stereo image pair obtained in step (2-11) is cg The context feature cube V corresponding to the stereo image pair obtained in the filtering step (2-12) c (ie, the two are multiplied element by element) to obtain the final context geometric cube V corresponding to the stereo image pair cgv ∈R C×D′×H / 4×W / 4 .

[0162] The specific formula is as follows:

[0163] V cgv =A cg ⊙V c

[0164] The advantage of this step is that the context feature cube is known according to the context geometric attention weight to obtain the cost volume that integrates rich context features and geometric features, so that the model can better capture the global information and context information in the image when calculating disparity, thereby improving the matching accuracy in complex scenes.

[0165] (2-14) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the context geometry cube obtained in step (2-13) is input into the first two encoder layers and the first two decoder layers of the hourglass network module in the 3D regularization network. cgv Cost aggregation is performed (the context geometric cube is aggregated along the disparity dimension, the channel dimension, and the feature dimension) to obtain the target cube corresponding to the stereo image pair.

[0166] (2-15) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), use a softmax function to weight all disparity candidate values ​​at each pixel of the target cube corresponding to the stereo image pair obtained in step (2-15) (i.e., the 12 disparities in step (2-11)) to obtain the disparity value at the pixel corresponding to the stereo image pair. The disparity values ​​of the target cube at all pixels are combined to obtain an initial disparity map corresponding to the stereo image pair with a disparity dimension of 1.

[0167] The advantages of steps (2-15) and (2-16) are that cost aggregation combines multi-dimensional matching costs to optimize the disparity estimation of each pixel to improve matching accuracy, remove noise, enhance robustness, improve computational efficiency, enhance boundary preservation, and adapt to complex geometric structures.

[0168] (2-16) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the initial disparity map corresponding to the stereo image pair obtained in step (2-15) and the 1 / 4 scale right view feature map obtained in step (2-2) are input into the warped disparity transformation layer of the disparity optimization module to obtain the 1 / 4 scale warped right view feature map f corresponding to the stereo image pair. rw,4 .

[0169] Specifically, the 1 / 4 scale distorted right view feature map f rw,4 The calculation formula is as follows:

[0170] f rw,4 =warping(f r,4 ,d0)

[0171] where f r,4is the right perspective feature map at 1 / 4 scale, d0 is the initial disparity value of the pixel point in the initial disparity map, and the warpping function is the warping disparity operation of the warped disparity transformation layer.

[0172] (2-17) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 4 scale left view feature map obtained in step (2-2) and the 1 / 4 scale distorted right view feature map f corresponding to the stereo image pair obtained in step (2-16) are converted into rw,4 Perform bilinear interpolation upsampling to obtain the left view feature map and the distorted right view feature map of the corresponding scale of the stereo image pair, and input the left view feature map and the distorted right view feature map into the distortion cost cube construction layer of the distortion parallax optimization module to obtain the distortion cost volume V corresponding to the stereo image pair. w .

[0173] Specifically, the distortion body V w The expression is as follows:

[0174]

[0175] where f l and f rw are the left view feature map and the distorted right view feature map of the corresponding scale of the stereo image pair, and △d represents the range of fine-grained residual search.

[0176] The advantage of this step is that the optimal disparity value is generally distributed within a certain range around the initial disparity. Therefore, a fine-grained residual search range △d is defined, which reduces the residual search range from the initial disparity search range (0, d) to the fine-grained search range (d-△d, d+△d). This reduces the difficulty of the network in finding the correct residual disparity in the unconstrained residual disparity search space and greatly speeds up the calculation time.

[0177] (2-18) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the left perspective feature map and the distorted right perspective feature map corresponding to the stereo image pair obtained in step (2-17) are subtracted to obtain the reconstruction error corresponding to the stereo image pair (which is used to indicate the incorrect area of ​​the initial disparity).

[0178] Specifically, the expression of reconstruction error is as follows

[0179] Error=f l (x,y,d)―f rw (x―d,y,d)

[0180] (2-19) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the initial disparity map corresponding to the stereo image pair obtained in step (2-15), the left view feature map corresponding to the stereo image pair obtained in step (2-17), and the distorted cost volume V w , and the reconstruction error corresponding to the stereo image pair obtained in step (2-18) is input into the distortion disparity optimization layer of the distortion disparity optimization module to obtain the fine-grained residual disparity value corresponding to the stereo image pair, and the fine-grained residual disparity value is added to the initial disparity to obtain the refined disparity value corresponding to the stereo image pair.

[0181] The advantage of steps (2-16) to (2-19) is that, in theory, the distorted right view feature map should perfectly match the left view feature map. Due to the existence of errors, the present invention requires further optimization of the disparity. The present invention obtains a 5D warped cost volume based on the cosine similarity between the left view feature and the distorted right view feature at each disparity level, guiding the network to learn the optimal residual disparity with the maximum similarity.

[0182] (2-20) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), if Figure 2 As shown, the 3D convolution operations in steps (2-4) to (2-10) and step (2-14) are modified to 3D depth-separable convolutions, and the above steps (2-4) to (2-19) are repeated to obtain the predicted disparity value of the model corresponding to the stereo image pair;

[0183] The advantage of this step is that, compared with conventional 3D convolution, 3D separable convolution not only ensures that most of the information output by the convolution is preserved, but also greatly reduces the number of parameters and calculation time, greatly improving the efficiency of model training and testing.

[0184] (2-21) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), all disparity values ​​of the context geometry attention weight cube obtained in (2-10) at each pixel are weighted using the softmax function to obtain the disparity value at the pixel corresponding to the stereo image pair, and the disparity values ​​of the context geometry attention weight cube at all pixels are combined to obtain the context feature geometry attention weight disparity map d corresponding to the stereo image pair with a disparity dimension of 1. att .

[0185] The advantage of this step is that the model's predicted disparity has a large error in the initial training. The refined training of steps (2-11) to (2-20) will cause a relatively large waste of video memory and computing time. This step first trains part of the model of steps (2-2) to (2-10), greatly reducing video memory consumption and improving computing efficiency.

[0186] (2-22) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), a loss function is constructed using the predicted disparity value of the model corresponding to the stereo image pair obtained in step (2-20).

[0187] Specifically, the loss function formula is as follows:

[0188]

[0189] where d att and d gt It is the disparity directly obtained from the geometric attention weight of the context feature and the real disparity map provided in the training set. att represents the coefficient of the predicted attention weight disparity, λ represents the coefficient of the final predicted disparity, λ att The value range of and λ is 0 to 1, and the sum of the two is 1. In the present invention, λ att The values ​​of and λ are 0.3 and 0.7, represents the smoothed L1 loss.

[0190] The advantage of this step is that the smoothed L1 loss is robust and has low sensitivity to outliers, ensuring the stability and efficiency of the training process.

[0191] (2-23) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the loss function corresponding to the stereo image pair obtained in step (2-22) is used, and the stereo matching model is iteratively trained using the back propagation method until the stereo matching model converges, thereby obtaining a preliminary trained stereo matching model corresponding to the stereo image pair.

[0192] (2-24) For each stereo image pair in the verification set of the stereo image pair dataset preprocessed in step (2-1), the stereo matching model preliminarily trained in step (3-23) is verified until the obtained disparity accuracy reaches the optimal level, thereby obtaining the trained stereo matching model corresponding to the stereo image pair.

[0193] The advantage of steps (2-23) and (2-24) is that they use backpropagation for iterative training and validation sets to help gradually optimize model parameters, thereby improving the performance of the final model. As training progresses, the model is able to adjust and optimize its feature representation with each iteration, gradually improving the accuracy of the matching.

[0194] Experimental results

[0195] The present invention is developed using PyTorch and executed on an RTX 3090 GPU. For pre-training on SceneFlow, the input image is normalized using standard deviation statistics and ImageNet mean, and then image enhancement is performed with a probability of 0.2. The enhancement includes color adjustment, brightness enhancement, and contrast modification to simulate different exposure conditions. In addition, random vertical and rotational shifts are applied to the right image, which is then randomly cropped to a resolution of 512×384. The present invention uses the Adam

[37] optimizer with β1=0.9, β2=0.999. The feature geometry attention network is trained for 20 epochs using the scene flow dataset, and then the entire complete network is trained for 20 epochs. The coefficients of the two outputs are set to λ att =0.5, λ=1, the initial learning rate is set to 0.001, and decays by a factor of 2 after epochs 10, 13, 16, and 18.

[0196] To demonstrate the accuracy and speed improvements of our method for disparity estimation, we conducted tests on multiple datasets. Taking the SceneFlow dataset as an example, we compared the test results obtained with currently used methods. For various state-of-the-art methods, such as AANet, GWCNet, and LEAstereo, our method improved runtime and accuracy by over 30% and 50%, respectively, while maintaining a low parameter count. This performance is unmatched in both speed and accuracy. For fast real-time methods like BGNet and CoEx, although our method does not offer significant advantages in terms of runtime and parameter count, it significantly outperforms other real-time methods in terms of EPE. Compared to Fast-ACVNet, our method significantly improves accuracy, reducing the EPE to 0.54, while maintaining comparable runtime. Furthermore, compared to HitNet, our method achieves comparable accuracy but reduces computation time by 26%. The experimental results on the scene flow dataset show that the method of the present invention has a good balance between efficiency and performance, and has achieved good results in terms of accuracy and speed. Figure 6The evaluation results are shown in Table 1. The present invention is also trained and tested on the KITTI dataset, and the comparison with the most advanced method is shown in the figure below. Figure 7 While ensuring model accuracy and speed, the model trained using only the SceneFlow dataset was tested on randomly selected stereo image pairs from the Middlebury dataset and the ETH3D dataset, and the results were compared with those of the advanced model Fast-ACVNet. The comparison chart is shown in the figure below. Figure 8 shown.

[0197]

[0198] Table 1

[0199] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A stereo matching method based on contextual geometric features and distortion parallax optimization, characterized in that: The steps include: (1) Obtain the left and right perspective images to be matched that have been calibrated by the binocular camera, and preprocess the left and right perspective images to be matched to obtain preprocessed left and right perspective images; (2) Inputting the pre-processed left and right view images obtained in step (1) into a pre-trained stereo matching model to obtain a final disparity map as a stereo matching result; the stereo matching model comprises four parts connected in sequence: a lightweight multi-scale feature extraction network, a 3D regularization network, a warped disparity optimization module, and a 3D depth-separable convolution module; The 3D regularization network includes an hourglass network module and a contextual geometry attention module; The hourglass network module consists of two parts: downsampling and upsampling; The downsampling part includes: The first layer decoder takes as input a five-dimensional tensor of shape (B, C, D / 4, H / 4, W / 4). It first feeds the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3, converts the number of channels of the five-dimensional tensor into 2C, and outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8). The five-dimensional tensor is then fed into a second 3D convolution kernel of dimension 3×3×3 and a BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8), where D represents the disparity between the left and right view images. The second layer decoder takes as input the five-dimensional tensor of the shape (B, 2C, D / 8, H / 8, W / 8) output by the first layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3, converts the number of channels of the five-dimensional tensor into 4C, and outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); then, the five-dimensional tensor is input into the second 3D convolution kernel of dimension 3×3×3 and the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); The third layer decoder takes as input the five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16) output by the second layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3, converts the number of channels of the five-dimensional tensor into 6C, and outputs a five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32); then, the five-dimensional tensor is input into the second 3D convolution kernel of dimension 3×3×3 and the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32); The upsampling part includes: The first layer encoder takes as input the five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32) output by the downsampled third layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4, converts the number of channels of the five-dimensional tensor into 4C, and outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); then, the five-dimensional tensor is input into the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); The second-layer encoder takes as input the five-dimensional tensor output by the first-layer decoder with a shape of (B, 4C, D / 16, H / 16, W / 16). It first inputs the five-dimensional tensor into a 3D convolution kernel with a dimension of 4×4×4, converts the number of channels of the five-dimensional tensor into 2C, and outputs a five-dimensional tensor with a shape of (B, 2C, D / 8, H / 8, W / 8). Then, the five-dimensional tensor is input into the BN+ReLU layer in sequence, and finally outputs a five-dimensional tensor with a shape of (B, 2C, D / 8, H / 8, W / 8). The third-layer encoder takes as input the five-dimensional tensor output by the second-layer decoder with a shape of (B, 2C, D / 8, H / 8, W / 8). It first inputs the five-dimensional tensor into a 3D convolution kernel with a dimension of 4×4×4, converts the number of channels of the five-dimensional tensor into C, and outputs a five-dimensional tensor with a shape of (B, C, D / 4, H / 4, W / 4). Then, the five-dimensional tensor is successively input into the BN+ReLU layer, and finally outputs a five-dimensional tensor with a shape of (B, C, D / 4, H / 4, W / 4). The contextual geometric attention module includes two inputs, one of which is the left view feature map F c , which is a four-dimensional tensor of shape (B, C, H, W) that contains rich context feature information; the other input is the geometric cube F G , a five-dimensional tensor of shape (B, C, D, H, W), which contains rich geometric information; The contextual geometric attention module first expands the left view feature map into a five-dimensional tensor of shape (B, C, D, H, W) according to the disparity dimension, namely the context feature cube F′ c , and the geometric cube F G and context feature cube F′ c Splicing is performed along the channel dimension to obtain a five-dimensional tensor with a shape of (B, 2C, D, H, W), namely the splicing cube; then, the five-dimensional tensor is input into a 3D convolution kernel with a dimension of 1×5×5, and the number of channels of the five-dimensional tensor is reduced to C, thereby outputting a five-dimensional tensor with a shape of (B, C, D, H, W); subsequently, the five-dimensional tensor is activated using the σ (Sigmoid) function, thereby outputting a five-dimensional tensor with a shape of (B, C, D, H, W), namely the splicing fusion cube A CF , the formula is as follows: TO CF =σ(conv3d(concat{F′ c ,F G })) Among them, concat means that the input tensor is spliced ​​along the channel dimension, and conv3d means 3D convolution; Then, by splicing the fused cube A CF ∈R B×C×D×H×W and context feature cube F′ c To multiply element by element to obtain the initial context set attention, which is a five-dimensional tensor with a shape of (B, C, D, H, W). Subsequently, the five-dimensional tensor is spliced ​​with the geometric cube along the channel dimension to output a five-dimensional tensor with a shape of (B, 2C, D, H, W); finally, the five-dimensional tensor is input into a 3D convolution kernel with a dimension of 1×5×5, and the number of channels of the five-dimensional tensor is reduced to C, thereby outputting a five-dimensional tensor with a shape of (B, C, D, H, W), namely the context geometric attention cube G, which fully integrates the context feature information and geometric information to obtain a context geometric attention cube G containing rich information. Its formula is as follows: G=conv3d(concat{F G ,A CF ⊙F′ c }) Where ⊙ represents element-by-element multiplication; The warped parallax optimization module includes: The first part is the warped disparity transformation layer, whose input is the right view feature map and the initial disparity map. The right view feature map is a four-dimensional tensor with a shape of (B, C, H, W), and the initial disparity map is a four-dimensional tensor with a shape of (B, 1, H, W). The warped disparity transformation layer first creates grid coordinates (xx, yy) according to the number of channels C, height H, and width W of the input right view feature map to represent the coordinates of each pixel in the right view feature map; then, the grid horizontal coordinate xx is distorted and adjusted according to the initial disparity map to obtain the updated grid horizontal coordinate, which is adjusted to the standardized range [-1, 1] and converted to the PyTorch grid format to obtain the warped grid coordinates after initial disparity adjustment (warped xx, warped yy); then, use the nn.functional.grid_sample function and use the distorted grid coordinates after initial disparity adjustment to perform bilinear interpolation sampling on the right view feature map to obtain a distorted image; finally, create a mask mask of the same size as the right view feature map with all 1s, use the nn.functional.grid_sample function and use the distorted grid coordinates after initial disparity adjustment to perform bilinear interpolation sampling on the mask mask to obtain the value of the mask at the target position of the distorted grid coordinates to determine the valid pixels in the right view feature map, and retain the valid pixels, thereby obtaining a four-dimensional tensor with a shape of (B, C, H, W) after parallax distortion, that is, the distorted right view feature map; The second part is the warped cost cube construction layer, whose input is the warped right view feature map, left view feature map and maximum disparity value maxdisp obtained by the warped disparity transformation layer. The warped cost cube construction layer first creates an all-zero tensor volume of the same data type as the left view feature map according to the shape of the left view feature map to store the correlation volume; then, the disparity value range [-maxdisp, maxdisp] is determined according to the maximum disparity value maxdisp, all integer disparities in the disparity value range are traversed, and the cosine similarity is used to calculate the correlation between the left view feature map and the warped right view feature map at each integer disparity; finally, the obtained correlations under all integer disparities are stored in the all-zero tensor volume to obtain the warped cost volume, which is a five-dimensional tensor with a shape of (B, C, D, H, W); The third part is the warped disparity optimization layer, whose input is the left view feature map, the initial disparity map, and the warped cost volume obtained in the second layer. The warped disparity optimization layer first subtracts the warped right view feature map from the left view feature map to obtain the reconstruction error, which is a four-dimensional tensor with a shape of (B, C, H, W); then, the reconstruction error, the left view feature map, the initial disparity map, and the warped cost volume are spliced ​​along the channel dimension to obtain a spliced ​​volume, which is a four-dimensional tensor with a shape of (B, C ′ , H, W) four-dimensional tensor; then, the spliced ​​body and the initial disparity are input into a sub-precision module consisting of 8 convolutional layers for disparity optimization to extract the residual disparity value disp, that is, the disparity dimension value.

2. The stereo matching method based on contextual geometric features and distortion parallax optimization according to claim 1, characterized in that: The preprocessing process in step (1) is specifically as follows: first, obtain the left and right view images and their image data taken by the binocular camera, including the left and right view image channel number C, the height H of the left and right view images, and the width W of the left and right view images; then, use the Sobel function in OpenCV to perform edge detection processing on the obtained left and right view images to obtain the edges in the left and right view images; then, perform convolution operations on the left and right view images in the horizontal and vertical directions and normalize them respectively to obtain the left and right view gradient intensity maps in the horizontal and vertical directions respectively; finally, according to the obtained left and right view gradient intensity maps, the irrelevant edge parts of the left and right view images are cropped respectively to make them meet the input image size required by the stereo matching model, so as to obtain the preprocessed left and right view images.

3. The stereo matching method based on contextual geometric features and distortion parallax optimization according to claim 1 or 2, characterized in that: The lightweight multi-scale feature extraction network consists of three layers: The first layer is a multi-scale deep feature extraction layer. Its input is the left and right view images, both of which are four-dimensional tensors of shape (B, C, H, W). The multi-scale deep feature extraction layer uses the pre-trained lightweight feature extraction network MobileNet-V3 in the timm library to perform deep feature extraction on the input left view image and right view image respectively, and changes the number of input channels to 48, 64, 160, and 192. It outputs four four-dimensional tensors corresponding to the left view image and the right view image respectively, whose feature scales are equal to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the feature scales of the left and right view images, and their shapes are (B, 48, H / 4, W / 4), (B, 96, H / 8, W / 8), (B, 160, H / 16, W / 16), and (B, 192, H / 32, W / 32), respectively, where B represents the batch size; The second layer is a multi-scale shallow feature extraction layer, whose input is also the left view image and the right view image, both of which are four-dimensional tensors of shape (B, C, H, W). The multi-scale shallow feature extraction layer inputs the four-dimensional tensors corresponding to the left view image and the right view image into the ordinary convolution layer respectively, and uses a 3×3 convolution kernel, a step size of 2 pixels and a padding of 1 pixel to change the number of input channels of the left view image and the right view image to 48, 64, 160, and 192, and outputs four four-dimensional tensors corresponding to the left and right view images respectively, whose feature scales are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the feature scales of the left and right view images, and their shapes are (B, 48, H / 4, W / 4), (B, 96, H / 8, W / 8), (B, 160, H / 16, W / 16), and (B, 192, H / 32, W / 32). The third layer is the multi-scale fusion layer, whose input is the four-dimensional tensor corresponding to the deep features of the left and right view images obtained in the first layer and the four-dimensional tensor corresponding to the shallow features of the left and right view images obtained in the second layer. The multi-scale fusion layer first concatenates the two four-dimensional tensors of corresponding scales according to the channel dimension to obtain the shapes of (B,92,H / 4,W / 4), (B,192,H / 8,W / 8), (B,320,H / 16,W / 16), (B,384, The concatenated 4D tensors of four different scales (H / 32, W / 32) are constructed. Then, ordinary convolution with a 3×3 kernel, a stride of 1, and a padding of 1 is used to reduce the number of channels of the four concatenated 4D tensors of different scales by 1 / 2, and the output multi-scale 4D tensors are (B, 48, H / 4, W / 4), (B, 96, H / 8, W / 8), (B, 160, H / 16, W / 16), and (B, 192, H / 32, W / 32).

4. The stereo matching method based on contextual geometric features and distortion parallax optimization according to claim 3, characterized in that: For the sub-precision module, its specific structure is: The first to fourth layers are four sequentially connected convolutional layers with the same structure. Each convolutional layer contains a sequence of convolution and activation functions. The input of the first layer is a concatenation, which has a shape of (B, C ′ , H, W), the first layer first inputs the four-dimensional tensor into a 3×3 convolution kernel, reduces the input channel of the four-dimensional tensor to C, and outputs a four-dimensional tensor with a shape of (B, C, H, W). Then, a nonlinear transformation is performed through the activation function Mish, and the output is a four-dimensional tensor with a shape of (B, C, H, W). The input of the second layer is the output of the first layer, and so on. Finally, the fourth layer outputs a four-dimensional tensor with a shape of (B, C, H, W); The fifth to seventh layers are three sequentially connected residual blocks with the same structure. Their input is the output of the fourth layer, a four-dimensional tensor with a shape of (B, C, H, W). The fifth layer first inputs the four-dimensional tensor into three 3x3 convolution kernels and connects the resulting output residuals. The number of input channels of the four-dimensional tensor is reduced to 96, 64, and 32 respectively, and a four-dimensional tensor with a shape of (B, 32, H, W) is output. The input of the sixth layer is the output of the fifth layer, and so on. Finally, the seventh layer outputs a four-dimensional tensor with a shape of (B, 32, H, W). The eighth layer is a separate convolutional layer. Its input is the four-dimensional tensor of shape (B, 32, H, W) output by the seventh layer. The eighth layer inputs the four-dimensional tensor into a 3x3 convolution kernel, reduces the number of channels of the four-dimensional tensor to 1, and outputs a four-dimensional tensor of shape (B, 1, H, W). The residual disparity value disp, that is, the disparity dimension value, is extracted from the four-dimensional tensor. The input of the 3D depth-wise separable convolution module is a five-dimensional tensor of shape (B, C, D, H, W). The 3D depth-wise separable convolution module first inputs the five-dimensional tensor into a 3D point convolution kernel of dimension 1×1×1, and increases the input channel of the five-dimensional tensor to tC, where t is the expansion factor; then, the output result is input into the BN+ReLU layer to output a five-dimensional tensor of shape (B, tC, D, H, W); then, the five-dimensional tensor is successively input into a 3D depth-wise convolution kernel of dimension 3×3×3 and a BN+ReLU layer to output a five-dimensional tensor of shape (B, tC, D, H, W); finally, the five-dimensional tensor is input into a 3D point convolution kernel of dimension 1×1×1 to reduce the output channel to C of arbitrary size. ′ And input the output result into the BN+ReLU layer, and finally output a shape (B, C ′ , D, H, W) five-dimensional tensor.

5. The stereo matching method based on contextual geometric features and distortion parallax optimization according to claim 4, characterized in that: The stereo matching model is trained through the following steps: (2-1) obtaining a stereo image pair dataset, and preprocessing the stereo image pair dataset to obtain a preprocessed stereo image pair dataset, where the stereo image pair dataset includes a training set, a validation set, and a test set; The training set also contains the disparity of each stereo image pair; (2-2) for each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the left and right view images in the stereo image pair are respectively input into a lightweight multi-scale feature extraction network to perform multi-scale feature extraction, so as to obtain four four-dimensional tensors corresponding to the stereo image pair, whose scales are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the size of the left and right view images, i.e., left and right view feature maps at scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32; (2-3) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 4 scale left and right view feature maps f corresponding to the stereo image pair obtained in step (2-2) are l and f r ∈R B×C×H / 4×W / 4 Construct a 1 / 4 scale 5D cost cube; (2-4) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the geometric cube corresponding to the stereo image pair obtained in step (2-3) is input into the downsampling part of the hourglass network module in the 3D regularization network to obtain initial geometric cubes at scales of 1 / 8, 1 / 16, and 1 / 32; (2-5) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 32 scale initial geometric cube corresponding to the stereo image pair obtained in step (2-4) is input into the first layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 16 scale geometric cube, and the 1 / 16 scale initial geometric cube obtained by downsampling in step (2-4) is spliced ​​according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 16 scale geometric cube corresponding to the stereo image pair; (2-6) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 16 scale geometric cube corresponding to the stereo image pair obtained in step (2-5) and the 1 / 16 scale left view feature map obtained in step (2-2) are input into the context geometry attention module in the 3D regularization network to obtain a 1 / 16 scale context geometry attention weight cube corresponding to the stereo image pair, which ultimately integrates the context feature information and geometric information; (2-7) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 16 scale context geometric attention weight cube corresponding to the stereo image pair obtained in (2-6) is input into the second layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 8 scale geometric cube, and the 1 / 8 scale initial geometric cube obtained by downsampling in step (2-4) is spliced ​​according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 8 scale geometric cube corresponding to the stereo image pair; (2-8) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale geometric cube corresponding to the stereo image pair obtained in step (2-7) and the 1 / 8 scale feature map of the left view obtained in step (2-2) are input into the context geometry attention module in the 3D regularization network to obtain a 1 / 8 scale context geometry attention weight cube corresponding to the stereo image pair, which ultimately integrates context feature information and geometric information; (2-9) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale context geometric attention weight cube corresponding to the stereo image pair obtained in step (2-8) is input into the third layer encoder of the upsampling part of the hourglass network module in the 3D regularization network to obtain a 1 / 4 scale geometric cube, and the 1 / 4 scale geometric cube is spliced ​​with the 1 / 4 scale geometric cube of step (2-3) according to the channel dimension, and fused through 3D convolution to obtain the final 1 / 4 scale geometric cube corresponding to the stereo image pair; (2-10) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 8 scale geometric cube corresponding to the stereo image pair obtained in step (2-9) and the 1 / 4 scale feature map of the left view obtained in step (2-2) are input into the context geometry attention module in the 3D regularization network to obtain the final 1 / 4 scale context geometry attention weight cube corresponding to the stereo image pair; (2-11) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), all weights in the 1 / 4 scale context geometric attention weight cube corresponding to the stereo image pair obtained in step (2-10) are sorted by the importance of the disparity value in its disparity dimension, and all weights corresponding to the top 12 disparities are selected as the final context geometric attention weight cube A corresponding to the stereo image pair. cg ∈R C ×D′×H / 4×W / 4 ; (2-12) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the 1 / 4 scale left view feature map f corresponding to the stereo image pair obtained in step (2-2) is l ∈R B×C×H / 4×W / 4 , also known as the context feature map, is indexed according to the disparity value in step (2-3), the context feature map is increased by the disparity dimension, and the context feature map is copied along the disparity dimension to obtain the context feature cube V corresponding to the stereo image pair with the same shape as the context geometric attention weight cube c ∈R B×C×D′×H / 4×W / 4 ; (2-13) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the context geometric attention weight cube A corresponding to the stereo image pair obtained in step (2-11) is cg The context feature cube V corresponding to the stereo image pair obtained in the filtering step (2-12) c , that is, the two are multiplied element by element to obtain the final context geometric cube V corresponding to the stereo image pair cgv ∈R C×D′×H / 4×W / 4 ; (2-14) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the context geometry cube obtained in step (2-13) is input into the first two encoder layers and the first two decoder layers of the hourglass network module in the 3D regularization network; the context geometry cube V cgv Perform cost aggregation to obtain the target cube corresponding to the stereo image pair; (2-15) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), all disparity candidate values ​​of the target cube corresponding to the stereo image pair obtained in step (2-15) at each pixel, i.e., the 12 disparities in step (2-11), are weighted using a softmax function to obtain the disparity value at the pixel corresponding to the stereo image pair; and the disparity values ​​of the target cube at all pixels are combined to obtain an initial disparity map corresponding to the stereo image pair with a disparity dimension of 1; (2-16) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the initial disparity map corresponding to the stereo image pair obtained in step (2-15) and the 1 / 4 scale right view feature map obtained in step (2-2) are input into the warped disparity transformation layer of the disparity optimization module to obtain the 1 / 4 scale warped right view feature map f corresponding to the stereo image pair. rw,4 ; (2-17) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), the 1 / 4 scale left view feature map obtained in step (2-2) and the 1 / 4 scale distorted right view feature map f corresponding to the stereo image pair obtained in step (2-16) are converted into rw,4 Perform bilinear interpolation upsampling to obtain the left view feature map and the distorted right view feature map of the corresponding scale of the stereo image pair, and input the left view feature map and the distorted right view feature map into the distortion cost cube construction layer of the distortion parallax optimization module to obtain the distortion cost volume V corresponding to the stereo image pair. w ; (2-18) for each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), subtracting the left view feature map and the distorted right view feature map corresponding to the stereo image pair obtained in step (2-17) to obtain a reconstruction error corresponding to the stereo image pair; (2-19) For each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), the initial disparity map corresponding to the stereo image pair obtained in step (2-14), the left view feature map corresponding to the stereo image pair obtained in step (2-17), and the distorted cost volume V w , and the reconstruction error corresponding to the stereo image pair obtained in step (2-18) is input into the warped parallax layer of the warped parallax optimization module to obtain a fine-grained residual parallax value corresponding to the stereo image pair, and the fine-grained residual parallax value is added to the initial parallax to obtain a refined parallax value corresponding to the stereo image pair; (2-20) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), modify the 3D convolution operations in steps (2-4) to (2-10) and step (2-14) to 3D depth-separable convolution, and repeat the above steps (2-4) to (2-19) to obtain the predicted disparity value of the model corresponding to the stereo image pair; (2-21) For each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), all disparity values ​​of the context geometry attention weight cube obtained in (2-10) at each pixel are weighted using the softmax function to obtain the disparity value at the pixel corresponding to the stereo image pair; and the disparity values ​​of the context geometry attention weight cube at all pixels are combined to obtain the context feature geometry attention weight disparity map d corresponding to the stereo image pair with a disparity dimension of 1. att ; (2-22) for each stereo image pair in the training set of the stereo image pair dataset preprocessed in step (2-1), constructing a loss function using the predicted disparity value of the model corresponding to the stereo image pair obtained in step (2-20); (2-23) for each stereo image pair in the training set of the stereo image pair data set preprocessed in step (2-1), applying the loss function corresponding to the stereo image pair obtained in step (2-22) and using a back-propagation method to iteratively train the stereo matching model until the stereo matching model converges, thereby obtaining a preliminarily trained stereo matching model corresponding to the stereo image pair; (2-24) For each stereo image pair in the verification set of the stereo image pair dataset preprocessed in step (2-1), the stereo matching model preliminarily trained in step (3-23) is verified until the obtained disparity accuracy reaches the optimal level, thereby obtaining the trained stereo matching model corresponding to the stereo image pair.

6. The stereo matching method based on contextual geometric features and distortion parallax optimization according to claim 5, characterized in that: The calculation formula of the geometric cube in steps (2-3) is as follows: Where x represents the horizontal coordinate of the pixel on the left or right view feature map, y represents the vertical coordinate of the pixel on the left or right view feature map, d represents the disparity value between the left view feature map and the right view feature map, <,> represents the dot product of the feature vector of the pixel on the left or right view feature map, which is used to measure the similarity between the pixel of the left view image and the corresponding pixel of the right view image, ‖‖2 represents the L2 norm of the feature vector of the pixel on the left or right view feature map, and f l (x, y, d) represents the feature vector at the pixel position of the left image feature map with disparity d and coordinates (x, y), f r (xd,y,d) represents the feature vector at the pixel position of the right image feature map with disparity d and coordinates (xd,y) after disparity offset. The 1 / 4 scale distorted right view feature map f in step (2-17) rw,4 The calculation formula is as follows: f rw,4 =warping(f r,4 ,d0) where f r,4 is the right view feature map at 1 / 4 scale, d0 is the initial disparity value of the pixel point in the initial disparity map, and the warpping function is the warping disparity operation of the warped disparity transformation layer; The distortion in step (2-17) is the original body V w The expression is as follows: where f l and f rw are the left view feature map and the distorted right view feature map of the corresponding scale of the stereo image pair, and △d represents the range of fine-grained residual search; The loss function formula in step (2-23) is as follows: where d att and d gt It is the disparity directly obtained by the geometric attention weight of the context feature and the real disparity map provided in the training set; λ att represents the coefficient of the predicted attention weight disparity, λ represents the coefficient of the final predicted disparity, λ att The values ​​of and λ range from 0 to 1, and their sum is 1. represents the smoothed L1 loss.

7. A stereo matching system based on contextual geometric features and distortion parallax optimization, characterized in that: include: The first module is used to obtain the left and right perspective images to be matched that have been calibrated by the binocular camera, and preprocess the left and right perspective images to be matched to obtain preprocessed left and right perspective images; The second module is used to input the preprocessed left and right view images obtained by the first module into a pre-trained stereo matching model to obtain a final disparity map as a stereo matching result; The stereo matching model consists of four sequentially connected parts: a lightweight multi-scale feature extraction network, a 3D regularization network, a warped parallax optimization module, and a 3D depth-separable convolution module. The 3D regularization network includes an hourglass network module and a contextual geometry attention module; The hourglass network module consists of two parts: downsampling and upsampling; The downsampling part includes: The first layer decoder takes as input a five-dimensional tensor of shape (B, C, D / 4, H / 4, W / 4). It first feeds the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3, converts the number of channels of the five-dimensional tensor into 2C, and outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8). The five-dimensional tensor is then fed into a second 3D convolution kernel of dimension 3×3×3 and a BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 2C, D / 8, H / 8, W / 8), where D represents the disparity between the left and right view images. The second layer decoder takes as input the five-dimensional tensor of the shape (B, 2C, D / 8, H / 8, W / 8) output by the first layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3, converts the number of channels of the five-dimensional tensor into 4C, and outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); then, the five-dimensional tensor is input into the second 3D convolution kernel of dimension 3×3×3 and the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); The third layer decoder takes as input the five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16) output by the second layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 3×3×3, converts the number of channels of the five-dimensional tensor into 6C, and outputs a five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32); then, the five-dimensional tensor is input into the second 3D convolution kernel of dimension 3×3×3 and the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32); The upsampling part includes: The first layer encoder takes as input the five-dimensional tensor of shape (B, 6C, D / 32, H / 32, W / 32) output by the downsampled third layer decoder. It first inputs the five-dimensional tensor into a 3D convolution kernel of dimension 4×4×4, converts the number of channels of the five-dimensional tensor into 4C, and outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); then, the five-dimensional tensor is input into the BN+ReLU layer, and finally outputs a five-dimensional tensor of shape (B, 4C, D / 16, H / 16, W / 16); The second-layer encoder takes as input the five-dimensional tensor output by the first-layer decoder with a shape of (B, 4C, D / 16, H / 16, W / 16). It first inputs the five-dimensional tensor into a 3D convolution kernel with a dimension of 4×4×4, converts the number of channels of the five-dimensional tensor into 2C, and outputs a five-dimensional tensor with a shape of (B, 2C, D / 8, H / 8, W / 8). Then, the five-dimensional tensor is input into the BN+ReLU layer in sequence, and finally outputs a five-dimensional tensor with a shape of (B, 2C, D / 8, H / 8, W / 8). The third-layer encoder takes as input the five-dimensional tensor output by the second-layer decoder with a shape of (B, 2C, D / 8, H / 8, W / 8). It first inputs the five-dimensional tensor into a 3D convolution kernel with a dimension of 4×4×4, converts the number of channels of the five-dimensional tensor into C, and outputs a five-dimensional tensor with a shape of (B, C, D / 4, H / 4, W / 4). Then, the five-dimensional tensor is successively input into the BN+ReLU layer, and finally outputs a five-dimensional tensor with a shape of (B, C, D / 4, H / 4, W / 4). The contextual geometric attention module includes two inputs, one of which is the left view feature map F c , which is a four-dimensional tensor of shape (B, C, H, W) that contains rich context feature information; the other input is the geometric cube F G , a five-dimensional tensor of shape (B, C, D, H, W), which contains rich geometric information; The contextual geometric attention module first expands the left view feature map into a five-dimensional tensor of shape (B, C, D, H, W) according to the disparity dimension, namely the context feature cube F ′ , and the geometric cube F and the context feature cube F ′ Splicing is performed along the channel dimension to obtain a five-dimensional tensor with a shape of (B, 2C, D, H, W), namely the splicing cube; then, the five-dimensional tensor is input into a 3D convolution kernel with a dimension of 1×5×5, and the number of channels of the five-dimensional tensor is reduced to C, thereby outputting a five-dimensional tensor with a shape of (B, C, D, H, W); subsequently, the five-dimensional tensor is activated using the σ (Sigmoid) function, thereby outputting a five-dimensional tensor with a shape of (B, C, D, H, W), namely the splicing fusion cube A CF , the formula is as follows: TO CF =σ(conv3d(concat{F′ G ,F G })) Among them, concat means that the input tensor is spliced ​​along the channel dimension, and conv3d means 3D convolution; Then, by splicing and fusing cube A CF ∈R B×C×D×H×W and context feature cube F′ c To multiply element by element to obtain the initial context set attention, which is a five-dimensional tensor with a shape of (B, C, D, H, W). Subsequently, the five-dimensional tensor is spliced ​​with the geometric cube along the channel dimension to output a five-dimensional tensor with a shape of (B, 2C, D, H, W); finally, the five-dimensional tensor is input into a 3D convolution kernel with a dimension of 1×5×5, and the number of channels of the five-dimensional tensor is reduced to C, thereby outputting a five-dimensional tensor with a shape of (B, C, D, H, W), namely the context geometric attention cube G, which fully integrates the context feature information and geometric information to obtain a context geometric attention cube G containing rich information. Its formula is as follows: G=conv3d(concat{F G ,A CF ⊙F′ c }) Where ⊙ represents element-by-element multiplication; The warped parallax optimization module includes: The first part is the warped disparity transformation layer, whose input is the right view feature map and the initial disparity map. The right view feature map is a four-dimensional tensor with a shape of (B, C, H, W), and the initial disparity map is a four-dimensional tensor with a shape of (B, 1, H, W). The warped disparity transformation layer first creates grid coordinates (xx, yy) according to the number of channels C, height H, and width W of the input right view feature map to represent the coordinates of each pixel in the right view feature map; then, the grid horizontal coordinate xx is distorted and adjusted according to the initial disparity map to obtain the updated grid horizontal coordinate, which is adjusted to the standardized range [-1, 1] and converted to the PyTorch grid format to obtain the warped grid coordinates after initial disparity adjustment (warped xx, warped yy); then, use the nn.functional.grid_sample function and use the distorted grid coordinates after initial disparity adjustment to perform bilinear interpolation sampling on the right view feature map to obtain a distorted image; finally, create a mask mask of the same size as the right view feature map with all 1s, use the nn.functional.grid_sample function and use the distorted grid coordinates after initial disparity adjustment to perform bilinear interpolation sampling on the mask mask to obtain the value of the mask at the target position of the distorted grid coordinates to determine the valid pixels in the right view feature map, and retain the valid pixels, thereby obtaining a four-dimensional tensor with a shape of (B, C, H, W) after parallax distortion, that is, the distorted right view feature map; The second part is the warped cost cube construction layer, whose input is the warped right view feature map, left view feature map and maximum disparity value maxdisp obtained by the warped disparity transformation layer. The warped cost cube construction layer first creates an all-zero tensor volume of the same data type as the left view feature map according to the shape of the left view feature map to store the correlation volume; then, the disparity value range [-maxdisp, maxdisp] is determined according to the maximum disparity value maxdisp, all integer disparities in the disparity value range are traversed, and the cosine similarity is used to calculate the correlation between the left view feature map and the warped right view feature map at each integer disparity; finally, the obtained correlations under all integer disparities are stored in the all-zero tensor volume to obtain the warped cost volume, which is a five-dimensional tensor with a shape of (B, C, D, H, W); The third part is the warped disparity optimization layer, whose input is the left view feature map, the initial disparity map, and the warped cost volume obtained in the second layer. The warped disparity optimization layer first subtracts the warped right view feature map from the left view feature map to obtain the reconstruction error, which is a four-dimensional tensor with a shape of (B, C, H, W); then, the reconstruction error, the left view feature map, the initial disparity map, and the warped cost volume are spliced ​​along the channel dimension to obtain a spliced ​​volume, which is a four-dimensional tensor with a shape of (B, C ′ , H, W) four-dimensional tensor; then, the spliced ​​body and the initial disparity are input into a sub-precision module consisting of 8 convolutional layers for disparity optimization to extract the residual disparity value disp, that is, the disparity dimension value.

Images