A stereo matching system and method based on wavelet transform and light large kernel convolution

Abstract

The application provides a stereo matching system based on wavelet transform and lightweight large kernel convolution, which comprises a data acquisition unit and a stereo matching unit based on wavelet transform and lightweight large kernel convolution; the method comprises the following steps: acquiring a scene image to be measured, wavelet processing, extracting context information, extracting features through lightweight large kernel convolution, constructing a cost volume and combining a coding volume to obtain a final disparity, so that the matching precision in the stereo matching field and the matching and generalization of sharp regions of an image are improved; wavelet transform can extract multi-frequency image information, lightweight large kernel convolution can increase a receptive field, capture long-distance dependence and extract multi-frequency global feature information, so that the precision of stereo matching is improved; meanwhile, lightweight large kernel convolution can reduce a large amount of parameter calculation and parameters caused by large kernel convolution, and the extracted high-frequency information can improve the matching problem of sharp regions; experimental results show that the method provided by the application can improve the precision and generalization of stereo matching.

Description

[Technical Field]

[0001] This invention relates to the field of computer vision, specifically to a stereo matching system and method based on wavelet transform and lightweight large kernel convolution. [Background Technology]

[0002] In recent years, convolutional neural networks (CNNs) have been successfully applied to tasks such as object detection and image classification due to their powerful feature representation and function fitting capabilities. Deep learning-based stereo matching algorithms are mainly divided into end-to-end and non-end-to-end stereo matching algorithms. Early stereo matching algorithms were primarily implemented in a non-end-to-end manner, calculating the matching cost in the stereo matching stage by acquiring image features. Their results on the KITTI benchmark were superior to traditional algorithms. Applying convolutional neural networks to some steps of stereo matching improved performance. However, these algorithms still require some post-processing steps. Generally, non-end-to-end stereo matching algorithms include both deep learning processing modules and traditional algorithm processing modules. Compared to traditional algorithms, non-end-to-end stereo matching algorithms offer improved performance, but because they also include some manually defined functions, it is difficult to uniformly optimize the entire algorithm process. End-to-end stereo matching algorithms, on the other hand, have learnable algorithm parameters, resulting in strong adaptive adjustment capabilities and high accuracy in the generated disparity maps. Researchers have improved the accuracy of stereo matching by constructing an end-to-end stereo matching algorithm, eliminating post-processing steps, and optimizing the matching cost using 3D convolution. Later, some researchers used spatial pyramid structures to extract contextual information in the feature extraction stage and cascaded hourglass structures to optimize the cost cube. In recent years, iterative network updates have been used to refine parallax, resulting in some improvement in accuracy. However, even with these algorithms enhancing scene recognition capabilities to improve matching accuracy, they still struggle to achieve consistent matching for ill-conditioned regions lacking prior information, such as textureless regions. Accuracy and generalization still require further improvement. [Summary of the Invention]

[0003] The purpose of this invention is to address the aforementioned technical shortcomings and deficiencies by providing a stereo matching system and method based on wavelet transform and lightweight large kernel convolution. This system and method is simple in structure and easy to implement, and it has high scene recognition accuracy and practical value.

[0004] The technical solution of this invention: A stereo matching system based on wavelet transform and lightweight large kernel convolution, characterized in that it includes a data acquisition unit and a stereo matching unit based on wavelet transform and lightweight large kernel convolution; wherein, the stereo matching unit based on wavelet transform and lightweight large kernel convolution consists of a wavelet transform processing module, a context extraction module, a lightweight large kernel convolution feature extraction module, a cost body construction module, a combined encoding body construction module, and an iterative update module; the data acquisition unit is used to acquire the left and right color images of the scene under test and the true disparity map of the left color image, and its output is connected to the context extraction module and the wavelet transform processing module respectively. The input terminal of the context extraction module is connected to the input terminal of the iterative update module; the output terminal of the wavelet transform processing module is connected to the input terminal of the lightweight large kernel convolution feature extraction module; the output terminal of the lightweight large kernel convolution feature extraction module is connected to the input terminals of the cost body construction module and the combined coding body construction module respectively; the output terminal of the cost body construction module is connected to the input terminals of the combined coding body construction module and the iterative update module respectively; the output terminal of the combined coding body construction module is connected to the input terminal of the iterative update module; the output terminal of the iterative update module outputs the final disparity map, and the loss is calculated using the final disparity map and the true disparity map.

[0005] The cost body construction module, combined encoding body construction module, and iterative update module adopt the IGEV (Iterative Geometry Encoding Volume) network structure.

[0006] A stereo matching method based on wavelet transform and lightweight large-kernel convolution, characterized by the following steps:

[0007] (1) The data acquisition unit acquires the left and right color images of the scene under test and the true disparity map of the left image, and inputs the obtained left and right color images into the wavelet transform processing module, and inputs the obtained left color image into the context extraction module.

[0008] (2) The wavelet transform processing module processes the left and right color images using wavelet functions to obtain image sets of 12 channels on each side;

[0009] In step (2), the wavelet transform processing module performs wavelet function processing on the left and right color images, specifically referring to: Figure 2As shown, the left and right color images received by the wavelet transform processing module are divided into three channels: R (Red), G (Green), and B (Blue). These three channels then perform wavelet function dwt2 processing on the left and right color images respectively, obtaining image information at four frequencies: LL, LH, HL, and HH. LL represents the approximate image information in the low-frequency region, while LH, HL, and HH represent the detailed image information in the high-frequency region. The image information at four frequencies (LL, LH, HL, HH) generated from the R channel, the G channel, and the B channel of the left color image are then processed together. The image information of four frequencies (LL, LH, HL, HH) of the right color image, totaling 12 multi-frequency image information, is concatenated using the Concat function according to the channels of the image information to obtain a left image set of 12 channels. The image information of four frequencies (LL, LH, HL, HH) generated from the R channel, the image information of four frequencies (LL, LH, HL, HH) generated from the G channel, and the image information of four frequencies (LL, LH, HL, HH) generated from the B channel of the right color image, totaling 12 multi-frequency image information, is concatenated using the Concat function according to the channels of the image information to obtain a right image set of 12 channels, as shown in formula (1):

[0010]

[0011] Where x is the RGB input image, i is the channel corresponding to the RGB image, dwt2 is the wavelet function, LL, LH, HL, and HH are the image information of four frequencies, Concat is the stitching operation, and p is the image set;

[0012] The resolution of the four frequency images acquired by each of the three channels R, G, and B is H / 2 x W / 2, and the number of channels is 1. Here, H and W are the height and width of the left and right color images received by the input terminal of the wavelet transform processing module, respectively.

[0013] (3) Use the context extraction module to extract context information from the left color image to obtain context information maps with scales of H / 4x W / 4x64, H / 4x H / 4x 128, H / 8x W / 8x 128 and H / 16x W / 16x 128;

[0014] The context extraction module in step (3) consists of one convolutional layer, one normalization layer, three residual layers, and two downsampling layers. Each of the three residual layers I, II, and III consists of two residual blocks I, for a total of six residual blocks I. Each of the downsampling layers I and II consists of one residual block I and one residual block II. Residual block I consists of two convolutions with a kernel size of 3x3 and a stride of 1, and two ReLU functions. Residual block II consists of one convolution with a kernel size of... The system consists of a 3x3 convolution with a stride of 2, a 3x3 convolution with a stride of 1, and two ReLU functions; the convolutional layer of the context extraction module is a 7x7 convolutional layer with a stride of 2, 3 input channels, and 64 output channels; the three residual layers are residual layer I, residual layer II, and residual layer III with 64, 96, and 128 output channels, respectively; the two downsampling layers are downsampling layer I and downsampling layer II, each with 128 output channels.

[0015] The process of extracting context information in step (3) consists of the following steps:

[0016] (3-1) Using the context extraction module, the left color image obtained in step (1) is processed through a convolutional layer and then through a normalization layer to obtain a context information map I with a scale of ;

[0017] (3-2) The context information graph I obtained in step (3-1) is passed through residual layer I, residual layer II and residual layer III to obtain context information graph II;

[0018] (3-3) The context information graph II obtained in step (3-2) is passed through the downsampling layer I to obtain the context information graph III;

[0019] (3-4) The context information graph III obtained in step (3-3) is passed through the downsampling layer II to obtain the context information graph IV.

[0020] The scales of the context information graphs I, II, III and IV obtained in step (3) are H / 4x W / 4x 64, H / 4x H / 4x 128, H / 8x W / 8x 128 and H / 16x W / 16x128, respectively.

[0021] (4) Use a lightweight large kernel convolution module to extract features from the image sets of the left and right 12 channels obtained in step (2) to obtain multi-scale feature maps. Where l represents the feature map of the left image and r represents the feature map of the right image;

[0022] Step (4) specifically refers to: the lightweight large kernel convolution module performing four stages of image data processing on the image sets of the left and right 12 channels obtained in step (2), with N being the number of lightweight large kernel basic modules used in the four stages. i The output channels are 3, 3, 5, and 2 in sequence, with C being the number of output channels. i The values ​​are 24, 32, 96, and 160 respectively, where i ∈ {1, 2, 3, 4}. The feature map scale obtained from the 12-channel image set after the first stage is H / 4x W / 4x C1, the scale after the second stage is H / 8x W / 8x C2, the scale after the third stage is H / 16x W / 16x C3, and the scale after the fourth stage is H / 32x W / 32x C4. In other words, a multi-scale feature map is finally obtained. Where l represents the feature map of the left image and r represents the feature map of the right image.

[0023] The four-stage image data processing process consists of the following steps:

[0024] (4-1) Divide each channel image of the 12 channels on the left and right sides obtained in step (2) into N patches of size 7x7, where N is H / 14 x W / 14. Downsample these patches using a convolution with a kernel size of 3 and a stride of 2, and then perform normalization as shown in formula (2) to obtain the output x:

[0025] x = Norm(Overl apPatch(p))(2)

[0026] Wherein, Norm is the normalization process, OverlapPatch is the image segmentation and downsampling process, and p is the image set mentioned in step (2);

[0027] (4-2) The output x obtained in step (4-1) is passed through N lightweight large-core base modules, and the lightweight large-core base block will be 21x in size. The large kernel convolution of 21 is decomposed into four convolutions: a 1x7 convolution, a 7x1 convolution, a 3x3 convolution, and a dilated convolution with a kernel size of 7 and a dilation rate of 3. The output x obtained in step (4-1) is normalized and then divided into four groups according to the average number of channels. Each group is then subjected to one of the four convolutions: the first group is processed by a 1x7 convolution, the second group by a 7x1 convolution, the third group by a 3x3 convolution, and the fourth group by a dilated convolution with a kernel size of 7 and a dilation rate of 3. The four groups of data after convolution are then concatenated according to their channels. The concatenated data is then element-wise added to the output x obtained in step (4-1) to obtain data y. Data y is then normalized and processed using an MLP (Multilayer Perceptron) module, a well-known technique. Finally, it is element-wise added to y to obtain the output z.

[0028] The steps of the lightweight large-core basic module in step (4-2) include the following:

[0029] (4-2-1) The output X obtained in step (4-1) is first standardized, as shown in formula (3):

[0030] x'=Norm(x)(3)

[0031] (4-2-2) Divide the output obtained in step (4-2-1) into 4 groups according to the channels, and process the image data using 4 convolutions of size 1x7, 7x1, and 3x3 and a dilated convolution with a kernel size of 7 and a dilation rate of 3 for each of the 4 groups to obtain the output x'. h ,x' w ,x' hw ,x' dl As shown in formula (4):

[0032]

[0033] Split represents the operation of dividing the data equally by channel; x h x w x hw x dl These are four sets of data evenly distributed by channel; DWConv 7x1 A 7x1 depth convolution; DWConv 1x7 A 1x7 depth convolution; DWConv 3x3 A 3x3 depthwise convolution; DLConv k＝7，d＝3 A dilated convolution with a kernel size of 7 and a dilation rate of 3; x' h ,x' w ,x'hw ,x' dl These are four sets of data processed by four different convolutions.

[0034] (4-2-3) Merge the four sets of data obtained from the four different convolution processes in step (4-2-2), that is, concatenate them by channel, and perform element-wise addition with the output x from step (4-1) as shown in formula (5) to obtain the output y:

[0035] y = Concat(x' h ,x' w ,x' hw ,x' dl )+x(5)

[0036] Concat is a concatenation operation based on channels;

[0037] (4-2-4) The output y in step (4-2-3) is normalized and processed by the MLP module to obtain the output y', as shown in formula (6);

[0038] y'=MLP(Norm(y))(6)

[0039] (4-2-5) Perform element-wise addition on the output y' in step (4-2-4) and the output y in step (4-2-3) as shown in formula (7) to obtain the output z.

[0040] z = y' + y(7)

[0041] (4-3) The output z in step (4-2) is standardized to obtain the feature map information, as shown in formula (8).

[0042] fmap = Norm(z)(8)

[0043] (5) The multi-scale feature map obtained in step (4) Input the cost volume construction module and the combined encoding volume construction module to obtain the cost volume and the combined encoding volume respectively; input the multi-scale context information map obtained in step (3) together with the cost volume and the combined encoding volume into the iterative update module to obtain the final disparity; wherein the cost volume construction module, the combined encoding volume construction module and the iterative update module adopt the network structure of IGEV (Iterative Geometry Encoding Volume), which is a well-known technology.

[0044] The advantages of this invention are as follows: The contribution of the wavelet transform and lightweight large-kernel convolution method to stereo matching is that this method can capture long-distance dependencies, increase the receptive field, and extract multi-frequency feature information. Low-frequency information can maintain the matching effect in textureless regions, while high-frequency information can improve the effect in sharp regions of the image. At the same time, it reduces the large number of parameters brought by large-kernel convolution, thus improving accuracy, computational cost, and generalization ability.

[0045] 1. This invention fuses information of different frequencies through wavelet transform, preserving low-frequency information and high-frequency details of the image.

[0046] 2. By using lightweight large-kernel convolution feature extraction, long-distance dependencies are captured, the receptive field is increased, and global multi-frequency matching information is obtained while reducing the large number of parameters brought by large-kernel convolution.

[0047] 3. This invention can accelerate inference speed while reducing the number of parameters, retain detailed contour information of the image, match images well, improve the accuracy of stereo matching, and has good generalization ability.

[0048] 4. Based on an iteratively updated stereo matching model, this method obtains multi-frequency image information through wavelet transform and extracts multi-frequency feature information through lightweight large-kernel convolution to capture long-distance dependencies and increase the receptive field, thereby enriching the matching information. This method solves the problem of some ill-conditioned regions. The average matching error on the Sceneflow dataset is 0.435, with a lower error rate. It also shows good generalization on the Eth3d and Middlebury datasets. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the structure of a stereo matching system based on wavelet transform and lightweight large kernel convolution according to the present invention.

[0050] Figure 2 This is a schematic diagram illustrating the principle structure of a wavelet transform processing module in a stereo matching system based on wavelet transform and lightweight large kernel convolution, according to the present invention.

[0051] Figure 3 This is a schematic diagram of the principle structure of the context extraction module in a stereo matching system based on wavelet transform and lightweight large kernel convolution, as well as the structure of residual block I and residual block II.

[0052] Figure 4 This is a schematic diagram illustrating the principle and structure of a lightweight large kernel convolution feature extraction module in a stereo matching system based on wavelet transform and lightweight large kernel convolution according to the present invention.

[0053] Figure 5This is a schematic diagram illustrating the principle structure of a lightweight large kernel basic module in a stereo matching system based on wavelet transform and lightweight large kernel convolution according to the present invention.

[0054] Figure 6 This is a schematic diagram of a test case comparing the generalization performance of the present invention and the IGEV method on the Middlebury dataset (wherein, Figure 6 -a represents the color image of the left image to be tested. Figure 6 -b represents the disparity map generated by the IGEV method. Figure 6 -c represents the disparity map generated by this method. Figure 6 -d represents the true disparity value of the left image). Detailed Implementation

[0055] The present invention will now be described in detail with reference to the accompanying drawings.

[0056] Example: A stereo matching system based on wavelet transform and lightweight large kernel convolution, such as... Figure 1 As shown, its features include a data acquisition unit and a stereo matching unit based on wavelet transform and lightweight large kernel convolution. The stereo matching unit based on wavelet transform and lightweight large kernel convolution consists of a lower wavelet transform processing module, a context extraction module, a lightweight large kernel convolution feature extraction module, a cost body construction module, a combined encoding body construction module, and an iterative update module. The data acquisition unit acquires left and right color images of the scene under test and their corresponding real disparity maps. Its output is connected to the input of the context extraction module and the wavelet transform processing module, respectively. The output of the context extraction module is connected to the input of the iterative update module. The output of the wavelet transform processing module is connected to the input of the lightweight large kernel convolution feature extraction module. The output of the lightweight large kernel convolution feature extraction module is connected to the input of the cost body construction module and the combined encoding body construction module, respectively. The output of the cost body construction module is connected to the input of the combined encoding body construction module and the iterative update module, respectively. The output of the combined encoding body construction module is connected to the output of the iterative update module. The output of the iterative update module outputs the final disparity map, and the loss is calculated using the final disparity map and the real disparity map.

[0057] In a further embodiment, the data acquisition unit acquires the left and right color images of the scene under test and the true disparity map of the left image, and inputs the obtained left and right color images into the wavelet transform processing module and the obtained left color image into the context extraction module; the context extraction module receives the left color image from the data acquisition unit and outputs multi-scale context features; the lightweight large-kernel convolution feature extraction module receives the left and right color images from the data acquisition unit and outputs multi-scale feature maps of the left and right images respectively; the cost body construction module and the combined encoding body construction module receive the left and right multi-scale feature maps output by the lightweight large-kernel convolution feature extraction module; the iterative update module receives the multi-scale context features output by the context extraction module, the initial disparity output by the cost body construction module, and the combined encoding body output by the combined encoding body construction module, iteratively updates the initial disparity, and outputs the final disparity map. During the training phase, the loss is calculated between the final disparity map and the true disparity map to optimize the model. During the inference phase, the final disparity map is output.

[0058] The cost body construction module, combined code body construction module, and iterative update module adopt the IGEV network structure.

[0059] In this embodiment, the data acquisition unit processes the Sceneflow, Eth3d, and Middlebury datasets. The Sceneflow dataset is used for model pre-training. This dataset contains 35,454 training image pairs and 4,370 test image pairs, providing a real disparity map for model training. Middlebury is an indoor dataset that provides 15 training pairs and 15 test pairs, some of which are under inconsistent lighting or color conditions, and all images have three different resolutions. Eth3d is a grayscale dataset with 27 training pairs and 20 test pairs. The Middlebury and Eth3d datasets are used to evaluate the cross-domain generalization performance of this method.

[0060] A stereo matching method based on wavelet transform and lightweight large-kernel convolution, characterized by the following steps:

[0061] (1) The data acquisition unit acquires the left and right color images of the scene under test and the true disparity map of the left image, and inputs the obtained left and right color images into the wavelet transform processing module, and inputs the obtained left color image into the context extraction module.

[0062] (2) The wavelet transform processing module processes the left and right color images using wavelet functions to obtain image sets of 12 channels on each side;

[0063] In step (2), the wavelet transform processing module performs wavelet function processing on the left and right color images, specifically referring to: Figure 2 As shown, the left and right color images received at the input of the wavelet transform processing module are divided into three channels: R, G, and B. These three channels then perform wavelet function dwt2 processing on the left and right color images respectively, obtaining image information at four frequencies for each channel: LL, LH, HL, and HH. LL represents the approximate image information for the low-frequency portion, while LH, HL, and HH represent the detailed image information for the high-frequency portion. The image information at frequencies LL, LH, HL, and HH generated from the R channel, the G channel, and the B channel of the left color image are then processed. The image information of four frequencies, HH, HL, and HH, totaling 12 multi-frequency image information, is concatenated using the Concat function according to the channels of the image information to obtain a left image set of 12 channels. The image information of four frequencies, LL, LH, HL, and HH, generated from the R channel of the right color image, the image information of four frequencies, LL, LH, HL, and HH, generated from the G channel, and the image information of four frequencies, LL, LH, HL, and HH, generated from the B channel, totaling 12 multi-frequency image information, is concatenated using the Concat function according to the channels of the image information to obtain a right image set of 12 channels, as shown in formula (1):

[0064]

[0065] Where x is the RGB input image, i is the channel corresponding to the RGB image, dwt2 is the wavelet function, LL, LH, HL, and HH are the image information of four frequencies, Concat is the stitching operation, and p is the image set;

[0066] The resolution of the four frequency images acquired by each of the three channels R, G, and B is H / 2 x W / 2, and the number of channels is 1. Here, H and W are the height and width of the left and right color images received by the input terminal of the wavelet transform processing module, respectively.

[0067] (3) Use the context extraction module to extract context information from the left color image to obtain context information maps with scales of H / 4x W / 4x64, H / 4x H / 4x 128, H / 8x W / 8x 128 and H / 16x W / 16x 128;

[0068] The context extraction module in step (3) specifically refers to: such as Figure 3As shown, the system consists of one convolutional layer, one normalization layer, three residual layers, and two downsampling layers. Each of the three residual layers (I, II, and III) comprises two residual blocks (I), for a total of six residual blocks (I). Each of the downsampling layers (I and II) comprises one residual block (I) and one residual block (II). Residual block I consists of two convolutions with a kernel size of 3x3 and a stride of 1, and two ReLU functions. Residual block II consists of one convolution with a kernel size of 3x3 and a stride of 1. The system consists of a 2x2 convolution, a 3x3 convolution with a stride of 1, and two ReLU functions; the context extraction module's convolutional layer is a 7x7 convolutional layer with a stride of 2, 3 input channels, and 64 output channels; the three residual layers are residual layer I, residual layer II, and residual layer III with 64, 96, and 128 output channels, respectively; the two downsampling layers are downsampling layer I and downsampling layer II, each with 128 output channels.

[0069] The process of extracting context information in step (3) consists of the following steps:

[0070] (3-1) Using the context extraction module, the left color image obtained in step (1) is processed through a convolutional layer and then through a normalization layer to obtain a context information map I with a scale of ;

[0071] (3-2) The context information graph I obtained in step (3-1) is passed through residual layer I, residual layer II and residual layer III to obtain context information graph II;

[0072] (3-3) The context information graph II obtained in step (3-2) is passed through the downsampling layer I to obtain the context information graph III;

[0073] (3-4) The context information graph III obtained in step (3-3) is passed through the downsampling layer II to obtain the context information graph IV.

[0074] The scales of the context information graphs I, II, III and IV obtained in step (3) are H / 4x W / 4x 64, H / 4x H / 4x 128, H / 8x W / 8x 128 and H / 16x W / 16x128, respectively.

[0075] (4) Use a lightweight large kernel convolution module to extract features from the image sets of the left and right 12 channels obtained in step (2) to obtain multi-scale feature maps. Where l represents the feature map of the left image and r represents the feature map of the right image;

[0076] Step (4) specifically refers to: Figure 4As shown, the image data processing for the left and right 12 channels of the image set obtained in step (2) is performed in four stages by the lightweight large kernel convolution module. The number of lightweight large kernel base modules used in the four stages is N. i The output channels are 3, 3, 5, and 2 in sequence, with C being the number of output channels. i The values ​​are 24, 32, 96, and 160 respectively, where i ∈ {1, 2, 3, 4}. The feature map scale obtained from the 12-channel image set after the first stage is H / 4x W / 4x C1, the scale after the second stage is H / 8x W / 8x C2, the scale after the third stage is H / 16x W / 16x C3, and the scale after the fourth stage is H / 32x W / 32x C4. In other words, a multi-scale feature map is finally obtained. Where l represents the feature map of the left image and r represents the feature map of the right image.

[0077] The four-stage image data processing process consists of the following steps:

[0078] (4-1) Divide each of the 12 channels in the image set obtained in step (2) into N patches of size 7x7, where N is H / 14 x W / 14. Downsample these patches using a convolution with a kernel size of 3 and a stride of 2, and then perform normalization as shown in formula (2) to obtain the output x:

[0079] x = Norm(Overl apPatch(p))(2)

[0080] Wherein, Norm is the normalization process, OverlapPatch is the image segmentation and downsampling process, and p is the image set mentioned in step (2);

[0081] (4-2) The output x obtained in step (4-1) is processed through N lightweight large kernel base modules. The lightweight large kernel base module decomposes the large kernel convolution of size 21x21 into 4 convolutions, namely: 1x7, 7x1, 3x3 convolutions and dilated convolution with kernel size 7 and dilation rate 3. After the output x obtained in step (4-1) is normalized, it is divided into 4 groups according to the average number of channels. The first group is processed by 1x7 convolution, the second group is processed by 7x1 convolution, the third group is processed by 3x3 convolution, and the fourth group is processed by dilated convolution with kernel size 7 and dilation rate 3. Then, the 4 groups of data after convolution are concatenated according to the number of channels. The concatenated data is element-wise added to the output x obtained in step (4-1) to obtain data y. Data y is normalized and processed by the MLP (Multilayer Perceptron) module, which is a well-known technology. Then, it is element-wise added to y to obtain the output z.

[0082] The steps of the lightweight large-core basic module in step (4-2) include the following: Figure 5 As shown:

[0083] (4-2-1) The output X obtained in step (4-1) is first standardized, as shown in formula (3):

[0084] x'=Norm(x)(3)

[0085] (4-2-2) Divide the output obtained in step (4-2-1) into 4 groups according to the channels, and process the image data using 4 convolutions of size 1x7, 7x1, and 3x3 and a dilated convolution with a kernel size of 7 and a dilation rate of 3 for each of the 4 groups to obtain the output x'. h ,x' w ,x' hw ,x' dl As shown in formula (4):

[0086]

[0087] Split represents the operation of dividing the data equally by channel; x h x w x hw x dl These are four sets of data evenly distributed by channel; DWConv 7x1 A 7x1 depth convolution; DWConv 1x7 A 1x7 depth convolution; DWConv 3x3 A 3x3 depthwise convolution; DLConv k＝7，d＝3 A dilated convolution with a kernel size of 7 and a dilation rate of 3; x' h ,x' w ,x' hw ,x' dl These are four sets of data processed by four different convolutions.

[0088] (4-2-3) Merge the four sets of data obtained from the four different convolution processes in step (4-2-2), that is, concatenate them by channel, and perform element-wise addition with the output x from step (4-1) as shown in formula (5) to obtain the output y:

[0089] y = Concat(x' h ,x' w ,x' hw ,x' dl )+x(5)

[0090] Concat is a concatenation operation based on channels;

[0091] (4-2-4) The output y in step (4-2-3) is normalized and processed by the MLP module to obtain the output y', as shown in formula (6);

[0092] y'=MLP(Norm(y))(6)

[0093] (4-2-5) Perform element-wise addition on the output y' in step (4-2-4) and the output y in step (4-2-3) as shown in formula (7) to obtain the output z.

[0094] z = y' + y(7)

[0095] (4-3) The output z in step (4-2) is standardized to obtain the feature map information, as shown in formula (8).

[0096] fmap = Norm(z)(8)

[0097] (5) The multi-scale feature map obtained in step (4) Input the cost body construction module and the combined coding body construction module to obtain the cost body and the combined coding body respectively; input the multi-scale context information graph obtained in step (3) together with the cost body and the combined coding body into the iterative update module to obtain the final disparity; wherein the cost body construction module, the combined coding body construction module and the iterative update module adopt the IGEV network structure, which is a well-known technology.

[0098] (6) The contribution of wavelet transform and lightweight large-kernel convolution to stereo matching is that this method can capture long-distance dependencies, increase the receptive field, and extract multi-frequency feature information. Low-frequency information can maintain the matching effect of textureless regions, while high-frequency information can improve the effect of sharp regions in the image. At the same time, it reduces the large number of parameters brought by large-kernel convolution, and improves accuracy, computational cost and generalization. As shown in Table 1, the number of parameters of this method is reduced to 12.38M, which is less than the number of parameters of IGEV. As shown in Table 2, the EPE of this method on the Sceneflow dataset is 0.435. As shown in Table 3, this method has good generalization on the Eth3d and Middlebury datasets. Figure 6 The example shown is a test case comparing the generalization performance of our method and the IGEV method on the Middlebury dataset.

[0099] Table 1 Comparison of parameters between this method and IGEV

[0100]

[0101] Table 2. Comparison of matching performance between this method and other methods on the Sceneflow dataset.

[0102]

[0103] Table 3. Comparison of generalization performance of this method with other methods.

[0104]

[0105] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A stereo matching system based on wavelet transform and lightweight large-kernel convolution, characterized in that... The system includes a data acquisition unit and a stereo matching unit based on wavelet transform and lightweight large kernel convolution. The stereo matching unit based on wavelet transform and lightweight large kernel convolution comprises a wavelet transform processing module, a context extraction module, a lightweight large kernel convolution feature extraction module, a cost body construction module, a combined encoding body construction module, and an iterative update module. The data acquisition unit acquires left and right color images of the test scene and a true disparity map of the left color image. Its output is connected to the inputs of the context extraction module and the wavelet transform processing module, respectively. The output of the context extraction module is connected to the input of the iterative update module. The output of the wavelet transform processing module is connected to the input of the lightweight large kernel convolution feature extraction module. The output of the lightweight large kernel convolution feature extraction module is connected to the inputs of the cost body construction module and the combined encoding body construction module, respectively. The output of the cost body construction module is connected to the inputs of the combined encoding body construction module and the iterative update module, respectively. The output of the combined encoding body construction module is connected to the input of the iterative update module. The iterative update module outputs the final disparity map, and the loss is calculated using the final disparity map and the true disparity map.

2. The stereo matching system based on wavelet transform and lightweight large kernel convolution as described in claim 1, characterized in that... The cost body construction module, combined code body construction module, and iterative update module adopt the IGEV network structure.

3. A stereo matching method based on wavelet transform and lightweight large-kernel convolution, characterized in that... Includes the following steps: (1) The data acquisition unit acquires the left and right color images of the scene under test and the true disparity map of the left image, and inputs the obtained left and right color images into the wavelet transform processing module, and inputs the obtained left color image into the context extraction module. (2) The wavelet transform processing module performs wavelet function processing on the left and right color images to obtain image sets of 12 channels on each side; (3) Use the context extraction module to extract context information from the left color image to obtain context information maps with scales of H / 4 x W / 4 x 64, H / 4 x H / 4 x 128, H / 8 x W / 8 x 128 and H / 16 x W / 16 x 128; (4) Use a lightweight large kernel convolution module to extract features from the image sets of the left and right 12 channels obtained in step (2) to obtain multi-scale feature maps. Where l represents the feature map of the left image, r represents the feature map of the right image; i represents the number of stages in image data processing; C i This represents the number of output channels in the i-th stage; (5) The multi-scale feature map obtained in step (4) Input the cost body construction module and the combined encoding body construction module to obtain the cost body and the combined encoding body respectively; input the multi-scale context information graph obtained in step (3) together with the cost body and the combined encoding body into the iterative update module to obtain the final disparity; The cost body construction module, the combined encoding body construction module, and the iterative update module all use the IGEV network structure.

4. The stereo matching method based on wavelet transform and lightweight large kernel convolution according to claim 3, characterized in that... In step (2), the wavelet transform processing module performs wavelet function processing on the left and right color images. Specifically, the left and right color images received by the input end of the wavelet transform processing module are divided into three channels: R, G, and B. These three channels then perform wavelet function dwt2 processing on the left and right color images respectively, obtaining image information at four frequencies: LL, LH, HL, and HH. LL represents the approximate image information in the low-frequency part, while LH, HL, and HH represent the detailed image information in the high-frequency part. The image information at four frequencies (LL, LH, HL, HH) generated from the R channel of the left color image and the image information at four frequencies (LL, LH, HL, HH) generated from the G channel are then processed. The image information of the four frequencies LL, LH, HL, and HH generated by the B channel, totaling 12 multi-frequency image information, is concatenated according to the channels of the image information using the Concat function to obtain a left image set of 12 channels. The image information of the four frequencies LL, LH, HL, and HH generated by the R channel, the four frequencies LL, LH, HL, and HH generated by the G channel, and the four frequencies LL, LH, HL, and HH generated by the B channel of the right color image, totaling 12 multi-frequency image information, is concatenated according to the channels of the image information using the Concat function to obtain a right image set of 12 channels, as shown in formula (1): (1) Where x is the RGB input image, i is the channel corresponding to the RGB image, dwt2 is the wavelet function, LL, LH, HL, and HH are the image information of four frequencies, Concat is the stitching operation, and p is the image set.

5. The stereo matching method based on wavelet transform and lightweight large kernel convolution according to claim 4, characterized in that... The resolution of the four frequency images acquired by each of the three channels R, G, and B is H / 2 x W / 2, and the number of channels is 1. Here, H and W are the height and width of the left and right color images received by the input terminal of the wavelet transform processing module, respectively.

6. The stereo matching method based on wavelet transform and lightweight large kernel convolution according to claim 3, characterized in that... The context extraction module in step (3) consists of one convolutional layer, one normalization layer, three residual layers, and two downsampling layers. Each of the three residual layers I, II, and III consists of two residual blocks I, for a total of six residual blocks I. Each of the downsampling layers I and II consists of one residual block I and one residual block II. Residual block I consists of two convolutions with a kernel size of 3x3 and a stride of 1, and two ReLU functions. Residual block II consists of one convolution with a kernel size of... The system consists of a 3x3 convolution with a stride of 2, a 3x3 convolution with a stride of 1, and two ReLU functions; the convolutional layer of the context extraction module is a 7x7 convolutional layer with a stride of 2, 3 input channels, and 64 output channels; the three residual layers are residual layer I, residual layer II, and residual layer III with 64, 96, and 128 output channels, respectively; the two downsampling layers are downsampling layer I and downsampling layer II, each with 128 output channels.

7. The stereo matching method based on wavelet transform and lightweight large kernel convolution according to claim 3, characterized in that... The process of extracting context information in step (3) consists of the following steps: (3-1) Using the context extraction module, the left color image obtained in step (1) is processed through a convolutional layer and then through a normalization layer to obtain the context information map I at the scale; (3-2) The context information graph I obtained in step (3-1) is passed through residual layer I, residual layer II and residual layer III to obtain context information graph II; (3-3) The context information graph II obtained in step (3-2) is passed through the downsampling layer I to obtain the context information graph III; (3-4) The context information graph III obtained in step (3-3) is passed through the downsampling layer II to obtain the context information graph IV.

8. The stereo matching method based on wavelet transform and lightweight large kernel convolution according to claim 7, characterized in that... The scales of the context information graphs I, II, III and IV obtained in step (3) are H / 4 x W / 4 x 64, H / 4 x H / 4 x 128, H / 8 x W / 8 x 128 and H / 16 x W / 16 x 128, respectively.

9. The stereo matching method based on wavelet transform and lightweight large kernel convolution according to claim 3, characterized in that... Step (4) specifically refers to: the lightweight large kernel convolution module performing four stages of image data processing on the image sets of the left and right 12 channels obtained in step (2), with N being the number of lightweight large kernel base modules used in the four stages. i The output channels are 3, 3, 5, and 2 in sequence, with C being the number of output channels. i The values ​​are 24, 32, 96, and 160 respectively, where i ∈ {1, 2, 3, 4}. The feature map scale obtained from the 12-channel image set after the first stage is H / 4 x W / 4 x C1, the scale after the second stage is H / 8 x W / 8 x C2, the scale after the third stage is H / 16 x W / 16 x C3, and the scale after the fourth stage is H / 32 x W / 32 x C4. In other words, a multi-scale feature map is finally obtained. , where l represents the feature map of the left image and r represents the feature map of the right image; The four-stage image data processing process consists of the following steps: (4-1) Divide each channel image of the 12 channels on the left and right sides obtained in step (2) into N patches of size 7x7, where N is H / 14 x W / 14. Downsample these patches using a convolution with a kernel size of 3 and a stride of 2, and then perform normalization as shown in formula (2) to obtain the output x: (2) Wherein, Norm is the normalization process, OverlapPatch is the image segmentation and downsampling process, and p is the image set mentioned in step (2); (4-2) The output x obtained in step (4-1) is processed through N lightweight large kernel base modules. The lightweight large kernel base module decomposes the large kernel convolution of size 21 x 21 into 4 convolutions, namely: 1x7 convolution, 7x1 convolution, 3x3 convolution and dilated convolution with kernel size of 7 and dilation rate of 3. After the output x obtained in step (4-1) is normalized, it is divided into 4 groups according to the average of the channels. The 4 groups are then processed by 4 convolutions respectively: the first group is processed by 1x7 convolution, the second group by 7x1 convolution, the third group by 3x3 convolution, and the fourth group by dilated convolution with kernel size of 7 and dilation rate of 3. Then, the 4 groups of data after convolution are concatenated according to the channels. The concatenated data is then element-wise added to the output x obtained in step (4-1) to obtain data y. Data y is then normalized and processed by the MLP module, and then element-wise added to y to obtain output z. (4-3) The output z in step (4-2) is standardized to obtain the feature map information, as shown in formula (8). (8)。 10. A stereo matching method based on wavelet transform and lightweight large kernel convolution as described in claim 9, characterized in that... The steps of the lightweight large-core basic module in step (4-2) include the following: (4-2-1) The output X obtained in step (4-1) is first standardized, as shown in formula (3): (3) (4-2-2) Divide the output obtained in step (4-2-1) into 4 groups according to the channels, and process the image data using 4 convolutions of size 1 x 7, 7 x 1, and 3 x 3 and a dilated convolution with a kernel size of 7 and a dilation rate of 3 for each of the 4 groups to obtain the output. As shown in formula (4): (4) Split represents the operation of dividing the data equally by channel; x h x w x hw x dl These are four sets of data evenly distributed by channel; DWConv 7x1 A 7 x 1 depth convolution; DWConv 1x7 A 1 x 7 depthwise convolution; DWConv 3x3 A 3x3 depthwise convolution; DLConv k=7，d=3 A dilated convolution with a kernel size of 7 and a dilation rate of 3; These are four sets of data processed by four different convolutions. (4-2-3) Merge the four sets of data obtained from the four different convolution processes in step (4-2-2), that is, concatenate them by channel, and perform element-wise addition with the output x in step (4-1) as shown in formula (5) to obtain the output y: (5) Concat is a concatenation operation based on channels; (4-2-4) The output y in step (4-2-3) is normalized and processed by the MLP module to obtain the output. As shown in formula (6); (6) (4-2-5) The output of step (4-2-4) The output z is obtained by performing element-wise addition with the output y from step (4-2-3) as shown in formula (7). (7)。

Images