Three-dimensional matching system and method based on wavelet transform and lightweight large kernel convolution

Through the stereo matching system of wavelet transformation and lightweight large-core convolution, the problem of insufficient matching consistency and generalization of the stereo matching algorithm in textureless areas is solved, and higher accuracy and generalization are achieved, especially in the non-standard area. Matching effect of texture areas.

CN119048568AActive Publication Date: 2024-11-29TIANJIN UNIVERSITY OF TECHNOLOGY

Patent Information

Application Number
CN202410981407.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-29
Estimated Expiration
2044-07-22

AI Technical Summary

Technical Problem

The existing stereo matching algorithms are difficult to achieve matching consistency in pathological areas that lack prior information, such as textureless areas, and the accuracy and generalization need to be improved.

Method used

Using a stereo matching system based on wavelet transformation and lightweight large-core convolution, multi-frequency information is extracted through the wavelet transformation processing module, combined with the lightweight large-core convolution feature extraction module to capture long-distance dependence, increase the receptive field, and use iterative updates. Module optimization parallax map.

Benefits of technology

The accuracy and generalization of stereo matching are improved, especially the matching effect in textureless areas, the number of parameters and calculations brought by large kernel convolution is reduced, and the generalization ability of the model is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048568A_ABST
    Figure CN119048568A_ABST
Patent Text Reader

Abstract

The invention provides a three-dimensional matching system based on wavelet transform and light-weight large kernel convolution. The three-dimensional matching system comprises a data acquisition unit and a three-dimensional matching unit based on wavelet transform and light-weight large kernel convolution, through a method of acquiring a scene image to be detected, performing wavelet processing, extracting context information, extracting features by lightweight large kernel convolution, and constructing a cost body and a combined coding body to obtain a final parallax, the matching precision in the stereo matching field and the matching and generalization problem of an image sharp region are improved. The multi-frequency image information can be extracted through wavelet transformation, the receptive field can be increased, long-distance dependence can be captured and multi-frequency global feature information can be extracted through light-weight large kernel convolution so as to improve the stereo matching precision, meanwhile, a large amount of parameter calculation and parameters caused by large kernel convolution can be reduced through light weight, and the matching precision is improved. The extracted high-frequency information can improve the matching problem of sharp regions; experimental results show that the method provided by the invention can improve the precision and generalization of stereo matching.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a stereo matching system and method based on wavelet transform and lightweight large kernel convolution. Background Art

[0002] In recent years, convolutional neural networks have been successfully applied to tasks such as target detection and image classification due to their powerful feature representation and function fitting capabilities. Stereo matching algorithms based on deep learning are mainly divided into end-to-end stereo matching algorithms and non-end-to-end stereo matching algorithms. Early stereo matching algorithms were mainly implemented in a non-end-to-end manner, and the matching cost of the stereo matching stage was calculated by obtaining image features. Its results on the KITTI benchmark were better than those of traditional algorithms. The performance of stereo matching was improved by applying convolutional neural networks to some steps of stereo matching. However, the above algorithms still require some post-processing steps. Generally speaking, non-end-to-end binocular stereo matching algorithms include both deep learning processing modules and traditional algorithm processing modules. Compared with traditional algorithms, the performance of non-end-to-end stereo matching algorithms has been improved, but since they also include some manually defined functions, it is difficult to uniformly optimize the entire algorithm process. End-to-end stereo matching algorithms have strong adaptive adjustment capabilities because their algorithm parameters can be learned, and the resulting disparity map has high accuracy. People have eliminated the post-processing steps of stereo matching by constructing an end-to-end stereo matching algorithm, and used 3D convolution to optimize the matching cost, further improving the accuracy of the stereo matching algorithm. Later, some people used a spatial pyramid structure to extract contextual information in the feature extraction stage, and used a cascaded hourglass structure to optimize the cost cube. In the past two years, people have used iterative update networks to refine disparity, which has improved the accuracy to a certain extent. However, even if the above algorithm improves matching accuracy by enhancing scene recognition capabilities, it is still difficult to achieve matching consistency for some pathological areas that lack prior information, such as textureless areas, and the accuracy and generalization need to be improved. Summary of the invention

[0003] The purpose of the present invention is to solve the above-mentioned technical shortcomings and deficiencies, and to provide a stereo matching system and method based on wavelet transform and lightweight large kernel convolution. The system and method are simple in structure and easy to implement. The method has high scene recognition accuracy and has certain practical value.

[0004] The technical solution of the present invention is as follows: a stereo matching system based on wavelet transform and light-weight large-kernel convolution, which is characterized in that it comprises a data acquisition unit and a stereo matching unit based on wavelet transform and light-weight large-kernel convolution; wherein the stereo matching unit based on wavelet transform and light-weight large-kernel convolution is composed of a wavelet transform processing module, a context extraction module, a light-weight large-kernel convolution feature extraction module, a cost body construction module, a combined coding body construction module and an iterative update module; the data acquisition unit is used to collect left and right color images of a scene to be tested and a real disparity map of the left color image, and its output end is respectively connected to the context extraction module and the wavelet transform processing module The output end of the context extraction module is connected to the input end of the iterative update module; the output end of the wavelet transform processing module is connected to the input end of the lightweight large-kernel convolution feature extraction module; the output end of the lightweight large-kernel convolution feature extraction module is respectively connected to the input ends of the cost body construction module and the combined coding body construction module; the output end of the cost body construction module is respectively connected to the combined coding body construction module and the iterative update module input end; the output end of the combined coding body construction is connected to the input end of the iterative update module; the output end of the iterative update module outputs the final disparity map, and the loss is calculated using the final disparity map and the true disparity map.

[0005] The cost volume construction module, the combined encoding volume construction module and the iterative update module adopt the IGEV (Iterative Geometry Encoding Volume) network structure.

[0006] A stereo matching method based on wavelet transform and light-weight large-kernel convolution, characterized in that it comprises the following steps:

[0007] (1) The data acquisition unit acquires the left and right color images of the scene to be tested and the true disparity map of the left image, and inputs the obtained left and right color images into the wavelet transform processing module, and inputs the obtained left color image into the context extraction module;

[0008] (2) The wavelet transform processing module performs wavelet function processing on the left and right color images to obtain an image set of 12 channels on the left and right sides;

[0009] In step (2), the wavelet transform processing module performs wavelet function processing on the left and right color images, specifically: Figure 2As shown, the left and right color images received at the input end of the wavelet transform processing module are divided into three channels, namely R (Red), G (Green), and B (Blue), and the three channels respectively perform wavelet function dwt2 processing on the left and right color images, and each channel obtains image information of four frequencies, namely LL, LH, HL, and HH, wherein LL is the approximate image information of the low-frequency part, and LH, HL, and HH are the detailed image information of the high-frequency part; the image information of the four frequencies of LL, LH, HL, and HH generated by the R channel of the left color image, the image information of the four frequencies of LL, LH, HL, and HH generated by the G channel, and the image information of the four frequencies of LL, LH, HL, and HH generated by the B channel are respectively processed. The image information of the four frequencies of LL, LH, HL, and HH generated by the R channel of the right color image, the image information of the four frequencies of LL, LH, HL, and HH generated by the G channel, and the image information of the four frequencies of LL, LH, HL, and HH generated by the B channel, a total of 12 multi-frequency image information, are spliced ​​using the Concat function according to the channels of the image information to obtain a left image set of 12 channels, as shown in formula (1):

[0010]

[0011] Where x is the RGB input image, i is the channel corresponding to the RGB image, dwt2 is the wavelet function, LL, LH, HL, HH are the image information of four frequencies, Concat is the concatenation operation, and p is the image set;

[0012] The resolution of the four frequency images obtained by each of the three channels R, G, and B is H / 2x W / 2, and the number of channels is 1, where H and W are the height and width of the left and right color images received by the input end of the wavelet transform processing module respectively.

[0013] (3) Use the context extraction module to extract context information from the left color image and obtain context information maps with scales of H / 4x W / 4x64, H / 4x H / 4x 128, H / 8x W / 8x 128, and H / 16x W / 16x 128;

[0014] The context extraction module in step (3) is composed of 1 convolution layer, 1 normalization layer, 3 residual layers and 2 downsampling layers; the 3 residual layers I, II and III are each composed of 2 residual blocks I, and the 3 residual layers have a total of 6 residual blocks I; the downsampling layers I and II are each composed of 1 residual block I and 1 residual block II; the residual block I is composed of 2 convolutions with a kernel size of 3x3 and a step size of 1 and two ReLU functions, and the residual block II is composed of 1 kernel size It consists of a 3x3 convolution with a step size of 2, a convolution with a kernel size of 3x3 and a step size of 1, and two ReLU functions; the convolution layer of the context extraction module is a convolution layer with a kernel size of 7x7, a step size of 2, 3 input channels and 64 output channels; the three residual layers are residual layer I, residual layer II and residual layer III, whose output channels are 64, 96 and 128 respectively; the two downsampling layers are downsampling layer I and downsampling layer II, whose output channels are both 128.

[0015] The process of extracting context information in step (3) is specifically composed of the following steps:

[0016] (3-1) Using the context extraction module, the left color image obtained in step (1) is subjected to a convolution layer and then a normalization layer to obtain a context information graph I of scale;

[0017] (3-2) The context information graph I obtained in step (3-1) is passed through residual layer I, residual layer II and residual layer III to obtain context information graph II;

[0018] (3-3) The context information graph II obtained in step (3-2) is subjected to downsampling layer I to obtain context information graph III;

[0019] (3-4) The context information graph III obtained in step (3-3) is passed through a downsampling layer II to obtain a context information graph IV.

[0020] The scales of the context information graph I, context information graph II, context information graph III and context information graph IV obtained in step (3) are H / 4x W / 4x 64, H / 4x H / 4x 128, H / 8x W / 8x 128 and H / 16x W / 16x128 respectively.

[0021] (4) Use a lightweight large-kernel convolution module to extract features from the image sets of 12 channels on the left and right obtained in step (2) to obtain a multi-scale feature map Where l represents the feature map of the left image, and r represents the feature map of the right image;

[0022] The step (4) specifically refers to: the light-weight large-core convolution module performs four stages of image data processing on the left and right 12-channel image sets obtained in step (2), and the number of light-weight large-core basic modules used in the four stages is N i 3, 3, 5, 2, the number of output channels is C i They are 24, 32, 96, and 160, i∈{1,2,3,4}; the scale of the feature map obtained by the first stage of the 12-channel image set is H / 4x W / 4x C1, the scale of the feature map obtained in the second stage is H / 8xW / 8x C2, the scale of the feature map obtained in the third stage is H / 16x W / 16x C3, and the scale of the feature map obtained in the fourth stage is H / 32x W / 32x C4, that is, the multi-scale feature map is finally obtained. Where l represents the feature map of the left image, and r represents the feature map of the right image.

[0023] The four-stage image data processing process specifically consists of the following steps:

[0024] (4-1) Each channel image of the left and right 12-channel image set obtained in step (2) is divided into N patches of size 7x7, where the value of N is H / 14x W / 14. These patches are downsampled using a convolution with a kernel size of 3 and a stride of 2, and then normalized as shown in formula (2) to obtain the output x:

[0025] x=Norm(OverlapPatch(p))(2)

[0026] Wherein, Norm is the normalization process, OverlapPatch is the image segmentation and downsampling process, and p is the image set described in step (2);

[0027] (4-2) The output x obtained in step (4-1) passes through N lightweight large core basic modules, and the lightweight large core basic block will be 21x The large kernel convolution of 21 is decomposed into 4 convolutions, namely: 1x7 convolution, 7x1 convolution, 3x3 convolution and dilated convolution with a kernel size of 7 and a dilation rate of 3; the output x obtained in step (4-1) is normalized and then divided into 4 groups by channel, and 4 convolutions are performed respectively, namely: the first group is processed by 1x7 convolution, the second group is processed by 7x1 convolution, the third group is processed by 3x3 convolution, and the fourth group is processed by dilated convolution with a kernel size of 7 and a dilation rate of 3, and then the 4 groups of data after convolution are spliced ​​by channel, and the spliced ​​data are element-added with the output x obtained in step (4-1) to obtain data y; the data y is normalized and subjected to an MLP (Multilayer Perceptron) module, which is a well-known technology, and then element-added with y to obtain output z;

[0028] The steps of the lightweight large core basic module of step (4-2) include the following:

[0029] (4-2-1) The output X obtained in step (4-1) is first standardized, as shown in formula (3):

[0030] x'=Norm(x)(3)

[0031] (4-2-2) The output obtained in step (4-2-1) is divided into 4 groups by channel, and the 4 groups are processed with 4 convolutions of 1x 7, 7x 1, 3x 3 and a dilated convolution with a kernel size of 7 and a dilation rate of 3, to obtain the output x' h ,x' w ,x' hw ,x' dl , as shown in formula (4):

[0032]

[0033] Where Split is an equal division operation by channel; h 、x w 、x hw 、x dl They are 4 groups of data divided equally by channels; DWConv 7x1 7x 1 depth convolution; DWConv 1x7 1x 7 depth convolution; DWConv 3x3 3x 3 depth convolution; DLConv k=7,d=3 is a dilated convolution with a kernel size of 7 and a dilation rate of 3; x' h ,x' w ,x'hw ,x' dl These are 4 sets of data after 4 different convolution processes;

[0034] (4-2-3) The four sets of data obtained after four different convolution processes in step (4-2-2) are merged, that is, they are spliced ​​by channel, and the element addition operation is performed with the output x in step (4-1) as shown in formula (5) to obtain the output y:

[0035] y=Concat(x' h ,x' w ,x' hw ,x' dl )+x(5)

[0036] Concat is a concatenation operation performed by channel;

[0037] (4-2-4) The output y in step (4-2-3) is subjected to normalization and MLP module to obtain the output y', as shown in formula (6);

[0038] y'=MLP(Norm(y))(6)

[0039] (4-2-5) Perform element-wise addition operation on the output y' in step (4-2-4) and the output y in step (4-2-3) as shown in formula (7) to obtain the output z.

[0040] z=y'+y(7)

[0041] (4-3) The output z in step (4-2) is normalized to obtain feature map information, as shown in formula (8).

[0042] fmap = Norm(z)(8)

[0043] (5) The multi-scale feature map obtained in step (4) Input the cost volume construction module and the combined encoding volume construction module to obtain the cost volume and the combined encoding volume respectively; input the multi-scale context information graph obtained in step (3) together with the cost volume and the combined encoding volume into the iterative update module to obtain the final disparity; wherein the cost volume construction module, the combined encoding volume construction module and the iterative update module adopt the IGEV (Iterative Geometry Encoding Volume, iterative geometry encoding volume) network structure, which is a well-known technology.

[0044] The advantages of the present invention are as follows: The contribution of the wavelet transform and lightweight large-kernel convolution method to the stereo matching effect lies in that the method can capture long-distance dependence, increase the receptive field, and extract multi-frequency feature information. Low-frequency information can maintain the matching effect of textureless areas, and high-frequency information can improve the effect of sharp areas of the image. At the same time, a large number of parameters brought by large-kernel convolution are reduced, and the accuracy, computational complexity and generalization are improved:

[0045] 1. The present invention fuses information of different frequencies through wavelet transform, thereby retaining the low-frequency information and high-frequency details of the image.

[0046] 2. Through lightweight large-kernel convolution feature extraction, long-distance dependencies are captured, the receptive field is increased, and global multi-frequency matching information is obtained while reducing the large number of parameters brought by large-kernel convolution.

[0047] 3. The present invention can speed up the reasoning speed while reducing the number of parameters, retain the detailed contour information of the image, can match the image well, improve the accuracy of stereo matching and has good generalization.

[0048] 4. Based on the iterative update stereo matching model, multi-frequency image information is obtained through wavelet transform, and multi-frequency feature information is extracted through lightweight large-kernel convolution, capturing long-distance dependencies and increasing the receptive field to enrich matching information. This method solves the problem of some pathological areas. The average matching error on the sceneflow dataset is 0.435, with a lower error rate. It has good generalization performance on the Eth3d and Middlebury datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a structural schematic diagram of a stereo matching system based on wavelet transform and lightweight large kernel convolution of the present invention.

[0050] Figure 2 It is a schematic diagram of the principle structure of a wavelet transform processing module in a stereo matching system based on wavelet transform and lightweight large kernel convolution of the present invention.

[0051] Figure 3 It is a schematic diagram of the principle structure of a context extraction module in a stereo matching system based on wavelet transform and lightweight large kernel convolution of the present invention, as well as the structure of residual block I and residual block II.

[0052] Figure 4 It is a schematic diagram of the principle structure of a lightweight large-kernel convolution feature extraction module in a stereo matching system based on wavelet transform and lightweight large-kernel convolution of the present invention.

[0053] Figure 5It is a schematic diagram of the principle structure of a lightweight large-kernel basic module in a stereo matching system based on wavelet transform and lightweight large-kernel convolution of the present invention.

[0054] Figure 6 Schematic diagram of a generalization comparison test example of the present invention and the IGEV method on the Middlebury dataset (wherein, Figure 6 -a is the left color image to be tested, Figure 6 -b is the disparity map generated by the IGEV method, Figure 6 -c is the disparity map generated by this method, Figure 6 -d is the actual disparity value of the left image). DETAILED DESCRIPTION

[0055] The present invention is described in detail below with reference to the accompanying drawings.

[0056] Embodiment: A stereo matching system based on wavelet transform and lightweight large kernel convolution, such as Figure 1 As shown, it is characterized in that it includes a data acquisition unit, a stereo matching unit based on wavelet transform and light large kernel convolution; wherein the stereo matching unit based on wavelet transform and light large kernel convolution is composed of a wavelet transform processing module, a context extraction module, a light large kernel convolution feature extraction module, a cost volume construction module, a combined coding volume construction module and an iterative update module; the data acquisition unit is used to collect the left and right color images of the scene to be tested and the corresponding real disparity map, and its output end is respectively connected to the input end of the context extraction module and the wavelet transform processing module; the output end of the context extraction module is connected to the input end of the iterative update module; the output end of the wavelet transform processing module is connected to the input end of the light large kernel convolution feature extraction module; the output end of the light large kernel convolution feature extraction module is respectively connected to the input end of the cost volume construction module and the combined coding volume construction module; the output end of the cost volume construction module is respectively connected to the input end of the combined coding volume construction module and the iterative update module; the output end of the combined coding volume construction is connected to the input end of the iterative update module; the output end of the iterative update module outputs the final disparity, and the loss is calculated using the final disparity map and the real disparity map.

[0057] In the embodiment, the data acquisition unit acquires the left and right color images of the scene to be tested and the real disparity map of the left image, and inputs the obtained left and right color images into the wavelet transform processing module, and inputs the obtained left color image into the context extraction module; the context extraction module receives the left color image of the data acquisition unit and outputs multi-scale context features; the lightweight large-kernel convolution feature extraction module receives the left and right color images of the data acquisition unit, and outputs multi-scale feature maps of the left image and the right image respectively; the cost body construction module and the combined encoding body construction module receive the left and right multi-scale feature maps output by the lightweight large-kernel convolution feature extraction module; the iterative update module receives the multi-scale context features output by the context extraction module, the initial disparity output by the cost body construction module, and the combined encoding body output by the combined encoding body construction, and after iteratively updating the initial disparity, outputs the final disparity map. In the training stage, the final disparity map and the real disparity map calculate the loss to optimize the model, and in the reasoning stage, the final disparity map is output.

[0058] The cost body construction module, the combined encoding body construction module and the iterative update module adopt the IGEV network structure.

[0059] In an embodiment of the present solution, the data acquisition unit processes Sceneflow, Eth3d and Middlebury datasets; the Sceneflow dataset is used for pre-training of the model, and the dataset contains 35,454 training image pairs and 4,370 test image pairs, and it provides real disparity maps for model training; Middlebury is an indoor dataset, which provides 15 training pairs and 15 test pairs, some of which are under inconsistent lighting or color conditions, and all images have three different resolutions; Eth3d is a grayscale dataset with 27 training pairs and 20 test pairs, and the Middlebury and Eth3d datasets are used to evaluate the cross-domain generalization performance of the present method.

[0060] A stereo matching method based on wavelet transform and light-weight large-kernel convolution, characterized in that it comprises the following steps:

[0061] (1) The data acquisition unit acquires the left and right color images of the scene to be tested and the true disparity map of the left image, and inputs the obtained left and right color images into the wavelet transform processing module, and inputs the obtained left color image into the context extraction module;

[0062] (2) The wavelet transform processing module performs wavelet function processing on the left and right color images to obtain an image set of 12 channels on the left and right sides;

[0063] The wavelet transform processing module in step (2) performs wavelet function processing on the left and right color images, specifically: Figure 2 As shown, the left and right color images received at the input end of the wavelet transform processing module are divided into three channels of R, G, and B, respectively. The three channels perform wavelet function dwt2 processing on the left and right color images respectively, and each channel obtains image information of four frequencies, namely LL, LH, HL, and HH, wherein LL is the approximate image information of the low-frequency part, and LH, HL, and HH are the detailed image information of the high-frequency part; the image information of the four frequencies of LL, LH, HL, and HH generated by the R channel of the left color image, the image information of the four frequencies of LL, LH, HL, and HH generated by the G channel, and the image information of LL, LH, HL, and HH generated by the B channel are respectively processed. The image information of four frequencies LL, LH, HL, and HH generated by the R channel of the right color image, the image information of four frequencies LL, LH, HL, and HH generated by the G channel, and the image information of four frequencies LL, LH, HL, and HH generated by the B channel, a total of 12 multi-frequency image information, are spliced ​​using the Concat function according to the channels of the image information to obtain a 12-channel right image set, as shown in formula (1):

[0064]

[0065] Where x is the RGB input image, i is the channel corresponding to the RGB image, dwt2 is the wavelet function, LL, LH, HL, HH are the image information of four frequencies, Concat is the concatenation operation, and p is the image set;

[0066] The resolution of the four frequency images obtained by each of the three channels R, G, and B is H / 2x W / 2, and the number of channels is 1, where H and W are the height and width of the left and right color images received by the input end of the wavelet transform processing module respectively.

[0067] (3) Use the context extraction module to extract context information from the left color image and obtain context information maps with scales of H / 4x W / 4x64, H / 4x H / 4x 128, H / 8x W / 8x 128, and H / 16x W / 16x 128;

[0068] The context extraction module in step (3) specifically refers to: Figure 3As shown in the figure, it consists of 1 convolution layer, 1 normalization layer, 3 residual layers and 2 downsampling layers; the 3 residual layers I, II and III are composed of 2 residual blocks I, and the 3 residual layers have a total of 6 residual blocks I; the downsampling layers I and II are composed of 1 residual block I and 1 residual block II; the residual block I is composed of 2 convolutions with a kernel size of 3x3 and a step size of 1 and two ReLU functions, and the residual block II is composed of 1 convolution with a kernel size of 3x3 and a step size of 2, a convolution with a kernel size of 3x3 and a step size of 1, and two ReLU functions; the convolution layer of the context extraction module is a convolution layer with a kernel size of 7x7, a step size of 2, 3 input channels and 64 output channels; the three residual layers are residual layer I, residual layer II and residual layer III, whose output channels are 64, 96 and 128 respectively; the two downsampling layers are downsampling layer I and downsampling layer II, whose output channels are both 128.

[0069] The process of extracting context information in step (3) is specifically composed of the following steps:

[0070] (3-1) Using the context extraction module, the left color image obtained in step (1) is subjected to a convolution layer and then a normalization layer to obtain a context information graph I of scale;

[0071] (3-2) The context information graph I obtained in step (3-1) is passed through residual layer I, residual layer II and residual layer III to obtain context information graph II;

[0072] (3-3) The context information graph II obtained in step (3-2) is subjected to downsampling layer I to obtain context information graph III;

[0073] (3-4) The context information graph III obtained in step (3-3) is passed through a downsampling layer II to obtain a context information graph IV.

[0074] The scales of the context information graph I, context information graph II, context information graph III and context information graph IV obtained in step (3) are H / 4x W / 4x 64, H / 4x H / 4x 128, H / 8x W / 8x 128 and H / 16x W / 16x128 respectively.

[0075] (4) Use a lightweight large-kernel convolution module to extract features from the image sets of 12 channels on the left and right obtained in step (2) to obtain a multi-scale feature map Where l represents the feature map of the left image, and r represents the feature map of the right image;

[0076] The step (4) specifically refers to: Figure 4As shown in the figure, the light-weight large-core convolution module performs four stages of image data processing on the left and right 12-channel image sets obtained in step (2). The number of light-weight large-core basic modules used in the four stages is N. i 3, 3, 5, 2, the number of output channels is C i They are 24, 32, 96, and 160, i∈{1,2,3,4}; the scale of the feature map obtained by the first stage of the 12-channel image set is H / 4x W / 4x C1, the scale of the feature map obtained in the second stage is H / 8xW / 8x C2, the scale of the feature map obtained in the third stage is H / 16x W / 16x C3, and the scale of the feature map obtained in the fourth stage is H / 32x W / 32x C4, that is, the multi-scale feature map is finally obtained. Where l represents the feature map of the left image, and r represents the feature map of the right image.

[0077] The four-stage image data processing process specifically consists of the following steps:

[0078] (4-1) Each channel image of the left and right 12-channel image set obtained in step (2) is divided into N patches of size 7x7, where the value of N is H / 14x W / 14. These patches are downsampled using a convolution with a kernel size of 3 and a stride of 2, and then normalized as shown in formula (2) to obtain the output x:

[0079] x=Norm(OverlapPatch(p))(2)

[0080] Wherein, Norm is the normalization process, OverlapPatch is the image segmentation and downsampling process, and p is the image set described in step (2);

[0081] (4-2) The output x obtained in step (4-1) is passed through N lightweight large core basic modules, and the lightweight large core basic module is to decompose the large core convolution of size 21x 21 into 4 convolutions, namely: 1x 7, 7x 1, 3x 3 convolutions and a dilated convolution with a kernel size of 7 and a dilation rate of 3; the output x obtained in step (4-1) is normalized and then divided into 4 groups by channel, the first group is processed by 1x7 convolution, the second group is processed by 7x1 convolution, the third group is processed by 3x3 convolution, and the fourth group is processed by a dilated convolution with a kernel size of 7 and a dilation rate of 3, and then the 4 groups of data after convolution are spliced ​​by channel, and the spliced ​​data are element-added with the output x obtained in step (4-1) to obtain data y; the data y is normalized and subjected to an MLP (Multilayer Perceptron) module, which is a well-known technology, and then element-added with y to obtain output z;

[0082] The steps of the lightweight large core basic module of step (4-2) include the following: Figure 5 As shown:

[0083] (4-2-1) The output X obtained in step (4-1) is first standardized, as shown in formula (3):

[0084] x'=Norm(x)(3)

[0085] (4-2-2) The output obtained in step (4-2-1) is divided into 4 groups by channel, and the 4 groups are processed with 4 convolutions of 1x 7, 7x 1, 3x 3 and a dilated convolution with a kernel size of 7 and a dilation rate of 3, to obtain the output x' h ,x' w ,x' hw ,x' dl , as shown in formula (4):

[0086]

[0087] Where Split is an equal division operation by channel; h 、x w 、x hw 、x dl They are 4 groups of data divided equally by channels; DWConv 7x1 7x 1 depth convolution; DWConv 1x7 1x 7 depth convolution; DWConv 3x3 3x 3 depth convolution; DLConv k=7,d=3 is a dilated convolution with a kernel size of 7 and a dilation rate of 3; x' h ,x' w ,x' hw ,x' dl These are 4 sets of data after 4 different convolution processes;

[0088] (4-2-3) The four sets of data obtained after four different convolution processes in step (4-2-2) are merged, that is, they are spliced ​​by channel, and the element addition operation is performed with the output x in step (4-1) as shown in formula (5) to obtain the output y:

[0089] y=Concat(x' h ,x' w ,x' hw ,x' dl )+x(5)

[0090] Concat is a concatenation operation performed by channel;

[0091] (4-2-4) The output y in step (4-2-3) is subjected to normalization and MLP module to obtain the output y', as shown in formula (6);

[0092] y'=MLP(Norm(y))(6)

[0093] (4-2-5) Perform element-wise addition operation on the output y' in step (4-2-4) and the output y in step (4-2-3) as shown in formula (7) to obtain the output z.

[0094] z=y'+y(7)

[0095] (4-3) The output z in step (4-2) is normalized to obtain feature map information, as shown in formula (8).

[0096] fmap = Norm(z)(8)

[0097] (5) The multi-scale feature map obtained in step (4) Input the cost body construction module and the combined coding body construction module to obtain the cost body and the combined coding body respectively; input the multi-scale context information graph obtained in step (3) together with the cost body and the combined coding body into the iterative update module to obtain the final disparity; wherein the cost body construction module, the combined coding body construction module and the iterative update module adopt the IGEV network structure, which is a well-known technology.

[0098] (6) The contribution of the wavelet transform and lightweight large kernel convolution method to the stereo matching effect lies in that the method can capture long-distance dependence, increase the receptive field, and extract multi-frequency feature information. Low-frequency information can maintain the matching effect of textureless areas, and high-frequency information can improve the effect of sharp areas of the image. At the same time, it reduces the large number of parameters brought by large kernel convolution, and improves the accuracy, calculation amount and generalization. As shown in Table 1, the number of parameters of this method is reduced to 12.38M, which is less than that of IGEV. As shown in Table 2, the EPE of this method on the Sceneflow dataset reaches 0.435. As shown in Table 3, this method has good generalization on the Eth3d and Middlebury datasets; Figure 6 The figure shows a generalization comparison test example between this method and the IGEV method on the Middlebury dataset.

[0099] Table 1 Parameter comparison between this method and IGEV

[0100]

[0101] Table 2 Comparison of matching effects of this method and other methods on sceneflow dataset

[0102]

[0103] Table 3 Generalization comparison of this method and other methods

[0104]

[0105] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A stereo matching system based on wavelet transform and lightweight large kernel convolution, characterized in that It includes a data acquisition unit, a stereo matching unit based on wavelet transform and light large kernel convolution; wherein the stereo matching unit based on wavelet transform and light large kernel convolution consists of a wavelet transform processing module, a context extraction module, a light large kernel convolution feature extraction module, a cost body construction module, a combined coding body construction module and an iterative update module; the data acquisition unit is used to collect the left and right color images of the scene to be tested and the real disparity map of the left color image, and its output end is respectively connected to the input end of the context extraction module and the input end of the wavelet transform processing module; the output end of the context extraction module is connected to the input end of the iterative update module; the output end of the wavelet transform processing module is connected to the input end of the light large kernel convolution feature extraction module; the output end of the light large kernel convolution feature extraction module is respectively connected to the input end of the cost body construction module and the combined coding body construction module; the output end of the cost body construction module is respectively connected to the input end of the combined coding body construction module and the iterative update module; the output end of the combined coding body construction is connected to the input end of the iterative update module; the output end of the iterative update module outputs the final disparity map, and the loss is calculated using the final disparity map and the real disparity map.

2. According to claim 1, a stereo matching system based on wavelet transform and lightweight large kernel convolution is characterized in that The cost body construction module, the combined encoding body construction module and the iterative update module adopt the IGEV network structure.

3. A stereo matching method based on wavelet transform and lightweight large kernel convolution, characterized in that It includes the following steps: (1) The data acquisition unit acquires the left and right color images of the scene to be tested and the true disparity map of the left image, and inputs the obtained left and right color images into the wavelet transform processing module, and inputs the obtained left color image into the context extraction module; (2) The wavelet transform processing module performs wavelet function processing on the left and right color images to obtain an image set of 12 channels on the left and right sides; (3) Use the context extraction module to extract context information from the left color image and obtain context information maps with scales of H / 4x W / 4x 64, H / 4x H / 4x 128, H / 8x W / 8x 128, and H / 16x W / 16x 128; (4) Use a lightweight large-kernel convolution module to extract features from the image sets of 12 channels on the left and right obtained in step (2) to obtain a multi-scale feature map Where l represents the feature map of the left image, and r represents the feature map of the right image; (5) The multi-scale feature map obtained in step (4) Input the cost body construction module and the combined encoding body construction module to obtain the cost body and the combined encoding body respectively; input the multi-scale context information graph obtained in step (3) together with the cost body and the combined encoding body into the iterative update module to obtain the final disparity; The cost body module, the combined encoding body module and the iterative update module adopt the IGEV network structure.

4. According to claim 3, a stereo matching method based on wavelet transform and light-weight large-kernel convolution is characterized in that Specifically, the wavelet transform processing module performs wavelet function processing on the left and right color images in step (2) as follows: the left and right color images received at the input end of the wavelet transform processing module are divided into three channels, R, G, and B, respectively, and the three channels respectively perform wavelet function dwt2 processing on the left and right color images, and each channel obtains image information of four frequencies, namely LL, LH, HL, and HH, wherein LL is the approximate image information of the low-frequency part, and LH, HL, and HH are the detailed image information of the high-frequency part; the image information of the four frequencies of LL, LH, HL, and HH generated by the R channel of the left color image and the image information of the four frequencies of LL, LH, HL, and HH generated by the G channel are processed. The image information of LL, LH, HL, HH frequencies generated by the R channel of the right color image, the image information of LL, LH, HL, HH frequencies generated by the G channel, and the image information of LL, LH, HL, HH frequencies generated by the B channel, a total of 12 multi-frequency image information, are spliced ​​using the Concat function according to the channels of the image information to obtain a left image set of 12 channels, and the image information of LL, LH, HL, HH frequencies generated by the R channel of the right color image, the image information of LL, LH, HL, HH frequencies generated by the G channel, and the image information of LL, LH, HL, HH frequencies generated by the B channel, a total of 12 multi-frequency image information, are spliced ​​using the Concat function according to the channels of the image information to obtain a right image set of 12 channels, as shown in formula (1): Among them, x is the RGB input image, i is the channel corresponding to the RGB image, dwt2 is the wavelet function, LL, LH, HL, HH are the image information of four frequencies, Concat is the splicing operation, and p is the image set.

5. According to claim 4, a stereo matching method based on wavelet transform and light-weight large-kernel convolution is characterized in that The resolution of the four frequency images obtained by each of the three channels R, G, and B is H / 2x W / 2, and the number of channels is 1, where H and W are the height and width of the left and right color images received by the input end of the wavelet transform processing module respectively.

6. According to claim 3, a stereo matching method based on wavelet transform and light-weight large-kernel convolution is characterized in that The context extraction module in step (3) is composed of 1 convolution layer, 1 normalization layer, 3 residual layers and 2 downsampling layers; the 3 residual layers I, II and III are each composed of 2 residual blocks I, and the 3 residual layers have a total of 6 residual blocks I; the downsampling layers I and II are each composed of 1 residual block I and 1 residual block II; the residual block I is composed of 2 convolutions with a kernel size of 3x3 and a step size of 1 and two ReLU functions, and the residual block II is composed of 1 kernel size It consists of a 3x3 convolution with a step size of 2, a convolution with a kernel size of 3x3 and a step size of 1, and two ReLU functions; the convolution layer of the context extraction module is a convolution layer with a kernel size of 7x7, a step size of 2, 3 input channels and 64 output channels; the three residual layers are residual layer I, residual layer II and residual layer III, whose output channels are 64, 96 and 128 respectively; the two downsampling layers are downsampling layer I and downsampling layer II, whose output channels are both 128.

7. According to claim 3, a stereo matching method based on wavelet transform and light-weight large-kernel convolution is characterized in that The process of extracting context information in step (3) is specifically composed of the following steps: (3-1) Using the context extraction module, the left color image obtained in step (1) is subjected to a convolution layer and then a normalization layer to obtain a context information graph I of scale; (3-2) The context information graph I obtained in step (3-1) is passed through residual layer I, residual layer II and residual layer III to obtain context information graph II; (3-3) The context information graph II obtained in step (3-2) is subjected to downsampling layer I to obtain context information graph III; (3-4) The context information graph III obtained in step (3-3) is passed through a downsampling layer II to obtain a context information graph IV.

8. A stereo matching method based on wavelet transform and light-weight large-kernel convolution according to claim 7, characterized in that The scales of the context information graph I, context information graph II, context information graph III and context information graph IV obtained in step (3) are H / 4x W / 4x 64, H / 4x H / 4x 128, H / 8x W / 8x 128 and H / 16x W / 16x 128 respectively.

9. The stereo matching method based on wavelet transform and light-weight large-kernel convolution according to claim 3, characterized in that The step (4) specifically refers to: the light-weight large-core convolution module performs four stages of image data processing on the left and right 12-channel image sets obtained in step (2), and the number of light-weight large-core basic modules used in the four stages is N i 3, 3, 5, 2, the number of output channels is C i They are 24, 32, 96, and 160, i∈{1,2,3,4}; the scale of the feature map obtained by the first stage of the 12-channel image set is H / 4x W / 4x C1, the scale of the feature map obtained in the second stage is H / 8xW / 8x C2, the scale of the feature map obtained in the third stage is H / 16x W / 16x C3, and the scale of the feature map obtained in the fourth stage is H / 32xW / 32x C4, that is, the multi-scale feature map is finally obtained. Where l represents the feature map of the left image, and r represents the feature map of the right image; The four-stage image data processing process specifically consists of the following steps: (4-1) Each channel image of the left and right 12-channel image set obtained in step (2) is divided into N patches of size 7x7, where the value of N is H / 14x W / 14. These patches are downsampled using a convolution with a kernel size of 3 and a stride of 2, and then normalized as shown in formula (2) to obtain the output x: x=Norm(OverlapPatch(p))(2) Wherein, Norm is the normalization process, OverlapPatch is the image segmentation and downsampling process, and p is the image set described in step (2); (4-2) The output x obtained in step (4-1) is passed through N lightweight large core basic modules, and the lightweight large core basic module is to decompose the large core convolution of size 21x 21 into 4 convolutions, namely: 1x7 convolution, 7x1 convolution, 3x3 convolution and dilated convolution with a kernel size of 7 and a dilation rate of 3; the output x obtained in step (4-1) is normalized and then divided into 4 groups by channel, and 4 convolutions are performed respectively, namely: the first group is processed by 1x7 convolution, the second group is processed by 7x1 convolution, the third group is processed by 3x3 convolution, and the fourth group is processed by dilated convolution with a kernel size of 7 and a dilation rate of 3, and then the 4 groups of data after convolution are spliced ​​by channel, and the spliced ​​data are element-added with the output x obtained in step (4-1) to obtain data y; the data y is normalized and subjected to the MLP module, and then element-added with y to obtain the output z; (4-3) The output z in step (4-2) is normalized to obtain feature map information, as shown in formula (8). fmap = Norm(z)(8) 10. A stereo matching method based on wavelet transform and light-weight large kernel convolution according to claim 9. The method is characterized in that The steps of the lightweight large core basic module of step (4-2) include the following: (4-2-1) The output X obtained in step (4-1) is first standardized, as shown in formula (3): x'=Norm(x)(3)(4-2-2) The output obtained in step (4-2-1) is divided into 4 groups by channel, and the 4 groups are processed with 4 convolutions of 1 x 7, 7 x 1, 3 x 3 and a dilated convolution with a kernel size of 7 and a dilation rate of 3, to obtain the output x' h ,x' w ,x' hw ,x' dl , as shown in formula (4): Where Split is an equal division operation by channel; h 、x w 、x hw 、x dl They are 4 groups of data divided equally by channels; DWConv 7x1 It is a 7 x 1 depth convolution; DWConv 1x7 1 x 7 depth convolution; DWConv 3x3 3 x 3 depth convolution; DLConv k=7,d=3 is a dilated convolution with a kernel size of 7 and a dilation rate of 3; x' h ,x' w ,x' hw ,x' dl These are 4 sets of data after 4 different convolution processes; (4-2-3) The four sets of data obtained after four different convolution processes in step (4-2-2) are merged, that is, they are spliced ​​by channel, and the element addition operation is performed with the output x in step (4-1) as shown in formula (5) to obtain the output y: y=Concat(x' h ,x' w ,x' hw ,x' dl )+x(5) where Concat is a concatenation operation performed by channel; (4-2-4) The output y in step (4-2-3) is subjected to normalization and MLP module to obtain the output y', as shown in formula (6); y'=MLP(Norm(y))(6)(4-2-5) Perform element-wise addition operation on the output y' in step (4-2-4) and the output y in step (4-2-3) as shown in formula (7) to obtain the output z. z=y′+y(7).

Citation Information

Patent Citations

  • Holographic stereoscopic image matching algorithm

    CN108805914A

  • Stereo matching method and system based on iterative geometric coding body

    CN116051739A

  • Stereo matching system and stereo matching method using the same

    US20090041336A1

Cited By

  • Lightweight wavelet convolution guide wire segmentation network model and double guide wire generation method

    CN120339271A