Stereo matching method based on Mangbar cost body

Abstract

The invention discloses a stereo matching method based on a Mangbar cost volume, and the method comprises the steps: obtaining a to-be-processed left-view image and a to-be-processed right-view image, the left-view image and the right-view image being images of the same object at different view angles; inputting the left view image and the right view image into an improved stereo matching network to obtain a target prediction disparity map; the improved stereo matching network at least comprises a feature extraction module, a cost body construction module, an aggregation module and a parallax regression module which are sequentially connected in series, and the feature extraction module is further connected with the parallax regression module in series. The cost body construction module comprises an initial cost body construction module, a residual vision Mangbar layer, a 3D attention module, a normalization module, a first fusion module and a second fusion module, wherein the initial cost body construction module and the residual vision Mangbar layer are sequentially connected in series; the 3D attention module and the normalization module are respectively connected with the output of the residual vision Mangbar layer in series; the first fusion module is connected with the output of the 3D attention module and the output of the normalization module in series; according to the invention, the residual vision Mangbar layer and the 3D attention module are introduced, so that the matching precision and robustness are improved.

Description

Technical Field

[0001] The present application generally relates to the field of computer vision technology. More specifically, the present application relates to a stereo matching method based on a Mamba cost volume. Background Art

[0002] Stereo matching is a key task in computer vision technology and is often used in different fields such as autonomous driving, 3D reconstruction and augmented reality. Stereo matching is to extract depth information from two or more images with different perspectives to restore the three-dimensional structure of the scene. Its basic principle is to use parallax, that is, the displacement of the same object under different perspectives, to infer the three-dimensional structure of the scene.

[0003] At present, stereo matching methods are divided into traditional stereo matching methods and deep learning-based stereo matching methods. Traditional stereo matching methods are divided into four steps: matching cost calculation, cost aggregation, disparity calculation, and disparity refinement. However, traditional stereo matching methods usually rely on manually designed basic features, such as pixel grayscale value differences or Census transformation, when calculating matching costs. This limits its ability to extract global information and integrate multi-scale features, so it has poor robustness when dealing with occlusion, weak texture, and reflective areas, and it is difficult to obtain accurate and reliable results in practical applications. The deep learning network architecture of stereo matching can usually be divided into four parts: feature extraction, cost volume construction, cost aggregation, and disparity regression. Due to the powerful feature learning ability of convolutional neural networks, the prediction accuracy of the network can be significantly improved. At the same time, it shows great potential to solve the problem that the above-mentioned traditional stereo matching methods have poor robustness when dealing with occlusion, weak texture, and reflective areas, and it is difficult to obtain accurate and reliable results in practical applications.

[0004] As examples, three deep learning-based stereo matching networks in the prior art are given, namely PSMNet (Pyramid Stereo Matching Network), GwcNet (Group-wise Cost Volumes Network) and ACVNet (Attention-Cost Volume Network). Among them, PSMNet integrates global information into the concatenated cost volume, so that the model can capture the matching relationship between distant pixels, thereby improving the performance in complex scenes, especially when dealing with occlusion and texture-rich areas. However, the cost volume generation of PSMNet relies on the traditional concatenation method, which can effectively capture global information but may ignore the fine-grained relationship between local pixels. Therefore, in some special cases, such as areas with scarce textures, PSMNet may face certain challenges. GwcNet provides a similarity measure for the cost volume by performing dot product operations on grouped features, thereby improving the efficiency and accuracy of cost volume generation. However, this dot product operation after grouping relies more on mathematical experience to measure similarity, and may not fully capture the potential relationship in the features. In particular, for those areas where there is a subtle connection between local and global features, this method may not fully tap these potential clues, thereby affecting the performance in weak textures or complex scenes. ACVNet improves the accuracy of the model by constructing a group-related cost body based on adaptive patch matching to generate attention weights for enhancing the concatenated cost body. However, the group-related cost body uses mathematical experience to measure this similarity and cannot fully utilize the potential clues in the features, which may be the relationship between the pixel and the surrounding pixels, or the connection between pixels that are far away.

[0005] Based on the above content, we can know that although the current deep learning-based stereo matching networks have made many innovations in improving accuracy and processing complex scenes, they still have room for improvement in fully exploring the potential feature relationships in images. In particular, there is still potential in how to utilize the local relationships between pixels and how to establish stronger connections between distant pixels. If this kind of information can be fully explored, it will help improve the network's prediction ability in occluded and weakly textured areas, thereby improving the overall matching accuracy and robustness.

[0006] In view of this, there is an urgent need to provide a stereo matching method based on the Mamba cost volume in order to improve the matching accuracy and robustness of stereo matching. Summary of the invention

[0007] In order to at least solve one or more of the technical problems mentioned above, the present application proposes a stereo matching method based on a Mamba cost volume in multiple aspects.

[0008] The present application provides a stereo matching method based on a Mamba cost volume, comprising: obtaining a left-view image and a right-view image to be processed, wherein the left-view image and the right-view image are images of the same object at different viewing angles; inputting the left-view image and the right-view image into an improved stereo matching network to obtain a target predicted disparity map; wherein the improved stereo matching network at least comprises a feature extraction module, a cost volume construction module, an aggregation module and a disparity regression module connected in series in sequence, and the feature extraction module is also connected in series with the disparity regression module; the cost volume construction module comprises an initial cost volume construction module, a residual visual Mamba layer, a 3D attention module and a normalization module connected in series with the output of the residual visual Mamba layer respectively, a first fusion module connected in series with the output of the 3D attention module and the output of the normalization module, and a second fusion module connected in series with the first fusion module.

[0009] Optionally, the left-view image and the right-view image share weight parameters of the feature extraction module, and the feature extraction module is a pre-trained Efficient-B3 network; the left feature map and the right feature map output by the 4th convolutional layer, the 5th convolutional layer, the 6th convolutional layer and the 8th convolutional layer in the feature extraction module are used as inputs of the cost volume construction module connected in series with the feature extraction module, and the left feature map and the right feature map output by the 4th convolutional layer in the feature extraction module are used as inputs of the disparity regression module connected in series with the feature extraction module; the sizes of the left feature map and the right feature map output by the 4th convolutional layer, the 5th convolutional layer, the 6th convolutional layer and the 8th convolutional layer are 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the left-view image and the right-view image, respectively, and the number of channels is 24, 32, 48 and 132, respectively.

[0010] Optionally, the initial cost volume construction module performs at least the following steps in sequence: eliminating the pixel values ​​of the left i columns and the right i columns of a set of input left feature maps and right feature maps, where i is the disparity level of the left feature map and the right feature map; splicing the left feature map and the right feature map after the pixel values ​​are eliminated in a staggered superposition manner in the channel latitude to obtain an initial cost volume; and upscaling the initial cost volume so that the latitude of the upscaled initial cost volume meets the input latitude requirement of the residual visual mamba layer, wherein the input latitude requirement of the residual visual mamba layer is four-dimensional, and the number of output channels of the residual visual mamba layer is C / 4.

[0011] Optionally, the residual visual Mamba layer performs at least the following steps in sequence: layer normalization of the initial cost volume output by the initial cost volume construction module; inputting the layer-normalized initial cost volume into the Mamba model; pixel-by-pixel addition of learnable parameters to the initial cost volume output by the Mamba model; layer normalization of the initial cost volume after pixel-by-pixel addition; and inputting the layer-normalized initial cost volume into a linear layer.

[0012] Optionally, the 3D attention module performs at least the following steps in sequence: performing maximum pooling and average pooling on the initial cost volume output by the residual visual mamba layer respectively; adding the initial cost volume after maximum pooling and the initial cost volume after average pooling pixel by pixel; performing a first Sigmoid normalization on the initial cost volume after pixel by pixel addition; performing a first pixel-by-pixel multiplication on the initial cost volume output by the residual visual mamba layer and the initial cost volume after the first Sigmoid normalization; performing at least three 3D convolutions on the initial cost volume after the first pixel-by-pixel multiplication, wherein the convolution kernel sizes of the three 3D convolutions are 1x7x7, 7x1x1 and 7x7x7, respectively, acting on the space, depth and overall dimensions respectively; performing a second Sigmoid normalization on the initial cost volume after at least three 3D convolutions; performing a second pixel-by-pixel multiplication on the initial cost volume after the second Sigmoid normalization and the initial cost volume after the first pixel-by-pixel addition;

[0013] The normalization module performs at least the following steps in sequence: performing layer normalization on the initial cost body output by the residual visual mamba layer; weighting the initial cost body output by the residual visual mamba layer based on a learnable parameter; and performing a residual connection between the weighted initial cost body and the layer-normalized initial cost body.

[0014] Optionally, the first fusion module at least performs the following steps in sequence: adding the initial cost volume output by the normalization module and the initial cost volume output by the 3D attention module pixel by pixel; adding i columns of 0 to the right side of the initial cost volume after pixel by pixel addition to obtain a set of cost volumes corresponding to the left feature map and the right feature map; wherein the cost volumes output by the first fusion module are of size of , size is of and size is of , where is obtained by taking a set of feature maps output by the 5th convolutional layer in the feature extraction module as input, It is obtained by taking a set of feature maps output by the 6th convolutional layer in the feature extraction module as input. It is obtained by taking a set of feature maps output by the 8th convolutional layer in the feature extraction module as input.

[0015] Optionally, the second fusion module at least sequentially performs the following steps: according to a preset fusion expression, the output of the first fusion module and Fusion is performed to obtain an initial fusion cost volume; the initial fusion cost volume is combined with the The fusion is performed to obtain a fusion cost body; wherein the preset fusion expression is:

[0016]

[0017] in, represents 3D deconvolution, Conv represents 3D convolution, x1 and x2 represent the number of channels of the output channel, represents group regularization and Activation function, Represents a concatenation operation on the channel dimension.

[0018] Optionally, the aggregation module includes a 3D residual block and a cost aggregation module based on partial 3D convolution connected in series, wherein the cost aggregation module based on partial 3D convolution includes at least a first hourglass network based on partial 3D convolution, a second hourglass network based on partial 3D convolution, and a third hourglass network based on partial 3D convolution connected in series; wherein,

[0019] The 3D residual block at least performs the following steps in sequence: performing 3D convolution on the fused cost volume output by the cost volume construction module; performing residual connection on the fused cost volume after the 3D convolution and the fused cost volume output by the cost volume construction module;

[0020] The first hourglass network based on partial 3D convolution, the second hourglass network based on partial 3D convolution, and the third hourglass network based on partial 3D convolution all include at least a first 3D convolution, a first partial 3D convolution, a second 3D convolution, a second partial 3D convolution, a third 3D convolution, and a fourth 3D convolution connected in series, wherein the third 3D convolution is residually connected to the first partial 3D convolution, and the fourth 3D convolution is residually connected to the first 3D convolution.

[0021] Optionally, the disparity regression module at least includes a first disparity regression module, a second disparity regression module, a third disparity regression module and a fourth disparity regression module; wherein the first disparity regression module, the second disparity regression module, the third disparity regression module and the fourth disparity regression module each include an initial disparity generation module and a disparity refinement module connected in series in sequence;

[0022] The initial disparity generation module performs at least the following steps in sequence: performing at least 2 3D convolutions on the input; performing trilinear interpolation on the input after the 3D convolution; performing probability weighted regression on the input after the trilinear interpolation to obtain an initial predicted disparity map, wherein the probability of the initial predicted disparity map is calculated by a Sigmod function;

[0023] The disparity refinement module at least performs the following steps in sequence: interpolating the input left feature map and the right feature map to the original view size; calculating the difference between the interpolated left feature map and the interpolated right feature map to obtain left and right feature errors; splicing the left and right feature errors and the input initial predicted disparity map in the channel dimension to obtain a first splicing result; performing convolution operations on the first splicing result using three 2D dilated convolutions respectively, wherein the dilation rates of the 2D convolutions are 1, 2, and 4 respectively;

[0024] Each output obtained after the convolution operation is spliced ​​in the channel dimension to obtain a second splicing result; a channel attention module is applied to the second splicing result, and a residual connection is performed between the obtained result and the initial predicted disparity map to obtain the target predicted disparity map;

[0025] The 3D residual block output in the aggregation module is connected in series with the initial disparity generation module of the first disparity regression module, the first partial 3D convolution-based hourglass network output in the aggregation module is connected in series with the initial disparity generation module of the second disparity regression module, the second partial 3D convolution-based hourglass network output in the aggregation module is connected in series with the initial disparity generation module of the third disparity regression module, and the third partial 3D convolution-based hourglass network output in the aggregation module is connected in series with the initial disparity generation module of the fourth disparity regression module.

[0026] Optionally, after the step of inputting the left view image and the right view image into an improved stereo matching network to obtain a target predicted disparity map, the step further includes: calculating the target predicted disparity map and the real disparity map. loss; The loss serves as a constraint during the training of the stereo matching network.

[0027] Through a stereo matching method based on a Mamba cost body as provided above, an embodiment of the present application introduces a residual visual Mamba layer and a 3D attention module into a cost body construction module, so as to fully exploit the relationship between deep and shallow features extracted in the feature extraction process through the residual visual Mamba layer, and provide richer contextual information for the cost body in combination with local enhancement, so that the stereo matching network can more easily cope with the difficult matching problem in pathological areas, thereby achieving the purpose of improving the matching accuracy and robustness of the stereo matching network. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0029] Figure 1 A flowchart of a stereo matching method based on a Mamba cost volume in some embodiments of the present application;

[0030] Figure 2 A schematic diagram of the architecture of a stereo matching network according to some embodiments of the present application;

[0031] Figure 3 A schematic diagram of a network architecture of a cost volume construction module in a stereo matching network in some embodiments of the present application;

[0032] Figure 4 Schematic diagram of an hourglass network based on partial 3D convolution and a network architecture of partial 3D convolution in a stereo matching network of some embodiments of the present application;

[0033] Figure 5 A schematic diagram of a network architecture of a disparity refinement module in a stereo matching network in some embodiments of the present application;

[0034] Figure 6 This is a qualitative comparison result diagram of this application on the SceneFlow dataset;

[0035] Figure 7 This is a comparison chart of the disparity prediction results of this application on the KITTI2015 dataset;

[0036] Figure 8 This is a qualitative comparison result diagram of this application on the KITTI2012 test set. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0038] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present application indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0039] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this application specification and claims, unless the context clearly indicates otherwise, the singular forms of "a", "an" and "the" are intended to include plural forms. It should also be further understood that the term "and / or" used in this application specification and claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0040] As used in this specification and claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0041] The specific implementation of the present application is described in detail below with reference to the accompanying drawings.

[0042] Embodiment 1

[0043] Reference Figure 1 , Figure 1 This is a flowchart of the steps of a stereo matching method based on a Mamba cost volume in some embodiments of the present application. In some embodiments, a stereo matching method based on a Mamba cost volume provided by the present application at least includes the following steps S10 and S20.

[0044] Step S10: Acquire a left-view image and a right-view image to be processed, wherein the left-view image and the right-view image are images of the same object at different viewing angles;

[0045] It should be noted that the present application does not limit the means of obtaining the left view image and the right view image, and those skilled in the art may select different means of obtaining according to specific needs. For example, in some embodiments, the left view image and the right view image to be processed may be obtained by a stereo camera.

[0046] In some embodiments, the maximum parallax range of the left view image and the right view image is Can be set to 192.

[0047] Step S20: inputting the left view image and the right view image into an improved stereo matching network to obtain a target predicted disparity map.

[0048] In some embodiments, reference Figure 2 , Figure 2 Schematic diagram of the architecture of the stereo matching network of some embodiments of the present application. Figure 2 As shown, the improved stereo matching network at least includes a feature extraction module, a cost volume construction module, an aggregation module and a disparity regression module connected in series in sequence, wherein the feature extraction module is also connected in series with the disparity regression module.

[0049] Further, refer to Figure 3 , Figure 3 Schematic diagram of the network architecture of the cost volume construction module in the stereo matching network of some embodiments of the present application. Figure 3 As shown in FIG. 1 , the cost volume construction module includes an initial cost volume construction module, a residual visual mamba layer, a 3D attention module and a normalization module connected in series with the output of the residual visual mamba layer, a first fusion module connected in series with the output of the 3D attention module and the output of the normalization module, and a second fusion module connected in series with the first fusion module. It should be noted that the second fusion module is connected in series with the output of the 3D attention module and the output of the normalization module. Figure 3 Not shown.

[0050] In the technical solution provided in this embodiment, the residual visual mamba layer and the 3D attention module are introduced into the cost body construction module, so that the relationship between the deep and shallow features extracted in the feature extraction process is fully explored through the residual visual mamba layer, and combined with the local enhancement method, richer contextual information is provided for the cost body, so that the stereo matching network can more easily cope with the difficult matching problem of pathological areas, thereby achieving the purpose of improving the matching accuracy and robustness of the stereo matching network.

[0051] Next, the specific implementation of the improved stereo matching network is described in more detail, but these descriptions do not mean that only this implementation is feasible. In other words, these specific technical details or implementation methods are only used to help understand the content of the technical improvement, and will not limit the scope of the patent or restrict the implementation method of the patent.

[0052] Embodiment 2

[0053] In some embodiments, the input left-view image and the input right-view image share weight parameters of the feature extraction module. Further, in some embodiments, the feature extraction module can be a pre-trained Efficient-B3 network.

[0054] As an example, the ImageNet dataset can be used to pre-train the Efficient-B3 network for image classification to obtain the pre-trained Efficient-B3 network, which can then be used as a feature extraction module of the stereo matching network.

[0055] In some embodiments, the left feature map and the right feature map output by the 5th convolutional layer, the 6th convolutional layer and the 8th convolutional layer in the feature extraction module are used as the input of the cost volume construction module connected in series with the feature extraction module, and the left feature map and the right feature map output by the 4th convolutional layer in the feature extraction module are used as the input of the disparity regression module connected in series with the feature extraction module.

[0056] In some specific embodiments, the sizes of the left feature map and the right feature map output by the 4th convolutional layer, the 5th convolutional layer, the 6th convolutional layer and the 8th convolutional layer in the feature extraction module are 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the left view image and the right view image, respectively, and the number of channels is 24, 32, 48 and 132, respectively.

[0057] In the technical solution provided in this embodiment, by using the pre-trained Efficient-B3 network as the feature extraction module and setting the weight parameters of the input left view and right view to share the feature extraction module, the potential information in the left view and the right view can be effectively extracted.

[0058] Embodiment 3

[0059] In some embodiments, Figure 3 As shown, the cost volume construction module includes an initial cost volume construction module, a residual visual mamba layer, a 3D attention module and a normalization module connected in series with the output of the residual visual mamba layer, respectively, a first fusion module connected in series with the output of the 3D attention module and the output of the normalization module, and a second fusion module connected in series with the first fusion module.

[0060] According to the previous description, the input of the cost volume construction module is the left feature map and the right feature map output by the 5th convolutional layer, the 6th convolutional layer and the 8th convolutional layer in the feature extraction module. It can be understood that the left feature map and the right feature map output by each convolutional layer are a group of feature maps, which are regarded as a group of inputs to the cost volume construction module. In the cost volume construction module, each group of input left feature maps and right feature maps need to be processed by the initial cost volume construction module, the residual visual mamba layer, the 3D attention module, the normalization module and the first fusion module in the cost volume construction module to obtain the cost volume corresponding to each group of left feature maps and right feature maps. Finally, each group of cost bodies will be fused by the second fusion module in the input cost volume construction module to obtain a fused cost body. Figure 2 The three cost volumes in the cost volume construction module shown are the cost volumes corresponding to each set of left feature maps and right feature maps output by the first fusion module. Figure 2 The fused cost volume in the cost volume construction module shown is the fused cost volume obtained by fusing three cost volumes through the second fusion module.

[0061] In some embodiments, three groups of parallel initial cost volume construction modules, residual visual mamba layers, 3D attention modules, normalization modules, and first fusion modules may be set in the cost volume construction module, and each group is used to process a group of left feature maps and right feature maps of the input to obtain the cost volume corresponding to the group of left feature maps and right feature maps. The second fusion module is connected in series with the first fusion modules in the three groups to fuse the cost volumes output by each group of the first fusion modules to generate a fused cost volume. Such a setting can process each group of inputs in parallel at the same time, thereby improving the matching rate.

[0062] In other embodiments, only one set of initial cost body construction modules, residual visual mamba layer, 3D attention module, normalization module and first fusion module may be set in the cost body construction module, and the second fusion module is connected in series with the first fusion module. Such a setting requires processing a set of left feature maps and right feature maps input in sequence, and after the processing of the previous set of left feature maps and right feature maps is completed, the next set of left feature maps and right feature maps are processed, and finally all cost bodies are fused in the second fusion module to obtain a fused cost body. It should be noted that when implementing the technical solution of this embodiment, those skilled in the art can flexibly select the corresponding solution according to the content and teaching disclosed in this embodiment.

[0063] Next, each module in the cost body construction module will be described in detail, but this does not limit the present application.

[0064] Specifically, in some embodiments, the initial cost volume construction module at least performs the following steps S21 to S23 in sequence:

[0065] Step S21: remove pixel values ​​in the left i columns and the right i columns of the input left feature map and right feature map, where i is the disparity level of the left feature map and the right feature map;

[0066] Step S22: splicing the left feature map and the right feature map after the pixel values ​​are removed in a staggered superposition manner in the channel latitude to obtain an initial cost volume;

[0067] Step S23: The initial cost volume is upscaled so that the latitude of the upscaled initial cost volume meets the input latitude requirement of the residual visual mamba layer.

[0068] In some embodiments, the input latitude requirement of the residual visual mamba layer can be set to four dimensions, and the number of output channels of the residual visual mamba layer can be set to C / 4. Therefore, in the above step S23, the initial cost volume is upgraded to four dimensions.

[0069] As an example, assume that the size of the input left feature map and right feature map is HxWxC, and the disparity level is i. Remove the pixel values ​​of the left i columns and the right i columns in the left and right feature maps. Then, splice the left and right feature maps in the channel latitude in an interlaced superposition manner to obtain the size The initial cost Finally, the initial cost body Upgrade it to four dimensions and output it to the residual visual Mamba layer in series with it.

[0070] After the initial cost volume is obtained by the initial cost volume construction module, the initial cost volume is input into the residual visual mamba module for further processing. Figure 3 As shown, the residual visual mamba module at least performs the following steps S24 to S28:

[0071] Step S24: performing layer normalization on the initial cost volume output by the initial cost volume construction module;

[0072] Step S25: inputting the layer-normalized initial cost volume into the Mamba model;

[0073] Step S26: adding the learnable parameters to the initial cost volume output by the Mamba model pixel by pixel;

[0074] Step S27: performing layer normalization on the initial cost volume after pixel-by-pixel addition;

[0075] Step S28: Input the layer-normalized initial cost volume into the linear layer.

[0076] After being processed by the above-mentioned residual visual Mamba module, the initial cost volume can be enhanced to fully explore the relationship between the deep and shallow features extracted during the feature extraction process.

[0077] The initial cost volume processed by the above-mentioned residual visual Mamba module is input into the 3D attention module and the normalization module in the cost volume construction module for further processing. Finally, the initial cost volumes processed by these two branches are fused by the first fusion module in the cost volume construction module to obtain a set of cost volumes corresponding to the left feature map and the right feature map.

[0078] In some embodiments, Figure 3 As shown, the first branch 3D attention module at least performs the following steps S29 to S35 in sequence:

[0079] Step S29: performing maximum pooling and average pooling on the initial cost volume output by the residual visual mamba layer respectively;

[0080] Step S30: adding the initial cost volume after maximum pooling and the initial cost volume after average pooling pixel by pixel;

[0081] Step S31: performing the first Sigmoid normalization on the initial cost volume after pixel-by-pixel addition;

[0082] Step S32: performing a first pixel-by-pixel multiplication on the initial cost volume output by the residual visual mamba layer and the initial cost volume after the first Sigmoid normalization;

[0083] Step S33: performing at least three 3D convolutions on the initial cost volume after the first pixel-by-pixel multiplication, wherein the convolution kernel sizes of the three 3D convolutions are 1x7x7, 7x1x1 and 7x7x7, respectively, acting on the spatial, depth and overall dimensions, respectively;

[0084] Step S34: performing a second Sigmoid normalization on the initial cost volume after at least three 3D convolutions;

[0085] Step S35: performing a second pixel-by-pixel multiplication on the initial cost volume after the second sigmoid normalization and the initial cost volume after the first pixel-by-pixel addition.

[0086] In some embodiments, Figure 3 As shown, the second branch normalization module at least performs the following steps S36 to S39 in sequence:

[0087] Step S37: performing layer normalization on the initial cost volume output by the residual visual mamba layer;

[0088] Step S38: weighting the initial cost volume output by the residual visual mamba layer based on a learnable parameter;

[0089] Step S39: performing a residual connection between the weighted initial cost volume and the layer-normalized initial cost volume.

[0090] After the initial cost volume output by the residual visual mamba layer is processed by the 3D attention module and the normalization module respectively, the initial cost volume output by the 3D attention module and the initial cost volume output by the normalization module are input to the first fusion module for fusion to obtain a set of cost volumes corresponding to the left feature map and the right feature map. In some embodiments, Figure 3 As shown, the first fusion module at least performs the following steps S40 and S41 in sequence:

[0091] Step S40: adding the initial cost volume output by the normalization module and the initial cost volume output by the 3D attention module pixel by pixel;

[0092] Step S41: add i columns of 0 to the right side of the initial cost volume after pixel-by-pixel addition to obtain a set of cost volumes corresponding to the left feature map and the right feature map.

[0093] As an example, based on the above example, the initial cost volume output by the initial cost volume construction module is After the first fusion module processes step S40, the initial cost volume is obtained. Then in the above step S41, in the initial cost volume Right side supplement Column 0, the final shape at the current scale is The cost body. is the parallax level The number of .

[0094] It can be understood that after the above steps of the first fusion module, a set of cost volumes corresponding to the left feature map and the right feature map can be obtained, that is, Figure 2 The cost volume in the cost volume construction module shown. It can be understood that since the input of the cost volume construction module is the left feature map and the right feature map output by the 5th convolutional layer, the 6th convolutional layer and the 8th convolutional layer in the feature extraction module, and the left feature map and the right feature map output by each convolutional layer are used as a set of inputs of the cost volume construction module, the cost volume construction module finally obtains 3 cost volumes.

[0095] Specifically, since the sizes of the left feature map and the right feature map output by the 5th, 6th and 8th convolutional layers as the input of the initial cost volume construction module are 1 / 4, 1 / 8 and 1 / 16 of the left view image and the right view image respectively, the cost volumes output by the first fusion module are of , size is of and size is of .in It is obtained by taking a set of feature maps output by the 5th convolutional layer in the feature extraction module as input. It is obtained by taking a set of feature maps output by the 6th convolutional layer in the feature extraction module as input. It is obtained by taking a set of feature maps output by the 8th convolutional layer in the feature extraction module as input.

[0096] All cost volumes output by the first fusion module will be fused by the second fusion module input into the cost volume construction module to obtain a fused cost volume. In some embodiments, the second fusion module at least performs the following steps S42 and S43 in sequence:

[0097] Step S42: According to a preset fusion expression, the output of the first fusion module and Perform fusion to obtain an initial fusion cost body;

[0098] Step S43: According to the preset fusion expression, the initial fusion cost volume and Perform fusion to obtain a fusion cost body;

[0099] The preset fusion expression is:

[0100]

[0101] in, represents 3D deconvolution, Conv represents 3D convolution, x1 and x2 represent the number of channels of the output channel, represents group regularization and Activation function, Represents a concatenation operation on the channel dimension.

[0102] As an example, the cost body , and After the second fusion module, the final size is The fusion cost , of which 16 are 2 times of that, and 48 is the parallax range.

[0103] In the technical solution provided in this embodiment, the residual visual mamba layer and the 3D attention module are introduced into the cost body construction module, so that the relationship between the deep and shallow features extracted in the feature extraction process is fully explored through the residual visual mamba layer, and combined with the local enhancement method, richer contextual information is provided for the cost body, so that the stereo matching network can more easily cope with the difficult matching problem of pathological areas, thereby achieving the purpose of improving the matching accuracy and robustness of the stereo matching network.

[0104] Embodiment 4

[0105] like Figure 2 As shown, after the cost volume construction module outputs the fused cost volume, the fused cost volume is input into the aggregation module for cost volume aggregation. Figure 2 As shown, the aggregation module includes a 3D residual block and a cost aggregation module based on partial 3D convolution connected in series. The cost aggregation module based on partial 3D convolution includes at least a first hourglass network based on partial 3D convolution, a second hourglass network based on partial 3D convolution, and a third hourglass network based on partial 3D convolution connected in series. In this embodiment, by replacing some ordinary 3D convolutions in the cost aggregation process with partial 3D convolutions in the aggregation module, the amount of parameters and calculations of the stereo matching network can be reduced, and the matching accuracy can be improved.

[0106] Specifically, in some embodiments, the 3D residual block at least performs the following steps S44 and S45 in sequence:

[0107] Step S44: performing 3D convolution on the fused cost volume output by the cost volume construction module;

[0108] Step S45: Perform residual connection on the fused cost volume after 3D convolution and the fused cost volume output by the cost volume construction module.

[0109] like Figure 2 As shown, the fusion cost body The output obtained after passing through the 3D residual block is represented as output 0.

[0110] Next, the output of the 3D residual block is further input into the hourglass network based on partial 3D convolution connected in series for processing. Specifically, in some embodiments, referring to Figure 4 , Figure 4 Schematic diagram of the hourglass network based on partial 3D convolution and the network architecture of partial 3D convolution in the stereo matching network of some embodiments of the present application. Figure 4 As shown, the first hourglass network based on partial 3D convolution, the second hourglass network based on partial 3D convolution, and the third hourglass network based on partial 3D convolution all include at least the first 3D convolution, the first partial 3D convolution, the second 3D convolution, the second partial 3D convolution, the third 3D convolution, and the fourth 3D convolution connected in series, wherein the third 3D convolution is connected to the residual of the first partial 3D convolution, and the fourth 3D convolution is connected to the residual of the first 3D convolution. Figure 2 As shown, the outputs of the first hourglass network based on partial 3D convolution, the second hourglass network based on partial 3D convolution, and the third hourglass network based on partial 3D convolution are represented as output 1, output 2, and output 3, respectively.

[0111] In the technical solution provided in this embodiment, some common 3D convolutions in the cost aggregation process are replaced by partial 3D convolutions to reduce the number of parameters and the amount of calculation of the stereo matching network while achieving a more accurate matching effect.

[0112] Embodiment 5

[0113] In some embodiments, the disparity regression module includes at least a first disparity regression module, a second disparity regression module, a third disparity regression module and a fourth disparity regression module. The first disparity regression module, the second disparity regression module, the third disparity regression module and the fourth disparity regression module each include an initial disparity generation module and a disparity refinement module connected in series. Further, the 3D residual block output in the aggregation module is connected in series with the initial disparity generation module of the first disparity regression module, the first hourglass network output based on partial 3D convolution is connected in series with the initial disparity generation module of the second disparity regression module, the second hourglass network output based on partial 3D convolution is connected in series with the initial disparity generation module of the third disparity regression module, and the third hourglass network output based on partial 3D convolution is connected in series with the initial disparity generation module of the fourth disparity regression module. According to such a setting, the output 0 of the 3D residual block in the aggregation module, the output 1, output 2, and output 3 of each hourglass network based on partial 3D convolution can be correspondingly input into the initial disparity generation modules of the first disparity regression module, the second disparity regression module, the third disparity regression module, and the fourth disparity regression module to generate an initial predicted disparity map.

[0114] Specifically, in some embodiments, the initial disparity generation module at least performs the following steps S46 to S48 in sequence:

[0115] Step S46: performing at least 2 3D convolutions on the input;

[0116] Step S47: performing trilinear interpolation on the input after 3D convolution;

[0117] Step S48: performing probability weighted regression on the input after trilinear interpolation to obtain an initial predicted disparity map, wherein the probability of the initial predicted disparity map is calculated by a Sigmod function.

[0118] Furthermore, after the initial disparity generation module outputs the initial predicted disparity map, the initial predicted disparity map is input into the disparity refinement module connected in series therewith, and the left feature map and the right feature map output by the 4th convolutional layer in the feature extraction module are also input into the disparity refinement module to refine the initial predicted disparity map and obtain the target predicted disparity map.

[0119] In some embodiments, reference Figure 5 , Figure 5 Schematic diagram of the network architecture of the disparity refinement module in the stereo matching network of some embodiments of the present application. Figure 5 As shown, the parallax refinement module at least performs the following steps S49 to S54 in sequence:

[0120] Step S49: interpolating the input left feature map and right feature map to the original view size;

[0121] Step S50: Calculate the difference between the interpolated left feature map and the interpolated right feature map to obtain a left-right feature error;

[0122] Step S51: splicing the left and right feature errors and the input initial predicted disparity map in the channel dimension to obtain a first splicing result;

[0123] Step S52: using three 2D dilated convolutions to perform convolution operations on the first splicing result respectively, wherein the dilation rates of the 2D dilated convolutions are 1, 2, and 4 respectively;

[0124] Step S53: splicing each output obtained after the convolution operation in the channel dimension to obtain a second splicing result;

[0125] Step S54: applying a channel attention module to the second splicing result, and performing a residual connection between the obtained result and the initial predicted disparity map to obtain the target predicted disparity map.

[0126] Therefore, each disparity regression module finally obtains a target predicted disparity map. Figure 2 As shown, the target predicted disparity map output by the first disparity regression module is represented as target predicted disparity map 0, and the target predicted disparity map output by the second disparity regression module is represented as target predicted disparity map Figure 1 , the target prediction disparity map output by the third disparity regression module is represented as the target prediction disparity Figure 2 , the target prediction disparity map output by the fourth disparity regression module is represented as the target prediction disparity Figure 3 .

[0127] In the technical solution provided in this embodiment, by combining the feature error and the target prediction disparity map generated by the initial prediction error map, the robustness of the stereo matching network in pathological areas such as reflection and occlusion can be enhanced.

[0128] Embodiment 6

[0129] In some embodiments, after the step of inputting the left view image and the right view image into an improved stereo matching network to obtain a target predicted disparity map, the following steps S55 and S56 are also included:

[0130] Step S55: Calculate the target predicted disparity map and the real disparity map loss;

[0131] Step S56: The loss serves as a constraint during the training of the stereo matching network.

[0132] In the technical solution provided in this embodiment, by calculating the target predicted disparity map and the real disparity map The loss is used as a constraint during the training of the stereo matching network, so that the model parameters of the stereo matching network are continuously updated during training, and a stereo matching network with high matching accuracy and robustness can be obtained with only a small number of batches of training.

[0133] According to the above embodiments, the stereo matching method based on the Mamba cost volume provided by the present application has at least the following beneficial effects:

[0134] (1) This application introduces the Mamba model to fully explore the relationship between deep and shallow features extracted during the feature extraction process, and combines it with local enhancement to provide richer contextual information for the cost body, making it easier for the model to cope with the difficult matching problem in pathological areas.

[0135] (2) This application uses partial convolution to replace some ordinary 3D convolutions in the cost aggregation process, thereby achieving a more accurate effect while reducing the number of parameters and the amount of calculation.

[0136] (3) This application designs a disparity refinement method that combines feature error and initial disparity, which can enhance the robustness of the model in pathological areas such as reflection and occlusion.

[0137] (4) The present application proposes a stereo matching method based on the Mamba cost body, which can achieve satisfactory prediction results with only a small number of training batches.

[0138] Embodiment 7

[0139] This embodiment will verify the effect of the stereo matching network proposed in the above embodiments.

[0140] In this embodiment, the data in SceneFlow and KITTI datasets are used to experiment the stereo matching method based on Mamba cost volume proposed in this application. The SceneFlow dataset is a large-scale synthetic dataset for training and evaluating visual tasks such as stereo matching and depth estimation, which consists of high-quality images synthesized by computers. The KITTI dataset is collected from real road scenes in Karlsruhe, Germany and its surrounding areas. The data is recorded by multiple sensors installed on a car.

[0141] In this embodiment, the stereo matching method based on the Mamba cost volume proposed in this application is compared with a series of advanced stereo matching methods in the prior art in terms of EPE indicators, and the corresponding calculation amount and parameter amount are also shown.

[0142] Table 1 shows the comparison results on the SceneFlow test set. As can be seen from Table 1, compared with PCWNet, which also uses transformation operations in the disparity refinement stage, although the number of parameters in this application is 16.19% more than that of PCWNet, the amount of calculation is greatly reduced by 65.59%. In addition, the EPE index of this application is 3.85% lower than that of PCWNet. Compared with ACVNet-fast's training strategy of three stages and 64 batches in each stage, we conducted a total of 18 batches of training, and with lower parameters and calculations, the stereo matching method based on the Mamba cost volume of this application has better accuracy.

[0143] Table 1. Comparison results on the SceneFlow test set

[0144] further, Figure 6 This is a qualitative comparison result diagram of this application on the SceneFlow dataset. Figure 6 As shown in the figure, (a) represents the input image, (b), (c), and (d) represent the disparity map and error map of GwcNet, ACVNet, and this application, respectively. Figure 6 It can be seen that the method of this application is better than GwcNet in predicting pathological areas, especially in textureless or weakly textured areas, and can generate smoother and more accurate results. Compared with ACVNet, the method of this application is more accurate in predicting the edges of objects. At the same time, it shows that a cost body containing rich information can use fewer parameters and have high accuracy in the aggregation process.

[0145] Table 2 shows the verification results on the KITTI2015 dataset. As can be seen from Table 2, with only 0.52M more parameters than AANet, the D1all index of all pixels of the proposed method is superior to most networks in the prior art, for example, 6.12% higher than PGNet and 2.13% higher than CFNet. This result shows that the proposed method can achieve relatively accurate prediction results in practical applications.

[0146] Table 2. Verification results on the KITTI2015 dataset

[0147] Figure 7 This is a comparison of the disparity prediction results of this application on the KITTI2015 dataset. Figure 7As shown, (a) represents the input image, (b), (c), and (d) represent the disparity map and error map of GwcNet, ACVNet, and this application, respectively. From left to right are the visualization results of details, holes, weak textures, and occluded areas, and the corresponding areas are marked with dotted boxes. The method of this application benefits from the powerful long-distance modeling capability of the Mamba cost body and the targeted optimization of the error area by the disparity optimization module. Compared with GwcNet, the prediction effect of this application method in these pathological areas is significantly improved, and the prediction effect in the hole area and weak texture exceeds that of ACVNet.

[0148] Table 3 shows the quantitative comparison results of the reflection area in the KITTI2012 dataset. It can be seen from Table 3 that the matching method based on the Mamba cost volume of the present application can well predict the disparity value of the reflection area, surpassing ACVNet in the px-3 index of all pixels and non-occluded pixels, and the number of parameters is 37.94% less than it. The computational complexity of the present application method is 31.14% less than that of GwcNet, and the number of parameters is reduced by 30.8% compared with GWcNet. Although the present application method uses the same stacked hourglass network as GwcNet in the cost aggregation module, the addition of some 3D convolutions in the present application reduces the number of parameters and computations, and Mamba reduces the quadratic complexity to linear when modeling, reducing the computational overhead. In the disparity refinement module, the present application method uses 2D convolution, which also avoids the increase in the number of parameters. In addition, the batch size set for our stereo matching network training is 2. Except for GANet and GCNet, which are 1, the batch sizes of the other models in Table 3 are all greater than 10 during training. Although smaller batch sizes are easily affected by noise, the stereo matching network of this application still achieves the same effect as ACVNet.

[0149] Table 3. Quantitative comparison results of reflection areas in the KITTI2012 dataset

[0150] Figure 8 This is a qualitative comparison result of this application on the KITTI2012 test set. Figure 8 As shown in the figure, (a) represents the input image, (b), (c), and (d) represent the disparity map and error map of GwcNet, ACVNet, and this application, respectively. From left to right are the occlusion, reflection, edge, and hole areas, and the corresponding areas are marked with dotted boxes. Figure 8 It can be seen that the present application shows better performance in pathological areas. For example, the effect of the reflective area of ​​the roof in the second picture is significantly better than GwcNet and ACVNet, and the error of the hole area on the right in the fourth picture is also significantly improved.

[0151] Although multiple embodiments of the present application have been shown and described herein, it is obvious to those skilled in the art that such embodiments are provided only by way of example. Those skilled in the art can think of many changes, modifications and alternatives without departing from the thought and spirit of the present application. It should be understood that in the process of practicing the present application, various alternatives to the embodiments of the present application described herein can be adopted. The attached claims are intended to limit the scope of protection of the present application, and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A stereo matching method based on Mamba cost volume, characterized in that: include: Acquire a left-view image and a right-view image to be processed, wherein the left-view image and the right-view image are images of the same object at different viewing angles; Inputting the left view image and the right view image into an improved stereo matching network to obtain a target predicted disparity map; The improved stereo matching network at least comprises a feature extraction module, a cost volume construction module, an aggregation module and a disparity regression module connected in series, and the feature extraction module is also connected in series with the disparity regression module; The cost volume construction module includes an initial cost volume construction module, a residual visual mamba layer, a 3D attention module and a normalization module respectively connected in series with the output of the residual visual mamba layer, a first fusion module connected in series with the output of the 3D attention module and the output of the normalization module, and a second fusion module connected in series with the first fusion module.

2. The method according to claim 1, characterized in that The left view image and the right view image share the weight parameters of the feature extraction module, and the feature extraction module is a pre-trained Efficient-B3 network; the left feature map and the right feature map output by the 4th convolutional layer, the 5th convolutional layer, the 6th convolutional layer and the 8th convolutional layer in the feature extraction module are used as the input of the cost volume construction module connected in series with the feature extraction module, and the left feature map and the right feature map output by the 4th convolutional layer in the feature extraction module are used as the input of the disparity regression module connected in series with the feature extraction module; the sizes of the left feature map and the right feature map output by the 4th convolutional layer, the 5th convolutional layer, the 6th convolutional layer and the 8th convolutional layer are 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the left view image and the right view image, respectively, and the number of channels is 24, 32, 48 and 132, respectively.

3. The method according to claim 1, characterized in that The initial cost volume construction module at least performs the following steps in sequence: Remove the pixel values ​​in the left i columns and the right i columns of a set of input left feature maps and right feature maps, where i is the disparity level of the left feature map and the right feature map; The left feature map and the right feature map after the pixel values ​​are removed are spliced ​​in the channel latitude in an interlaced superposition manner to obtain an initial cost volume; The initial cost body is upscaled so that the latitude of the upscaled initial cost body meets the input latitude requirement of the residual visual mamba layer, wherein the input latitude requirement of the residual visual mamba layer is four-dimensional, and the number of output channels of the residual visual mamba layer is C / 4.

4. The method according to claim 3, characterized in that The residual visual mamba layer at least performs the following steps in sequence: Performing layer normalization on the initial cost volume output by the initial cost volume construction module; Inputting the layer-normalized initial cost volume into the Mamba model; Adding the learnable parameters to the initial cost volume output by the Mamba model pixel by pixel; Performing layer normalization on the initial cost volume after pixel-by-pixel addition; The initial cost volume after layer normalization is input into the linear layer.

5. The method according to claim 4, characterized in that The 3D attention module performs at least the following steps in sequence: Performing maximum pooling and average pooling on the initial cost volume output by the residual visual mamba layer respectively; Adding the initial cost volume after maximum pooling and the initial cost volume after average pooling pixel by pixel; Performing a first Sigmoid normalization on the initial cost volume after pixel-by-pixel addition; Performing a first pixel-by-pixel multiplication on the initial cost volume output by the residual visual mamba layer and the initial cost volume after the first sigmoid normalization; Perform at least three 3D convolutions on the initial cost volume after the first pixel-by-pixel multiplication, wherein the convolution kernel sizes of the three 3D convolutions are 1x7x7, 7x1x1, and 7x7x7, respectively, acting on the spatial, depth, and overall dimensions; Perform a second Sigmoid normalization on the initial cost volume after at least three 3D convolutions; Multiplying the initial cost volume after the second sigmoid normalization by the initial cost volume after the first pixel-by-pixel addition for a second time; The normalization module at least performs the following steps in sequence: Performing layer normalization on the initial cost volume output by the residual visual mamba layer; Weighting the initial cost volume output by the residual visual mamba layer based on the learnable parameters; The weighted initial cost volume and the layer-normalized initial cost volume are residually connected.

6. The method according to claim 5, characterized in that The first fusion module at least performs the following steps in sequence: Adding the initial cost volume output by the normalization module to the initial cost volume output by the 3D attention module pixel by pixel; After pixel-by-pixel addition, i columns of 0 are added to the right of the initial cost volume to obtain a set of cost volumes corresponding to the left feature map and the right feature map; wherein the cost volumes output by the first fusion module are of size of , size is of and size is of , where is obtained by taking a set of feature maps output by the 5th convolutional layer in the feature extraction module as input, It is obtained by taking a set of feature maps output by the 6th convolutional layer in the feature extraction module as input. It is obtained by taking a set of feature maps output by the 8th convolutional layer in the feature extraction module as input.

7. The method according to claim 6, characterized in that The second fusion module at least performs the following steps in sequence: The output of the first fusion module is converted into and Perform fusion to obtain an initial fusion cost body; According to the preset fusion expression, the initial fusion cost volume and the The fusion is performed to obtain a fusion cost body; wherein the preset fusion expression is: ； in, represents 3D deconvolution, Conv represents 3D convolution, x1 and x2 represent the number of channels of the output channel, represents group regularization and Activation function, Represents a concatenation operation on the channel dimension.

8. The method according to claim 7, characterized in that The aggregation module includes a 3D residual block and a cost aggregation module based on partial 3D convolution connected in series, wherein the cost aggregation module based on partial 3D convolution includes at least a first hourglass network based on partial 3D convolution, a second hourglass network based on partial 3D convolution, and a third hourglass network based on partial 3D convolution connected in series; wherein, The 3D residual block at least performs the following steps in sequence: Performing 3D convolution on the fused cost volume output by the cost volume construction module; Performing residual connection on the fused cost volume after 3D convolution and the fused cost volume output by the cost volume construction module; The first hourglass network based on partial 3D convolution, the second hourglass network based on partial 3D convolution, and the third hourglass network based on partial 3D convolution all include at least a first 3D convolution, a first partial 3D convolution, a second 3D convolution, a second partial 3D convolution, a third 3D convolution, and a fourth 3D convolution connected in series, wherein the third 3D convolution is residually connected to the first partial 3D convolution, and the fourth 3D convolution is residually connected to the first 3D convolution.

9. The method according to claim 8, characterized in that The disparity regression module at least includes a first disparity regression module, a second disparity regression module, a third disparity regression module and a fourth disparity regression module; wherein the first disparity regression module, the second disparity regression module, the third disparity regression module and the fourth disparity regression module each include an initial disparity generation module and a disparity refinement module connected in series in sequence; The initial disparity generation module at least performs the following steps in sequence: Perform at least 2 3D convolutions on the input; Perform trilinear interpolation on the input after 3D convolution; Performing probability weighted regression on the input after trilinear interpolation to obtain an initial predicted disparity map, wherein the probability of the initial predicted disparity map is calculated by a Sigmod function; The parallax refinement module at least performs the following steps in sequence: Interpolate the input left and right feature maps to the original view size; Calculate the difference between the interpolated left feature map and the interpolated right feature map to obtain a left-right feature error; Splicing the left and right feature errors and the input initial predicted disparity map in a channel dimension to obtain a first splicing result; Using three 2D dilated convolutions to perform convolution operations on the first splicing results respectively, wherein the dilation rates of the 2D convolutions are 1, 2, and 4 respectively; Each output obtained after the convolution operation is concatenated in the channel dimension to obtain a second concatenated result; Applying a channel attention module to the second splicing result, and performing a residual connection between the obtained result and the initial predicted disparity map to obtain the target predicted disparity map; The 3D residual block output in the aggregation module is connected in series with the initial disparity generation module of the first disparity regression module, the first partial 3D convolution-based hourglass network output in the aggregation module is connected in series with the initial disparity generation module of the second disparity regression module, the second partial 3D convolution-based hourglass network output in the aggregation module is connected in series with the initial disparity generation module of the third disparity regression module, and the third partial 3D convolution-based hourglass network output in the aggregation module is connected in series with the initial disparity generation module of the fourth disparity regression module.

10. The method according to claim 1, characterized in that After the step of inputting the left view image and the right view image into an improved stereo matching network to obtain a target predicted disparity map, the step further includes: Calculate the target predicted disparity map and the real disparity map loss; The The loss serves as a constraint during the training of the stereo matching network.

Images