A stereo matching method based on attention mechanism and noise suppression GRU

Abstract

The application discloses a stereo matching method based on an attention mechanism and noise suppression GRU, and comprises the following steps: feature extraction, obtaining N high-channel feature maps; performing homography transformation on the N high-channel feature maps according to corresponding depths to obtain a high-channel feature body; performing regularization processing on high-channel cost bodies C(0)~C(D-1) by using a GRU module to obtain D cost maps, and aggregating the D cost maps to obtain a probability body; and converting the probability body into a depth map to complete stereo matching. The application combines the two modes of a hollow convolution and an attention mechanism to improve the quality of feature extraction; and proposes a noise filtering GRU cost body construction mode to replace the original 3D CNN, thereby reducing the amount of calculation and the memory requirement; the method of filtering the cost body multiple times by using a spatial attention mechanism reduces the influence of edge and background noise on depth estimation, enables the model to better reconstruct a point cloud, and improves the final point cloud reconstruction effect.

Description

Technical Field

[0001] This invention relates to the field of multi-view stereo matching, and more particularly to a stereo matching method based on attention mechanism and noise suppression GRU. Background Technology

[0002] Multi-view stereo matching (MVS) is a core problem in computer science. MVS reconstruction can be viewed as the inverse process of taking photographs of a given scene. Just as taking a photograph maps a 3D scene to 2D using a camera, MVS reconstruction aims to reconstruct the true 3D scene from images taken from different viewpoints. Recent advances in convolutional neural network research have sparked interest in improving stereo matching reconstruction. Conceptually, learning-based methods can learn global semantic information; for example, prior knowledge of highlights and reflections can lead to more robust matching results. Currently, there are some attempts at two-view stereo matching, replacing hand-designed similarity measures or regularization methods with neural networks. Unlike binocular stereo matching, MVS inputs any number of views, which is a challenging problem that deep learning methods need to address.

[0003] Currently, the mainstream deep learning-based multi-view stereo matching method uses MVSNet (Multi-View Stereo Network) for 3D reconstruction tasks. The main features and working principles of MVSNet are: 1. End-to-end deep learning: MVSNet uses an end-to-end deep learning approach, taking images from multiple perspectives as input (one reference image and multiple source images) and outputting a depth estimate for each pixel, thus generating a 3D point cloud. 2. 3D-CNN architecture in the cost volume regularization stage: MVSNet uses a 3D CNN to process images from multiple perspectives, constructing a 3D cost volume. This cost volume captures the similarity information of pixels between different perspectives, which helps in depth estimation. 3. Multi-view information fusion: By learning image features and similarities between different perspectives, MVSNet can effectively fuse information from multiple perspectives, thereby improving the quality of 3D reconstruction. 4. Depth estimation and point cloud generation: MVSNet first estimates the depth of each pixel, and then generates a dense 3D point cloud containing geometric information of the scene based on these depth estimates.

[0004] However, existing deep learning-based multi-view stereo vision (MVS) methods almost exclusively focus on improving reconstruction quality. Besides quality, efficiency is also an ideal characteristic for MVS in real-world scenarios. MVSNet uses 3D-CNN to construct the cost volume. Since the 3D cost volume needs to store the similarity score of each pixel, it requires a large amount of memory, which may lead to performance degradation. Furthermore, the effectiveness of the 3D cost volume depends on the disparity range; if the disparity range is too large or too small, it may reduce the accuracy of depth estimation. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a stereo matching method based on attention mechanism and noise suppression GRU.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A first aspect of the present invention provides a stereo matching method based on an attention mechanism and a noise-suppressing GRU, comprising the following steps:

[0008] The reference image is used to extract features from the N-1 source images to obtain N high-channel feature maps;

[0009] The N high-channel feature maps are subjected to homography transformation at the corresponding depth to obtain D groups of N high-channel feature bodies in each group. Then, the high-channel feature bodies in each group are aggregated into corresponding high-channel cost bodies C(0)~C(D-1) using a cost index based on variance, where D is the number of depths.

[0010] The high-channel cost bodies C(0) to C(D-1) are regularized using the GRU module to obtain D cost maps. The D cost maps are then aggregated to obtain a probability body.

[0011] The probabilistic volume is converted into a depth map to complete the stereo matching;

[0012] Feature extraction is performed on the N input 2D images using the reference image to obtain N high-channel feature maps, specifically:

[0013] Feature extraction is performed using a feature extraction network that combines dilated convolution and attention mechanisms. The feature extraction network includes N sequentially connected dilated convolution and attention modules, a convolution concatenation module, and a two-dimensional convolutional layer. Each dilated convolution and attention module includes a sequentially connected dilated convolutional layer and a CBAM module.

[0014] The process of using the GRU module to regularize the high-channel cost body C(0) to C(D-1) to obtain the probability body is as follows:

[0015] The high-channel cost volumes C(0) to C(D-1) are optimized using a GRU network with added noise suppression. Finally, the 1-channel cost volumes Cr(0) to Cr(D-1) are normalized using a softmax operation to obtain the probability volumes. The GRU network with added noise suppression includes: D GRU processing links, D-1 first spatial attention weighting modules A1, D-1 first weight filtering modules P1, D-1 second spatial attention weighting modules A2, and D-1 second weight filtering modules P2. Each GRU processing link includes a 2D convolution module, a first GRU processing module, a second GRU processing module, and a third GRU processing module.

[0016] The first 2D convolution module maps the high-channel cost volume C(0) to the middle channel and then inputs it into the first GRU processing module of the first channel to obtain the first middle channel cost volume feature; the first middle channel cost volume feature is input into the second GRU processing module of the first channel to obtain the first low channel cost volume feature; the first low channel cost volume feature is input into the third GRU processing module of the first channel to obtain the first 1-channel cost volume Cr(0);

[0017] After the Mth-way 2D convolutional module maps the high-channel cost volume C(M) to the middle channel, it inputs it into the first GRU processing module of the Mth-way to obtain the Mth middle channel cost volume feature; the M-1th middle channel cost volume feature and the Mth channel cost volume feature are respectively used as the first unfiltered cost volume feature W. u1 and the first filtering cost body feature W f1 The M1 weight grid is calculated by inputting it into the corresponding first spatial attention weighting module A1; the M1 weight grid and the Mth channel cost volume feature are input into the corresponding first weight filtering module P1 for weight filtering to obtain the Mth channel filtering cost volume feature;

[0018] The filtered cost body feature of the Mth middle channel is input into the second GRU processing module of the Mth channel to obtain the cost body feature of the Mth low channel; the cost body feature of the (M-1)th low channel and the cost body feature of the Mth low channel are respectively used as the second unfiltered cost body feature W. u2 Second filter cost body feature W f2 The M2 weight grid is calculated by inputting it into the corresponding second spatial attention weighting module A2; the M2 weight grid and the M low-channel cost volume feature are input into the corresponding second weight filtering module P2 for weight filtering to obtain the M low-channel filtered cost volume feature;

[0019] The cost body features of the (M-1)th low channel and the filtering cost body features of the Mth low channel are input into the third GRU processing module of the Mth channel to obtain the cost body feature Cr(M) of the M1st channel;

[0020] M ranges from 1 to D-1.

[0021] Furthermore, the CBAM module includes a channel attention module and a spatial attention module connected in sequence.

[0022] Furthermore, the N dilated convolution and attention modules are specifically three.

[0023] Furthermore, both the first spatial attention weighting module A1 and the second spatial attention weighting module A2 include:

[0024] The first compression module is used to compress the cost volume features of two adjacent high-frequency paths into unfiltered cost volume features W. u ;

[0025] The second compression module is used to compress the cost volume features of two adjacent low-to-medium paths into filtered cost volume features W. f ;

[0026] The matrix interpolation module is used to calculate the unfiltered cost volume features W pixel by pixel. u and filtering cost body features W f The feature difference weight grid G ​​is used to obtain the semantic information difference at each pixel point, i.e. This indicates that the feature matrix difference is calculated pixel by pixel;

[0027] The activation function calculation module is used to activate the feature difference weight grid G, i.e., Relu(G);

[0028] The complement calculation module is used to calculate the complement R of the activated feature difference weight grid G. The complement R represents the feature similarity, i.e., R = W. c (Relu(G)), W c This means finding the complement of the matrix with respect to 1, and adding a local minimum value;

[0029] The feature matrix multiplication module is used to multiply the complement R with the filter cost volume features W. f Performing a multiplication operation yields the weight grid F, i.e., F = W. f ⊙R.

[0030] Furthermore, both the first weight filtering module P1 and the second weight filtering module P2 include:

[0031] The multiplication calculation module is used to multiply the weight grid and the corresponding channel cost volume feature to obtain the weighted filtered cost volume feature of the corresponding channel.

[0032] Furthermore, the high channel has 32 channels, the medium channel has 16 channels, and the low channel has 4 channels.

[0033] The beneficial effects of this invention are:

[0034] In an exemplary embodiment of the present invention, the combination of dilated convolution and attention mechanisms can compensate for each other's shortcomings, enabling the feature extraction network in this exemplary embodiment to have a larger receptive field while more accurately focusing on task-related (target scene) information, thereby improving the quality of feature extraction.

[0035] Simultaneously, a noise-filtering GRU cost body construction method is proposed to replace the original 3D CNN. This transforms the process of ordinary regularization, performed all at multiple depths at once, into a process performed depth-by-depth, utilizing the output of the previous depth (treating the depth scale as the time scale of the recurrent neural network). This reduces the memory required for D depth samples to T / D, thus reducing computational cost and memory requirements. Furthermore, by employing a spatial attention mechanism to filter the cost body multiple times, the model can selectively focus on and weight information among different regions and depth candidates. This reduces the impact of edge and background noise on depth estimation, enabling the model to better reconstruct the point cloud and improving the final point cloud reconstruction effect. Attached Figure Description

[0036] Figure 1 The flowchart illustrates a stereo matching method based on an attention mechanism and a noise suppression GRU provided in an exemplary embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram of the structure of a feature extraction network provided in an exemplary embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of the structure of a GRU network with added noise suppression provided in an exemplary embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram of the spatial attention weighting module provided in an exemplary embodiment of the present invention. Detailed Implementation

[0040] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] In the description of this invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0042] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0043] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0044] See Figure 1 , Figure 1 The flowchart shown illustrates an exemplary embodiment of the present invention providing a stereo matching method based on an attention mechanism and a noise-suppressing GRU, comprising the following steps:

[0045] The reference image is used to extract features from the N-1 source images to obtain N high-channel feature maps;

[0046] The N high-channel feature maps are subjected to homography transformation at the corresponding depth to obtain D groups of N high-channel feature bodies in each group. Then, the high-channel feature bodies in each group are aggregated into corresponding high-channel cost bodies C(0)~C(D-1) using a cost index based on variance, where D is the number of depths.

[0047] The high-channel cost bodies C(0) to C(D-1) are regularized using the GRU module to obtain D cost maps. The D cost maps are then aggregated to obtain a probability body.

[0048] The probabilistic volume is converted into a depth map to complete the stereo matching;

[0049] Feature extraction is performed on the N input 2D images using the reference image to obtain N high-channel feature maps, specifically:

[0050] Feature extraction is performed using a feature extraction network that combines dilated convolution and attention mechanisms, such as... Figure 2 As shown, the feature extraction network includes N sequentially connected dilated convolutional and attention modules, a convolutional concatenation module, and a two-dimensional convolutional layer. Each dilated convolutional and attention module includes a sequentially connected dilated convolutional layer and a CBAM module.

[0051] The process of using the GRU module to regularize the high-channel cost body C(0) to C(D-1) to obtain the probability body is as follows:

[0052] The high-channel cost volumes C(0) to C(D-1) are optimized using a GRU network with added noise suppression. The final output 1-channel cost volumes Cr(0) to Cr(D-1) are then normalized using a softmax operation to obtain the probability volumes. The GRU network with added noise suppression is shown below. Figure 3 As shown, it includes: D GRU processing links, D-1 first spatial attention weighting modules A1, D-1 first weight filtering modules P1, D-1 second spatial attention weighting modules A2, and D-1 second weight filtering modules P2. Each GRU processing link includes a 2D convolution module, a first GRU processing module, a second GRU processing module, and a third GRU processing module.

[0053] The first 2D convolution module maps the high-channel cost volume C(0) to the middle channel and then inputs it into the first GRU processing module of the first channel to obtain the first middle channel cost volume feature; the first middle channel cost volume feature is input into the second GRU processing module of the first channel to obtain the first low channel cost volume feature; the first low channel cost volume feature is input into the third GRU processing module of the first channel to obtain the first 1-channel cost volume Cr(0);

[0054] After the Mth-way 2D convolutional module maps the high-channel cost volume C(M) to the middle channel, it inputs it into the first GRU processing module of the Mth-way to obtain the Mth middle channel cost volume feature; the M-1th middle channel cost volume feature and the Mth channel cost volume feature are respectively used as the first unfiltered cost volume feature W. u1 and the first filtering cost body feature W f1 The M1 weight grid is calculated by inputting it into the corresponding first spatial attention weighting module A1; the M1 weight grid and the Mth channel cost volume feature are input into the corresponding first weight filtering module P1 for weight filtering to obtain the Mth channel filtering cost volume feature;

[0055] The filtered cost body feature of the Mth middle channel is input into the second GRU processing module of the Mth channel to obtain the cost body feature of the Mth low channel; the cost body feature of the (M-1)th low channel and the cost body feature of the Mth low channel are respectively used as the second unfiltered cost body feature W. u2 Second filter cost body feature W f2The M2 weight grid is calculated by inputting it into the corresponding second spatial attention weighting module A2; the M2 weight grid and the M low-channel cost volume feature are input into the corresponding second weight filtering module P2 for weight filtering to obtain the M low-channel filtered cost volume feature;

[0056] The cost body features of the (M-1)th low channel and the filtering cost body features of the Mth low channel are input into the third GRU processing module of the Mth channel to obtain the cost body feature Cr(M) of the M1st channel;

[0057] M ranges from 1 to D-1.

[0058] Specifically, in this exemplary embodiment, preferred examples are given for the feature extraction part, as well as the cost body construction and regularization part of the existing MVSNet. First, for the entire stereo matching method, it includes the following steps: using the reference image to extract features from the input N-1 source images to obtain N high-channel feature maps (that is, N feature maps obtained by passing the N source images and the reference image through the feature extraction network); then, the N high-channel feature maps are subjected to homography transformation at the corresponding depth to obtain D groups of N high-channel feature bodies each, and each group of high-channel feature bodies is aggregated into corresponding high-channel cost bodies C(0) to C(D-1) using a variance-based cost index, where D is the number of depths (homography transformation and the variance cost index for generating cost bodies are conventional techniques in this field and will not be elaborated here). Figure 3 Circle M in the middle); use the GRU module to perform regularization processing on the high channel cost volume C(0)~C(D-1) to obtain D cost maps, aggregate the D cost maps to obtain a probability volume; convert the probability volume into a depth map to complete stereo matching.

[0059] Meanwhile, in this exemplary embodiment, the preferred scheme for the feature extraction part mainly proposes, as follows: Figure 2 The feature extraction network shown combines dilated convolution and attention mechanisms. Dilated convolution is based on convolutional neural networks, so it has the advantage of a relatively simple network structure compared to the feature pyramids in existing technologies.

[0060] More specifically, the main function of dilated convolution is to increase the receptive field without increasing the number of parameters. This allows the network to capture features over a larger area, improving performance in handling long-range dependencies or specific tasks (such as segmentation); it primarily enables the network to better capture global features. Meanwhile, the attention mechanism allows the model to dynamically focus on different parts of the input, which is very useful for processing important information in the input. The attention mechanism helps improve the model's attention capabilities, enabling it to better learn the relevance and contextual information of a specific task. For example, in a stereo matching task, the attention mechanism helps the network better focus on the target scene region in 3D space when processing features. In this exemplary embodiment, combining dilated convolution and the attention mechanism can compensate for each other's shortcomings, allowing the feature extraction network in this exemplary embodiment to have a larger receptive field while more accurately focusing on task-related (target scene) information, thus improving the quality of feature extraction.

[0061] Furthermore, in this exemplary embodiment, a preferred scheme is proposed for the cost body construction and regularization part. Specifically, for the cost body construction and regularization part, a noise-filtering GRU cost body construction method (i.e., the GRU network with added noise suppression) is first proposed to replace the original 3D CNN. This transforms the ordinary regularization process, which is performed all at multiple depths at once, into a process performed depth by depth, utilizing the output of the previous depth (i.e., treating the depth scale as the time scale of the recurrent neural network). This reduces the memory required for the original D depth samples to T / D, thus reducing computational load and memory requirements. Furthermore, in the cost body regularization stage, such as... Figure 3 As shown, the noise suppression layer (D-1 first spatial attention weighting modules A1 and D-1 first weight filtering modules P1 between the first GRU processing module and the second GRU processing module, and D-1 second spatial attention weighting modules A2 and D-1 second weight filtering modules P2 between the second GRU processing module and the third GRU processing module) effectively filters boundary noise and background noise, achieving a good suppression effect on edge noise and background noise, making the reconstructed point cloud more complete.

[0062] More specifically, for the GRU network with added noise suppression, let's take the preferred embodiment as an example: "32 high channels, 16 medium channels, and 4 low channels." First, a 2D convolutional module is used to reduce the number of channels from 32 to 16. Then, the first, second, and third GRU processing modules reduce the number of channels to 16, 4, and 1 respectively, ultimately outputting a regularized cost map. Furthermore, for D depth samples, the depth is calculated in the order of "0 to D-1," meaning that after calculating the regularized cost body at depth "0," the regularized cost body at depth "1" is calculated. It's important to note that the input to each GRU processing module at this depth not only includes the cost body at that depth but also utilizes the outputs of each GRU processing module at the previous depth, reflecting the concept of a recurrent neural network.

[0063] Furthermore, during the cost volume regularization stage, a noise suppression layer (D-1 first spatial attention weighting modules A1 and D-1 first weight filtering modules P1 between the first GRU processing module and the second GRU processing module) is used. Figure 3 The circles A and P between the first and second GRU processing modules, and the D-1 second spatial attention weighting modules A2 and D-1 second weight filtering modules P2 between the second and third GRU processing modules. Figure 3 The circles A and P between the second and third GRU processing modules in the network are used to effectively filter boundary noise and background noise. Areas with similar spatial information are more likely to be reconstructed target regions, while areas with different spatial information are more likely to be weakly correlated or noisy regions. Specifically, features of the same scale and adjacent depths in the network are compared pixel-by-pixel, and a spatial information weight grid is calculated. This spatial information weight grid is then applied to the input of the next layer. Referring to the spatial information, the weight grid filters the cost volume again.

[0064] Due to issues such as the object surface not conforming to an ideal Lambertian volume structure, network noise in the feature extraction network, and occlusion between different views, the cost volume calculated from the loss cost includes edge noise and background noise. In the cost volume regularization stage, this paper combines a spatial attention module and a GRU structure to more effectively filter boundary noise and background noise, thereby improving the quality of the final reconstructed point cloud.

[0065] Spatial attention mechanism: The spatial attention mechanism allows the model to dynamically focus on different regions in the input, enhancing or reducing information in different regions as needed for the task. This is extremely useful for processing complex image data, especially when edge noise and background noise are present. By utilizing the spatial attention mechanism, the model can focus on regions containing useful information, thereby reducing the impact of noise. This helps improve the processing of cost volumes and reduces unnecessary noise signals.

[0066] Multiple cost volume filtering: Cost volumes are commonly used in stereo matching tasks, where the model needs to select the optimal disparity to generate a depth map or point cloud. In the cost volume, each pixel or image region's depth candidate has an associated cost value. These costs can be affected by noise and interference, especially in edge and background regions. By filtering the cost volume multiple times, the model can gradually reduce the influence of noise and select a more accurate depth estimate. Multiple filtering can progressively improve the quality of the depth estimate.

[0067] Combining these two concepts, the method of using spatial attention to filter the cost volume multiple times allows the model to selectively focus on and weight information among different regions and depth candidates. This reduces the impact of edges and background noise on depth estimation, enabling the model to reconstruct the point cloud better and improving the final point cloud reconstruction results.

[0068] More preferably, in an exemplary embodiment, the CBAM module includes a channel attention module and a spatial attention module connected in sequence. More preferably, in an exemplary embodiment, the N dilated convolution and attention modules are specifically three.

[0069] More preferably, in an exemplary embodiment, such as Figure 4 As shown, both the first spatial attention weighting module A1 and the second spatial attention weighting module A2 include:

[0070] The first compression module is used to compress the cost volume features of two adjacent high-frequency paths into unfiltered cost volume features W. u ;

[0071] The second compression module is used to compress the cost volume features of two adjacent low-to-medium paths into filtered cost volume features W. f ;

[0072] The matrix interpolation module is used to calculate the unfiltered cost volume features W pixel by pixel. u and filtering cost body features W f The feature difference weight grid G ​​is used to obtain the semantic information difference at each pixel point, i.e. This indicates that the feature matrix difference is calculated pixel by pixel;

[0073] The activation function calculation module is used to activate the feature difference weight grid G, i.e., Relu(G);

[0074] The complement calculation module is used to calculate the complement R of the activated feature difference weight grid G. The complement R represents the feature similarity, i.e., R = W. c (Relu(G)), W c This means finding the complement of the matrix with respect to 1, and adding a local minimum value;

[0075] The feature matrix multiplication module is used to multiply the complement R with the filter cost volume features W. f Performing a multiplication operation yields the weight grid F, i.e., F = W. f ⊙R.

[0076] Specifically, in this exemplary embodiment, areas with identical spatial information are more likely to be the target region for reconstruction, while areas with different spatial information are more likely to be weakly correlated regions or noisy regions. Therefore, the spatial attention mechanism is implemented as follows: features of the same scale and adjacent depths in the network are compared pixel by pixel, a spatial information weight grid is calculated, and the spatial information weight grid is applied to the input of the next layer. Referring to the spatial information, the weight grid F filters the cost volume again.

[0077] More preferably, in an exemplary embodiment, both the first weight filtering module P1 and the second weight filtering module P2 include:

[0078] The multiplication calculation module is used to multiply the weight grid and the corresponding channel cost volume feature to obtain the weighted filtered cost volume feature of the corresponding channel.

[0079] Having the same inventive concept as the exemplary embodiments described above, another exemplary embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores computer instructions executable on the processor, and the processor executes the steps of the stereo matching method based on attention mechanism and noise suppression GRU when executing the computer instructions.

[0080] Electronic devices are manifested in the form of general-purpose computing devices. Components of electronic devices may include, but are not limited to: at least one processing unit, at least one storage unit, and a bus connecting different system components (including storage units and processing units).

[0081] The storage unit stores program code that can be executed by the processing unit, causing the processing unit to perform the steps described in the "Exemplary Methods" section above, based on various exemplary embodiments of the present invention. For example, the processing unit can perform actions such as... Figure 1 The method shown in the figure.

[0082] The storage unit may include readable media in the form of volatile storage units, such as random access memory (RAM) and / or cache storage units, and may further include read-only memory (ROM).

[0083] The storage unit may also include a program / utility having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0084] A bus can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus that uses any of the various bus structures.

[0085] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0086] Through the above description, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to this exemplary embodiment.

[0087] Having the same inventive concept as the above exemplary embodiments, another exemplary embodiment of the present invention provides a storage medium storing computer instructions thereon, which, when executed, perform the steps of the stereo matching method based on attention mechanism and noise suppression GRU.

[0088] Based on this understanding, the technical solution of this embodiment, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product (program product). The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0089] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0090] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0091] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0092] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0093] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A stereo matching method based on attention mechanism and noise suppression GRU, comprising the following steps: The reference image is used to extract features from the N-1 source images to obtain N high-channel feature maps; The N high-channel feature maps are subjected to homography transformation at the corresponding depth to obtain D groups of N high-channel feature bodies in each group. Then, the high-channel feature bodies in each group are aggregated into corresponding high-channel cost bodies C(0)~C(D-1) using a cost index based on variance, where D is the number of depths. The high-channel cost bodies C(0)~C(D-1) are regularized using the GRU module to obtain D cost maps, and the D cost maps are aggregated to obtain a probability body; Convert the probabilistic volume into a depth map to complete stereo matching; Its characteristic is that: features are extracted from N input 2D images using a reference image to obtain N high-channel feature maps, specifically: Feature extraction is performed using a feature extraction network that combines dilated convolution and attention mechanisms. The feature extraction network includes N sequentially connected dilated convolution and attention modules, a convolution concatenation module, and a two-dimensional convolutional layer. Each dilated convolution and attention module includes a sequentially connected dilated convolutional layer and a CBAM module. The process of using the GRU module to regularize the high-channel cost body C(0)~C(D-1) to obtain the probability body is as follows: The high-channel cost volumes C(0) to C(D-1) are optimized using a GRU network with added noise suppression. The final output 1-channel cost volumes Cr(0) to Cr(D-1) are normalized using a softmax operation to obtain the probability volume. The GRU network with added noise suppression includes: D GRU processing links, D-1 first spatial attention weighting modules A1, D-1 first weight filtering modules P1, D-1 second spatial attention weighting modules A2, and D-1 second weight filtering modules P2. Each GRU processing link includes a 2D convolution module, a first GRU processing module, a second GRU processing module, and a third GRU processing module. The first 2D convolution module maps the high-channel cost volume C(0) to the middle channel and then inputs it into the first GRU processing module of the first channel to obtain the first middle channel cost volume feature; the first middle channel cost volume feature is input into the second GRU processing module of the first channel to obtain the first low channel cost volume feature; the first low channel cost volume feature is input into the third GRU processing module of the first channel to obtain the first 1-channel cost volume Cr(0); After the Mth-way 2D convolutional module maps the high-channel cost volume C(M) to the middle channel, it inputs it into the first GRU processing module of the Mth-way to obtain the Mth middle channel cost volume feature; the M-1th middle channel cost volume feature and the Mth channel cost volume feature are respectively used as the first unfiltered cost volume feature W. u1 and the first filtering cost body feature W f1 The M1 weight grid is calculated by inputting it into the corresponding first spatial attention weighting module A1; the M1 weight grid and the Mth channel cost volume feature are input into the corresponding first weight filtering module P1 for weight filtering to obtain the Mth channel filtering cost volume feature; The filtered cost body feature of the Mth middle channel is input into the second GRU processing module of the Mth channel to obtain the cost body feature of the Mth low channel; the cost body feature of the (M-1)th low channel and the cost body feature of the Mth low channel are respectively used as the second unfiltered cost body feature W. u2 Second filter cost body feature W f2 The M2 weight grid is calculated by inputting it into the corresponding second spatial attention weighting module A2; the M2 weight grid and the M low-channel cost volume feature are input into the corresponding second weight filtering module P2 for weight filtering to obtain the M low-channel filtered cost volume feature; The cost body features of the (M-1)th low channel and the filtering cost body features of the Mth low channel are input into the third GRU processing module of the Mth channel to obtain the cost body feature Cr(M) of the M1st channel; M takes values ​​from 1 to D-1; Both the first spatial attention weighting module A1 and the second spatial attention weighting module A2 include: The first compression module is used to compress the cost volume features of two adjacent high-frequency paths into unfiltered cost volume features W. u ; The second compression module is used to compress the cost volume features of two adjacent low-to-medium paths into filtered cost volume features W. f ; The matrix interpolation module is used to calculate the unfiltered cost volume features W pixel by pixel. u and filtering cost body features W f The feature difference weight grid G ​​is used to obtain the semantic information difference at each pixel, i.e., G = W. u ⊕W f ⊕ indicates that the difference of the feature matrix is ​​calculated pixel by pixel; The activation function calculation module is used to activate the feature difference weight grid G, i.e., Relu(G); The complement calculation module is used to calculate the complement R of the activated feature difference weight grid G. The complement R represents the feature similarity, i.e., R = W. c (Relu(G)), W c This means finding the complement of the matrix with respect to 1, and adding a local minimum value; The feature matrix multiplication module is used to multiply the complement R with the filter cost volume features W. f Performing a multiplication operation yields the weight grid F, i.e., F = W. f ⊙R.

2. The stereo matching method based on attention mechanism and noise suppression GRU according to claim 1, characterized in that: The CBAM module includes a channel attention module and a spatial attention module connected in sequence.

3. The stereo matching method based on attention mechanism and noise suppression GRU according to claim 1, characterized in that: The N dilated convolution and attention modules are specifically three.

4. The stereo matching method based on attention mechanism and noise suppression GRU according to claim 1, characterized in that: Both the first weight filtering module P1 and the second weight filtering module P2 include: The multiplication calculation module is used to multiply the weight grid and the corresponding channel cost volume feature to obtain the weighted filtered cost volume feature of the corresponding channel.

5. The stereo matching method based on attention mechanism and noise suppression GRU according to claim 1, characterized in that: The high channel has 32 channels, the medium channel has 16 channels, and the low channel has 4 channels.

Images