A binocular disparity estimation method based on a twin network

Abstract

The application discloses a binocular disparity estimation method based on a twin network, and comprises the following steps: firstly, a binocular disparity estimation model is constructed; secondly, left and right images are input into the binocular disparity estimation model; the binocular disparity estimation model comprises a feature extraction network, a frequency domain attention module, an initial disparity estimation network and an iterative optimization network; the feature extraction network comprises a feature network and a context network; the output obtained after the left and right images are input into the feature network is input into the initial disparity estimation network; the initial disparity estimation network comprises an aggregated cost volume, a sandglass network and an n-layer grid search module L; the output of the feature extraction network is input into the frequency domain attention module; the iterative optimization network is composed of n-layer gate logic units; the frequency domain attention mechanism is introduced, and the processing capability of the context network for high-frequency information is significantly enhanced; meanwhile, the coverage range and accuracy of the algorithm are improved by improving a disparity regression strategy and introducing the concept of local maximum.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a binocular disparity estimation method based on Siamese networks. Background Technology

[0002] Binocular depth estimation, as an extension of target tracking, is crucial for acquiring the target's three-dimensional coordinate information and is a key step in achieving stereo tracking. By combining binocular parallax estimation and target tracking technology, more accurate and comprehensive target localization and tracking can be achieved, providing a solid foundation for intelligent vision systems and autonomous navigation.

[0003] Meanwhile, binocular disparity estimation technology, by analyzing image information acquired by binocular cameras, achieves 3D reconstruction and precise ranging of underwater objects, further enhancing the spatial perception capabilities of underwater operations. Therefore, with the increasing sophistication of marine scientific research, computer vision technologies such as target tracking and binocular disparity estimation will gradually become core technologies in the marine field, driving more innovation and breakthroughs in underwater operations. However, binocular disparity estimation still faces many challenges, such as illumination variations, texture loss, occlusion, and disparity discontinuities. These problems limit its performance and robustness in practical applications. Among them, illumination variations cause changes in image pixel intensity, affecting the accuracy of disparity estimation; texture loss makes feature points scarce and difficult to match; occlusion and disparity discontinuities further increase the complexity of depth information acquisition.

[0004] Underwater target depth estimation is the process of inferring the distance or depth information of objects in an underwater scene using underwater images or sensor data. This technology has significant applications in underwater robotics, underwater navigation, and marine science.

[0005] Vision-based depth estimation algorithms are a type of target depth estimation algorithm, which can be divided into monocular and binocular methods depending on the number of cameras. In monocular depth estimation, only one camera is needed to estimate the distance between the object and the camera from a single image, making it suitable for fields such as in-vehicle intelligent systems and smart homes. Since the true depth of underwater scenes is difficult to obtain directly, traditional supervised learning methods face challenges in training the network. To address this, Zhong et al. proposed a method that does not rely on supervised learning, inferring depth maps from real underwater stereo images. Their proposed learning process combines style adaptation in appearance space and feature adaptation in semantic space, progressively adjusting the depth estimation model to adapt to the special requirements of the underwater environment. However, monocular depth estimation is an ill-posed problem, essentially relying on fitting a dataset for depth inference, which leads to relatively weak scalability.

[0006] Binocular depth estimation, also known as stereo matching, is a technique that uses disparity information between images captured by two cameras to calculate the distance to objects. Compared to monocular depth estimation, binocular depth estimation typically provides more accurate depth information and is suitable for fields such as robot navigation and visual ranging systems. To ensure the accuracy of disparity estimation, binocular depth estimation usually requires the use of camera equipment of the same model and focal length. Wang Yiqiang et al. proposed the FRCNN-3D algorithm based on the Stereo R-CNN algorithm for 3D object detection in binocular vision. This algorithm is based on a stereo region convolutional neural network and solves the contradiction between the feature extraction network and the receptive field by introducing an attention-guided feature pyramid structure. It also establishes a new constraint relationship between the 3D center projection and the 2D center, thereby improving the accuracy of 3D object detection. In the area of ​​unsupervised learning, Skinner et al. proposed an underwater binocular depth estimation and color correction deep neural network based on unsupervised learning. This research, leveraging expertise in image processing, computer vision, and underwater light propagation, successfully completed the two tasks of disparity estimation and color correction in an unsupervised manner.

[0007] Inspired by the aforementioned literature and methods, we propose a binocular disparity estimation model based on Siamese networks. The main innovations are as follows:

[0008] This method utilizes a context network to extract additional features from either the left or right image during the feature extraction stage. The extracted feature maps are then used to iteratively optimize the network input. To enhance the context network's focus on high-frequency information, this study introduces a frequency domain attention mechanism, assigning different weights to information of different frequencies within a window, enabling the network to more effectively focus on key details. Furthermore, this study improves the Topk disparity regression strategy by introducing additional maximum disparity, resulting in better regression performance for the original Topk disparity regression strategy. Summary of the Invention

[0009] This invention proposes a binocular disparity estimation method based on Siamese networks. The algorithm framework consists of three core parts: a feature extraction network, an initial disparity estimation network, and an iterative optimization network. In the feature extraction network, a method using...

[0010] The ImageNet pre-trained MobileNetv2 backbone network serves as the feature extractor, ensuring size consistency when processing binocular image pairs. In the initial disparity estimation network, a cost volume is constructed using group correlation, and cost aggregation is performed through an hourglass network. An improved Topk disparity regression strategy is also introduced to enhance the accuracy of the initial disparity estimation. In the iterative optimization network, multi-level gating logic units are used to fine-tune the initial disparity. The implementation details of each component will be described in detail later.

[0011] The purpose of this invention is to provide a binocular disparity estimation method based on Siamese networks, comprising the following steps:

[0012] Obtain the left and right images; construct a binocular disparity estimation model; input the left and right images into the binocular disparity estimation model;

[0013] The binocular disparity estimation model includes a feature extraction network, a frequency domain attention module, an initial disparity estimation network, and an iterative optimization network;

[0014] The feature extraction network includes a feature network and a context network; the output obtained after processing by the feature network in the left and right images is fed into the initial disparity estimation network.

[0015] The initial disparity estimation network includes an aggregate cost body, an hourglass network, and an n-layer grid retrieval module L. The output of the feature extraction network enters the frequency domain attention module.

[0016] The iterative optimization network consists of n layers of gated logic units;

[0017] The left and right images are input into the feature network to obtain the left and right image features I. l,r The initial disparity map is obtained by aggregating the cost volume through the initial disparity estimation network.

[0018] Left and right image features I l,r After being processed by the aggregated cost body of the initial disparity estimation network, it enters the n-layer grid retrieval module L to obtain the relevant features of the n layers;

[0019] The input context network in either the left or right figure obtains multi-layer context features, which are then input into the frequency domain attention module for processing.

[0020] Using the iterative optimization network, based on the output of the frequency domain attention module and the relevant features after the n-layer grid retrieval module L, the initial disparity map is iteratively optimized to obtain the final disparity map.

[0021] Preferably, the feature network uses MobileNetv2 as the backbone network, extracting features at 1 / 4, 1 / 8, and 1 / 16 resolution of the input image in the first three stages of MobileNetv2, respectively; the adaptive spatial feature fusion (ASFF) method is used to fuse the features at the above three resolutions, and the number of channels is adjusted by 1×1 convolution to obtain the left and right image features I. l,r ;

[0022] The context network uses MobileNetv2 as the backbone network to extract multi-layer context features at 1 / 4, 1 / 8, and 1 / 16 resolutions from any one of the left or right images.

[0023] Preferably, the process of obtaining the initial disparity map specifically includes the following steps;

[0024] (i) Set the disparity value d, and combine the left and right image features I l,r Move d pixels in opposite directions respectively;

[0025] (ii) Perform correlation calculations on the features of the left and right images after displacement to obtain the matching cost under disparity d. The calculation formula is as follows:

[0026] C(d,x,y)= l (x,y),I g (xd,y)>

[0027] Where <.,.> represents the inner product operation, I l (x,y) represents the features of the left image, I g (xd,y) represents the right image feature moving d pixels in the opposite direction, where d represents the disparity value, and the range of d is from 0 to the set maximum disparity range.

[0028] (iii) The matching cost C(d,x,y) is divided into two groups according to the number of groups preset by the program. We perform a reshape operation and average the feature maps in each group to obtain the disparity matching cost for each disparity d.

[0029] The process from (i) to (iii) is repeated until the matching cost at all depths is obtained. Since the translation operation reduces the width of the image intersection, zero padding is required on the right side of the matching cost. Finally, all matching costs are concatenated to form an initial cost volume with a size of G×V×H / 4×W / 4, where G is the number of groups and V is the disparity range. The initial cost volume is processed by an hourglass network to effectively extract and integrate feature information, resulting in an aggregated cost volume with the same size as the initial cost volume.

[0030] After obtaining the aggregated cost body, a correlation pyramid is constructed based on the aggregated cost body, and a correlation pyramid is obtained through three average pooling operations with a disparity dimension step size of 2. In subsequent iterative optimization, the current state information required for each iteration is obtained by querying the correlation pyramid. Based on the aggregated cost body, an initial disparity map is obtained by processing it with an improved Topk disparity regression strategy.

[0031] Preferably, the iterative optimization network uses multi-level gated logic units (MGRUs) to finely adjust the initial disparity, and the process of obtaining the final disparity map includes the following steps:

[0032] ​The relevance pyramid obtained through the aggregated cost body is subjected to grid search to obtain the relevance features of all levels of the relevance pyramid as input. The contextual relevance features processed by the frequency domain attention module are also used as input. These two inputs are fed into the MGRU for processing to obtain the disparity compensation Δd, which is compared with the disparity d obtained in the previous iteration. t-1 Add them together to obtain the disparity d for this iteration. t After n iterations of MGRU optimization, the final disparity map is obtained.

[0033] Preferably, the processing procedure of the frequency domain attention module includes the following steps:

[0034] The frequency domain attention module first expands the input features into an 8×8 patch block. Then, it transforms the patch block to the frequency domain using FFT. In the frequency domain, a gated layer with learnable weight parameters is used to determine the weight of each frequency component to effectively capture lines and edges in the image. The weights are updated through backpropagation. Next, the weighted patch block is transformed back to the spatial domain using IFFT and folded back to its original size. Finally, the absolute value of the complex features is calculated to obtain the corresponding signal amplitude.

[0035] Preferably, the feature extraction network includes a residual layer, a 1×1 convolutional layer, and an activation function layer.

[0036] Preferably, in the process of iteratively optimizing the initial disparity map to obtain the final disparity map, the loss of the initial disparity map is calculated using the Smooth L1 loss function;

[0037] For the disparity map in the iterative optimization process, the L1 loss function is used to calculate the loss; the total loss function is shown in the following formula:

[0038]

[0039] Where N is the number of disparity estimation maps iteratively optimized, determined by the set number of iterations; λ N-i This represents the loss weight for the i-th iteration of optimization, with an initial value of 0.9, which gradually decreases as the number of iterations increases;

[0040] The Smooth L1 loss function is defined as follows:

[0041]

[0042] Preferably, in step (iii), the pre-set number of groups is two. This effectively improves the calculation...

[0043] Calculate efficiency.

[0044] Preferably, both the left and right images are pictures of underwater currents.

[0045] The effective effects of this invention are as follows:

[0046] This invention achieves significant results in addressing the challenges of binocular disparity estimation, aiming to substantially improve the accuracy and stability of stereo tracking. By combining deep learning and traditional computer vision methods, this research designs an efficient and robust binocular disparity estimation algorithm to address issues such as varying illumination, missing textures, occlusion, and disparity discontinuities. This invention is also particularly suitable for image processing of ocean currents or hydrothermal plumes. Compared to existing technologies, this invention is more adaptable to underwater environments with poor lighting, turbidity caused by seabed sediments, and occlusion.

[0047] This invention innovatively introduces a frequency domain attention mechanism, significantly enhancing the context network's ability to process high-frequency information, thereby improving the quality of feature extraction. Simultaneously, by improving the disparity regression strategy and introducing the concept of local maxima, the algorithm's coverage and accuracy are both improved. Extensive experimental comparisons on three authoritative datasets—SceneFlow, KITTI 2015, and ETH3D—fully validate the effectiveness and superior performance of the proposed algorithm. This research not only provides new research ideas and methods for the fields of depth perception and stereo vision but also lays a solid foundation for achieving more accurate and reliable target localization and tracking. Attached Figure Description

[0048] Figure 1 This is a network structure diagram of the binocular disparity estimation algorithm proposed in this invention;

[0049] Figure 2 This is the method for constructing the group-related cost body in this invention;

[0050] Figure 3 This is a visualization of a portion of the SceneFlow test set images in this invention;

[0051] Figure 4 This is a visualization of a portion of the KITTI 2015 test set images in this invention;

[0052] Figure 5 This is a visualization of a portion of the ETH3D test set images in this invention; Detailed Implementation

[0053] The invention will now be further described with reference to the accompanying drawings.

[0054] This invention proposes a binocular disparity estimation algorithm based on Siamese networks and gated recurrent units, the structure of which is as follows: Figure 1As shown, the algorithm framework consists of three core parts: a feature extraction network, an initial disparity estimation network, and an iterative optimization network. In the feature extraction network, a MobileNetv2 backbone network pre-trained on ImageNet is used as the feature extractor, ensuring size consistency when processing binocular image pairs. In the initial disparity estimation network, a cost volume is constructed using group correlation, and cost aggregation is performed through an hourglass network. An improved Topk disparity regression strategy is also introduced to enhance the accuracy of the initial disparity estimation. In the iterative optimization network, multi-level gating logic units are used to fine-tune the initial disparity. The implementation details of each component will be described in detail later.

[0055] The method includes the following steps:

[0056] Step 1: Feature Extraction Network

[0057] The feature extraction network structure in this study is similar to that of the GRU-based binocular disparity estimation algorithm, also comprising a feature network and a context network. In the feature network, the first three stages of MobileNetv2 are used to extract features at 1 / 4, 1 / 8, and 1 / 16 resolutions, respectively. To effectively fuse features of different sizes output from different layers, this study employs an adaptive spatial fusion method, fusing features at three different scales and adjusting the number of channels through 1×1 convolutions. Finally, the left and right image features I are obtained. l,r (where I∈R) C×H / 4×W / 4 (C=32).

[0058] Although the context network and feature network use the same MobileNetv2 architecture as their backbone, their functions differ. The context network focuses on extracting multi-layer contextual features at 1 / 4, 1 / 8, and 1 / 16 resolutions of the left image. These features do not directly participate in the construction of the cost volume but play a role in the subsequent GRU disparity iterative optimization. Furthermore, this study also improves some of the MV2 block structures in the context network.

[0059] The feature network and context network differ significantly in their tasks, therefore their weights are independent rather than a twin structure. This means they each learn feature representations adapted to their respective tasks without interfering with each other. The feature network focuses on extracting features suitable for constructing the cost volume, while the context network focuses on extracting contextual information from the left image for subsequent disparity iteration optimization. This independent learning approach ensures that they can optimize for their respective tasks without interfering with or confusing the task objectives.

[0060] Step 2: Frequency Domain Attention Mechanism

[0061] In image analysis, the Fast Fourier Transform (FFT) is often used to transform images from the spatial domain to the frequency domain for in-depth frequency domain analysis. The approach to handling high-frequency and low-frequency information depends on the specific needs: high-pass filters are used to enhance high-frequency features such as texture, details, and edges; conversely, low-pass filters are preferred to eliminate high-frequency noise. In visual images, high-frequency information typically refers to rapidly changing, detail-rich parts, such as texture and edges. In contextual networks, the extracted features are used as input to the iteratively optimized GRU as information for the hidden state, which is crucial for disparity thinning. Therefore, contextual networks should focus more on and enhance the high-frequency components of the features.

[0062] To enhance the attention given to high-frequency features by the context network, this study introduces a frequency domain attention mechanism. This research draws on the idea of ​​frequency-domain-based discriminative feedforward networks and transforms it into a frequency domain attention mechanism. This mechanism is applied to the MV2 block in the MobileNetV2 backbone structure of the context network, and the MV2 block with the frequency domain attention mechanism is called the FMV2 block. The FMV2 block adds a frequency domain attention module (FA) to the original MV2 block. The FA consists of a Fast Fourier Transform (FFT) layer, a weighted gating layer, and an Inverse Fourier Transform (IFFT) layer. Operationally, the input features are first patched and expanded to a size of 8×8. Then, the patch block is transformed to the frequency domain using FFT. In the frequency domain, a gating layer with learnable weight parameters is used to determine the weight of each frequency component to effectively capture lines and edges in the image. These weights are updated through backpropagation learning of the network. Next, the weighted patch block is transformed back to the spatial domain using IFFT and folded back to its original size. Finally, the absolute value of the complex features is calculated to obtain the corresponding signal amplitude.

[0063] Step 3: Initial disparity estimation

[0064] Obtaining an initial disparity estimate is a crucial step before performing iterative disparity optimization. This involves three core steps: cost volume construction, cost volume aggregation, and disparity regression. This process is similar to that of MobileStereoNet-2D, and we will explore these steps in detail below.

[0065] (3.1) Cost Body Construction

[0066] In the binocular disparity estimation problem, the cost volume is used to measure the degree of matching and similarity between different pixels, and its construction method is a key technique in the binocular disparity problem. This study uses a group correlation method to construct the initial cost volume, and the specific process is as follows: Figure 2 As shown.

[0067] First, a disparity value d is set, and the left and right feature maps are shifted by d pixels in opposite directions. Then, relevant calculations are performed on the shifted feature maps to obtain the matching cost under the disparity d. The specific formulas for these calculations are as follows:

[0068] C(d,x,y)= l (x,y),I g (xd,y)> (1)

[0069] Where <.,.> represents the inner product operation, I l (x,y) represents the features of the left image, I g (xd,y) represents the right image feature shifted d pixels in the opposite direction, where d represents the disparity value, ranging from 0 to the set maximum disparity range. The matching cost C(d,x,y) is divided into two groups according to the pre-defined number of groups, and reshaped. The feature maps within each group are averaged to obtain the disparity matching cost at each depth. This process is repeated until the matching cost for all disparities d is obtained. Since the shift operation reduces the width of the image intersections, zero-padding is needed on the right side of the matching cost. Finally, all matching costs are concatenated to form the initial cost volume, with a size of G×V×H / 4×W / 4, where G is the number of groups and V is the disparity range. This cost volume will be further refined in subsequent iterations to optimize the network.

[0070] After obtaining the initial disparity cost volume, this study, drawing inspiration from the RAFT-Stereo method, constructs a relevance pyramid based on this cost volume. A three-layer relevance pyramid is generated through three average pooling operations with a disparity dimension step size of 2. This structure plays a crucial role in subsequent iterative optimizations, as the current state information required for each iteration is obtained by querying the relevance pyramid.

[0071] (3.2) Cost aggregation

[0072] While the cost volume constructed through group correlation integrates features from both left and right images and information at different disparity levels, it is limited to a local context and lacks non-local contextual information. To strengthen the connections between adjacent pixels and achieve effective cost aggregation, this study introduces a Stacked Hourglass network. With its multi-level structure, the Stacked Hourglass network can process input multiple times at different scales, accurately capturing the correlation information between global and local data, thereby significantly improving the cost aggregation effect.

[0073] ​The hourglass network consists of a pre-hourglass block and an hourglass block. The pre-hourglass block is implemented by a 3×3×3 3D convolutional layer with a stride of 1 and the same input and output channels. This network performs multi-level feature extraction and abstract representation on the input data, laying the foundation for the efficient utilization of the hourglass network. The structure of the hourglass block is similar to UNet, being an encoding-decoding structure. In the encoding part, the hourglass network downsamples high-resolution features to low resolution through 3D convolutions with a stride of 2. In the decoding part, the hourglass network concatenates features from the same layer through skip connections and then upsamples them through 3D deconvolution. Through such bottom-up and top-down operations, the hourglass network can effectively extract and integrate feature information. Finally, the hourglass cost aggregation module outputs an aggregated cost volume with the same size as the input cost volume, providing strong support for subsequent disparity regression tasks.

[0074] (3.3) Parallax Regression

[0075] Before performing disparity regression, this study first reduces the number of channels to 1 using a 3×3×3 3D convolution. Regarding the disparity regression method, this study improves upon CoEx's Topk disparity regression strategy by introducing an additional maximum confidence level. Specifically, this study uses 1D max pooling to find local maxima confidence levels along the disparity dimension and compares them with the unpooled cost volume to filter out the true maxima. Subsequently, the Topk algorithm is used to select the k largest maxima, and the largest value among them is excluded (the maximum maxima is also the global maximum). To ensure that the newly introduced maximum confidence level is neither too small nor overlaps with the original Topk confidence level, this study sets a specific confidence level range, namely [0.5×c k ,c k ), where c k This is the k-th maximum confidence score. If no maximum confidence score that meets the criteria can be found, the algorithm degenerates into the traditional Topk method, introducing the (k+1)-th maximum value. This improvement aims to enhance the accuracy and robustness of disparity regression.

[0076] Step 4: Iteratively optimize the network

[0077] Iterative optimization starts with an initial disparity estimate d0 and progressively estimates a series of disparity values. The task of updating the disparity values ​​is implemented by a multi-layer gated logic unit (MGRU). Structurally, the MGRU consists of three convolutional GRUs at different levels, which process feature maps with input image resolutions of 1 / 16, 1 / 8, and 1 / 4 of the original image, respectively.

[0078] (4.1) Input

[0079] In the first iteration of optimization, the multi-layer context features extracted by the context feature network are processed by the tanh activation function and used as the initial hidden state h of the GRU. t-1 Simultaneously, the reset gating weights W are calculated using three 3×3 convolution operations. r Update the gating weights W z and hidden state weight W h Initial input information x t This is achieved by analyzing the hidden states h of the adjacent GRU layers. t-1 The hidden state h is obtained through downsampling or upsampling. In subsequent iterative optimizations, the hidden state h... t-1 It will be continuously updated, and W r W z and W h It remains unchanged after initialization.

[0080] During the disparity update process, relevant features are also obtained as input by retrieving all levels of the relevant pyramid through a grid. To construct a grid with integer offsets, the offsets are determined based on the current level of the feature pyramid. For example, for a pixel u = (x, y) in the left image at 1 / 4 resolution, the horizontal axis offset of the grid can be defined as:

[0081]

[0082] Among them, f 1 (x, y) represents the disparity value of pixel u under the current disparity estimation, and l represents the pyramid level. Furthermore, the deeper the current pyramid level, the more 1D average pooling layers it passes through, and the smaller the corresponding offset.

[0083] (4.2) Update

[0084] In each iteration, the GRU-16 is first updated, and the hidden state h is obtained. t16 The data will be upsampled and concatenated with the downsampled initial hidden states of GRU-04 and GRU-08 to form the hidden state input h of GRU-08. t8-1 For GRU-04, it first uses an encoder to combine disparity features, correlation features, and the initial hidden state h. t4-1 Generate motion features m t Next, m t Compared with the upsampled hidden state h of GRU-08 t8 And the hidden state of GRU-04 itself h t4-1 The concatenation is used as the input for the hidden state of GRU-04. Finally, the hidden state h output by GRU-04 is... t4 The disparity compensation Δd is obtained after convolution, and compared with the disparity d obtained in the previous iteration.t-1 Add them together to obtain the disparity d for this iteration. t .

[0085] The internal implementation of GRU is consistent across different levels. It receives the current input information x. t The hidden state h passed from the previous node t-1 As input, output the current hidden node y. t and the hidden state h that needs to be passed to the next node t The main calculations within a single-layer GRU are shown in equations (3) to (6).

[0086] r t =σ(W r ·Conv([h t-1 ,x t ])) (3)

[0087] z t =σ(W z ·Conv([h t-1 ,x t ])) (4)

[0088] h ′ t =tanh(W h ·Conv([r t ⊙h t-1 ,x t ])) (5)

[0089] h t =(1-z) t )⊙h t-1 +z t ⊙h ′ t (6)

[0090] Here, σ is the sigmoid function, whose output range is [-1, 1], similar to the tanh function. Conv represents the convolution operation, replacing the original fully connected layer. First, through the hidden state h... t-1 and input information x t Calculate the gate r for control reset separately t and the gating z that controls updates t As shown in formulas (3) and (4). Next, using the reset gating r calculated in the previous step... t "Reset" hidden state h t ′ That is, ignoring some historical information and then comparing it with the current input information x. t The concatenation is followed by the tanh activation function to obtain the candidate hidden state h. ′As shown in formula (5). Finally, using z t For h ′ and h t-1 Perform weighted fusion to obtain the updated hidden state h. t As shown in formula (6), where (1-z) t )⊙h t-1 This represents selective "forgetting" of the previous hidden state, z t ⊙h t ′ This represents a selective "memory" of the current candidate hidden state.

[0091] Step 5: Loss Function

[0092] In this study, the Smooth L1 loss function was used to calculate the loss of the initial disparity estimation map, while the L1 loss function was used to calculate the loss for all iteratively optimized disparity estimation maps. The total training loss function is shown in Equation (7):

[0093]

[0094] Where N is the number of disparity estimation maps iteratively optimized, determined by the set number of iterations. λ N-i Let represent the loss weight for the i-th iteration, initially set to 0.9, and gradually decrease as the iteration number increases. The Smooth L1 loss function is defined as follows:

[0095]

[0096] Step 6: Comparative Experiment

[0097] This study conducted comparative experiments on the Scene Flow, KITTI 2015, and ETH3D datasets, comparing the proposed binocular disparity estimation algorithm with nine existing high-performing algorithms: GCNet, GwcNet, PSMNet, MobileStereoNet-2D, MobileStereoNet-3D, PCW-Net, RAFT-Stereo, IGEV, and DiffuVolume. On the ETH3D dataset, the comparison method was adjusted, selecting six high-performing algorithms for comparison: CFNet_RVC, FC-GANet, GwcNet, DANet, RAFT-Stereo, and IGEV. Among these, RAFT-Stereo and IGEV are two high-performing stereo matching algorithms based on GRU iterative optimization.

[0098] (1) Scene Flow

[0099] This study conducted tests on the test dataset in the Scene Flow dataset, and the specific results are shown in Table 1. Considering that the original RAFT-Stereo paper did not provide test results on the Flythings3D dataset, this study re-performed the experiments based on the training weights provided in the original paper.

[0100] As shown in Table 1, due to the ample training samples provided by the Scene Flow dataset, the endpoint errors of all algorithms remained within 2 pixels. Among them, the DiffuVolume disparity estimation algorithm achieved the best estimation performance on the Scene Flow dataset, with an endpoint error of only 0.46 pixels. In comparison, the disparity estimation algorithm proposed in this paper, while slightly inferior, still achieved an endpoint error of 0.58 pixels on the same dataset, demonstrating commendable performance. Furthermore, the lightweight disparity estimation algorithms MobileStereoNet-2D and MobileStereoNet-3D had endpoint errors of 1.14 pixels and 0.80 pixels, respectively, showing a significant difference compared to the algorithm presented in this paper. Overall, the binocular disparity estimation algorithm proposed in this paper exhibits a good level of disparity estimation.

[0101] Table 1 Evaluation results of different binocular disparity estimation algorithms on the SceneFlow and KITTI 2015 datasets.

[0102]

[0103] (2) KITTI 2015

[0104] Table 1 presents the test results of each binocular disparity estimation algorithm on the KITTI 2015 dataset. The results show that the DiffuVolume and IGEV algorithms still perform well in disparity estimation. Compared with the RAFT-Stereo algorithm, which also uses GRU optimization, the proposed disparity estimation method demonstrates superior performance on several key evaluation metrics, specifically a 0.19% reduction in D1-bg, a 0.63% reduction in D1-fg, and a 0.29% reduction in D1-all. This result fully demonstrates the effectiveness and significant performance advantage of the proposed method on the KITTI 2015 dataset.

[0105] (3)ETH3D

[0106] Table 2 details the disparity estimation performance of our proposed algorithm on the ETH3D dataset. The results show that our algorithm ranks fourth in the comparison. On the key metric of mean disparity error (AvgErr), our algorithm achieves 0.22 pixels, which is only 0.08 pixels less than the best IGEV algorithm. Simultaneously, our algorithm also performs well on the Bad metric, with three evaluation values ​​of 2.88%, 0.85%, and 0.28%, respectively. The RAFT-Stereo algorithm also achieved impressive results, successfully controlling its mean disparity error to within 0.20 pixels.

[0107] Table 2 Evaluation results of different binocular disparity estimation algorithms on the ETH3D dataset

[0108]

[0109] Step 7: Qualitative Analysis

[0110] The disparity estimation results of the RAFT-Stereo, IGEV, and the binocular disparity estimation algorithm proposed in this study are visualized. The scenarios used in these visualizations are derived from the three datasets used in the comparative experiments.

[0111] Because the Scene Flow dataset contains a large number of training samples, it provides ample training opportunities for the models. In addition, the image styles of its training set and test set are consistent, so all three algorithms show excellent disparity estimation performance on this dataset. Figure 3 The five rows of images represent the original image from the left camera, the true disparity map, the IGEV predicted disparity map, the RAFT-Stereo predicted disparity map, and the disparity map predicted by the algorithm in this paper. It is noteworthy that in the third column of images, the disparity values ​​predicted by all three methods are generally higher than the true disparity, especially in the object region in the middle of the left eye image.

[0112] Visualization of the KITTI 2015 dataset Figure 4 The four rows of images in the image represent the original image from the left camera, the IGEV predicted disparity map, the RAFT-Stereo predicted disparity map, and the disparity map predicted by the algorithm in this paper, respectively. It can be observed that the RAFT-Stereo algorithm is slightly lacking in handling details, such as wires. For the ETH3D dataset (…),… Figure 5 The differences among the three algorithms are more pronounced. In the first set of images, RAFT-Stereo's disparity estimation for seats and statues is lower than the other two algorithms, indicating a performance degradation under changing lighting conditions. In the second set of images, the IGEV algorithm underestimates the disparity of objects near walls. In the third set of images, RAFT-Stereo exhibits a similar problem to the first set. Overall, the method proposed in this invention demonstrates greater stability and robustness across various scenarios.

Claims

1. A binocular disparity estimation method based on Siamese networks, characterized in that, Includes the following steps: Obtain the left and right images; construct a binocular disparity estimation model; input the left and right images into the binocular disparity estimation model; The binocular disparity estimation model includes a feature extraction network, a frequency domain attention module, an initial disparity estimation network, and an iterative optimization network; The feature extraction network includes a feature network and a context network; the output obtained after processing by the feature network in the left and right images is fed into the initial disparity estimation network. The initial disparity estimation network includes an aggregate cost body, an hourglass network, and an n-layer grid retrieval module L. The output of the feature extraction network enters the frequency domain attention module. The iterative optimization network consists of n layers of gated logic units; The left and right images are input into the feature network to obtain the left and right image features. The initial disparity map is obtained by aggregating the cost volume through the initial disparity estimation network. Left and right image features After being processed by the aggregated cost body of the initial disparity estimation network, it enters the n-layer grid retrieval module L to obtain the relevant features of the n layers; The input context network in either the left or right figure obtains multi-layer context features, which are then input into the frequency domain attention module for processing. Using the iterative optimization network, based on the output of the frequency domain attention module and the relevant features after the n-layer grid retrieval module L, the initial disparity map is iteratively optimized to obtain the final disparity map; The process of obtaining the initial disparity map specifically includes the following steps; (i) Set the disparity value d and combine the features of the left and right images. Move d pixels in opposite directions respectively; (ii) Perform correlation calculations on the features of the left and right images after displacement to obtain the matching cost under disparity d. The calculation formula is as follows: ； in, This represents the inner product operation. Indicate the features of the left image, This indicates that the right image feature moves d pixels in the opposite direction, where d represents the disparity value, and the range of d is from 0 to the set maximum disparity range. (iii) Matching cost Based on the number of groups preset by the program, a reshape operation is performed, and the feature maps within each group are averaged to obtain the disparity matching cost for each d disparity. The process from (i) to (iii) is repeated until the matching cost at all depths is obtained; zero-padding is performed on the right side of the matching cost; finally, all matching costs are concatenated to form an initial cost volume with a size of G×V×H / 4×W / 4, where G is the number of groups and V is the disparity range; after the initial cost volume is processed by an hourglass network, an aggregated cost volume with the same size as the initial cost volume is obtained. After obtaining the aggregated cost body, a correlation pyramid is constructed based on the aggregated cost body, and a correlation pyramid is obtained through three average pooling operations with a disparity dimension step size of 2. In subsequent iterative optimization, the current state information required for each iteration is obtained by querying the correlation pyramid. Based on the aggregated cost body, an initial disparity map is obtained by processing it with an improved Topk disparity regression strategy. The processing procedure of the frequency domain attention module includes the following steps: The frequency domain attention module first expands the input features into an 8×8 patch block. Then, it transforms the patch block to the frequency domain using an FFT. In the frequency domain, a gated layer with learnable weight parameters is used to determine the weight of each frequency component. The weights are updated through backpropagation. Next, the weighted patch block is transformed back to the spatial domain using an IFFT and folded back to its original size. Finally, the absolute value of the complex features is calculated to obtain the corresponding signal amplitude.

2. The binocular disparity estimation method based on Siamese networks as described in claim 1, characterized in that, The feature network uses MobileNetv2 as the backbone network, and extracts features of 1 / 4, 1 / 8 and 1 / 16 resolution of the input image in the first three stages of MobileNetv2, respectively. The adaptive spatial feature fusion (ASFF) method is used to fuse the features from the three resolutions mentioned above, and the number of channels is adjusted through 1×1 convolution to obtain the features of the left and right images. ; The context network uses MobileNetv2 as the backbone network to extract multi-layer context features at 1 / 4, 1 / 8, and 1 / 16 resolutions from any one of the left or right images.

3. The binocular disparity estimation method based on Siamese networks as described in claim 1, characterized in that, The iterative optimization network uses multi-level gated logic units (MGRUs) to finely adjust the initial disparity. The process of obtaining the final disparity map includes the following steps: The relevance pyramid obtained through aggregated cost volume is subjected to grid search to obtain the relevance features of all levels of the relevance pyramid as input. The contextual relevance features processed by the frequency domain attention module are also used as input. These two inputs are fed into the MGRU for processing to obtain disparity compensation. The disparity obtained from the previous iteration Add them together to get the disparity for this iteration. After n iterations of MGRU optimization, the final disparity map is obtained.

4. The binocular disparity estimation method based on Siamese networks as described in claim 1, characterized in that, The feature extraction network includes a residual layer, a 1×1 convolutional layer, and an activation function layer.

5. The binocular disparity estimation method based on Siamese networks as described in claim 1, characterized in that, In step (iii), the pre-set number of groups is two.

6. The binocular disparity estimation method based on Siamese networks as described in claim 1, characterized in that, Both the left and right images are pictures of underwater currents.

Images