A large disparity lightweight depth estimation method and system based on global search

Abstract

This invention discloses a lightweight depth estimation method and system based on global search with high disparity. The method first acquires and preprocesses left and right views to form a stereo image pair. Next, the stereo image pair is input into a feature extraction network to extract multi-scale feature tensors, including 1 / 8 and 1 / 4 scale feature tensors, and a context feature tensor. Then, based on the 1 / 8 scale feature tensor, a global matching cost volume is constructed, and a coarse disparity map is calculated. The initial disparity map is then repaired and upsampled to the 1 / 4 scale. A three-scale matching cost volume is constructed with reference to the 1 / 4 scale feature tensor and input into a bilateral aggregation network for regression optimization. Finally, a convex upsampling module restores the regression-optimized disparity map to the original image resolution, and the depth image is calculated by combining the stereo parameters. This invention provides smooth repair for physically occluded and weakly textured regions, outputting high-precision estimation results with rich details and sharp depth edges.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and graphics technology, and in particular to a lightweight depth estimation method and system based on global search with large disparity. Background Technology

[0002] In computer vision and graphics, depth estimation is one of the key technologies for achieving 3D spatial perception and reconstruction. In a binocular vision system, images of the same object are captured by two cameras placed at different horizontal positions. The disparity is obtained based on the difference in the object's projected position on the imaging planes of the two cameras. Then, based on the principle of triangulation, combined with known camera baseline distances and disparity information, the precise depth information of the object is calculated. This technology has strong application prospects in robotics, autonomous driving, and augmented reality.

[0003] With the development of deep learning, current stereo matching methods are mainly divided into cost volume aggregation-based methods and iterative optimization-based methods. The former (such as GCNet and PSMNet) uses 3D convolution to regularize the cost volume. Although the accuracy is high, the computational cost and memory usage are huge, making it difficult to run in real time. The latter (such as RAFT-Stereo) uses recurrent units (GRUs) to gradually update the disparity. Although the efficiency is improved, it still consumes considerable computational resources on high-resolution images.

[0004] For scenarios with limited computing resources, while the industry has proposed lightweight networks such as MobileStereoNet, attempting to trade speed for reduced resolution or simplified network structure, these methods often have significant limitations when facing complex scenarios. Specifically, existing lightweight methods typically lack an effective global search mechanism, relying mainly on matching based on local correlations within a limited range. This local search strategy has inherent flaws when facing scenarios with large parallax. Once the disparity value of the target exceeds the preset local search window, the network cannot capture the correct correspondence, easily getting trapped in local optima, leading to widespread matching failures or incorrect depth estimations, making it difficult to meet the perception requirements of large dynamic range scenarios in practical applications. Summary of the Invention

[0005] To address this, the present invention provides a lightweight depth estimation method and system for large disparity based on global search, which solves the problems of disparity truncation caused by the limited local search range in existing lightweight depth estimation methods, and the problem of easily getting trapped in local optima in large disparity scenarios, resulting in matching failure and inaccurate disparity estimation.

[0006] To address the aforementioned problems, this invention provides a lightweight depth estimation method with large disparity based on global search, the method comprising the following steps: S1. Use a binocular image acquisition device to capture the target scene, obtain left and right view images, and preprocess the left and right view images to form a binocular stereo image pair.

[0007] S2. Input the stereo image to the feature extraction network to extract the multi-scale feature tensor for matching and the context feature tensor for guidance.

[0008] S3. Based on the multi-scale feature tensor, construct the global matching cost volume and calculate the coarse disparity map. Use the gated axial attention module in combination with the context feature tensor to repair the coarse disparity map and generate the initial disparity map.

[0009] S4. Upsample the initial disparity map to a 1 / 4 scale, construct a three-scale matching cost body covering different search ranges by referring to the multi-scale feature tensor, and input it into the bilateral aggregation network for residual regression optimization, and output the optimized disparity map.

[0010] S5. Use the convex upsampling module to restore the optimized disparity map to the resolution of the left view image to obtain the full-resolution final disparity map. Combine the binocular parameters to calculate the depth image.

[0011] Further, in step S1, the preprocessing of the left and right view images specifically includes: The original left and right views are subjected to pixel distribution normalization processing; Based on the downsampling factor of the subsequent feature extraction network, the normalized image tensor is zero-padded at the edges to ensure that its target size meets the divisibility requirement of the receptive field of the feature extraction network, thereby avoiding the loss of edge features.

[0012] Further, in step S2, the feature extraction network specifically includes: A lightweight feature extraction network based on the inverse residual module is constructed using MobileNetV2 to extract original image features at different levels.

[0013] The FPN network is used to perform multi-scale information fusion on the original image features at different levels. Deep semantics are injected into shallow features through a top-down path, enhancing the discriminative power for weakly textured regions and eliminating pixel aliasing. A 1 / 8-scale feature tensor for global search and a 1 / 4-scale feature tensor for local fine-tuning are obtained from the features extracted by the FPN network, forming a multi-scale feature tensor set. Similarly, 1 / 16-scale, 1 / 8-scale, and 1 / 4-scale feature tensors corresponding to the left view are obtained from the features extracted by the FPN network, forming a context feature tensor set.

[0014] Further, in step S4, the construction of a three-scale matching cost body covering different search ranges using the reference multi-scale feature tensor specifically includes: First, the initial disparity map generated in step S3 is upsampled to a 1 / 4 scale as a prior disparity to guide subsequent local searches; Secondly, based on the 1 / 4 scale feature tensor in the multi-scale feature tensor, with the prior disparity as the center, a group correlation algorithm is used and feature matching is performed by setting different sparse sampling step sizes under three sampling index radii from small to large, to construct a three-scale cost volume tensor that respectively covers fine alignment, medium bias correction and wide recall. Finally, the three-scale cost volume tensors corresponding to all step sizes are concatenated along the channel dimension to obtain a high-dimensional aggregated input tensor that reflects multi-scale matching information.

[0015] Further, in step S4, the input bilateral aggregation network performs residual regression optimization and outputs an optimized disparity map, specifically including: The feature space of the high-dimensional aggregated input tensor is decoupled into detail branches and smooth branches using a scale-aware spatial attention mechanism; The detail branches and smooth branches are processed by independent aggregation sub-networks. The aggregation sub-networks all adopt the U-Net network structure that includes an encoder, a decoder and skip connections, and introduce a guided excitation module at each scale level to modulate the cost features using image texture features. After the two branches are independently optimized and fused using spatial attention tensors, the Top-k unified soft minimum algorithm is used to select several candidate points with the highest probability in the disparity dimension, perform local normalization to obtain the expectation, regress to obtain the disparity residual, and combine it with the prior disparity to output the optimized disparity map.

[0016] Further, in step S5, the process of restoring the optimized disparity map to the resolution of the left view image using the convex upsampling module to obtain the full-resolution final disparity map, and calculating the depth image by combining the binocular parameters, specifically includes: First, the disparity map optimized in step S4 is input into a convex combination upsampling network based on local spatial structure feature learning and used to predict pixel mapping weights, so as to obtain the upsampling weight tensor corresponding to the full resolution of the left view image. Then, using the upsampling weight tensor, each pixel value in the full-resolution disparity map is subjected to convex weighted sum processing through its corresponding neighborhood pixel set in the low-resolution space, so as to accurately reconstruct the full-resolution disparity value and thereby overcome the depth edge blurring phenomenon caused by traditional interpolation. Then, based on the principle of triangulation, the obtained full-resolution final disparity map is converted into a depth image tensor in physical space according to the physical baseline and equivalent focal length of the binocular camera.

[0017] Based on the same inventive concept as the aforementioned method, this invention also provides a lightweight depth estimation system with large disparity based on global search. The system is used to execute the above-mentioned depth estimation method and includes: a binocular stereo image pair module, a feature extraction network module, an initial disparity map module, a disparity map optimization module, and a depth image estimation and output module. The binocular stereo image pair module uses a binocular image acquisition device to capture the target scene, obtain left and right view images, and preprocess the left and right view images to form a binocular stereo image pair.

[0018] The feature extraction network module is used to input the stereo image into the feature extraction network and extract multi-scale feature tensors for matching and context feature tensors for guidance.

[0019] The initial disparity map module is used to construct a global matching cost volume and calculate a coarse disparity map based on a multi-scale feature tensor. The gated axial attention module is used in conjunction with the context feature tensor to repair the coarse disparity map and generate the initial disparity map.

[0020] The disparity map optimization module is used to upsample the initial disparity map to a 1 / 4 scale, construct a three-scale matching cost body covering different search ranges with reference to multi-scale feature tensors, and input it into a bilateral aggregation network for residual regression optimization, outputting the optimized disparity map.

[0021] The depth image estimation and output module uses the convex upsampling module to restore the optimized disparity map to the resolution of the left view image, obtaining the full-resolution final disparity map. It then calculates the depth image by combining the binocular parameters and transmits the result to the display terminal for display.

[0022] This invention provides a lightweight depth estimation method and system based on global search with large disparity, which has the following advantages: 1. By performing global matching at a low resolution of 1 / 8 and combining it with a gated axial attention module, the system effectively captures geometry across the entire parallax range. The system can adaptively and smoothly repair physically occluded and weakly textured areas, significantly reducing the probability of large-area matching failures in complex scenes.

[0023] 2. A lightweight backbone network is used for feature extraction and multi-scale fusion, which greatly reduces the computational overhead and memory usage of the model, enabling the system to be easily deployed on embedded hardware with limited computing resources and meet the requirements of real-time inference.

[0024] 3. In the residual regression stage, a Top-k unified soft minimum algorithm is introduced to accurately filter long-tailed noise interference; in the post-processing stage, a convex combined upsampling network is used to utilize local dynamic weighting and recover full-resolution parallax. This combination effectively overcomes the edge blurring phenomenon during depth map magnification, and finally outputs high-precision prediction results with rich details and sharp depth edges. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly described below. Referring to the accompanying drawings will provide a clearer understanding of the features and advantages of the present invention. The drawings are illustrative and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort. Wherein: Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the device system structure according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the overall network structure according to an embodiment of the present invention; Figure 4 This is the qualitative result of the Middlebury3 test set in the embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] To address the problem that current lightweight stereo matching methods are prone to getting stuck in local optima when faced with large parallax scenes, this embodiment provides a lightweight depth estimation method and system for large parallax based on global search.

[0028] In practical physical deployments, this system can run on resource-constrained embedded devices or edge computing platforms. The system front-end connects to a binocular image acquisition device, and the physical baseline of the binocular camera is set to... The equivalent focal length is set to The system backend connects to a display terminal to output and render high-precision 3D depth information.

[0029] like Figure 1The diagram shown is a flowchart of a lightweight depth estimation method with large disparity based on global search provided in an embodiment of the present invention. Simultaneously, combined with... Figure 3 The diagram shown illustrates the overall network structure of the present invention. The depth estimation method in this embodiment specifically includes the following steps: S1. Use a binocular image acquisition device to capture the target scene, obtain left and right view images, and preprocess the left and right view images to form a binocular stereo image pair.

[0030] Specifically, the scene is captured using a stereo camera with epipolar correction to obtain the original left view. and right view The image is normalized to ensure its pixel distribution follows a standard normal distribution. Then, to accommodate the receptive field requirements of the subsequent feature extraction network (32x downsampling) and to avoid edge feature loss, the normalized tensor is zero-padded. Let the original image size be... Target size after filling satisfy and The preprocessed network input tensor is obtained. .

[0031] S2. Input the stereo image to the feature extraction network to extract the multi-scale feature tensor for matching and the context feature tensor for guidance.

[0032] Specifically, a lightweight feature extraction network based on inverse residual modules is constructed using MobileNetV2 to simultaneously extract high-level semantic features and low-level texture features on computationally limited embedded devices. This network is designed for the input feature tensor of any layer. First, a nonlinear mapping function consisting of up-dimensional convolution, depthwise separable convolution, and down-dimensional convolution is used. The specific feature transformation process is represented as follows: In the formula, the first For dimensionality-increasing convolution operations used to increase feature dimensions, For depthwise separable convolution operations used to extract spatial features, the second For dimensionality reduction convolution operations used to compress dimensions, and This indicates the ReLU6 activation function. Subsequently, a Feature Pyramid Network (FPN) is used to fuse multi-scale information, injecting deep semantics into shallow features through a top-down path to enhance the discriminative power for weakly textured regions. The formula is as follows: In the formula, To unify the number of channels in convolution operations for shallow and deep features, Convolution operations are used to smooth the fused features and eliminate pixel aliasing caused by upsampling; and These are the shallow and deep feature tensors, respectively. This is a bilinear upsampling operation.

[0033] After the multi-scale fusion process described above, the feature extraction network finally generates three sets of key features to serve subsequent tasks: the first is the left feature tensor at the 1 / 8 scale, which has a large number of channels and a large receptive field. and left feature tensor It will be sent to step S3 for a global search; the next step is to use the left feature tensor at a higher resolution and richer texture details at a 1 / 4 scale. and right feature tensor This is specifically used for the local fine-grained matching in step S4; finally, to provide rich structural guidance for the subsequent attention module, features from each level are integrated to construct a context feature tensor set. .

[0034] S3. Based on the feature tensor at a 1 / 8 scale, construct the global matching cost volume and calculate the coarse disparity map. Then, use the gated axial attention module in combination with the context feature tensor to repair the coarse disparity map and generate the initial disparity map.

[0035] Specifically, to address the issue of local searches easily getting trapped in local extrema in high parallax scenes, we first utilize global matching to calculate the overall image correlation at low resolution (1 / 8 scale). In the multi-scale feature tensor at 1 / 8 scale, the left and right feature tensors corresponding to the left and right view images are respectively used. Epipolar global matching is then used to calculate the overall image correlation by analyzing the pixels in the left feature tensor. Features pixels in the right feature tensor Features Flatten out the matrix and calculate the matching score matrix across the entire disparity range. Capture large-scale geometric structures: In the formula and Let be the pixel indices in the horizontal direction of the left and right feature tensors. The number of feature channels, This represents matrix multiplication.

[0036] To effectively remove interference from occluded regions and non-matching points during the global search, a learnable truncation parameter is introduced. Construct an augmented probability distribution. First, calculate the global matching score matrix. With parameters The last dimension is stitched together, and the temperature coefficient is utilized. Perform distribution sharpening and calculate the normalized matching probability distribution tensor. : In the formula, This is a learnable truncation parameter used to represent a no-match state; This is a temperature coefficient used to adjust the sharpness of the probability distribution; The operation will represent the parameter "no match". After expansion, it is appended to the end of the matching score; After the operation, the previous section is captured. Each channel serves as the probability of assigning a valid pixel. This represents the column coordinate index of the right-hand image. It indicates the score of all matching pixels at a given point. All below At that time, the probability mass will be concentrated in This automatically suppresses the probability of noise within the effective parallax range.

[0037] Truncation parameters By incorporating the non-match probability, ensuring that the sum of the effective match probabilities is not 1, this invention first calculates the total probability of effective matches, which serves as the spatial matching confidence score for each pixel in the left view. : Subsequently, the effective probability distribution is renormalized using this confidence level, and the expected matching coordinates of each pixel in the left view in the right view are calculated, thus obtaining a physically accurate coarse disparity tensor. : In the formula, and These are the pixel indices in the horizontal direction of the left and right feature tensors; The width of the input feature tensor. This step uses the expected difference in probabilities across the entire image width as disparity, providing an accurate initial estimate in regions with large disparity.

[0038] To address common issues in stereo matching such as occlusion and weak texture holes, a gated axial attention module is used for global context repair to generate an initial disparity map. The gated axial attention module is divided into an axial attention section and a gating section. The axial attention section is obtained by concatenating lateral attention and longitudinal attention, and the lateral attention features... Longitudinal attention characteristics and repairing the parallax tensor The calculation formula is as follows: In the formula, is the scaling factor for the feature channel dimension; Indicates matrix transpose; This is a convolutional operation that aggregates local original features and global repaired features; To compress high-dimensional features into a single-channel convolution operation; This is a preset physical disparity scaling constant used to remap the network's normalized output values ​​back to the true physical disparity scale. For the disparity value features to be repaired, the coarse disparity tensor obtained from the aforementioned calculation is used. Confidence tensor With 1 / 8 scale context feature tensor The concatenation is performed along the channel dimension, obtained through dimensionality reduction convolution; the query matrix is ​​used in both horizontal and vertical attention. and Key matrix and Value matrix and Each through the wizard features Disparity features to be repaired Layer normalization operation Then multiply by the independently learnable weight tensor ( and , and , and The calculation formula is as follows: The wizard feature in the above formula It contains image texture and absolute position information, and is composed of context feature tensors. With spatial location encoding tensor After concatenation, feature mapping convolution operation is performed. Obtained. Spatial location encoding tensor and guide features The calculation formula is as follows: In the formula, and For the scale is Pixel x and y coordinates in the feature tensor and The results were obtained by performing normalization operations on each item.

[0039] The gating part introduces an adaptive fusion weight tensor. The original measured values ​​and the repaired predicted values ​​are dynamically weighted and fused to obtain the initial disparity map. : In the formula, This represents the Hadamard product. The adaptive fusion weight tensor... The formula for generating it is: In the formula, Represents the upper and lower feature tensors at a 1 / 8 scale; Used to map concatenated high-dimensional features into single-channel convolution operations; Use the Sigmoid activation function; Confidence level for spatial matching; This is a learnable bias term for the convolution operation. This mechanism ensures that original physical measurements are preserved in high-confidence regions by learning pixel-level confidence levels. In the low-confidence region, the transition to the attention-based repair result is smooth. .

[0040] S4. Upsample the initial disparity map to a 1 / 4 scale, construct a three-scale matching cost body covering different search ranges by referring to the multi-scale feature tensor, and input it into the bilateral aggregation network for residual regression optimization, and output the optimized disparity map.

[0041] Specifically, Upsampled to a 1 / 4 scale as prior disparity to guide fine-grained local matching Next, to preserve diverse feature information while reducing computational complexity, a group correlation algorithm is used to uniformly divide the left and right feature tensors at a 1 / 4 scale along the channel dimension. Subgroups, of which the first The eigenvectors corresponding to the eigensubgroups are represented as follows: In this embodiment, .

[0042] Subsequently, using the predicted position pointed to by the prior disparity as the center, corresponding matching points are found in the horizontal direction and similarity is calculated, thereby constructing the cost volume at a single scale. Specifically, regarding the pixels in the left image... The characteristics are determined by setting different sparse sampling step sizes. In the right figure, with the predicted matching point as the center and a radius of... Features are extracted from the neighborhood of the first element and a dot product is performed. This is for the first element... The formula for calculating the feature subgroup at the offset is: In the formula, This is the horizontal index of the pixel. For a fixed sampling index and the value range satisfies This means that for any scale, data is collected only near the predicted location. Discrete sample points (in this embodiment) This ensures the alignment of the feature tensors along the channel dimension. The stride here... It serves to adjust the search field of view: through The product operation, without increasing the number of computing channels, expands the physical search range to... Pixels (fine alignment) Pixels (medium correction) and Pixel-level (wide recall) achieves coverage of different degrees of parallax bias. Finally, all step sizes are... The corresponding cost volumes are concatenated along the channel dimension to form a high-dimensional aggregated input tensor capable of simultaneously capturing local details and global biases. .

[0043] Next, to address the foreground-background adhesion problem caused by depth map edge dilation, the aggregated feature tensor is... Input a bilateral aggregation network. Use fusion convolution to perform channel compression on contextual features, generating a single-channel scale-aware spatial attention tensor. : In the formula, Use the Sigmoid activation function; To fuse convolution operations, this attention tensor... As a soft threshold, the feature space is decoupled into detail branches and smooth branches, which are then processed through independent aggregation sub-networks. and After regularization, the output feature tensor is obtained by weighted fusion. The bilateral processing procedure is as follows: In the formula, Use the Sigmoid activation function; This represents the Hadamard product. In this model, Regions approaching 1 are defined as texture edges, with sharpness maintained by detail subnetworks; Regions approaching 0 are classified as smooth backgrounds, and noise is suppressed by smoothing subnetworks.

[0044] The aggregate subnetwork and All adopt a U-Net structure that includes an encoder, decoder, and skip connections. To guide the cost volume perception of local texture in the image, the spatial resolution scale set of the network feature layers is defined as follows: For any scale Let the cost body characteristics of the current level be... The corresponding context-guided features are The network introduces a guidance and incentive module. Modulate the cost features using image texture: In the formula, This is a mapping convolution operation. This mechanism dynamically injects contextual priors at the corresponding scale during the encoder's progressive downsampling stage, effectively suppressing matching noise.

[0045] Finally, to shield against long-tailed noise interference in the matching cost volume, a Top-k unified soft minimum algorithm is used at the tensor level to regress the disparity residual tensor. First, in the feature tensor Local disparity candidate dimensions Above, select the one with the highest probability value. Each element constitutes a disparity index set. : Subsequently, the feature tensors are locally normalized and their expectations are calculated only within the disparity channels contained in this set, and the disparity residual tensors are then computed. : In the formula, Representation of characteristic tensor In the disparity candidate index is Channel slice at the location; This indicates that the non-negative index coordinates of the local cost volume are remapped back to the actual physical disparity offset centered at 0.

[0046] After calculating the disparity residual tensor, it is added element-wise to the prior disparity tensor: This outputs a complete, high-precision optimized disparity tensor. This tensor-level operation significantly improves the noise resistance of depth measurements while maintaining a large parallax search range.

[0047] S5. The optimized disparity map is restored to the original image resolution using the convex upsampling module. The depth image is calculated by combining the binocular parameters, and the result is transmitted to the display terminal for display.

[0048] Specifically, to overcome the depth edge blurring caused by bilinear interpolation, a convex composite upsampling network is used to predict the upsampling weight tensor. Full-resolution parallax map Each high-resolution pixel in The parallax value is determined by its corresponding low-resolution size. Spatial neighborhood pixel set Each low-resolution pixel within The disparity value is obtained through convex weighted summation and precise calculation: In the formula, To restore the spatial pixel coordinate indexes on the disparity map to full resolution; Spatial pixel coordinate indexes on the optimized low-resolution disparity map; For high resolution pixels Projected onto a low-resolution scale Spatial neighborhood set; and Satisfy normalization constraints .

[0049] Ultimately, based on the principle of triangulation and according to the physical baseline of the stereo camera... and equivalent focal length Convert pixel-level parallax into a depth image tensor in physical space. : The calculated depth image Pseudo-color mapping is performed and transmitted to the display terminal for real-time display, completing a lightweight and high-precision reconstruction from two-dimensional image pairs to three-dimensional depth information.

[0050] Furthermore, to enable the aforementioned depth estimation network to possess accurate disparity prediction capabilities, this embodiment also provides a specific model training process. The training process employs a scheme combining a joint loss function with a specific parameter optimization strategy, as detailed below: The training process of the deep estimation network is specifically implemented using a multi-scale joint loss function: constructing a multi-scale joint loss function, selecting a basic error metric function, and constructing a total loss function that includes a main loss term and an auxiliary loss term; The main loss term is calculated for the initial disparity map, the optimized disparity map, and the final disparity map at different resolution scales to constrain the convergence of the global structure and enhance the matching accuracy of local textures. At the same time, the auxiliary loss term is calculated for the repair disparity tensor output by the attention module to force the network to learn the repair logic for occluded and weak texture regions. During the backpropagation phase, the AdamW optimizer is used to update the network parameters to prevent overfitting. At the same time, the OneCycleLR learning rate scheduling strategy is introduced, which linearly warms up the learning rate to the maximum value in the early stage of training, and then uses the cosine annealing strategy to decay the learning rate in order to finely search for the global optimum until the model converges.

[0051] Construct a multi-scale joint loss function. To supervise the learning performance of the network at different resolution levels, a joint loss function containing the main loss term is constructed. and auxiliary loss items Total loss function .

[0052] Smooth L1 Loss is chosen as the basic error metric function for predicting disparity. and true parallax Its definition is: It is important to emphasize that, to ensure the effectiveness of error calculation, all disparity tensors at low resolution scales must be upsampled to the full resolution of the original image via bilinear interpolation before being used in loss calculation, and then multiplied by the corresponding scale factor to ensure accuracy with the true disparity labels. Strict alignment is achieved between spatial resolution and physical parallax scale. This upsampling operation with scale restoration is defined as follows: ,in This represents the upsampling factor. Based on this, the total loss function... It consists of a weighted sum of four parts: In the formula, This indicates the calculation of this term and the true disparity. Smooth L1 error between; For the initial disparity map at a 1 / 8 scale generated in step S3 The loss term calculated after upsampling by 8 times. Weighting coefficients. , used to constrain the initial convergence of a large-scale global structure; For the disparity map optimized at a 1 / 4 scale for the bilateral aggregation network generated in step S4 The loss term calculated after a 4-fold upsampling. Weighting coefficients. This is used to enhance the fine-grained matching accuracy of local textures; For the full-resolution final disparity map generated by the convex upsampling module in step S5 The calculated loss term. Weighting coefficients. , serving as the core supervisory signal that dominates the network gradient; For the 1 / 8 scale repaired disparity tensor output by the gated axial attention module in step S3 The auxiliary loss term is calculated after upsampling by 8 times. Weighting coefficients. This project aims to force the cross-attention mechanism to independently learn adaptive patching logic for occluded and weakly textured regions using the left view, preventing the network from over-relying on physical measurement residuals during training.

[0053] Parameter Iterative Optimization Strategy. During the backpropagation phase, the AdamW optimizer is used to update the network parameters, leveraging its weight decay characteristic to prevent overfitting. Simultaneously, the OneCycleLR learning rate scheduling strategy is introduced, the specific process of which is as follows: In the first 30% of iterations during training, the learning rate is linearly warmed up from its initial value to its maximum value to quickly traverse the flat loss surface. In the remaining 70% of iterations, a cosine annealing strategy is used to decay the learning rate to a small value to finely search for the global optimum until the model converges.

[0054] To verify the effectiveness and robustness of the method proposed in this invention, it was deployed and evaluated on a standard stereo matching dataset. Figure 4 The image shows the test results of an embodiment of the present invention on the Middlebury test set. The results clearly demonstrate that the present invention can output high-precision depth maps, significantly overcoming the matching failure problem of existing lightweight networks in high parallax scenarios.

[0055] Based on the same inventive concept as the aforementioned global search-based large disparity lightweight depth estimation method, this embodiment also provides a global search-based large disparity lightweight depth estimation system. For example... Figure 2 As shown, the system includes: The binocular stereo image pair module uses a binocular image acquisition device to capture the target scene, obtain left and right view images, and preprocess the left and right view images to form a binocular stereo image pair.

[0056] The feature extraction network module is used to input the stereo image into the feature extraction network and extract multi-scale feature tensors for matching and context feature tensors for guidance.

[0057] The initial disparity map module is used to construct a global matching cost volume and calculate a coarse disparity map based on a multi-scale feature tensor. The gated axial attention module is used in conjunction with the context feature tensor to repair the coarse disparity map and generate the initial disparity map.

[0058] The disparity map optimization module is used to upsample the initial disparity map to a 1 / 4 scale, construct a three-scale matching cost body covering different search ranges with reference to multi-scale feature tensors, and input it into a bilateral aggregation network for residual regression optimization, outputting the optimized disparity map.

[0059] The depth image estimation and output module uses the convex upsampling module to restore the optimized disparity map to the resolution of the left view image, obtaining the full-resolution final disparity map. It then calculates the depth image by combining the binocular parameters and transmits the result to the display terminal for display.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions claimed by the present invention.

Claims

1. A lightweight depth estimation method with large disparity based on global search, characterized in that, Includes the following steps: S1. Use a binocular image acquisition device to capture the target scene, obtain the left view image and the right view image, and preprocess the left and right view images to form a binocular stereo image pair; S2. Input the stereo images into the feature extraction network to extract multi-scale feature tensors for matching and context feature tensors for guidance; S3. Based on the multi-scale feature tensor, construct the global matching cost volume and calculate the coarse disparity map. Use the gated axial attention module combined with the context feature tensor to repair the coarse disparity map and generate the initial disparity map. S4. Upsample the initial disparity map to a 1 / 4 scale, construct a three-scale matching cost body covering different search ranges by referring to the multi-scale feature tensor, and input it into the bilateral aggregation network for residual regression optimization, and output the optimized disparity map. S5. Use the convex upsampling module to restore the optimized disparity map to the resolution of the left view image to obtain the full-resolution final disparity map. Combine the binocular parameters to calculate the depth image.

2. The lightweight depth estimation method with large disparity based on global search according to claim 1, characterized in that, The preprocessing specifically includes: performing pixel distribution normalization processing on the original left view image and right view image; Based on the downsampling factor of the subsequent feature extraction network, the normalized image tensor is zero-padded at the edges to ensure that the target size meets the divisibility requirement of the receptive field of the feature extraction network.

3. The lightweight depth estimation method with large disparity based on global search according to claim 1, characterized in that, The feature extraction network specifically includes: A lightweight feature extraction network based on the inverse residual module is constructed using MobileNetV2 to extract original image features at different levels; The FPN network is used to perform multi-scale information fusion on the original image features at different levels. Deep semantics are injected into shallow features through a top-down path. The 1 / 8 scale feature tensor for global search and the 1 / 4 scale feature tensor for local fine matching are obtained from the features extracted by the FPN network to form a multi-scale feature tensor set. The 1 / 16 scale feature tensor, 1 / 8 scale feature tensor and 1 / 4 scale feature tensor corresponding to the left view are obtained from the features extracted by the FPN network to form a context feature tensor set.

4. The lightweight depth estimation method with large disparity based on global search according to claim 3, characterized in that, The specific implementation process of step S3 is as follows: In the multi-scale feature tensor at a 1 / 8 scale, the left and right feature tensors corresponding to the left and right view images, respectively, are used to calculate the global image correlation using epipolar global matching. This is achieved by analyzing the pixels in the left feature tensor. Features pixels in the right feature tensor Features Flatten out the matrix and calculate the matching score matrix across the entire disparity range. : In the formula and Let be the pixel indices in the horizontal direction of the left and right feature tensors. The number of feature channels, Represents matrix multiplication; Introducing learnable cutoff parameters Construct an augmented probability distribution; convert the global matching score matrix With parameters The last dimension is stitched together, and the temperature coefficient is utilized. Perform distribution sharpening and calculate the normalized matching probability distribution tensor. : After the operation, extract the previous section. Each channel is used as the matching probability of an effective pixel to obtain... ; Calculate the total probability of a valid match as the spatial match confidence score for each pixel in the left view. : Subsequently, the effective probability distribution is renormalized using this confidence level, and the expected matching coordinates of each pixel in the left view on the right view are calculated to obtain a physically accurate coarse disparity map. : For coarse disparity maps Global context repair is performed using a gated axis attention module to generate an initial disparity map. .

5. The lightweight depth estimation method with large disparity based on global search according to claim 4, characterized in that, The gated axial attention module is divided into an axial attention section and a gated section; The axial attention component is obtained by concatenating lateral attention and longitudinal attention; lateral attention features... Longitudinal attention characteristics and repairing the parallax tensor The calculation formula is as follows: In the formula, is the scaling factor for the feature channel dimension; Indicates matrix transpose; Convolution operation for aggregated features; To compress high-dimensional features into a single-channel convolution operation; This is a preset physical disparity scaling constant used to remap the normalized output values ​​back to the true physical disparity scale. The disparity value features to be repaired are derived from the coarse disparity tensor. Confidence tensor With 1 / 8 scale context feature tensor The concatenation is performed along the channel dimension, obtained through dimensionality reduction convolution; the query matrix is ​​used in both horizontal and vertical attention. and Key matrix and Value matrix and Each through the wizard features Disparity features to be repaired Multiply by an independently learnable weight tensor after layer normalization. and , and , and Get; Guide features It contains image texture and absolute position information, and is composed of context feature tensors. With spatial location encoding tensor After concatenation, feature mapping convolution operation is performed. We obtain the spatial location encoding tensor. The calculation formula is as follows: In the formula, and For the scale is Pixel x and y coordinates in the feature tensor and The results were obtained by performing normalization operations on each item. Adaptive fusion weight tensor is introduced in the gating part The original measured values ​​and the repaired predicted values ​​are dynamically weighted and fused to obtain the initial disparity map. : In the formula, Represents the Hadamard product, adaptive fusion weight tensor The formula for generating it is: In the formula, Represents the upper and lower feature tensors at a 1 / 8 scale; This is a convolution operation; Use the Sigmoid activation function; Spatial matching confidence; This is a learnable bias term for the convolution operation.

6. The lightweight depth estimation method with large disparity based on global search according to claim 4, characterized in that, The specific implementation process of constructing the three-scale matching cost body covering different search ranges is as follows: First, the initial disparity map generated in step S3 is upsampled to a 1 / 4 scale as a prior disparity to guide subsequent local searches; Secondly, based on the 1 / 4 scale feature tensor in the multi-scale feature tensor, with the prior disparity as the center, a group correlation algorithm is used and feature matching is performed by setting different sparse sampling step sizes under three sampling index radii from small to large, to construct a three-scale cost volume tensor that respectively covers fine alignment, medium bias correction and wide recall. Finally, the three-scale cost volume tensors corresponding to all step sizes are concatenated along the channel dimension to obtain a high-dimensional aggregated input tensor that reflects multi-scale matching information.

7. The lightweight depth estimation method with large disparity based on global search according to claim 6, characterized in that, Step S5 also includes the training process of the depth estimation network, which is specifically implemented using a multi-scale joint loss function as follows: Construct a multi-scale joint loss function, select a basic error metric function, and construct a total loss function that includes main loss terms and auxiliary loss terms; The main loss term is calculated for the initial disparity map, the optimized disparity map, and the final disparity map at different resolution scales; at the same time, the auxiliary loss term is calculated for the repaired disparity tensor output by the attention module.

8. A lightweight depth estimation system for large disparity based on global search, used to implement the lightweight depth estimation method for large disparity as described in any one of claims 1 to 7, characterized in that, Includes the following modules: The binocular stereo image pair module uses a binocular image acquisition device to capture the target scene, obtain left and right view images, and preprocesses the left and right view images to form a binocular stereo image pair. The feature extraction network module is used to input the stereo image into the feature extraction network and extract multi-scale feature tensors for matching and context feature tensors for guidance. The initial disparity map module is used to construct a global matching cost volume and calculate a coarse disparity map based on a multi-scale feature tensor. The gated axial attention module is used in conjunction with the context feature tensor to repair the coarse disparity map and generate the initial disparity map. The disparity map optimization module is used to upsample the initial disparity map to a 1 / 4 scale, construct a three-scale matching cost body covering different search ranges with reference to the multi-scale feature tensor, and input it into the bilateral aggregation network for residual regression optimization, and output the optimized disparity map. The depth image estimation and output module uses the convex upsampling module to restore the optimized disparity map to the resolution of the left view image, obtaining the full-resolution final disparity map. It then calculates the depth image by combining the binocular parameters and transmits the result to the display terminal for display.

Images