Binocular depth estimation method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a binocular depth estimation method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of computer vision. The method comprises the following steps: acquiring a binocular image; performing multi-scale feature extraction on the binocular image to obtain a multi-scale left image feature, a multi-scale right image feature and a multi-scale context feature; performing feature fusion on the left image features and the right image features of each scale to obtain a multi-scale parallax cost volume; performing line-by-line global coding on the parallax cost volume of the first scale and the context feature to obtain an initial parallax; performing line-by-line global coding on the parallax cost volume of the second scale and the context features to obtain fine parallax; according to the initial parallax and the fine parallax, sampling to obtain a plurality of candidate parallax; predicting a final parallax error according to the plurality of candidate parallax errors; and estimating the binocular depth according to the final parallax. According to the scheme, the precision requirement can be met while the power consumption is reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and specifically to a binocular depth estimation method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Binocular depth estimation is a key technology in computer vision. It simulates the principle of binocular parallax in humans and calculates the depth information of each point in the scene by analyzing the left and right images of the same scene. With the rapid development of deep learning technology in computer vision, deep learning-based binocular depth estimation methods have improved in accuracy and robustness.

[0003] However, deep learning-based binocular depth estimation methods still face significant challenges on end devices with limited computing resources. When running on mobile or embedded devices, there is often a trade-off between accuracy and speed, which can have a certain impact on practical application results, making it difficult to meet the speed and power consumption requirements of real-time applications. Summary of the Invention

[0004] The embodiments of the present application provide a binocular depth estimation method, device, electronic device, and computer-readable storage medium, which can reduce power consumption while meeting accuracy requirements.

[0005] In a first aspect, an embodiment of the present application provides a binocular depth estimation method, comprising:

[0006] Acquire binocular images;

[0007] Performing multi-scale feature extraction on the binocular images to obtain multi-scale left image features, multi-scale right image features, and multi-scale context features; wherein the multi-scale includes a first scale and a second scale;

[0008] The features of the left image and the right image at each scale are fused separately to obtain the multi-scale disparity cost volume;

[0009] Performing row-by-row global encoding on the disparity cost volume and context features of the first scale to obtain an initial disparity;

[0010] Performing row-by-row global encoding on the disparity cost volume and context features of the second scale to obtain fine disparity;

[0011] Sampling a plurality of candidate disparities according to the initial disparity and the refined disparity;

[0012] Predicting the final disparity based on multiple candidate disparities;

[0013] The binocular depth is estimated according to the final disparity.

[0014] In one embodiment, the performing multi-scale feature extraction on the binocular images to obtain multi-scale left image features, multi-scale right image features, and multi-scale context features includes:

[0015] Performing multi-scale feature extraction on the binocular images respectively to obtain the multi-scale left image features and the multi-scale right image features;

[0016] The multi-scale left image features are respectively input into the reverse residual module to obtain the multi-scale context features.

[0017] In one embodiment, the performing multi-scale feature extraction on the binocular images to obtain multi-scale left image features, multi-scale right image features, and multi-scale context features includes:

[0018] Using a lightweight visual encoder, multi-scale feature extraction is performed on the binocular images to obtain the multi-scale left image features, the multi-scale right image features and the multi-scale context features;

[0019] The lightweight visual encoder is a model that performs feature distillation based on a self-supervised learning model.

[0020] In one embodiment, the feature fusion of the left image features and the right image features at each scale to obtain a multi-scale disparity cost volume includes:

[0021] Correlating the features of the left image and the features of the right image at the first scale row by row to obtain a disparity cost volume at the first scale;

[0022] The features of the left image and the features of the right image at the second scale are correlated row by row to obtain a disparity cost volume at the second scale.

[0023] In one embodiment, globally encoding the disparity cost volume and context features at the first scale line by line to obtain the initial disparity includes:

[0024] Globally encoding the disparity cost volume of the first scale and the context features of the first scale row by row to obtain an initial disparity estimate;

[0025] The initial disparity estimate is upsampled to obtain the initial disparity at the second scale.

[0026] In one embodiment, sampling to obtain a plurality of candidate disparities according to the initial disparity and the refined disparity includes:

[0027] Performing local motion coding on the initial disparity to obtain local motion features;

[0028] fusing the local motion feature and the fine disparity to obtain a disparity to be sampled;

[0029] The left and right sides of the to-be-sampled disparity are uniformly sampled to obtain a plurality of candidate disparities.

[0030] In one embodiment, predicting the final disparity based on the plurality of candidate disparities includes:

[0031] Predicting weights, disparity increments, and upsampling masks for each of the plurality of candidate disparities;

[0032] performing weighted processing on the disparity increments of the plurality of candidate disparities according to respective weights of the plurality of candidate disparities to obtain an overall disparity increment;

[0033] performing weighted processing on the upsampling masks of the plurality of candidate disparities according to respective weights of the plurality of candidate disparities to obtain an overall upsampling mask;

[0034] The global disparity increment is upsampled based on the global upsampling mask to obtain the final disparity.

[0035] In a second aspect, an embodiment of the present application provides a binocular depth estimation device, comprising:

[0036] An image acquisition module, used to acquire binocular images;

[0037] A feature extraction module is used to perform multi-scale feature extraction on the binocular images to obtain multi-scale left image features, multi-scale right image features and multi-scale context features; wherein the multi-scale includes a first scale and a second scale;

[0038] The feature fusion module is used to fuse the features of the left image and the right image at each scale to obtain a multi-scale disparity cost volume;

[0039] an initial disparity acquisition module, configured to globally encode the disparity cost volume and context features of the first scale line by line to obtain an initial disparity;

[0040] a fine disparity acquisition module, configured to globally encode the disparity cost volume and context features of the second scale line by line to obtain fine disparity;

[0041] a sampling module, configured to sample and obtain a plurality of candidate disparities according to the initial disparity and the refined disparity;

[0042] A final prediction module, configured to predict a final disparity based on multiple candidate disparities;

[0043] The depth estimation module is configured to estimate the binocular depth according to the final disparity.

[0044] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the computer program is executed by the processor, the steps in the above-mentioned binocular depth estimation method are implemented.

[0045] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned binocular depth estimation method are implemented.

[0046] In a fifth aspect, embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described in the embodiments of the present application.

[0047] The embodiments of the present application have the following beneficial effects:

[0048] Determined multi-scale features ensure both the capture of detailed local information and the ability to adapt to global variations in large-scale disparity. Predicting the initial disparity narrows the search space for subsequent optimization, reducing the lengthy full-disparity search range in traditional algorithms to a small, high-probability region. This not only reduces unnecessary search calculations but also provides a more reliable initial estimate for subsequent fine-grained optimization, significantly improving the efficiency and accuracy of the overall algorithm. Based on multiple candidate disparities, efficient multi-branch parallel optimization can be achieved, resulting in a more accurate final disparity, significantly improving inference speed while ensuring optimization quality. This significantly reduces computational complexity and power consumption while ensuring depth estimation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 This is a schematic diagram of the steps of a binocular depth estimation method provided by an embodiment of the present application;

[0051] Figure 2 Schematic diagram of the architecture of a binocular depth estimation framework provided by an embodiment of the present application;

[0052] Figure 31 is a flow chart of a binocular depth estimation method provided by an embodiment of the present application;

[0053] Figure 4 1 is a schematic structural diagram of a binocular depth estimation device provided in one embodiment of the present application;

[0054] Figure 5 It is a structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0056] Binocular depth estimation is a key technology in visual perception. By matching consistent observations from two cameras, it can accurately calculate the depth of a scene. Due to its unique advantages, this technology is widely used in fields such as autonomous driving, robotics, and extended reality. Compared with active optical depth estimation, binocular depth estimation has the advantages of longer range and lower power consumption. It also offers superior scale accuracy compared to monocular depth estimation.

[0057] Traditional binocular depth estimation methods mainly rely on scan line search and block matching techniques based on geometric characteristics. Common methods include Semi-Global Matching (SGM) and PatchMatch (an algorithm based on local block matching and propagation optimization). These two types of methods have demonstrated high computational efficiency in many scenarios thanks to their mature theoretical framework and good terminal acceleration performance. However, since these methods rely on manually designed feature extraction and matching mechanisms, their feature representation capabilities and noise resistance are relatively limited. They are prone to mismatch problems in texture loss, illumination changes, and dynamic scenes, thus affecting the accuracy of depth estimation. In addition, traditional methods usually require parameter tuning for specific scenarios, or rely on the assistance of active light projection to improve matching results, which to a certain extent limits their versatility and adaptability.

[0058] With the rapid development of deep learning technology in the field of computer vision, deep learning-based binocular depth estimation methods have surpassed traditional methods in terms of accuracy and robustness. These methods automatically learn feature representations through neural networks, avoiding the limitations of traditional methods that rely on manually designed features and better adapting to complex and dynamic environments. Recently, algorithmic frameworks based on iterative optimization have become increasingly mainstream, such as end-to-end network models based on parallax cost volume construction and optimization, which have significantly improved the accuracy and robustness of depth estimation.

[0059] However, deep learning-based binocular depth estimation methods still face significant challenges on end devices with limited computing resources. Because deep learning models typically require significant computing resources and storage space, these algorithms struggle to meet the speed and power consumption requirements of real-time applications. When running on mobile or embedded devices, a trade-off between accuracy and speed often occurs, which can have a significant impact on practical application effectiveness. This is especially true in real-time 3D reconstruction and virtual-reality hybrid scenarios, where algorithm performance requirements are even more stringent.

[0060] Binocular depth estimation methods in related technologies primarily fall into two categories: algorithms based on iterative optimization and end-to-end direct regression. While the former offers high estimation accuracy, it suffers from high computational complexity, making it difficult to meet the real-time processing requirements of mobile devices. The latter offers higher computational efficiency but is prone to significant errors in complex scenarios, making depth estimation accuracy insufficient for demanding applications.

[0061] In response to the above problems, a binocular depth estimation method is proposed in an embodiment of the present application. This binocular depth estimation method proposes an innovative binocular depth estimation framework that integrates global and local optimization strategies. By introducing a framework design of a lightweight visual encoder based on feature distillation, global initialization and multi-candidate weighted optimization mechanism, it can significantly reduce the amount of calculation and power consumption while ensuring the accuracy of depth estimation, and significantly improve computing efficiency.

[0062] The binocular depth estimation method proposed in the embodiment of the present application can be implemented in real time on resource-constrained terminal devices, meeting the actual needs of real-time three-dimensional reconstruction, virtual and real occlusion and other related algorithms, thereby providing a more efficient and practical solution for the application expansion of binocular depth estimation on mobile terminals.

[0063] In one embodiment, Figure 1As shown, a binocular depth estimation method is provided. Although a logical order is shown in the step diagram, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings. The binocular depth estimation method optimizes the computational complexity and memory usage of the algorithm, reduces the use requirements of hardware resources, and thus reduces overall power consumption. Therefore, the binocular depth estimation method can be applied to a terminal. The terminal may include one or more of a drone, an extended reality device, and other embedded terminal devices.

[0064] Extended reality includes virtual reality, augmented reality, and mixed reality. Extended reality devices include mobile handheld terminals such as mobile phones and tablets, including heads-up displays such as car heads-up displays, including augmented reality (AR) head-mounted display devices, virtual reality (VR) head-mounted display devices, and near-eye display terminals worn on the human head, such as glasses-shaped displays and helmet-shaped displays. In the embodiment of the present application, the glasses-shaped display may include an optical see-through glasses-shaped display.

[0065] The display in the form of glasses includes a frame body, namely the temples and the frame, to support the extended reality glasses to be worn on the human head; includes an optical display module mainly composed of a micro display screen, optical lenses and optical waveguides, to display virtual content to the user; includes an audio module mainly composed of a microphone and a speaker, to collect and play sounds; includes sensors mainly composed of cameras, gyroscopes, barometers and infrared light transmitters and receivers, to collect information data related to the human body, the glasses body and the external environment; includes an integrated processor with a microcontroller unit (MCU) or a central processing unit (CPU) as the core to perform data processing and calculation; includes a circuit board, flexible or rigid, to connect other electronic components to form an electronic circuit system, which is generally placed in the inner cavity of the glasses frame and the temples; and may also include a battery power supply.

[0066] It should be noted that the order of description of the following embodiments does not limit the priority order of the embodiments.

[0067] according to Figure 1 The binocular depth estimation method shown in FIG. 1 includes at least steps S110 to S180. The binocular depth estimation method can be based on Figure 2 The binocular depth estimation framework shown is implemented. Figure 2 This is a schematic diagram of the architecture of a binocular depth estimation framework provided by an embodiment of the present application. The binocular depth estimation method is described in detail as follows:

[0068] In step S110 , a binocular image is acquired.

[0069] The terminal can have a binocular camera, or a combination of multiple cameras equivalent to binocular cameras. A binocular camera can capture the same target, producing a binocular image. The binocular image consists of a left-eye image and a right-eye image. The left-eye image is a two-dimensional image captured by the left camera in the binocular vision system, simulating the perspective of the human left eye. The right-eye image is a two-dimensional image captured by the right camera in the binocular vision system, simulating the perspective of the human right eye.

[0070] In step S120 , multi-scale feature extraction is performed on the binocular images to obtain multi-scale left image features, multi-scale right image features, and multi-scale context features.

[0071] The multi-scale includes a first scale and a second scale. The features of the second scale are finer than the features of the first scale. Optionally, the first scale is 1 / 8 of the resolution of the original image, and the second scale is 1 / 4 of the resolution of the original image.

[0072] The binocular images can be input into the feature extraction module, which can perform multi-scale feature extraction on each binocular image. The feature extraction module performs multi-scale feature extraction on the left image to obtain first-scale left image features and second-scale left image features. The feature extraction module performs multi-scale feature extraction on the right image to obtain first-scale right image features and second-scale right image features.

[0073] Multi-scale contextual features can be obtained by performing a reverse residual operation on the multi-scale left image features or the multi-scale right image features. Contextual features refer to semantic and spatial association information extracted from the image that is related to the surrounding environment of an object or pixel. Contextual features are not local features of a single pixel or object (such as edges or corners), but rather features extracted based on global or local association information such as the relationship between multiple regions in the image, scene structure, and positional dependencies between objects.

[0074] Optionally, a reverse residual operation is performed on the left image features of the first scale to obtain a left image feature difference of the first scale, and the left image feature difference of the first scale is fused with the left image features of the first scale to obtain a context feature of the first scale.

[0075] Optionally, a reverse residual operation is performed on the right image features of the first scale to obtain a right image feature difference of the first scale, and the right image feature difference of the first scale is fused with the right image features of the first scale to obtain the context features of the first scale.

[0076] Optionally, a reverse residual operation is performed on the left image features of the second scale to obtain a left image feature difference of the second scale, and the left image feature difference of the second scale is fused with the left image features of the second scale to obtain a context feature of the second scale.

[0077] Optionally, a reverse residual operation is performed on the right image features of the second scale to obtain a right image feature difference of the second scale, and the right image feature difference of the second scale is fused with the right image features of the second scale to obtain a context feature of the second scale.

[0078] The contextual features at each scale are determined based solely on the features of a single image (the left image features at the first scale, the left image features at the second scale, the right image features at the first scale, or the right image features at the second scale), rather than on both the left and right image features. This reduces computational complexity, improves real-time performance, and reduces reliance on binocular synchronization, enhancing robustness.

[0079] In step S130 , feature fusion is performed on the left image features and the right image features at each scale to obtain a multi-scale disparity cost volume.

[0080] The disparity cost volume is used to quantify the matching cost (i.e., similarity or difference) of pixels in a binocular image at different disparities. The disparity cost volume is a key intermediate structure connecting feature extraction and disparity calculation, and directly affects the accuracy and efficiency of depth estimation.

[0081] The first-scale left image features and the first-scale right image features are input into a feature fusion module. The feature fusion module can calculate the correlation between the first-scale left image features and the first-scale right image features row by row to perform feature fusion on the first-scale left image features and the first-scale right image features, thereby obtaining a disparity cost volume of the first scale.

[0082] The second-scale left image features and the second-scale right image features are input into a feature fusion module. The feature fusion module can calculate the correlation between the second-scale left image features and the second-scale right image features row by row to perform feature fusion on the second-scale left image features and the second-scale right image features, thereby obtaining a second-scale disparity cost volume.

[0083] In step S140 , the disparity cost volume and context features of the first scale are globally encoded row by row to obtain an initial disparity.

[0084] Row-by-row global encoding refers to modeling and encoding global information along the image row dimension; dividing the image or feature map into independent units by row (horizontally) and processing them row by row (rather than processing all pixels globally at once); in processing each row, the global context of all pixels in that row is considered, rather than the local neighborhood. Optionally, global modeling of pixels within a row can be performed using one-dimensional convolution, recurrent neural networks, or self-attention mechanisms. Optionally, the similarity of all pixel features within the same row can be calculated based on the row-level self-attention of the Transformer model, dynamically aggregating the global context.

[0085] The disparity cost volume and context features at the first scale are globally encoded row by row to obtain the initial disparity. The initial disparity is a coarse-grained abstraction of the original image disparity, which significantly improves computational efficiency by compressing spatial resolution.

[0086] In step S150 , the disparity cost volume and context features of the second scale are globally encoded row by row to obtain fine disparity.

[0087] The second-scale disparity cost volume and the second-scale context features are globally encoded row by row to obtain the fine disparity. The fine disparity is a slightly finer-grained abstraction of the original image disparity, which can achieve a balance between accuracy and efficiency.

[0088] In step S160 , a plurality of candidate disparities are sampled based on the initial disparity and the refined disparity.

[0089] The initial disparity can provide high-confidence initial candidate regions. Combined with the refined processing of local iterative optimization, the refined disparity can improve the accuracy.

[0090] The initial disparity and the refined disparity are fused to obtain the disparity to be sampled. The disparity to be sampled is sampled to obtain multiple candidate disparities.

[0091] In step S170 , a final disparity is predicted based on a plurality of candidate disparities.

[0092] By weighted fusion of multiple candidate disparities, the accurate final disparity can be obtained.

[0093] In step S180 , the binocular depth is estimated according to the final disparity.

[0094] According to the geometric relationship between the final disparity and the binocular depth, the binocular depth can be estimated based on the triangulation principle.

[0095] By adopting the technical solution of the embodiment of the present application, the multi-scale features determined can ensure that both fine local information can be captured and global changes in a large range of disparity can be adapted. Predicting the initial disparity can narrow the search space for subsequent optimization, reducing the lengthy full disparity search range in the traditional algorithm to a small high-probability area. This not only reduces unnecessary search calculations, but also provides a more reliable initial estimate for subsequent fine optimization, significantly improving the efficiency and accuracy of the overall algorithm. Based on multiple candidate disparities, efficient multi-branch parallel optimization can be achieved to obtain a more accurate final disparity, significantly improving the inference speed while ensuring the optimization quality. In this way, while ensuring the accuracy of depth estimation, the amount of calculation and power consumption are significantly reduced.

[0096] Figure 3 1 is a flow chart of a binocular depth estimation method provided in one embodiment of the present application.

[0097] On the basis of the above technical solution, as an embodiment, refer to Figure 3 , respectively performing multi-scale feature extraction on the binocular images to obtain multi-scale left image features, multi-scale right image features, and multi-scale context features may include: respectively performing multi-scale feature extraction on the binocular images to obtain the multi-scale left image features and the multi-scale right image features; and respectively inputting the multi-scale left image features into a reverse residual module to obtain the multi-scale context features.

[0098] The binocular images are input into the feature extraction module, which performs multi-scale feature extraction on each binocular image. The feature extraction module performs multi-scale feature extraction on the left image, obtaining first-scale and second-scale left image features. The feature extraction module performs multi-scale feature extraction on the right image, obtaining first-scale and second-scale right image features.

[0099] The multi-scale left image features are input into the reverse residual module and subjected to reverse residual operation to obtain multi-scale context features. The multi-scale context features include the context features of the first scale and the context features of the second scale.

[0100] The Inverse Residual operation is widely used in lightweight networks. By adjusting the order in which feature map dimensions change, it optimizes the balance between computational complexity and feature expression capabilities.

[0101] Traditional residual blocks first compress the feature map dimensions (dimensionality reduction), then extract features through convolution, and finally restore the dimensions. The inverse residual operation reverses this order. Given a low-dimensional input, the inverse residual operation first expands it to a high-dimensional space, then extracts features in the high-dimensional space, and finally compresses it back to a low-dimensional space. This avoids directly extracting features in the low-dimensional space, which would result in information loss due to the lower feature dimensionality.

[0102] The reverse residual operation uses deep convolution and an expansion-then-compression structure to significantly reduce the number of parameters and floating-point operations while maintaining feature expressiveness. The reverse residual operation leverages high-dimensional space to enhance feature expressiveness. Deep convolution and dimensionality transformation reduce computational costs, making it suitable for real-time inference on mobile devices.

[0103] The left image features of the first scale are input into the reverse residual module, and a reverse residual operation is performed on the left image features of the first scale to obtain the left image feature difference of the first scale. The left image feature difference of the first scale is fused with the left image features of the first scale to obtain the context feature of the first scale.

[0104] Performing a reverse residual operation on the left image features of the second scale, performing a reverse residual operation on the left image features of the second scale, and obtaining a left image feature difference of the second scale. The left image feature difference of the second scale is fused with the left image features of the second scale to obtain a context feature of the second scale.

[0105] By adopting the technical solution of the embodiment of the present application, the context features at each scale are determined based only on the features of a single image (the left image features at the first scale or the left image features at the second scale), rather than on both the left and right image features. This can reduce computational complexity and improve real-time performance. In addition, it can also reduce dependence on binocular synchronization and enhance robustness.

[0106] On the basis of the above technical solution, as an embodiment, refer to Figure 3 , the said performing multi-scale feature extraction on the binocular images respectively to obtain multi-scale left image features, multi-scale right image features and multi-scale context features can include: using a lightweight visual encoder to perform multi-scale feature extraction on the binocular images respectively to obtain the multi-scale left image features, the multi-scale right image features and the multi-scale context features; wherein, the lightweight visual encoder is based on a model for feature distillation of a self-supervised learning model.

[0107] The feature extraction module may include a lightweight visual encoder, which may be a model for feature distillation based on a self-supervised learning model (Data-efficient Image Net training with Object-Level Supervision, DINO).

[0108] The lightweight visual encoder is a neural network architecture designed for scenarios with limited computing resources. By optimizing the network structure (such as depthwise separable convolution and channel pruning) or introducing efficient components (such as bottleneck structures and knowledge distillation), the lightweight visual encoder can significantly reduce the number of parameters and computation while retaining the ability to extract semantic features from images. The lightweight visual encoder enables real-time inference on low-power platforms such as mobile devices and edge devices, while maintaining the performance of downstream tasks.

[0109] DINO is a self-supervised learning framework that uses a teacher-student network distillation mechanism to learn visual feature representations with dense semantic information from unlabeled image data. DINO utilizes dynamic clustering to generate pseudo-labels. Combined with a multi-crop training strategy, this model captures robust features across scales, significantly improving transfer performance and data efficiency for downstream tasks.

[0110] Feature distillation is a knowledge transfer technique that improves the performance of smaller models by minimizing the differences in intermediate-layer features between the student and teacher models, enabling the student model to learn the teacher's feature expression patterns. Feature distillation efficiently transfers knowledge from large models (such as feature distribution and semantic structure) to a lightweight architecture, maintaining or approaching the capabilities of the teacher model in scenarios with limited parameters and computational complexity.

[0111] The binocular image is fed into the feature extraction module, where the lightweight visual encoder extracts multi-scale features from the binocular image, generating multi-scale left image features and multi-scale right image features. The multi-scale left image features are fed into the reverse residual module to generate multi-scale contextual features.

[0112] The technical solutions of the embodiments of this application enable a lightweight visual encoder to reduce computational costs and model size through a streamlined architecture, making it suitable for resource-constrained scenarios such as mobile devices and edge devices, enabling low-power real-time inference. DINO, through self-supervised learning, can compensate for the limited feature representation of the lightweight visual encoder due to its small number of parameters, enhancing its ability to understand complex scenes.

[0113] On the basis of the above technical solution, as an embodiment, refer to Figure 3 The feature fusion of the left image features and the right image features at each scale to obtain a multi-scale disparity cost volume may include: correlating the left image features and the right image features at the first scale row by row to obtain the disparity cost volume at the first scale; and correlating the left image features and the right image features at the second scale row by row to obtain the disparity cost volume at the second scale.

[0114] Linewise correlation refers to breaking down the feature graph into rows and calculating the correlation row by row.

[0115] The first-scale left image features and the first-scale right image features are correlated row by row to obtain the first-scale disparity cost volume. The second-scale left image features and the second-scale right image features are correlated row by row to obtain the second-scale disparity cost volume.

[0116] By adopting the technical solution of the embodiment of the present application, line-by-line processing can achieve dimensionality reduction, significantly reduce the amount of calculation, adapt to scenes with high real-time requirements, focus on the local correlation of pixels within the row, and more accurately capture the disparity changes of objects in the same scan line, thereby improving the local consistency of the disparity map.

[0117] On the basis of the above technical solution, as an embodiment, refer to Figure 3 The globally encoding the disparity cost volume and the context features of the first scale line by line to obtain the initial disparity may include: globally encoding the disparity cost volume and the context features of the first scale line by line to obtain an initial disparity estimate; and upsampling the initial disparity estimate to obtain the initial disparity of the second scale.

[0118] Alternatively, row-by-row global encoding can be performed through one-dimensional convolution, recurrent neural networks, or self-attention mechanisms; or row-level self-attention of the Transformer can be used for row-by-row global encoding.

[0119] The disparity cost volume of the first scale and the context features of the first scale are globally encoded row by row to obtain the initial disparity estimate of the first scale. The initial disparity estimate of the first scale is upsampled by a factor of 2 to obtain the initial disparity of the second scale.

[0120] Using the technical solution of the embodiments of this application, the initial disparity is obtained by upsampling the initial disparity estimate at the first scale by a factor of 2, providing a coarse-grained abstraction of the original image disparity. This first-scale initial disparity estimate significantly improves computational efficiency by compressing spatial resolution. Furthermore, this first-scale initial disparity estimate narrows the search space for subsequent optimization, reducing the lengthy full disparity search range in traditional algorithms to a smaller, high-probability region. This not only reduces unnecessary search computations but also provides a more reliable initial estimate for subsequent refined optimization, significantly improving the efficiency and accuracy of the overall algorithm.

[0121] On the basis of the above technical solution, as an embodiment, refer to Figure 3The sampling to obtain multiple candidate disparities based on the initial disparity and the fine disparity may include: performing local motion encoding on the initial disparity to obtain local motion features; fusing the local motion features with the fine disparity to obtain the disparity to be sampled; and uniformly sampling the left and right sides of the disparity to be sampled to obtain multiple candidate disparities.

[0122] The initial disparity reflects the spatial position difference between objects in the left and right images. Therefore, local motion encoding can be performed on the initial disparity to obtain local motion features. Local motion features are used to characterize the motion information of a local region.

[0123] Alternatively, the initial disparity can be converted into more semantic local motion features through local motion coding methods such as differential operations (such as the difference of adjacent pixel disparities) or directional histograms (such as disparity gradient directional statistics).

[0124] The local motion features and fine disparity are globally encoded line by line to obtain the disparity to be sampled. The left and right sides of the disparity to be sampled are uniformly sampled to obtain multiple candidate disparities. For each candidate disparity value, the local features at the corresponding position are extracted for subsequent fine prediction.

[0125] Using the technical solutions of the embodiments of this application, the sampled disparity combines local details with motion features, encompassing multi-scale features of spatial position and motion dynamics. A lightweight, row-by-row global encoding approach reduces computational complexity while preserving motion semantics, providing a compact, spatiotemporally correlated feature representation for real-time analysis.

[0126] On the basis of the above technical solution, as an embodiment, refer to Figure 3 The predicting of the final disparity based on multiple candidate disparities may include: predicting the weights, disparity increments and upsampling masks of the multiple candidate disparities; performing weighted processing on the disparity increments of the multiple candidate disparities according to the weights of the multiple candidate disparities to obtain an overall disparity increment; performing weighted processing on the upsampling masks of the multiple candidate disparities according to the weights of the multiple candidate disparities to obtain an overall upsampling mask; and upsampling the overall disparity increment based on the overall upsampling mask to obtain the final disparity.

[0127] A lightweight prediction network is introduced to combine the local row features of each candidate disparity position to concurrently predict the disparity increment, weight, and upsampling mask corresponding to each candidate disparity. Using the weights of multiple candidate disparities, the disparity increments corresponding to the multiple candidates are weighted summed to obtain the overall disparity increment. Using the weights of multiple candidate disparities, the upsampling masks corresponding to the multiple candidates are weighted summed to obtain the overall upsampling mask.

[0128] The final disparity is obtained by upsampling the global disparity delta based on the guidance of the global upsampling mask. The upsampling mask is used to restore the low-resolution disparity map to a high resolution consistent with the original image, further improving the edge preservation and spatial accuracy of the disparity map.

[0129] By adopting the technical solution of the embodiment of the present application, the fusion of multi-branch candidate disparity results can offset the mismatch caused by local noise or texture loss in a single branch, suppress outliers through weight distribution, and enhance the adaptability of disparity estimation to complex scenes; the weighting mechanism can dynamically allocate weights according to the prediction confidence of each branch in different areas, thereby retaining detailed features while correcting overall deviations; with limited increase in computational cost, accuracy improvement is achieved through information complementarity, adapting to application scenarios with high real-time requirements.

[0130] To facilitate better implementation of the binocular depth estimation method of the present application, the present application also provides a binocular depth estimation device based on the above binocular depth estimation method. The meanings of the terms herein are the same as those in the above binocular depth estimation method, and the specific implementation details can be referred to the description in the method embodiment.

[0131] See also Figure 4 , Figure 4 : is a schematic diagram of the structure of a binocular depth estimation device provided in an embodiment of the present application, wherein the binocular depth estimation device includes:

[0132] Image acquisition module 401, used to acquire binocular images;

[0133] A feature extraction module 402 is configured to perform multi-scale feature extraction on the binocular images to obtain multi-scale left image features, multi-scale right image features, and multi-scale context features; wherein the multi-scale includes a first scale and a second scale;

[0134] The feature fusion module 403 is used to fuse the left image features and the right image features at each scale to obtain a multi-scale disparity cost volume;

[0135] An initial disparity acquisition module 404 is configured to globally encode the disparity cost volume and context features of the first scale line by line to obtain an initial disparity;

[0136] A fine disparity acquisition module 405 is configured to globally encode the disparity cost volume and context features of the second scale line by line to obtain fine disparity;

[0137] A sampling module 406 is configured to sample and obtain a plurality of candidate disparities according to the initial disparity and the refined disparity;

[0138] A final prediction module 407, configured to predict a final disparity based on a plurality of candidate disparities;

[0139] The depth estimation module 408 is configured to estimate the binocular depth according to the final disparity.

[0140] In one embodiment, the feature extraction module 402 is specifically configured to:

[0141] Performing multi-scale feature extraction on the binocular images respectively to obtain the multi-scale left image features and the multi-scale right image features;

[0142] The multi-scale left image features are respectively input into the reverse residual module to obtain the multi-scale context features.

[0143] In one embodiment, the feature extraction module 402 is specifically configured to:

[0144] Using a lightweight visual encoder, multi-scale feature extraction is performed on the binocular images to obtain the multi-scale left image features, the multi-scale right image features and the multi-scale context features;

[0145] The lightweight visual encoder is a model that performs feature distillation based on a self-supervised learning model.

[0146] In one embodiment, the feature fusion module 403 is specifically configured to:

[0147] Correlating the features of the left image and the features of the right image at the first scale row by row to obtain a disparity cost volume at the first scale;

[0148] The features of the left image and the features of the right image at the second scale are correlated row by row to obtain a disparity cost volume at the second scale.

[0149] In one embodiment, the initial disparity acquisition module 404 is specifically configured to:

[0150] Globally encoding the disparity cost volume of the first scale and the context features of the first scale row by row to obtain an initial disparity estimate;

[0151] The initial disparity estimate is upsampled to obtain the initial disparity at the second scale.

[0152] In one embodiment, the sampling module 406 is specifically configured to:

[0153] Performing local motion coding on the initial disparity to obtain local motion features;

[0154] fusing the local motion feature and the fine disparity to obtain a disparity to be sampled;

[0155] The left and right sides of the to-be-sampled disparity are uniformly sampled to obtain a plurality of candidate disparities.

[0156] In one embodiment, the final prediction module 407 is specifically configured to:

[0157] Predicting weights, disparity increments, and upsampling masks for each of the plurality of candidate disparities;

[0158] performing weighted processing on the disparity increments of the plurality of candidate disparities according to respective weights of the plurality of candidate disparities to obtain an overall disparity increment;

[0159] performing weighted processing on the upsampling masks of the plurality of candidate disparities according to respective weights of the plurality of candidate disparities to obtain an overall upsampling mask;

[0160] The global disparity increment is upsampled based on the global upsampling mask to obtain the final disparity.

[0161] By adopting the technical solution of the embodiment of the present application, the multi-scale features determined can ensure that both fine local information can be captured and global changes in a large range of disparity can be adapted. Predicting the initial disparity can narrow the search space for subsequent optimization, reducing the lengthy full disparity search range in the traditional algorithm to a small high-probability area. This not only reduces unnecessary search calculations, but also provides a more reliable initial estimate for subsequent fine optimization, significantly improving the efficiency and accuracy of the overall algorithm. Based on multiple candidate disparities, efficient multi-branch parallel optimization can be achieved to obtain a more accurate final disparity, significantly improving the inference speed while ensuring the optimization quality. In this way, while ensuring the accuracy of depth estimation, the amount of calculation and power consumption are significantly reduced.

[0162] For the specific definition of the binocular depth estimation device, please refer to the definition of the binocular depth estimation method above, which will not be repeated here. Each module in the above-mentioned binocular depth estimation device can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0163] In addition, the present application also provides an electronic device, such as Figure 5 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0164] The electronic device may include one or more processing core processors 501 and one or more computer readable storage media memories 502 and other components. Those skilled in the art will understand that Figure 5The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0165] The processor 501 is the control center of the electronic device. It connects the various parts of the entire electronic device using various interfaces and lines. By running or executing software programs and / or modules stored in the memory 502 and accessing data stored in the memory 502, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interface, and application programs, and the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 501.

[0166] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0167] In one embodiment, the electronic device further includes a power supply 503 for supplying power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 503 can also include one or more DC or AC power supplies, a recharging system, a power supply device debugging circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0168] In one embodiment, the electronic device may further include an input unit 504, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0169] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 501 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 502 according to the following instructions, and the processor 501 will run the application programs stored in the memory 502, thereby implementing the steps in any of the binocular depth estimation methods provided in the embodiments of the present application.

[0170] When the specific electronic device is wearable optical see-through smart glasses, in addition to the above structure, it also includes at least a glasses body frame, an optical display component, an electronic circuit component, a sensor, etc. The sensors built into the glasses include a heart rate monitor, a blood glucose meter, a microphone, a camera and / or an eye tracker.

[0171] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0172] In one embodiment, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method described in any embodiment of the present application is implemented.

[0173] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in any embodiment of the present application is implemented.

[0174] In some embodiments, a computer program product is also proposed, including a computer program or instructions, which implements the method described in any embodiment of the present application when executed by a processor.

[0175] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0176] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0177] To this end, the present application provides a computer-readable storage medium having a computer program stored thereon. The computer program can be loaded by a processor to execute the steps in any binocular depth estimation method provided in the present application.

[0178] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0179] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0180] Since the instructions stored in the computer-readable storage medium can execute the steps in any binocular depth estimation method provided in the present application, the beneficial effects that can be achieved by any binocular depth estimation method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0181] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements that are inherent to such process, method, article, or terminal device. In the absence of further restrictions, an element defined by the phrase "comprises a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0182] The above is a detailed introduction to a binocular depth estimation method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A binocular depth estimation method, characterized in that: The method comprises: Acquire binocular images; Performing multi-scale feature extraction on the binocular images to obtain multi-scale left image features, multi-scale right image features, and multi-scale context features; wherein the multi-scale includes a first scale and a second scale; The features of the left image and the right image at each scale are fused separately to obtain the multi-scale disparity cost volume; Performing row-by-row global encoding on the disparity cost volume and context features of the first scale to obtain an initial disparity; Performing row-by-row global encoding on the disparity cost volume and context features of the second scale to obtain fine disparity; Sampling a plurality of candidate disparities according to the initial disparity and the refined disparity; Predicting the final disparity based on multiple candidate disparities; The binocular depth is estimated according to the final disparity.

2. The method according to claim 1, characterized in that The multi-scale feature extraction is performed on the binocular images respectively to obtain multi-scale left image features, multi-scale right image features and multi-scale context features, including: Performing multi-scale feature extraction on the binocular images respectively to obtain the multi-scale left image features and the multi-scale right image features; The multi-scale left image features are respectively input into the reverse residual module to obtain the multi-scale context features.

3. The method according to claim 1, characterized in that The multi-scale feature extraction is performed on the binocular images respectively to obtain multi-scale left image features, multi-scale right image features and multi-scale context features, including: Using a lightweight visual encoder, multi-scale feature extraction is performed on the binocular images to obtain the multi-scale left image features, the multi-scale right image features and the multi-scale context features; The lightweight visual encoder is a model that performs feature distillation based on a self-supervised learning model.

4. The method according to claim 1, wherein The feature fusion of the left image features and the right image features at each scale is performed to obtain a multi-scale disparity cost volume, including: Correlating the features of the left image and the features of the right image at the first scale row by row to obtain a disparity cost volume at the first scale; The features of the left image and the features of the right image at the second scale are correlated row by row to obtain a disparity cost volume at the second scale.

5. The method according to claim 1, wherein The globally encoding the disparity cost volume and the context features of the first scale line by line to obtain the initial disparity includes: Globally encoding the disparity cost volume of the first scale and the context features of the first scale row by row to obtain an initial disparity estimate; The initial disparity estimate is upsampled to obtain the initial disparity at the second scale.

6. The method according to claim 1, characterized in that The sampling to obtain a plurality of candidate disparities according to the initial disparity and the refined disparity includes: Performing local motion coding on the initial disparity to obtain local motion features; fusing the local motion feature and the fine disparity to obtain a disparity to be sampled; The left and right sides of the to-be-sampled disparity are uniformly sampled to obtain a plurality of candidate disparities.

7. The method according to claim 1, characterized in that The predicting of the final disparity according to the plurality of candidate disparities includes: Predicting weights, disparity increments, and upsampling masks for each of the plurality of candidate disparities; performing weighted processing on the disparity increments of the plurality of candidate disparities according to respective weights of the plurality of candidate disparities to obtain an overall disparity increment; performing weighted processing on the upsampling masks of the plurality of candidate disparities according to respective weights of the plurality of candidate disparities to obtain an overall upsampling mask; The global disparity increment is upsampled based on the global upsampling mask to obtain the final disparity.

8. A binocular depth estimation device, characterized in that: The device comprises: An image acquisition module, used to acquire binocular images; A feature extraction module is used to perform multi-scale feature extraction on the binocular images to obtain multi-scale left image features, multi-scale right image features and multi-scale context features; wherein the multi-scale includes a first scale and a second scale; The feature fusion module is used to fuse the features of the left image and the right image at each scale to obtain a multi-scale disparity cost volume; an initial disparity acquisition module, configured to globally encode the disparity cost volume and context features of the first scale line by line to obtain an initial disparity; a fine disparity acquisition module, configured to globally encode the disparity cost volume and context features of the second scale line by line to obtain fine disparity; a sampling module, configured to sample and obtain a plurality of candidate disparities according to the initial disparity and the refined disparity; A final prediction module, configured to predict a final disparity based on multiple candidate disparities; The depth estimation module is configured to estimate the binocular depth according to the final disparity.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the binocular depth estimation method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the binocular depth estimation method according to any one of claims 1 to 7 are implemented.

Images