Depth estimation method based on structured light phase guiding

Abstract

The application provides a depth estimation method based on structured light phase guidance, comprising: image acquisition of a to-be-measured object by a binocular imaging device to obtain a left-eye image and a right-eye image, wherein the left-eye image and the right-eye image contain a structured light pattern formed by a grating structure; phase feature coding of the left-eye image and the right-eye image based on stripe information in the structured light pattern to obtain a left-eye phase feature map and a right-eye phase feature map; feature fusion of the left-eye image and the left-eye phase feature map and the right-eye image and the right-eye phase feature map based on a phase attention mechanism to correspondingly obtain left-eye depth features and right-eye depth features; feature fusion of the left-eye depth features and the right-eye depth features based on matching confidence between the left-eye depth features and the right-eye depth features to obtain fused depth features; generation of a disparity map based on the fused depth features; and obtaining depth information of the to-be-measured object based on the disparity map.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more specifically, to a depth estimation method based on structured light phase guidance. Background Technology

[0002] Depth estimation is one of the core tasks in computer vision and 3D perception, and it is widely used in fields such as medical minimally invasive surgery navigation, industrial robot grasping, autonomous driving environmental perception, scene modeling, and remote sensing 3D mapping. Binocular stereo vision depth estimation refers to a computer vision technique that simulates the principle of human binocular vision by using two cameras at fixed positions to acquire binocular image pairs of the same scene, and then using pixel-level disparity calculation and geometric triangulation to infer the 3D spatial depth information of the scene. Binocular stereo vision depth estimation transforms binocular images of various scenes, affected by factors such as noise, distortion, reflection, and occlusion, into pixel-level accurate depth maps, providing core support for subsequent 3D reconstruction, navigation, and positioning tasks.

[0003] In recent years, the rapid development of deep learning in computer vision has enabled it to achieve results far exceeding traditional matching methods in binocular depth estimation tasks, leading to the proposal of an increasing number of deep learning-based depth estimation methods. However, most methods either rely on complex feature extraction networks and matching strategies, resulting in a large number of parameters and high computational costs; or they simplify the model structure in pursuit of real-time performance, sacrificing matching robustness. This makes it difficult to apply depth estimation tasks to edge devices with limited computing resources. To address these issues, a large number of lightweight depth estimation methods have been proposed.

[0004] Most current lightweight methods reduce inference time by simplifying network structure and reducing feature dimensions. However, this leads to decreased adaptability to complex scenes such as weakly textured and reflective regions, and depth maps are prone to issues like holes, blurred edges, and disparity jumps. Simply put, most current lightweight methods fail to achieve a good balance between real-time inference and depth estimation accuracy and robustness. Therefore, a binocular stereo vision depth estimation method is needed that can meet the low-latency deployment requirements of edge devices while maintaining accuracy and completeness in complex scenes. Summary of the Invention

[0005] In view of this, this application provides a depth estimation method based on structured light phase guidance.

[0006] One aspect of this application provides a depth estimation method based on structured light phase guidance, comprising: acquiring images of a test object using a binocular imaging device with a grating structure attached, obtaining a left-eye image and a right-eye image, wherein the left-eye image and the right-eye image contain a structured light pattern projected through the grating structure; encoding phase features of the left-eye image and the right-eye image based on stripe information in the structured light pattern, obtaining a left-eye phase feature map and a right-eye phase feature map; fusing features of the left-eye image and the left-eye phase feature map, and the right-eye image and the right-eye phase feature map respectively, based on a phase attention mechanism, to obtain left-eye depth features and right-eye depth features respectively; fusing features of the left-eye depth features and the right-eye depth features based on the matching confidence between the left-eye depth features and the right-eye depth features, obtaining fused depth features; generating a disparity map based on the fused depth features; and obtaining depth information of the test object based on the disparity map.

[0007] According to an embodiment of this application, the above-mentioned phase feature encoding of the left-eye image and the right-eye image based on the stripe information in the structured light pattern to obtain a left-eye phase feature map and a right-eye phase feature map includes: performing noise reduction processing on the left-eye image and the right-eye image respectively to obtain a first noise-reduced image and a second noise-reduced image; based on depthwise separable convolution, concatenating the left-eye image and the first noise-reduced image in the channel dimension and then performing feature encoding to obtain an initial left-eye feature map; and using a column attention mechanism, processing the initial left-eye feature map based on the stripe information in the structured light pattern. Phase feature encoding is performed to obtain an initial left-eye phase feature map; feature extraction and phase value normalization are performed on the initial left-eye phase feature map to obtain a left-eye phase feature map; based on depthwise separable convolution, the right-eye image and the second denoised image are concatenated in the channel dimension and then feature encoded to obtain an initial right-eye feature map; through a column attention mechanism, phase feature encoding is performed on the initial right-eye feature map based on the stripe information in the structured light pattern to obtain an initial right-eye phase feature map; feature extraction and phase value normalization are performed on the initial right-eye phase feature map to obtain a right-eye phase feature map.

[0008] According to an embodiment of this application, the above-mentioned feature fusion of the left eye image and left eye phase feature map, the right eye image and right eye phase feature map based on the phase attention mechanism to obtain left eye depth features and right eye depth features respectively includes: concatenating the left eye phase feature map and the first denoised image in the channel dimension and then performing feature encoding to obtain a first feature; concatenating the right eye phase feature map and the second denoised image in the channel dimension and then performing feature encoding to obtain a second feature; obtaining a first spatial attention mask corresponding to the left eye phase feature map and a second spatial attention mask corresponding to the right eye phase feature map based on the phase attention mechanism; weighting the feature residuals of the first feature based on the first spatial attention mask to obtain the left eye depth feature; and weighting the feature residuals of the second feature based on the second spatial attention mask to obtain the right eye depth feature.

[0009] According to an embodiment of this application, the above-mentioned feature fusion of the left-eye depth features and the right-eye depth features based on the matching confidence between the left-eye depth features and the right-eye depth features to obtain fused depth features includes: obtaining a matching confidence map based on the left-eye depth features, the right-eye depth features, the left-eye phase feature map, and the right-eye phase feature map; and performing feature fusion of the left-eye depth features and the right-eye depth features based on the matching confidence map to obtain fused depth features.

[0010] According to an embodiment of this application, obtaining a matching confidence map based on the left eye depth feature, the right eye depth feature, the left eye phase feature map, and the right eye phase feature map includes: concatenating the left eye depth feature and the right eye depth feature along the channel dimension and then performing feature encoding to obtain a feature similarity map; calculating the cosine similarity of the left eye phase feature map and the right eye phase feature map to obtain a cosine similarity feature; performing feature encoding on the cosine similarity feature to obtain a phase consistency map; dividing the left eye depth feature and the right eye depth feature into multiple feature groups according to the channel dimension; for any feature group among the multiple feature groups, performing normalized cross-correlation calculation on the left eye depth feature and the right eye depth feature to obtain a cross-correlation map; fusing the multiple cross-correlation maps to obtain a single correlation map; and obtaining a matching confidence map based on the feature similarity map, the phase consistency map, and the single correlation map.

[0011] According to an embodiment of this application, obtaining a matching confidence map based on the feature similarity map, the phase consistency map, and the single correlation map includes: concatenating the feature similarity map, the phase consistency map, and the single correlation map and then performing feature encoding to obtain a weighted feature map. The weighted feature map is divided according to the channel dimension and includes a first weighted feature corresponding to the feature similarity map, a second weighted feature corresponding to the phase consistency map, and a third weighted feature corresponding to the single correlation map. The feature similarity map, the phase consistency map, and the single correlation map are weighted based on the first weighted feature, the second weighted feature, and the third weighted feature to obtain the matching confidence map.

[0012] According to an embodiment of this application, generating a disparity map based on the aforementioned fusion depth features includes: obtaining an initial disparity center and an initial disparity width based on the aforementioned fusion depth features through multi-scale analysis; generating channel modulation coefficients based on the aforementioned initial disparity center and the aforementioned initial disparity width; using the aforementioned channel modulation coefficients to perform feature modulation on the channel dimensions of the aforementioned fusion depth features to obtain enhanced depth features; and generating a disparity map based on the aforementioned enhanced depth features.

[0013] According to an embodiment of this application, the above-mentioned method of obtaining the initial disparity center and initial disparity width based on the fusion depth features through multi-scale analysis includes: performing global average pooling on the fusion depth features to obtain a first pooling feature; performing feature encoding on the first pooling feature to obtain a first disparity feature; dividing the fusion depth features into multiple sub-features based on a grid; performing pooling processing on the multiple sub-features to obtain multiple second pooling features; performing feature encoding on the multiple second pooling features to obtain multiple second initial disparity features; performing feature fusion on the multiple second initial disparity features based on a spatial attention mechanism to obtain a second disparity feature; performing global max pooling on the fusion depth features to obtain a third pooling feature; concatenating the first pooling feature and the third pooling feature and then performing feature encoding to obtain a third disparity feature; performing feature fusion on the first disparity feature, the second disparity feature, and the third disparity feature to obtain an initial disparity feature; and performing feature decoding on the initial disparity feature to obtain the initial disparity center and initial disparity width.

[0014] According to an embodiment of this application, the above-mentioned generation of a disparity map based on the enhanced depth features includes: obtaining a first disparity map based on the enhanced depth features; obtaining a fusion feature based on the left eye phase feature map and the first disparity map; performing feature enhancement processing on the first disparity map based on the fusion feature through a gating mechanism and residual learning to obtain a second disparity map; performing normalization processing on the second disparity map to obtain a third disparity map; and performing mapping processing on the third disparity map according to an initial disparity range determined by the initial disparity center and the initial disparity width to obtain a disparity map.

[0015] According to an embodiment of this application, obtaining fused features based on the left eye phase feature map and the first disparity map includes: extracting features from the left eye phase feature map to obtain a first feature map; extracting features from the first disparity map to obtain a second feature map; and fusing the first feature map and the second feature map based on depthwise separable convolution to obtain fused features.

[0016] According to embodiments of this application, a binocular imaging device with a grating structure is used to acquire images of the object under test, obtaining a left-eye image and a right-eye image containing a structured light pattern projected through the grating structure. Based on the stripe information in the structured light pattern, phase feature encoding is performed on the left-eye and right-eye images to obtain left-eye phase feature maps and right-eye phase feature maps. Then, based on a phase attention mechanism and matching confidence, feature fusion is performed on the left-eye and right-eye phase feature maps to obtain fused depth features. This avoids the problem of weak texture structure features in conventional binocular acquired images, thereby obtaining more accurate fused depth features and determining the depth information of the object under test based on the fused depth features, improving the accuracy of binocular depth estimation. Attached Figure Description

[0017] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 A schematic diagram of a binocular imaging device according to an embodiment of this application is shown;

[0019] Figure 2 A flowchart illustrating a depth estimation method based on structured light phase guidance according to an embodiment of this application is shown schematically.

[0020] Figure 3 A schematic diagram of the structure of a depth information estimation network according to an embodiment of this application is shown;

[0021] Figure 4 A block diagram of an electronic device according to an embodiment of this application is shown schematically. Detailed Implementation

[0022] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0023] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms "comprising," "including," etc., as used herein indicate the presence of the above-described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0024] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0025] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0026] In the embodiments of this application, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0027] In the embodiments of this application, the user's authorization or consent was obtained before obtaining or collecting the user's personal information.

[0028] In the embodiments of this application, the depth estimation method based on structured light phase guidance can be applied to medical surgical scenarios. Figure 1 A schematic diagram of a binocular imaging device according to an embodiment of this application is shown.

[0029] like Figure 1As shown, the binocular imaging device according to an embodiment of this application includes an endoscope cold light source 101, a beam guide 102, a condenser lens group 103, an optical fiber 104, a micro-projection lens 105, an endoscope left lens 1061, an endoscope right lens 1062, and a grating sheet 107.

[0030] The endoscope cold light source 101 provides stable and high-brightness cold light. The light is transmitted to the condenser lens group 103 via the beam guide 102. The condenser lens group 103 converges, collimates, and optimizes the light, which is then efficiently transmitted to the micro-projection lens 105 via the optical fiber 104. Finally, the micro-projection lens 105 accurately projects the light onto the observation area of ​​the endoscope. The grating plate 107 is located in front of the micro-projection lens 105. After passing through the grating plate 107, the light can generate a high-brightness, high-contrast striped projection in the observation area of ​​the endoscope. The left endoscope lens 1061 is used to acquire images of the object under test, obtaining a left-eye image containing the structured light pattern formed by the grating plate 107. The right endoscope lens 1062 is used to acquire images of the object under test, obtaining a right-eye image containing the structured light pattern formed by the grating plate 107. The aforementioned binocular imaging device combines miniaturization with high-quality projection, which can significantly enhance the structural features of weakly textured tissue surfaces in the surgical field of endoscopy, thereby improving the input image quality of the back-end depth estimation network.

[0031] Figure 2 A flowchart illustrating a depth estimation method based on structured light phase guidance according to an embodiment of this application is shown schematically.

[0032] like Figure 2 As shown, the depth estimation method based on structured light phase guidance according to the embodiments of this application includes operations S210 to S260.

[0033] In operation S210, the binocular imaging device with a grating structure is used to acquire images of the object under test, and obtain left and right eye images. The left and right eye images contain structured light patterns projected through the grating structure.

[0034] A binocular imaging device, also known as a binocular stereo vision system, is a system that uses two parallel cameras (mimicking human eyes) to acquire three-dimensional geometric information of objects. Simply put, the core principle of a binocular imaging device is to use two cameras to capture the same scene from different angles, and then calculate the depth (distance) information of the object by calculating the positional difference (parallax) of corresponding pixels in the two images. For example, when you look at the same object with your left and right eyes respectively, the object's position on the retinas of your left and right eyes is slightly different. This difference in position is called parallax. In a binocular system, the closer the object is to the camera, the greater its horizontal offset (parallax) in the left and right eye images; the farther away the object is, the smaller the parallax (parallax is zero at infinity).

[0035] Structured light patterns refer to light spots, stripes, or grids with specific coding rules that are actively projected onto the surface of an object using lasers or projectors. Structured light patterns can be designed according to actual needs, and their forms include, but are not limited to: discrete light spots, bar coding, black and white stripes of different widths, Gray code, and shifting stripes.

[0036] In operation S220, based on the stripe information in the structured light pattern, phase feature encoding is performed on the left and right eye images to obtain the left eye phase feature map and the right eye phase feature map.

[0037] Phase feature coding is a feature extraction method that transforms the phase information of an image in the frequency domain into a discrete, stable binary code form to describe the structure of textures, edges, and orientations. It is extremely robust to changes in illumination, blur, and contrast, and is one of the core technologies for texture analysis, edge detection, orientation estimation, and image matching.

[0038] In operation S230, based on the phase attention mechanism, feature fusion is performed on the left eye image and left eye phase feature map, and the right eye image and right eye phase feature map, respectively, to obtain the left eye depth feature and right eye depth feature.

[0039] Traditional self-attention mechanisms calculate similarity using only features such as amplitude, ignoring phase information in the image. In contrast, phase attention mechanisms use phase as the core matching criterion, leveraging phase's strong representational ability of structure, temporal sequence, and periodicity to improve robustness against noise, degradation, and phase shift. Based on the phase attention mechanism, feature fusion is performed on the left eye image and left eye phase feature map to obtain left eye depth features, and similarly, feature fusion is performed on the right eye image and right eye phase feature map to obtain right eye depth features.

[0040] In operation S240, based on the matching confidence between the left and right eye depth features, the left and right eye depth features are fused to obtain fused depth features.

[0041] Matching confidence represents the degree of certainty that a pixel in the left-eye image and a pixel in the right-eye image belong to the same spatial point. In the embodiments of this application, stereo matching can be performed on the left-eye depth features and right-eye depth features first, and the matching confidence corresponding to each pair of pixels can be calculated. When the matching confidence is higher, the feature of that pixel has a greater weight in feature fusion; when the matching confidence is lower, the feature of that pixel has a smaller weight in feature fusion, or is even suppressed, ultimately resulting in a more robust, more accurate, and less noisy fused depth feature.

[0042] In operation S250, a disparity map is generated based on the fused depth features.

[0043] In embodiments of this application, a disparity map can be generated by inputting fused depth features into a pre-trained disparity map model.

[0044] In operation S260, the depth information of the object under test is obtained based on the disparity map.

[0045] In the embodiments of this application, the three-dimensional coordinates of each point on the surface of the object can be calculated by using geometric formulas based on the triangulation principle in stereo vision and the parameters of the calibrated binocular imaging device (such as baseline distance, focal length, etc.), using the disparity value corresponding to the disparity map, and finally obtaining the depth information of the object.

[0046] Through the embodiments of this application, a binocular imaging device with a grating structure is used to acquire images of the object under test, obtaining a left-eye image and a right-eye image containing a structured light pattern projected through the aforementioned grating structure. Based on the stripe information in the structured light pattern, phase feature encoding is performed on the left-eye and right-eye images to obtain left-eye phase feature maps and right-eye phase feature maps. Then, based on a phase attention mechanism and matching confidence, feature fusion is performed on the left-eye and right-eye phase feature maps to finally obtain fused depth features. This avoids the problem of weak texture structure features in conventional binocular acquired images, thereby obtaining more accurate fused depth features and determining the depth information of the object under test based on the fused depth features, improving the accuracy of binocular depth estimation.

[0047] According to an embodiment of this application, phase feature encoding is performed on the left and right eye images based on the stripe information in the structured light pattern to obtain a left eye phase feature map and a right eye phase feature map. This includes: performing noise reduction processing on the left and right eye images respectively to obtain a first denoised image and a second denoised image; concatenating the left eye image and the first denoised image along the channel dimension using depthwise separable convolution and then performing feature encoding to obtain an initial left eye feature map; and performing phase encoding on the initial left eye feature map using a column attention mechanism based on the stripe information in the structured light pattern. Feature encoding is performed to obtain the initial left-eye phase feature map; feature extraction and phase value normalization are performed on the initial left-eye phase feature map to obtain the left-eye phase feature map; based on depthwise separable convolution, the right-eye image and the second denoised image are concatenated in the channel dimension and then feature encoded to obtain the initial right-eye feature map; through a column attention mechanism, the stripe information in the structured light pattern is used as a reference to perform phase feature encoding on the initial right-eye feature map to obtain the initial right-eye phase feature map; feature extraction and phase value normalization are performed on the initial right-eye phase feature map to obtain the right-eye phase feature map.

[0048] In the embodiments of this application, the feature encoding processes for the left and right eye images can be completely identical or may differ to some extent. The left eye image is used as the original image I. raw , will I raw The denoised image I is obtained after processing by a lightweight denoising network. denoised Lightweight denoising networks are a class of deep learning models specifically designed for resource-constrained scenarios. Their core objective is to significantly reduce the number of parameters, computational cost, and memory usage while maintaining denoising effectiveness, achieving low power consumption, low latency, and real-time inference. The original image I... raw and noise reduction image I denoised After concatenation along the channel dimension, a subnetwork f consisting of depthwise separable convolutions and column attention is applied. pre The initial feature map is obtained as follows:

[0049] Feature=f pre ([I raw ;I denoised ]) (1).

[0050] Subnetwork f pre It is a feature extraction submodule. As a small, functionally independent neural network unit, it is responsible for receiving the input image and then filtering and extracting useful information from the input image through two core operations: depthwise separable convolution and column attention. Finally, it outputs an initial feature map that represents the key features of the image.

[0051] Input the feature map into the phase head The output normalized phase value is mapped to [0,2] through a linear transformation. The phase feature map can be obtained from the interval:

[0052] (2);

[0053] Where tanh is the hyperbolic tangent function.

[0054] Phase Head (hereinafter referred to as) The `phase-predicting-head` is a dedicated sub-network module in deep learning (especially computer vision, structured light phase estimation, and stereo matching) specifically designed to predict phase information from a general feature map. Essentially, it's a lightweight prediction head that takes a general feature map as input and outputs position-by-position phase values. Through this process, the left-eye phase feature map corresponding to the left-eye image can be obtained.

[0055] In parallel, the initial feature map can be input into the confidence head. The confidence plot is obtained as follows:

[0056] (3);

[0057] Where σ is the Sigmoid function.

[0058] Confidence head The feature prediction branch, containing at least one convolutional layer, is used to predict the confidence value corresponding to each feature location based on the input feature map. This confidence value is then mapped to the [0, 1] interval using the Sigmoid function to obtain the confidence map c, which characterizes the reliability of the prediction results at each location. Through this process, the left-eye confidence map corresponding to the left-eye image can be obtained.

[0059] Based on the same processing procedure, the right eye image can be used as the original image I. raw Input is taken, and the right eye phase feature map and right eye confidence map are finally obtained. The left eye phase feature map and right eye phase feature map reflect the periodicity or gradient direction information of the structure in the image and are not sensitive to changes in illumination.

[0060] According to embodiments of this application, based on a phase attention mechanism, feature fusion is performed on the left-eye image and left-eye phase feature map, and the right-eye image and right-eye phase feature map, respectively, to obtain left-eye depth features and right-eye depth features. This includes: concatenating the left-eye phase feature map and the first denoised image along the channel dimension and then performing feature encoding to obtain a first feature; concatenating the right-eye phase feature map and the second denoised image along the channel dimension and then performing feature encoding to obtain a second feature; based on the phase attention mechanism, obtaining a first spatial attention mask corresponding to the left-eye phase feature map and a second spatial attention mask corresponding to the right-eye phase feature map; weighting the feature residuals of the first feature based on the first spatial attention mask to obtain the left-eye depth feature; and weighting the feature residuals of the second feature based on the second spatial attention mask to obtain the right-eye depth feature.

[0061] In the embodiments of this application, the processes of fusing features from the left eye image and the left eye phase feature map to obtain left eye depth features, and fusing features from the right eye image and the right eye phase feature map to obtain right eye depth features, can be completely identical or different.

[0062] Taking the processing of the left eye image and the left eye phase feature map as an example, the denoised image I corresponding to the left eye image is... denoised After being concatenated with the left eye phase feature map Φ along the channel dimension, the initial feature F is obtained by extracting features through a pre-set lightweight feature extraction network. init And obtains the initial features F through a preset residual learning module. init The corresponding feature residual ResBlock(F) init Simultaneously, through a phase attention mechanism, a spatial attention mask Att(Φ) is generated using the left eye phase feature map Φ. This mask is used to weight the feature residuals, thereby obtaining the depth feature F. out :

[0063] F out =F init +ResBlock(F init )⊙Att(Φ) (4;

[0064] Here, ⊙ represents element-wise multiplication.

[0065] Through the above process, the left eye depth feature F corresponding to the left eye image can be obtained. L Based on the same processing procedure, the right eye depth feature F corresponding to the right eye image can be obtained. R In embodiments of this application, the obtained depth features can be further enhanced using efficient channel attention.

[0066] According to an embodiment of the present application, based on the matching confidence between the left-eye depth feature and the right-eye depth feature, feature fusion is performed on the left-eye depth feature and the right-eye depth feature to obtain a fused depth feature, including: obtaining a matching confidence map based on the left-eye depth feature, the right-eye depth feature, the left-eye phase feature map, and the right-eye phase feature map; performing feature fusion on the left-eye depth feature and the right-eye depth feature based on the matching confidence map to obtain a fused depth feature.

[0067] The matching confidence map W measures the reliability of the matching between the left-eye depth feature and the right-eye depth feature pixel by pixel. The closer its value is to 1, the more credible the matching is, and the closer it is to 0, the more likely the matching may be ambiguous (such as occlusion, weak texture, or reflection area). The fusion of the left and right features can be guided in the following way to obtain the fused depth feature F fused :

[0068] F fused = F L + W ⊙ |F L - F R | (5);

[0069] In the high-confidence region (W ≈ 1), if the difference |F L - F R | between the left-eye depth feature and the right-eye depth feature is small, the fused feature F fused is approximately equal to the left-eye depth feature F L , retaining the original information; if the difference |F L - F R | between the left-eye depth feature and the right-eye depth feature is large, the right-eye depth feature is introduced by weighted difference to supplement information.

[0070] In the low-confidence region (W ≈ 0), even if the difference between the left-eye depth feature and the right-eye depth feature is large, its contribution is suppressed, and the fused depth feature is still mainly the left-eye depth feature, avoiding the introduction of noise by unreliable differences.

[0071] According to an embodiment of this application, a matching confidence map is obtained based on left-eye depth features, right-eye depth features, left-eye phase feature map, and right-eye phase feature map, including: concatenating the left-eye depth features and right-eye depth features along the channel dimension and then performing feature encoding to obtain a feature similarity map; calculating the cosine similarity of the left-eye phase feature map and right-eye phase feature map to obtain a cosine similarity feature; performing feature encoding on the cosine similarity feature to obtain a phase consistency map; dividing the left-eye depth features and right-eye depth features into multiple feature groups according to the channel dimension; for any feature group among the multiple feature groups, performing normalized cross-correlation calculation on the left-eye depth features and right-eye depth features to obtain a cross-correlation map; fusing the multiple cross-correlation maps to obtain a single correlation map; and obtaining a matching confidence map based on the feature similarity map, phase consistency map, and single correlation map.

[0072] In the embodiments of this application, the matching confidence map can be generated through multiple matching cues. Taking matching cues for three pathways as an example, three feature maps corresponding to feature similarity, phase consistency, and group correlation can be generated respectively.

[0073] Left eye depth feature F L And right eye depth feature F R After being concatenated along the channel dimension, the features are encoded using a pre-defined similarity extraction network to obtain the feature similarity map S. phase .

[0074] Calculate the left eye phase feature map Φ L and right eye phase feature map Φ R The cosine similarity is used to obtain the similarity Sim=cos(Φ). L -Φ R The similarity value Sim and its complementary value 1-Sim are concatenated according to the channel dimension, and then feature encoding is performed to obtain the phase consistency map S. phase .

[0075] The left and right eye depth features are divided into G groups according to the channel dimension. The normalized cross-correlation within each group is calculated to obtain G cross-correlation maps. These are then projected into a single cross-correlation map C through convolution. proj Where G is a positive integer. Taking a left-eye depth feature with a channel dimension of 32 and G of 4 as an example, after grouping, the channel dimensions of the left-eye and right-eye depth features in each group are both 8. The left-eye and right-eye depth features with 8 channel dimensions in each group are normalized and cross-correlated to obtain cross-correlation maps. Since there are 4 groups, 4 cross-correlation maps are obtained in the end. Feature fusion is performed on these 4 cross-correlation maps to obtain a single cross-correlation map C. proj .

[0076] According to an embodiment of this application, a matching confidence map is obtained based on a feature similarity map, a phase consistency map, and a single correlation map. This includes: concatenating the feature similarity map, the phase consistency map, and the single correlation map and then performing feature encoding to obtain a weighted feature map. The weighted feature map is divided by channel dimension and includes a first weighted feature corresponding to the feature similarity map, a second weighted feature corresponding to the phase consistency map, and a third weighted feature corresponding to the single correlation map. The feature similarity map, the phase consistency map, and the single correlation map are weighted based on the first weighted feature, the second weighted feature, and the third weighted feature to obtain the matching confidence map.

[0077] S feat S phase C proj After concatenation based on channel dimensions, a lightweight fusion convolutional layer generates a weighted feature map. This weighted feature map is divided by channel dimension, including a first weighted feature corresponding to the feature similarity map, a second weighted feature corresponding to the phase consistency map, and a third weighted feature corresponding to the single correlation map. Since the weight values ​​represented by the first, second, and third weighted features can be arbitrary real numbers, and these weight values ​​lack a uniform scale, it's impossible to directly determine the respective proportions of each weighted feature. Therefore, a normalized exponential function can be used to normalize the weighted feature map, obtaining adaptive first weighted feature W1, adaptive second weighted feature W2, and adaptive third weighted feature W3, such that W1 + W2 + W3 = 1. Finally, the matching confidence map W can be represented as:

[0078] W=W1⊙S feat +W2⊙S phase +W3⊙C proj (6);

[0079] According to an embodiment of this application, generating a disparity map based on fused depth features includes: obtaining an initial disparity center and an initial disparity width based on the fused depth features through multi-scale analysis; generating channel modulation coefficients based on the initial disparity center and the initial disparity width; using the channel modulation coefficients to perform feature modulation on the channel dimensions of the fused depth features to obtain enhanced depth features; and generating a disparity map based on the enhanced depth features.

[0080] In the embodiments of this application, the initial disparity center represents the mean disparity prediction of the current pixel or region, and the initial disparity width represents the disparity distribution range. The channel modulation coefficients correspond to the channel dimensions of the fused depth features and are used to characterize the contribution of different feature channels to disparity prediction. Using the generated channel modulation coefficients, feature modulation operations such as weighting or scaling are performed on the fused depth features along the channel dimensions to suppress feature channels that contribute less to disparity estimation and enhance feature channels that are sensitive and effective for disparity estimation, thereby obtaining enhanced depth features. Finally, based on the enhanced depth features, a final disparity map is generated through disparity regression or classification, achieving high-precision depth estimation.

[0081] According to embodiments of this application, an initial disparity center and an initial disparity width are obtained through multi-scale analysis based on fused depth features, including: performing global average pooling on the fused depth features to obtain a first pooling feature; performing feature encoding on the first pooling feature to obtain a first disparity feature; dividing the fused depth features into multiple sub-features based on a grid; performing pooling processing on the multiple sub-features respectively to obtain multiple second pooling features; performing feature encoding on the multiple second pooling features respectively to obtain multiple second initial disparity features; performing feature fusion on the multiple second initial disparity features based on a spatial attention mechanism to obtain a second disparity feature; performing global max pooling on the fused depth features to obtain a third pooling feature; concatenating the first pooling feature and the third pooling feature and then performing feature encoding to obtain a third disparity feature; performing feature fusion on the first disparity feature, the second disparity feature, and the third disparity feature to obtain an initial disparity feature; and performing feature decoding on the initial disparity feature to obtain the initial disparity center and the initial disparity width.

[0082] The fused deep features F can be processed from three branches using a pre-defined predictor module. fused The process yields three prediction results, each containing two parameter values: disparity center and disparity width.

[0083] In the global statistics branch, global average pooling can be used to fuse the depth features F. fused The first pooling feature is obtained by processing, and then the first pooling feature is encoded to obtain the first disparity feature. The first disparity feature is divided according to the channel dimension, including the first disparity center feature Cg corresponding to the disparity center and the first disparity width feature Rg corresponding to the disparity width.

[0084] In the regional statistics branch, the fused deep features F can be based on a 2×2 grid. fusedThe features are divided into four sub-features, and each of the four sub-features is pooled to obtain four second pooled features. Then, each of the four second pooled features is encoded to obtain four second initial disparity features. Finally, the four second initial disparity features are weighted and aggregated through a spatial attention mechanism to obtain the second disparity features. The second disparity features are divided according to the channel dimension, including the second disparity center feature Cr corresponding to the disparity center and the second disparity width feature Rr corresponding to the disparity width.

[0085] In the extreme value statistics branch, global max pooling can be used to fuse the deep features F. fused The third pooling feature is obtained through processing. Then, the first pooling feature and the third pooling feature are concatenated according to the channel dimension and then feature encoded to obtain the third disparity feature. The third disparity feature is divided according to the channel dimension into the third disparity center feature Ce corresponding to the disparity center and the third disparity width feature Re corresponding to the disparity width.

[0086] The first disparity center feature Cg and the third disparity center feature Ce can be fused to obtain the initial disparity center C. The first disparity width feature Rg, the second disparity width feature Rr and the third disparity width feature Re can be fused to obtain the initial disparity width R.

[0087] The predictor module obtains the initial disparity center C and the initial disparity width R, thereby dynamically determining the initial disparity range of the current scene as [CR / 2, C+R / 2]. This initial disparity range is used to fuse the depth features F. fused By modulating the channel dimension, the network's ability to perceive the deep structure of the scene is enhanced, thereby obtaining enhanced deep features F. enhanced .

[0088] According to an embodiment of this application, generating a disparity map based on enhanced depth features includes: obtaining a first disparity map based on enhanced depth features; obtaining fusion features based on the left eye phase feature map and the first disparity map; performing feature enhancement processing on the first disparity map based on the fusion features through a gating mechanism and residual learning to obtain a second disparity map; performing normalization processing on the second disparity map to obtain a third disparity map; and performing mapping processing on the third disparity map based on an initial disparity range determined by an initial disparity center and an initial disparity width to obtain a disparity map.

[0089] In embodiments of this application, enhanced depth features F can be used. enhanced Input a pre-trained disparity map model to obtain the first disparity map D. init Subsequently, the left eye phase feature map Φ was used. L And the first disparity map D initThe fusion feature F is obtained. Gate(F) is obtained by processing the fusion feature F through a gating mechanism, and ResBlock(F) is obtained by processing the fusion feature F through residual learning. Then the second disparity map D is obtained. refined It can be represented as:

[0090] D refined =D init +Gate(F)⊙ResBlock(F) (7);

[0091] Finally, using the Sigmoid function and the initial disparity range [CR / 2, C+R / 2] obtained above, D is... refined The disparity map is obtained by mapping.

[0092] In the embodiments of this application, the initial disparity is finely adjusted by using the phase information through the gated residual learning mechanism. The core idea is that the phase information provides clues to the geometric consistency of the scene, and the network determines the position and magnitude of the initial disparity that needs to be corrected based on this information.

[0093] According to an embodiment of this application, a fused feature is obtained based on the left eye phase feature map and the first disparity map, including: extracting features from the left eye phase feature map to obtain a first feature map; extracting features from the first disparity map to obtain a second feature map; and fusing the first feature map and the second feature map based on depthwise separable convolution to obtain the fused feature.

[0094] The fusion feature F refers to using depthwise separable convolutional layers to process the first disparity map D separately. init and left eye phase feature map Φ L Feature extraction is performed, and the resulting feature maps are concatenated according to channel dimension. These concatenated maps are then fused using consecutive depthwise separable convolutional blocks to obtain a joint feature representation. The fused feature F and the fused depth feature F are then discussed. fused These are two completely different features, and the ways in which they are obtained are also fundamentally different.

[0095] This application also provides a disparity map training model that can obtain a first disparity map based on fused deep features. The disparity map training model includes an encoder and a decoder. The encoder can be composed of multiple downsampling modules stacked together; at the bottleneck layer, spatial self-attention is introduced: the feature map is reduced to a low resolution through adaptive pooling, efficient multi-head self-attention is performed on this low-resolution feature sequence, and then the result is upsampled and the residuals are concatenated back to the original features. Simultaneously, a global feature vector can be extracted and transformed into a modulation vector M matching the number of channels in each decoding layer through a series of independent projection layers. i , where i represents the i-th decoding layer. During decoding, the output of the i-th layer is modulated as:

[0096] Out i =Decode i (·)⊙(1+M i (8);

[0097] Among them, Decode i (·) represents the original output of the i-th decoding layer.

[0098] This global-local feature co-modulation mechanism greatly enhances expressiveness. The decoder can consist of multiple upsampling modules, recovering resolution through bilinear upsampling and skip connections.

[0099] In the embodiments of this application, the dataset used to train the disparity map training model may include synthetic datasets and real datasets. Due to the scarcity of stripe structured light depth estimation datasets for surgical scenarios, this application proposes a data augmentation and simulation method based on structured light projection to generate a high-quality synthetic dataset suitable for training binocular stereo matching algorithms. This method is based on the publicly available endoscopic depth dataset SCARED, and its core lies in converting depth information into surface deformation cues to simulate the geometric deformation generated by structured light projection on non-planar surfaces, enriching the surgical scenario dataset for training the network. The core process of this method includes: ① Depth data preprocessing and restoration: removing outliers, suppressing noise, filling holes, and optimizing connected regions in the original depth map; ② Structured stripe pattern generation: creating a projection stripe template with sinusoidal periodicity; ③ Depth-based stripe deformation simulation: simulating the deformation effect of the projection stripes on the object surface based on the physical characteristics of the restored depth map; ④ Multimodal data fusion and distortion correction: fusing the deformed stripes with the original RGB image and performing camera distortion correction.

[0100] To optimize the performance of the disparity map training model in real-world scenarios, this application also created a real-world dataset containing animal tissue and human organ models. The real-world dataset was acquired using synchronized striped structured light projection via a commercial projector and a binocular electronic endoscope. A twelve-step phase-shifting method and complementary Gray code method were employed for phase resolution to obtain a high-precision disparity map, which served as the ground truth for training the disparity map training model, thereby improving the robustness of the model network in real-world scenarios.

[0101] In embodiments of this application, the training strategy for the disparity map training model can employ a unified hybrid supervised loss function, combining supervised loss with self-supervised loss based on phase consistency. The loss function can be expressed as:

[0102] Ltotal=λ reg (Lreg+λ) gard Lgrad) + λ phase Lphase+λsmooth Lsmooth (9);

[0103] Where Lreg is the regression loss, and λ reg Here, λ represents the regression loss coefficient, Lgrad represents the gradient loss, and λ represents the λ-value. gard λ is the gradient loss coefficient, Lgrad is the phase consistency loss, and λ is the gradient loss coefficient. phase λ is the phase consistency loss coefficient, Lsmooth is the smoothness loss, and λ is the phase consistency loss coefficient. smooth This is the smoothness loss coefficient.

[0104] In the embodiments of this application, the predicted disparity D can be predicted based on the phase position confidence c to obtain a more accurate regression loss result. The phase position confidence regression loss can be expressed as:

[0105] L reg-conf= |D-Dgt|c-αlog(c) (10;

[0106] Where Dgt is the true disparity, c is the phase position confidence corresponding to the predicted disparity D, and α is a manually set hyperparameter.

[0107] Introducing a phase position confidence level (c) can help reduce the confidence level of the trained model in high error regions. Furthermore, in the embodiments of this application, the gradient loss Lgrad is added to calculate the L1 loss between the first-order gradients of the predicted disparity and the true disparity along the coordinate axes of the disparity map, thus maintaining sharp edges.

[0108] In the embodiments of this application, the predicted disparity D can be used to transform the right eye phase feature map Φ R The image is warped in the opposite direction to the coordinate system of the left eye image, resulting in Φ. R→L And calculate the left eye phase feature map Φ. L With the distorted phase diagram Φ R→L The cosine similarity yields the phase consistency loss as:

[0109] Lphase=E[1-cos(Φ L -Φ R→L )] (11;

[0110] Here, E is used to characterize mathematical expectation.

[0111] Phase consistency loss does not require true disparity; it emphasizes that the disparity predicted by the trained model must satisfy the phase consistency between the left and right eye images, providing a strong geometric constraint for matching texture-deficient regions.

[0112] The smoothness loss Lsmooth is used to characterize the edge-aware smoothing loss based on image gradients, and allows for disparity discontinuities in areas with large image gradients.

[0113] In embodiments of the present invention, the training process of the disparity map training model can be implemented based on the AdamW optimizer, and cosine annealing can be used to increase the learning rate during model training. Optionally, exponential moving average (EMA) can be used to smooth the model weights and improve generalization. Mixed precision training can also be employed to accelerate the training process.

[0114] This application also provides a depth information estimation network. Figure 3 A schematic diagram of the structure of a depth information estimation network according to an embodiment of this application is shown. Figure 3 As shown, the depth information estimation network according to the embodiments of this application includes a preprocessing module 310, a feature extraction module 320, a binocular interaction module 330, a feature fusion module 340, a multi-scale disparity range predictor 350, a lightweight encoder-decoder 360, and a disparity optimization output module 370.

[0115] The preprocessing module 310 is used to acquire the left and right eye images and perform noise reduction processing on the left and right eye images through a lightweight denoising network to obtain the first denoised image and the second denoised image.

[0116] The feature extraction module 320 is used to obtain left eye depth features and right eye depth features based on the left eye image, the right eye image, the first denoised image, and the second denoised image.

[0117] The binocular interaction module 330 is used to generate a matching confidence map based on three pathway matching cues.

[0118] The feature fusion module 340 is used to fuse the left and right eye depth features based on the matching confidence map to obtain fused depth features.

[0119] The multi-scale disparity range predictor 350 is used to obtain the initial disparity center and initial disparity width from the fused depth features, and to perform feature enhancement on the fused depth features based on the initial disparity center and initial disparity width to obtain enhanced depth features.

[0120] The lightweight encoder-decoder 360 is used to obtain the first disparity map based on enhanced deep features. The lightweight encoder-decoder 360 can be implemented by training a model using the disparity map described above.

[0121] The disparity optimization output module 370 is used to optimize and normalize the first disparity map to obtain the final disparity map.

[0122] Figure 4 A block diagram of an electronic device according to an embodiment of this application is shown schematically. Figure 4The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0123] like Figure 4 As shown, an electronic device 400 according to an embodiment of this application includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage portion 408 into a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0124] RAM 403 stores various programs and data required for the operation of electronic device 400. Processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Processor 401 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 402 and / or RAM 403. It should be noted that programs may also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in one or more memories.

[0125] According to embodiments of this application, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to a bus 404. The electronic device 400 may also include one or more of the following components connected to the input / output (I / O) interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0126] According to embodiments of this application, the method flow according to embodiments of this application can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by processor 401, it performs the functions defined in the system of embodiments of this application. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0127] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0128] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 402 and / or RAM 403 and / or one or more memories other than ROM 402 and RAM 403 described above.

[0129] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this application. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the depth estimation method based on structured light phase guidance provided in the embodiments of this application.

[0130] When the computer program is executed by the processor 401, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0131] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via communication section 409, and / or installed from removable medium 411. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0132] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments of this application can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of this application.

[0134] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A depth estimation method based on structured light phase guidance, characterized in that, include: Images of the object under test are acquired by a binocular imaging device with a grating structure attached, resulting in a left eye image and a right eye image. The left eye image and the right eye image contain structured light patterns projected through the grating structure. Based on the stripe information in the structured light pattern, phase feature encoding is performed on the left eye image and the right eye image to obtain the left eye phase feature map and the right eye phase feature map; Based on the phase attention mechanism, feature fusion is performed on the left eye image and the left eye phase feature map, and the right eye image and the right eye phase feature map, respectively, to obtain the left eye depth feature and the right eye depth feature; Based on the matching confidence between the left and right eye depth features, feature fusion is performed on the left and right eye depth features to obtain fused depth features; Based on the fused depth features, a disparity map is generated; and The depth information of the object under test is obtained based on the disparity map.

2. The method according to claim 1, characterized in that, The step of encoding phase features of the left and right eye images based on the stripe information in the structured light pattern to obtain left and right eye phase feature maps includes: The left eye image and the right eye image are respectively subjected to noise reduction processing to obtain the first noise-reduced image and the second noise-reduced image; Based on depthwise separable convolution, the left eye image and the first denoised image are concatenated in the channel dimension and then feature encoded to obtain the initial left eye feature map. Using a column attention mechanism, the initial left eye feature map is encoded with phase features based on the stripe information in the structured light pattern to obtain the initial left eye phase feature map. The initial left eye phase feature map is subjected to feature extraction and phase value normalization to obtain the left eye phase feature map; Based on depthwise separable convolution, the right eye image and the second denoised image are concatenated in the channel dimension and then feature encoded to obtain the initial right eye feature map. Using a column attention mechanism, the initial right eye feature map is encoded with phase features based on the stripe information in the structured light pattern to obtain the initial right eye phase feature map. The initial right eye phase feature map is subjected to feature extraction and phase value normalization to obtain the right eye phase feature map.

3. The method according to claim 2, characterized in that, The phase attention mechanism involves fusing features from the left eye image and left eye phase feature map, and the right eye image and right eye phase feature map, respectively, to obtain left eye depth features and right eye depth features, including: The left eye phase feature map and the first denoised image are concatenated along the channel dimension and then feature encoded to obtain the first feature. The right eye phase feature map and the second denoised image are concatenated along the channel dimension and then feature encoded to obtain the second feature. Based on the phase attention mechanism, a first spatial attention mask corresponding to the left eye phase feature map and a second spatial attention mask corresponding to the right eye phase feature map are obtained; Based on the first spatial attention mask, the feature residuals of the first feature are weighted to obtain the left eye depth feature; Based on the second spatial attention mask, the feature residuals of the second feature are weighted to obtain the right eye depth feature.

4. The method according to claim 1, characterized in that, The step of fusing the left and right eye depth features based on the matching confidence between the left and right eye depth features to obtain fused depth features includes: Based on the left eye depth feature, the right eye depth feature, the left eye phase feature map, and the right eye phase feature map, a matching confidence map is obtained; Based on the matching confidence map, the left eye depth features and the right eye depth features are fused to obtain fused depth features.

5. The method according to claim 4, characterized in that, The process of obtaining a matching confidence map based on the left eye depth feature, the right eye depth feature, the left eye phase feature map, and the right eye phase feature map includes: The left and right eye depth features are concatenated along the channel dimension and then encoded to obtain a feature similarity map. Cosine similarity is calculated on the left eye phase feature map and the right eye phase feature map to obtain cosine similarity features; The cosine similarity features are encoded to obtain a phase consistency map; The left eye depth features and the right eye depth features are divided into multiple feature groups according to the channel dimension; For any one of the multiple feature groups, normalized cross-correlation calculation is performed on the left eye depth feature and the right eye depth feature to obtain a cross-correlation map; Feature fusion is performed on multiple cross-correlation maps to obtain a single cross-correlation map; Based on the feature similarity map, the phase consistency map, and the single correlation map, a matching confidence map is obtained.

6. The method according to claim 5, characterized in that, The process of obtaining a matching confidence map based on the feature similarity map, the phase consistency map, and the single correlation map includes: The feature similarity map, the phase consistency map, and the single correlation map are concatenated and then feature encoded to obtain a weighted feature map. The weighted feature map is divided according to the channel dimension and includes a first weighted feature corresponding to the feature similarity map, a second weighted feature corresponding to the phase consistency map, and a third weighted feature corresponding to the single correlation map. The feature similarity map, the phase consistency map, and the single correlation map are weighted based on the first weight feature, the second weight feature, and the third weight feature to obtain the matching confidence map.

7. The method according to claim 1, characterized in that, The step of generating a disparity map based on the fused depth features includes: Based on the fusion depth features, the initial disparity center and initial disparity width are obtained through multi-scale analysis. Based on the initial disparity center and the initial disparity width, channel modulation coefficients are generated; Using the channel modulation coefficients, feature modulation is performed on the channel dimension of the fused depth features to obtain enhanced depth features; Based on the enhanced depth features, a disparity map is generated.

8. The method according to claim 7, characterized in that, The process of obtaining the initial disparity center and initial disparity width through multi-scale analysis based on the fusion depth features includes: The fused depth features are subjected to global average pooling to obtain the first pooled features; The first pooling feature is encoded to obtain the first disparity feature; The fused deep features are divided into multiple sub-features based on a grid. The multiple sub-features are each subjected to pooling processing to obtain multiple second pooled features; Each of the second pooling features is feature-encoded to obtain multiple second initial disparity features; Based on the spatial attention mechanism, multiple second initial disparity features are fused to obtain second disparity features; The fused depth features are subjected to global max pooling to obtain the third pooling feature; The first pooling feature and the third pooling feature are concatenated and then encoded to obtain the third disparity feature; The first disparity feature, the second disparity feature, and the third disparity feature are fused to obtain the initial disparity feature; The initial disparity features are decoded to obtain the initial disparity center and the initial disparity width.

9. The method according to claim 7, characterized in that, The step of generating a disparity map based on the enhanced depth features includes: Based on the enhanced depth features, a first disparity map is obtained; Based on the left eye phase feature map and the first disparity map, the fusion feature is obtained; By using a gating mechanism and residual learning, the first disparity map is enhanced based on the fused features to obtain a second disparity map; The second disparity map is normalized to obtain the third disparity map; The third disparity map is mapped based on the initial disparity range determined by the initial disparity center and the initial disparity width to obtain a disparity map.

10. The method according to claim 9, characterized in that, The fusion features obtained based on the left eye phase feature map and the first disparity map include: Feature extraction is performed on the left eye phase feature map to obtain the first feature map; Feature extraction is performed on the first disparity map to obtain the second feature map; The first feature map and the second feature map are fused using depthwise separable convolution to obtain fused features.

Images