A pseudo binocular depth estimation method and system

Abstract

The present application relates to a kind of pseudo binocular depth estimation methods, comprising: the single left view of being photographed to the object to be measured continuously obtains;Single left view is extracted and reconstructs image feature to obtain pseudo right view;Receive the single left view from camera and pseudo right view from generator, align the features of single left view and pseudo right view and optimize its feature expression, and then output the depth information of depth map containing object to be measured.Pseudo binocular depth estimation method of the present application can realize efficient and high-precision depth estimation.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a pseudo-binocular depth estimation method. Background Technology

[0002] In traditional depth estimation methods, stereo cameras or LiDAR are typically used to acquire depth information of a scene, but these methods often face high hardware costs and complex calibration requirements. In contrast, monocular depth estimation only requires a single camera, but using only one camera leads to a lack of parallax information, resulting in lower accuracy, especially the problem of scale blur. Scale blur causes objects of the same size to appear in similar image quality at different distances. Furthermore, moving objects in dynamic scenes also affect the performance of monocular depth estimation, making it difficult to meet high accuracy requirements.

[0003] Therefore, generative models and depth estimation techniques have gradually developed in the field of depth map generation in recent years. Generative models can generate pseudo-binocular views based on monocular images, thereby forming disparity information and realizing pseudo-binocular depth estimation. By extracting disparity information, pseudo-right views can be generated based on monocular images, and depth information can be further estimated. This not only has strong adaptability in dynamic scenes, but also generates depth maps with limited computing resources.

[0004] However, in practical applications, the depth maps obtained by existing pseudo-binocular depth estimation methods still have poor accuracy and low image clarity, and the actual depth estimation results still cannot meet the requirements for high precision. Summary of the Invention

[0005] Based on this, the purpose of this invention is to provide a pseudo-binocular depth estimation method, which combines multi-level feature fusion and disparity information extraction, and fuses the generation network and depth estimation network to achieve efficient and high-precision depth estimation.

[0006] A pseudo-binocular depth estimation system, comprising:

[0007] The camera continuously takes pictures of the object under test to obtain a monocular left view;

[0008] The generator extracts and reconstructs image features from the monocular left view to obtain a pseudo right view;

[0009] The depth estimation unit receives a monocular left view from the camera and a pseudo right view from the generator, aligns the features of the monocular left view and the pseudo right view and optimizes their feature representation, and then outputs a depth map containing the depth information of the object under test.

[0010] Furthermore, it also includes a discriminator that compares and analyzes the features of the pseudo right view and the real right view from the generator to obtain an adversarial optimization value, and inputs the adversarial optimization value into the generator to optimize the generation of the pseudo right view.

[0011] Furthermore, the discriminator uses a joint loss calculation method to compare and analyze the features of the pseudo right view and the real right view to obtain an adversarial optimization value, which includes adversarial loss, feature matching loss and L2 loss;

[0012] The adversarial loss is used to measure the authenticity of the generated pseudo-right view, and is obtained through adversarial optimization of the generator and discriminator:

[0013] L adv =E[logD(right)]+E[log)1-D(G(left)))]

[0014] Among them, L adv To counteract the loss, D is the discriminator, E is the expected value of the distribution function, G is the generator, right is the true right view, left is the true left view, D(right) is the true right view input to the discriminator, and G(left) is the pseudo right view generated after the true left view is input to the generator.

[0015] The feature matching loss is used to optimize the difference between the generated pseudo right view and the real right view at the feature layer of the discriminator:

[0016]

[0017] Among them, L feature_match For feature matching loss, D (i) This represents the features extracted by the discriminator at the i-th layer, where L is the number of feature layers used for matching.

[0018] The L2 loss is used to constrain the pixel-level differences between the generated image and the real image:

[0019] L L2 =||right-G(left)||2

[0020] Among them, L L2 For L2 loss;

[0021] The adversarial optimization value is:

[0022] L gen =λ adv L adv +λ feature_match L feature_match +λ L2 L L2

[0023] Among them, L gen To combat the optimization value, λ adv , λ feature_match , λ L2 These are the weights corresponding to each item.

[0024] Furthermore, the depth estimation unit includes a multi-scale feature alignment module, a two-dimensional attention module, and a depth map construction module; the multi-scale feature alignment module is used to align equivalent features in the monocular left view and the pseudo right view to obtain multi-scale features; the two-dimensional attention module is used to calculate the attention distribution in the multi-scale feature space, enhance the depth estimation unit's ability to perceive salient regions, and improve performance in preserving boundary details and local information to obtain optimized features; the depth map construction module constructs a depth map containing depth information of the object under test based on the multi-scale features and the optimized features.

[0025] Furthermore, the multi-scale feature alignment module achieves feature alignment through three stages of convolution processing, including initial convolution, feature mask generation, and feature optimization.

[0026] The initial convolution stage includes: the input features are first extracted using a 3×3 convolution kernel, and a mask is generated by calculating the depth difference between the features of the two views;

[0027] The feature mask generation stage includes: optimizing and adjusting the region within the mask based on a flexible threshold.

[0028] The feature optimization stage includes: progressively extracting and optimizing feature information within the mask using two convolutional layers, one a 3×3 convolution and the other a 1×1 convolution;

[0029] The number of input feature channels in each stage is 32, 64 and 128 respectively, and the number of output feature channels is kept consistent with the number of input feature channels. The number of feature channels in the intermediate layer is set to 64, 128 and 256 respectively.

[0030] After three stages of layer-by-layer alignment, multi-scale pseudo-right view features are obtained, and a multi-scale fusion feature with 320 channels is obtained through a stitching operation.

[0031] Furthermore, the two-dimensional attention module is used to receive multi-scale fused features and perform attention-guided optimization to obtain optimized features, including:

[0032] The input features are passed through a 1×1 convolutional layer for initial feature transformation.

[0033] Attention weights are calculated using both channel and spatial dimensions. Channel attention calculates global context information through global pooling and generates the weight distribution for each channel. Spatial attention obtains the spatial weight distribution of salient regions through max pooling and average pooling of the feature maps.

[0034] By combining channel weights and spatial weights, a two-dimensional attention map is obtained, which is used to perform weighted operations on input features to obtain optimized features.

[0035] Furthermore, the multi-scale feature alignment module uses a joint loss calculation method to optimize features and obtain optimized feature values, which include feature alignment loss, reprojection loss, and L1 loss.

[0036] The reprojection loss is used to combine reprojection error and structural similarity to ensure the geometric consistency of disparity estimation.

[0037]

[0038] Among them, L photo For reprojection loss, I left Indicates a monocular left view, I reproj This represents the reprojected image of the monocular left view, where α is the weight controlling the SSIM loss;

[0039] The L1 loss is used to constrain the smoothness of the depth map, ensuring that the neighborhood depth is consistent.

[0040]

[0041] Among them, L smooth_L1 For L1 loss, This represents a gradient operation that restricts local changes in the depth map;

[0042] The feature alignment loss is used to represent the features of the true right view and the pseudo right view in multiple layers of the discriminator, improving the accuracy of depth estimation.

[0043]

[0044] Among them, L align For feature alignment loss, This represents the characteristics of the true right view at the i-th layer of the discriminator. This represents the feature of the pseudo-right view in the i-th layer of the discriminator, where N is the number of feature layers used for alignment.

[0045] The optimized feature value is:

[0046] L depth =L photo +λ smooth L smooth+λ align L align

[0047] Among them, L depth For the feature optimization value, λ smooth , λ align These are the weights corresponding to each item.

[0048] Furthermore, the depth map construction module is used to receive the adversarial loss, feature matching loss and L2 loss used by the discriminator and the feature alignment loss, reprojection loss and L1 loss used by the depth estimation unit, and continuously update and optimize the depth information of the depth map.

[0049] This invention also provides a pseudo-binocular depth estimation method, comprising:

[0050] Continuously photograph the object under test to obtain a monocular left view;

[0051] A pseudo right view is obtained by extracting and reconstructing image features from a monocular left view;

[0052] It receives a monocular left view from the camera and a pseudo right view from the generator, aligns the features of the monocular left view and the pseudo right view and optimizes their feature representation, and then outputs a depth map containing the depth information of the object under test.

[0053] Furthermore, it also includes: obtaining adversarial optimization values ​​by comparing and analyzing the features of the pseudo right view and the real right view, and optimizing the generation quality of the pseudo right view through the adversarial optimization values.

[0054] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the pseudo-binocular depth estimation system structure of this application.

[0056] Figure 2 This is a flowchart of the pseudo-binocular depth estimation method of this application. Detailed Implementation

[0057] The applicant carefully analyzed existing pseudo-binocular depth estimation methods and found that their low depth estimation accuracy stemmed from the fact that the pseudo right view generated based on the monocular image was not a true image. This pseudo right view lacked depth information distinct from the monocular image, resulting in significant errors in the disparity information obtained, thus leading to low final depth estimation accuracy. To address this, the applicant attempted a multi-step generation strategy to obtain a monocular left view and a pseudo right view incorporating pseudo-binocular disparity information. This strategy fused multi-level feature information to capture rich depth information, and a complex loss structure was introduced across multiple layers to optimize depth estimation.

[0058] Please see Figure 1 and Figure 2 Based on the above-mentioned inventive concept, the applicant designed a pseudo-binocular depth estimation system, including: a camera, a generator, a discriminator, and a depth estimation unit.

[0059] The camera continuously takes pictures of the object under test to obtain a monocular left view. The camera's shooting point is offset to the right from the surface of the object under test by a certain distance to obtain the same monocular left view as in binocular shooting.

[0060] The generator extracts and reconstructs image features from a monocular left view to obtain a pseudo right view. This is achieved through a multi-step generation process based on a diffusion model using a multi-layer convolutional network. The generator includes multiple extraction modules and upsampling modules. Each extraction module includes a convolutional layer, a ReLU activation layer, and a batch normalization layer to extract image features at different scales. Specifically, the multiple extraction modules learn image features based on the input monocular left view, guide the image features to gradually approximate the target distribution of the pseudo right view through diffusion noise, restore the image resolution through a progressive upsampling module, and fuse the image features to ensure the structural and texture consistency of the output pseudo right view.

[0061] The discriminator receives a pseudo-right view from the generator and a known true right view. It compares and analyzes the features of the pseudo-right view and the true right view to obtain an adversarial optimization value, which is then input into the generator to improve the generation quality of the pseudo-right view. The discriminator includes a feature extraction module and a feature alignment module. The feature extraction module extracts features from the pseudo-right view and the true right view to obtain feature distributions. The feature alignment module compares and analyzes the feature distributions of the pseudo-right view and the true right view to obtain the adversarial optimization value. The feature alignment module acts as an adversarial mechanism within the discriminator, obtaining the adversarial optimization value through cross-domain feature comparison and inputting it into the generator. This causes the feature distribution of the pseudo-right view to gradually approach that of the true right view, thereby improving the generation quality of the generator.

[0062] Specifically, the feature alignment module employs joint loss for comparative analysis to obtain adversarial optimization values, including adversarial loss and feature matching loss. The adversarial loss enhances the realism of the pseudo-right view through adversarial game between the generator and discriminator; the feature matching loss optimizes the depth information of the pseudo-right view through supervision by the feature alignment module. Furthermore, to further improve the generator's stability, L2 loss is introduced to ensure consistency in structural information and visual effect between the generated pseudo-right views.

[0063] The L2 loss directly optimizes the difference between the pseudo-right view and the real right view in pixel space, ensuring high fidelity of the output image. The feature matching loss guides the generator to progressively optimize the output image by comparing the distribution differences between the generated image and the real image in the discriminator's multi-layer feature spaces, making it closer to the real right view in high-level feature representation. Specifically, in the discriminator's multi-layer feature extraction, the feature matching loss calculates the feature differences between the generated image and the real image using mean squared error (MSE), providing the generator with a deep-level supervisory signal. Therefore, this design significantly improves the detail consistency and texture representation of the pseudo-right view.

[0064] The adversarial loss is used to measure the authenticity of the generated pseudo-right view, and is obtained through adversarial optimization of the generator and discriminator:

[0065] L adv =E[logD(right)]+E[log(1-D(G(left)))]

[0066] Among them, L adv To combat the loss, D is the discriminator, E is the expected value of the distribution function, G is the generator, right is the true right view, left is the true left view, D(right) is the true right view input to the discriminator, and G(left) is the pseudo right view generated after the true left view is input to the generator.

[0067] The feature matching loss is used to optimize the difference between the generated pseudo right view and the real right view in the feature layer of the discriminator, thereby enhancing the consistency of the deep features learned by the generator.

[0068]

[0069] Among them, L feature_match For feature matching loss, D (i) This represents the features extracted by the discriminator at the i-th layer, where L is the number of feature layers used for matching.

[0070] The L2 loss is used to constrain the pixel-level differences between the generated image and the real image, thereby improving the visual effect of the generator output.

[0071] L L2 =||right-G(left)||2

[0072] Among them, L L2 This is the L2 loss.

[0073] The depth estimation unit receives a monocular left view from the camera and a pseudo-right view from the generator, aligns the features of the monocular left view and the pseudo-right view, and optimizes their feature representations to output a depth map containing the depth information of the object under test. The depth estimation unit includes a multi-scale feature alignment module, a two-dimensional attention module, and a depth map construction module. The multi-scale feature alignment module uses a multi-scale downsampling strategy to progressively align equivalent features in the monocular left view and the pseudo-right view, making the features of the monocular left view and the pseudo-right view more consistent in terms of depth-related information, thus obtaining multi-scale features. The two-dimensional attention module calculates the attention distribution in the multi-scale feature space, enhancing the depth estimation unit's ability to perceive salient regions and improving performance in preserving boundary details and local information, thus obtaining optimized features. The depth map construction module receives the multi-scale features from the multi-scale feature alignment module and the optimized features from the two-dimensional attention module, and constructs a depth map containing the depth information of the object under test, thereby greatly improving the accuracy and stability of the depth map.

[0074] Specifically, the multi-scale feature alignment module is used to flexibly align the features of the monocular left view and the pseudo right view to reduce the impact of significant differences between the features of the monocular left view and the pseudo right view on subsequent depth estimation, while ensuring the stability of the overall structural information. The multi-scale feature alignment module achieves feature alignment through three stages of convolution processing: initial convolution, feature mask generation, and feature optimization. The initial convolution stage includes: the input features are first extracted using a 3×3 convolution kernel, and a mask is generated by calculating the depth difference between the features of the two views; the feature mask generation stage includes: using a flexible threshold as a condition, only the region within the mask is optimized and adjusted, while the features outside the mask remain unchanged; the feature optimization stage includes: using two convolutional layers, one 3×3 and the other 1×1, to progressively extract and optimize the feature information within the mask to obtain optimized feature values, while avoiding unnecessary interference with the overall feature distribution of the pseudo right view. The number of input feature channels in each stage is 32, 64, and 128, respectively, and the number of output feature channels remains consistent with the number of input feature channels. The number of feature channels in the intermediate layers is set to 64, 128, and 256, respectively. After three stages of layer-by-layer alignment, the generated multi-scale pseudo-right view features are integrated into a multi-scale fusion feature with 320 channels through a stitching operation. This feature includes optimized feature values, which serve as the input basis for subsequent operations of the depth estimation unit. This achieves the gradual alignment of the monocular left view and pseudo-right view feature distributions in local areas and the compatibility optimization of the overall structure.

[0075] A two-dimensional attention module enhances the depth estimation network's ability to perceive salient feature regions, thereby improving the prediction accuracy and stability of the final depth map. This two-dimensional attention module operates on the fusion layer of pseudo-binocular features, specifically optimizing the joint features of the monocular left view and the pseudo-right view through attention-guided processing. The input features to the two-dimensional attention module are aligned multi-scale fusion features with 320 channels. After passing through the two-dimensional attention module, the number of output feature channels remains unchanged. The implementation of the two-dimensional attention module includes three steps: First, the input features undergo preliminary feature transformation through a 1×1 convolutional layer to ensure the distribution of the input features is adapted to the attention operation. Then, attention weights are calculated using both channel and spatial dimensions. Channel attention calculates global context information through global pooling and generates the weight distribution for each channel; spatial attention obtains the spatial weight distribution of salient regions through max pooling and average pooling of the feature map. Finally, the channel weights and spatial weights are combined to form the final two-dimensional attention map, which is used to perform weighted operations on the input features to obtain optimized features. The final optimized features enhance the perception of salient regions at both the channel and spatial levels, thereby improving the performance of the depth estimation unit in preserving boundary details and local information. The features optimized by the 2D attention module will be directly input into subsequent layers of the depth estimation unit to further improve the accuracy and consistency of the pseudo-binocular depth estimation results.

[0076] Specifically, the multi-scale feature alignment module employs a joint loss calculation method to optimize features and obtain optimized feature values, which include feature alignment loss, reprojection loss, and L1 loss. These losses collectively optimize the performance of the depth estimation unit. Combined with the supervision information from the discriminator feature alignment module, the depth estimation results are more stable in dynamic scenes and boundary regions. The L1 loss of the monocular left view and the pseudo right view is used to maintain the geometric alignment relationship between the real left view and the pseudo right view. By directly calculating the pixel-by-pixel L1 loss on the monocular left view and the pseudo right view, the basic accuracy of disparity estimation can be optimized.

[0077] The reprojection loss is used to combine reprojection error and structural similarity (SSIM) to ensure geometric consistency of disparity estimation:

[0078]

[0079] Among them, L photo For reprojection loss, I left Indicates a monocular left view, I reproj This represents the reprojected image of the monocular left view, where α is the weight controlling the SSIM loss.

[0080] The L1 loss is used to constrain the smoothness of the depth map and prevent excessive noise from affecting the consistency of neighborhood depth.

[0081]

[0082] Among them, L smooth_L1 For L1 loss, This represents a gradient operation that restricts local changes in the depth map.

[0083] The feature alignment loss is used to represent the features of the true right view and the pseudo right view across multiple layers in the discriminator, thereby improving the accuracy of depth estimation.

[0084]

[0085] Among them, L align For feature alignment loss, This represents the characteristics of the true right view at the i-th layer of the discriminator. This represents the feature of the pseudo-right view at the i-th layer of the discriminator, where N is the number of feature layers used for alignment.

[0086] The final loss structure includes adversarial losses for the generator and discriminator, L2 loss, and feature matching loss, as well as reprojection loss, L1 loss, and feature alignment loss for the depth estimation unit. In the depth estimation unit, the introduction of feature alignment loss not only improves the detail consistency of the generated depth map but also enhances the model's expressive power in high-level feature spaces. This design significantly optimizes the depth map generation performance by establishing a link between geometric consistency and high-level representation.

[0087] The total loss of the generator and discriminator, i.e., the adversarial optimization value, is:

[0088] L gen =λ adv L adv +λ feature_match L feature_match +λ L2 L L2

[0089] Among them, L gen To combat the optimization value, λ adv , λ feature_match , λ L2 The weights are for each item.

[0090] The total loss of the depth estimation unit, i.e., the feature optimization value, is:

[0091] L depth =L photo +λ smooth L smooth +λ align L align

[0092] Among them, Ldepth For the feature optimization value, λ smooth , λ align These are the weights corresponding to each item.

[0093] The embodiments described above merely illustrate the preferred implementation of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and the present invention also intends to include these modifications and variations.

Claims

1. A pseudo-binocular depth estimation system, characterized in that: include: The camera continuously takes pictures of the object under test to obtain a monocular left view; The generator extracts and reconstructs image features from the monocular left view to obtain a pseudo right view; The depth estimation unit receives a monocular left view from the camera and a pseudo right view from the generator, aligns the features of the monocular left view and the pseudo right view and optimizes their feature representation, and then outputs a depth map containing the depth information of the object under test. The depth estimation unit includes a multi-scale feature alignment module, a two-dimensional attention module, and a depth map construction module. The multi-scale feature alignment module is used to align equivalent features in the monocular left view and pseudo right view to obtain multi-scale features. The two-dimensional attention module is used to calculate the attention distribution in the multi-scale feature space, enhance the depth estimation unit's ability to perceive salient regions, and improve performance in preserving boundary details and local information to obtain optimized features. The depth map construction module constructs a depth map containing depth information of the object under test based on the multi-scale features and optimized features. The multi-scale feature alignment module achieves feature alignment through three stages of convolution processing, including initial convolution, feature mask generation, and feature optimization. The initial convolution stage includes: the input features are first extracted using a 3×3 convolution kernel, and a mask is generated by calculating the depth difference between the features of the two views; The feature mask generation stage includes: optimizing and adjusting the region within the mask based on a flexible threshold. The feature optimization stage includes: progressively extracting and optimizing feature information within the mask using two convolutional layers, one a 3×3 convolution and the other a 1×1 convolution; The number of input feature channels in each stage is 32, 64 and 128 respectively, and the number of output feature channels is kept consistent with the number of input feature channels. The number of feature channels in the intermediate layer is set to 64, 128 and 256 respectively. After three stages of layer-by-layer alignment, multi-scale pseudo-right view features are obtained, and a multi-scale fusion feature with 320 channels is obtained through a stitching operation.

2. The pseudo-binocular depth estimation system according to claim 1, characterized in that: It also includes a discriminator that compares and analyzes the features of the pseudo right view and the real right view from the generator to obtain an adversarial optimization value, and inputs the adversarial optimization value into the generator to optimize the generation of the pseudo right view.

3. The pseudo-binocular depth estimation system according to claim 2, characterized in that: The discriminator uses a joint loss calculation method to compare and analyze the features of the pseudo right view and the real right view to obtain the adversarial optimization value, which includes adversarial loss, feature matching loss and L2 loss. The adversarial loss is used to measure the authenticity of the generated pseudo-right view, and is obtained through adversarial optimization of the generator and discriminator: in, To combat the loss, D is the discriminator, E is the expected value of the distribution function, G is the generator, right is the true right view, and left is the true left view. The true right view of the input discriminator. A pseudo-right view generated after inputting the real left view into the generator; The feature matching loss is used to optimize the difference between the generated pseudo right view and the real right view at the feature layer of the discriminator: in, For feature matching loss, This represents the features extracted by the discriminator at the i-th layer, where L is the number of feature layers used for matching. The L2 loss is used to constrain the pixel-level differences between the generated image and the real image: in, For L2 loss; The adversarial optimization value is: in, To combat the optimization value, , , These are the weights corresponding to each item.

4. The pseudo-binocular depth estimation system according to claim 3, characterized in that: The two-dimensional attention module is used to receive multi-scale fused features and perform attention-guided optimization to obtain optimized features, including: The input features are passed through a 1×1 convolutional layer for initial feature transformation. Attention weights are calculated using both channel and spatial dimensions. Channel attention calculates global context information through global pooling and generates the weight distribution for each channel. Spatial attention obtains the spatial weight distribution of salient regions through max pooling and average pooling of the feature maps. By combining channel weights and spatial weights, a two-dimensional attention map is obtained, which is used to perform weighted operations on input features to obtain optimized features.

5. A pseudo-binocular depth estimation method performed by the pseudo-binocular depth estimation system of claim 1, characterized in that: include: Continuously photograph the object under test to obtain a monocular left view; A pseudo right view is obtained by extracting and reconstructing image features from a monocular left view; It receives a monocular left view from the camera and a pseudo right view from the generator, aligns the features of the monocular left view and the pseudo right view and optimizes their feature representation, and then outputs a depth map containing the depth information of the object under test.

6. The pseudo-binocular depth estimation method according to claim 5, characterized in that: Also includes: By comparing and analyzing the characteristics of the pseudo right view and the real right view, an adversarial optimization value is obtained, and the generation quality of the pseudo right view is improved by using the adversarial optimization value.

Images