A binocular depth estimation method based on pre-fusion monocular affine depth
By directly fusing monocular affine depth information in a binocular disparity neural network, the problems of information loss, feature bias, and algorithm adaptability are solved, achieving more efficient disparity estimation accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHENSHENZHI WEILAICO LTD
- Filing Date
- 2026-04-27
- Publication Date
- 2026-07-21
AI Technical Summary
In existing technologies, monocular depth images suffer severe information loss and feature bias in binocular disparity estimation, information fusion is lagging, and algorithm adaptability is limited, resulting in insufficient accuracy and efficiency of disparity estimation.
By fusing monocular affine depth information in the early stages, the binocular disparity neural network utilizes the prior knowledge of the DepthAnything model to directly stitch the monocular depth map and RGB image at the very beginning of the data stream for feature extraction and disparity estimation.
It significantly improves the accuracy and robustness of disparity estimation, especially the accuracy of object depth and scale determination in complex scenes, thereby improving the network's adaptability and computational efficiency.
Smart Images
Figure CN122435003A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and more specifically to a binocular depth estimation method based on pre-fusion monocular affine depth. Background Technology
[0002] In the field of computer vision, binocular parallax estimation is an important means of acquiring scene depth information and can be widely used in autonomous driving, robot navigation, 3D reconstruction, and other fields. In autonomous driving, it is used to perceive the depth information of the vehicle's surrounding environment in real time and to assist the vehicle in obstacle avoidance and path planning; in robot navigation, it helps the robot identify the distance and position of obstacles to achieve autonomous movement; in the field of 3D reconstruction, it can quickly acquire scene depth data and build high-precision 3D models.
[0003] In recent years, with the development of deep learning, significant progress has been made in binocular disparity estimation methods based on neural networks. Meanwhile, the DepthAnything model, trained on a large amount of synthetic data, possesses prior knowledge of object scale, and its output monocular depth information can provide depth semantic information for the stereo matching process in binocular algorithms, gradually being applied to assist binocular disparity neural networks. In existing technical solutions using DepthAnything output to assist binocular disparity neural networks, the monocular depth image is typically used as input after feature extraction from the original image via a convolutional neural network (CNN), or as reference input to a gated recurrent unit (GRU). Specifically, in these methods, the original binocular image first enters the CNN network, undergoing multiple convolutional and pooling operations to extract image features, and then the monocular depth image output by the DepthAnything model is integrated into the network, possibly for some operation or concatenation with the feature maps extracted by the CNN; or the monocular depth image is used as reference information when the GRU processes feature sequences. This approach means that monocular depth information has already undergone many intermediate processing steps when participating in disparity estimation calculations, making it impossible to fully utilize its original information characteristics and maximize the prior ability of the DepthAnything model to measure object scale. This results in a bottleneck in improving the accuracy and efficiency of binocular disparity estimation in complex scenes.
[0004] It is evident that existing algorithms using DepthAnything to output monocular depth-assisted binocular parallax neural networks have the following defects and problems: Information Loss and Feature Bias: Using monocular depth images as input after CNN feature extraction of the original image results in a loss and bias in the original depth information, such as object scale, because CNNs primarily focus on the visual semantic features of the image during feature extraction. Furthermore, this fusion method leads to insufficient integration between the monocular depth information and the original image's RGB information in the feature space, affecting the effective utilization of depth information in subsequent disparity estimation.
[0005] Information fusion lag: When using monocular depth images as input references for the GRU, information fusion occurs relatively late in the network processing. At this point, the network has already performed extensive feature extraction and processing based on the RGB information of the original image. Monocular depth information cannot guide the network's computational direction and focus in the early stages of data processing. This results in the network not fully utilizing the object scale prior capability of the DepthAnything model, leading to insufficient accuracy in judging object depth and scale in complex scenes, thus affecting the accuracy of binocular disparity estimation.
[0006] Limitations in algorithm adaptability: Existing methods rely on specific network structures such as CNN-GRU for processing and fusing monocular depth images, resulting in poor adaptability to different network architectures. Changing the network structure may require significant adjustments to the fusion location and method of monocular depth images, thus limiting the algorithm's versatility and scalability.
[0007] Therefore, how to provide a binocular depth estimation method based on pre-fusion monocular affine depth that can effectively solve the problems existing in the above-mentioned prior art, make the utilization of depth information more direct and efficient, absorb prior capabilities more fully, have stronger network adaptability and scalability, and effectively improve the accuracy and efficiency of disparity estimation is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0008] In view of this, the present invention provides a binocular depth estimation method based on pre-fusion monocular affine depth. By pre-fusion of monocular affine depth information, the present invention enables the binocular disparity neural network to utilize prior knowledge from the DepthAnything model in the early stages, thereby improving the accuracy and robustness of disparity estimation. By inheriting depth features, the global consistency between the semantics of each segmented image is significantly improved, especially in challenging scenarios such as texture loss and illumination variations, where the algorithm of the present invention exhibits significant advantages.
[0009] To achieve the above objectives, the present invention adopts the following technical solution: A binocular depth estimation method based on pre-fusion monocular affine depth includes: Step 1: Preprocessing is used to adapt the monocular depth map to the RGB dimensions of the stereo image in terms of channel count and resolution. Step 2: Perform a concat operation on the RGB dimensions of the processed monocular depth map and the stereo image along the channel dimension to obtain the fused image of the left and right views; Step 3: Input the fused images of the left and right views into the binocular disparity neural network. The binocular disparity neural network first extracts features from the fused image through multi-layer convolution and pooling operations, and then performs binocular disparity matching based on the extracted features. Through cost calculation, cost aggregation, disparity estimation and optimization, it outputs the disparity map of the scene.
[0010] Optionally, in step 1, preprocessing includes: denoising and normalizing the binocular images; scaling and adjusting the dimensions of the monocular depth map according to the size and data format of the binocular images.
[0011] Optionally, in step 1, based on the size and data format of the binocular image, bilinear interpolation is used to scale and adjust the dimensions of the monocular depth map, as follows:
[0012] in, This is the adjusted monocular depth map; It is a bilinear interpolation; This is the monocular depth map before adjustment; and These represent the height and width of the image, respectively.
[0013] Optionally, in step 2, the processed monocular depth map and the stereo image are concatted along the channel dimension along their RGB dimensions to obtain a fused image of the left and right views, as follows:
[0014] in, , The images are fused together from the left and right views, respectively. , These are the left and right view images before merging; This is the adjusted monocular depth map; , These are the i-th color channels of the left and right views, respectively; and These represent the height and width of the image, respectively.
[0015] Optionally, in step 3, feature extraction is performed as follows:
[0016] in, , These are the extracted features of the left and right views, respectively. For feature extraction networks; , The images are fused together from the left and right views, respectively. , The height and width of the feature map; This represents the number of channels.
[0017] Optionally, in step 3, the cost is calculated as follows:
[0018] in, The result of the cost calculation; It is one of the following: squared difference, absolute difference, and learnable cost function; Features of the left view; To incorporate right view features Horizontal movement 1 pixel; This represents the maximum parallax search range.
[0019] Optionally, in step 3, the cost aggregation is as follows:
[0020] in, Aggregate results at a cost; This represents the disparity value. It is a convergent network; The result of the cost calculation; This represents the maximum parallax search range.
[0021] Optionally, in step 3, disparity estimation is performed as follows:
[0022] in, This is the result of disparity estimation; This represents the disparity value. Aggregate results at a cost.
[0023] Optionally, in step 3, the disparity estimation uses a soft argmin operation, as follows:
[0024] in, The results are disparity estimations using the soft argmin operation. This represents the maximum parallax search range. This represents the disparity value. Aggregate results at a cost.
[0025] Optionally, in step 3, parallax optimization is performed as follows:
[0026] in, Optimize results for parallax; The results are disparity estimations using the soft argmin operation. This includes: left-right consistency check, subpixel thinning, and median filtering operations.
[0027] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a binocular depth estimation method based on pre-fusion monocular affine depth, which achieves the following beneficial effects: Depth information is utilized more directly and efficiently: Existing technologies integrate monocular depth images after CNN feature extraction or as GRU input references. This invention directly concatenates monocular depth information with the RGB dimensions at the very beginning of the data stream. This technical feature eliminates the need for complex intermediate processing steps, allowing depth information to participate more faithfully in subsequent binocular disparity calculations. This significantly reduces the loss and bias of depth information during transmission, enabling binocular images to fully integrate depth dimension information from the very beginning of input. This significantly improves the auxiliary effect of depth information on binocular disparity estimation, allowing for more accurate determination of object depth and scale in complex scenes (such as low-texture, low-light environments).
[0028] More comprehensive absorption of prior knowledge: Leveraging the object scale prior knowledge obtained by training the DepthAnything model with a large amount of synthetic data, this invention, through front-end fusion of depth information, allows the binocular disparity neural network to absorb this prior knowledge more directly and fully. Compared with existing technologies, it can utilize object scale prior knowledge more efficiently in the disparity estimation process, making the disparity calculation more accurate for objects of different sizes and types, improving the algorithm's understanding of the relationship between object depth and scale, and making the output disparity map more consistent with the real scene in terms of object scale reconstruction.
[0029] Enhanced Network Adaptability and Extensibility: This invention offers greater flexibility in the design of the binocular disparity neural network. Only the data input front-end splicing interface needs adaptation; subsequent network structures can be easily replaced (e.g., by changing different convolutional layers or feature matching modules). Existing technologies rely on specific intermediate network structures (such as specific CNN-GRU structures) for deep information fusion, resulting in high adjustment costs when adapting to different networks. This invention reduces network structure coupling, improves the applicability and extensibility of the algorithm across different network architectures, and facilitates optimization of disparity estimation performance by incorporating new network structures.
[0030] The accuracy and efficiency of disparity estimation are effectively improved: Due to the advantages of front-end fusion of depth information, the depth information processing steps are reduced and the difficulty of network structure adaptation is lowered. In actual operation, it can accelerate the network's capture and utilization of effective information, and improve the computational efficiency of binocular disparity estimation. At the same time, because the utilization of depth information is more direct and the absorption of prior capabilities is more complete, disparity estimation errors can be effectively reduced, the accuracy of disparity maps can be improved, and the algorithm can provide more reliable depth data support for system decision-making in practical applications (such as autonomous driving and 3D reconstruction). Attached Figure Description
[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0032] Figure 1 This is a schematic diagram of the method flow of the present invention.
[0033] Figure 2 This is a schematic diagram of the binocular depth estimation algorithm based on pre-fusion monocular affine depth of the present invention.
[0034] Figure 3 This is a schematic diagram of the GRU structure of the present invention. Detailed Implementation
[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0036] Example 1: Embodiment 1 of this invention discloses a binocular depth estimation method based on pre-fusion monocular affine depth, such as... Figure 1 As shown, it includes: Step 1: Preprocessing is used to adapt the monocular depth map to the RGB dimensions of the binocular image in terms of channel number and resolution, in preparation for subsequent stitching operations.
[0037] The left and right view image data collected by the binocular vision system are acquired separately, and the DepthAnything model is used to generate monocular depth maps for the same scene.
[0038] Preprocessing includes: denoising and normalizing the binocular images; scaling and adjusting the dimensions of the monocular depth map according to the size and data format of the binocular images.
[0039] Binocular images can be represented as ;in, and These represent the height and width of the image, respectively. The monocular depth map generated by the DepthAnything model is represented as... .
[0040] Based on the size and data format of the stereo image, bilinear interpolation is used to scale and adjust the dimensions of the monocular depth map to match the size of the stereo image, as follows:
[0041] in, This is the adjusted monocular depth map; It is a bilinear interpolation; This is the monocular depth map before adjustment; and These represent the height and width of the image, respectively.
[0042] Step 2: Binocular depth estimation algorithm based on pre-fusion monocular affine depth, such as... Figure 2 As shown. At the very beginning of the data stream, the processed monocular depth map and the RGB dimensions of the stereo image are concatenated along the channel dimensions to obtain a fused image of the left and right views. Specifically, if the RGB data dimension of the stereo image is (H, W, 3) (H is the image height, W is the image width), and the monocular depth information dimension is adjusted to (H, W, 1), then after the concatenation operation, fused data with a dimension of (H, W, 4) is formed. This operation directly integrates the monocular depth information into the original image data without changing the affine properties, so that the stereo image has depth dimension information at the beginning of input to the neural network, while absorbing the prior ability of the DepthAnything model to determine the object scale.
[0043] The processed monocular depth map and the stereo image are concatenated along the channel dimension using their RGB dimensions to obtain a fused image of the left and right views, as shown below:
[0044] in, , The images are fused together from the left and right views, respectively. , These are the left and right view images before merging; This is the adjusted monocular depth map; , These are the i-th color channels of the left and right views, respectively; and These represent the height and width of the image, respectively.
[0045] Step 3: As Figure 2 As shown, after the pre-fusion operation in step 2, the fused images of the left and right views are input into a binocular disparity neural network. The interface of this neural network is adapted to the dimensions and format of the fused data, and its internal structure is flexible and can be replaced according to actual needs, such as replacing convolutional layers of different numbers or types, or adjusting the feature matching module. The binocular disparity neural network first extracts features from the fused image through multi-layer convolution and pooling operations, then performs binocular disparity matching based on the extracted features. Through cost calculation, cost aggregation, disparity estimation, and optimization, it outputs a disparity map of the scene.
[0046] Feature extraction, as follows:
[0047] in, , These are the extracted features of the left and right views, respectively. For feature extraction networks; , The images are fused together from the left and right views, respectively. , The height and width of the feature map; This represents the number of channels.
[0048] The cost is calculated as follows:
[0049] in, The result of the cost calculation; It is one of the following: Squared Difference (SAD), Absolute Difference (MAD), and a learnable cost function; Features of the left view; To incorporate right view features Horizontal movement 1 pixel; This represents the maximum parallax search range.
[0050] Cost aggregation, as follows:
[0051] in, Aggregate results at a cost; This represents the disparity value. It is a convergent network; The result of the cost calculation; This represents the maximum parallax search range.
[0052] Parallax estimation is as follows:
[0053] in, This is the result of disparity estimation; This represents the disparity value. Aggregate results at a cost.
[0054] To improve the accuracy of disparity estimation, a soft argmin operation is used, as follows:
[0055] in, The results are disparity estimations using the soft argmin operation. This represents the maximum parallax search range. This represents the disparity value. Aggregate results at a cost.
[0056] Parallax optimization, as follows:
[0057] in, Optimize results for parallax; The results are disparity estimations using the soft argmin operation. This includes left-right consistency checks, subpixel thinning, and median filtering operations, while in the field of neural network binocular depth estimation, this processing is generally called GRU.
[0058] like Figure 3 As shown, the GRU structure consists of reset gates. Update door Candidate hidden state Final hidden state As a simplified version of LSTM, the GRU structure uses fewer gates to remember key information and forget redundant content for iterative optimization. Combined with multi-resolution features, GRU can gradually refine disparity prediction during iteration, dynamically adjust feature weights using gating, and integrate depth cues from different stages and scales to make the final disparity map more detailed.
[0059] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0060] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A binocular depth estimation method based on pre-fusion monocular affine depth, characterized in that, include: Step 1: Preprocessing is used to adapt the monocular depth map to the RGB dimensions of the stereo image in terms of channel count and resolution. Step 2: Perform a concat operation on the RGB dimensions of the processed monocular depth map and the stereo image along the channel dimension to obtain the fused image of the left and right views; Step 3: Input the fused images of the left and right views into the binocular disparity neural network. The binocular disparity neural network first extracts features from the fused images through multi-layer convolution and pooling operations, then performs binocular disparity matching based on the extracted features, and outputs the disparity map of the scene through cost calculation, cost aggregation, disparity estimation and optimization.
2. The binocular depth estimation method based on pre-fusion monocular affine depth as described in claim 1, characterized in that, In step 1, the preprocessing includes: denoising and normalizing the binocular image; and scaling and adjusting the dimensions of the monocular depth map according to the size and data format of the binocular image.
3. The binocular depth estimation method based on pre-fusion monocular affine depth according to claim 2, characterized in that, In step 1, based on the size and data format of the stereo image, bilinear interpolation is used to scale and adjust the dimensions of the monocular depth map, as follows: in, This is the adjusted monocular depth map; It is a bilinear interpolation; This is the monocular depth map before adjustment; and These represent the height and width of the image, respectively.
4. A binocular depth estimation method based on pre-fusion monocular affine depth according to claim 1, characterized in that, In step 2, the processed monocular depth map and the stereo image are concatted along the channel dimension along their RGB dimensions to obtain a fused image of the left and right views, as shown below: in, , The images are fused together from the left and right views, respectively. , These are the left and right view images before merging; This is the adjusted monocular depth map; , These are the i-th color channels of the left and right views, respectively; and These represent the height and width of the image, respectively.
5. A binocular depth estimation method based on pre-fusion monocular affine depth according to claim 1, characterized in that, In step 3, the feature extraction is as follows: in, , These are the extracted features of the left and right views, respectively. For feature extraction networks; , The images are fused together from the left and right views, respectively. , The height and width of the feature map; This represents the number of channels.
6. A binocular depth estimation method based on pre-fusion monocular affine depth according to claim 1, characterized in that, In step 3, the cost calculation is as follows: in, The result of the cost calculation; It is one of the following: squared difference, absolute difference, and learnable cost function; Features of the left view; To incorporate right view features Horizontal movement 1 pixel; This represents the maximum parallax search range.
7. A binocular depth estimation method based on pre-fusion monocular affine depth according to claim 1, characterized in that, In step 3, the cost aggregation is as follows: in, Aggregate results at a cost; This represents the disparity value. It is a convergent network; The result of the cost calculation; This represents the maximum parallax search range.
8. A binocular depth estimation method based on pre-fusion monocular affine depth according to claim 1, characterized in that, In step 3, the disparity estimation is as follows: in, This is the result of disparity estimation; This represents the disparity value. Aggregate results at a cost.
9. A binocular depth estimation method based on pre-fusion monocular affine depth according to claim 1, characterized in that, In step 3, the disparity estimation uses a soft argmin operation, as follows: in, The results are disparity estimations using the soft argmin operation. This represents the maximum parallax search range. This represents the disparity value. Aggregate results at a cost.
10. A binocular depth estimation method based on pre-fusion monocular affine depth according to claim 1, characterized in that, In step 3, parallax optimization is performed as follows: in, Optimize results for parallax; The results are disparity estimations using the soft argmin operation. This includes: left-right consistency check, subpixel thinning, and median filtering operations.