A Depth Estimation Method Based on Binocular Polarization Camera and Atmospheric Scattering Model

Abstract

The present invention discloses a depth estimation method based on a binocular polarization camera and an atmospheric scattering model, which relates to the technical field of image processing and includes: calculating a first close-range dense depth map and a first far-scene sparse depth map based on the atmospheric scattering model; aligning and normalizing the first close-range dense depth map and the first far-scene sparse depth map to obtain a second close-range dense depth map and a second far-scene sparse depth map; verifying the sparse depth values of the atmospheric scattering model according to the second close-range dense depth map to mark the credible depth points in the second far-scene sparse depth map, generating a confidence mask; inputting the second close-range dense depth map, the second far-scene sparse depth map and the confidence mask into a depth estimation network model, and outputting a full-scene dense depth map. The present invention enhances the robustness of depth estimation in complex environments, realizes the optimization of depth continuity of the full scene, and improves the adaptability and scalability of complex scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a depth estimation method based on a binocular polarization camera and an atmospheric scattering model. Background Art

[0002] With the continuous development of autonomous driving, robot navigation, and three-dimensional modeling technologies, depth estimation has become one of the key issues. Traditional binocular cameras obtain depth information of nearby scenes through disparity calculation, but they face great challenges in long-distance depth estimation, mainly reflected in the baseline length limitation and disparity attenuation. To overcome this problem, in recent years, researchers have proposed methods using lidar, monocular vision, and other sensors for long-distance depth estimation. However, these methods usually require high equipment costs or cannot effectively cope with the atmospheric scattering effect in complex environments. The atmospheric scattering effect, especially under environmental conditions such as haze and rain, significantly affects the attenuation of light propagation and further reduces the accuracy of long-distance depth estimation. Summary of the Invention

[0003] The present invention proposes a depth estimation method based on a binocular polarization camera and an atmospheric scattering model to solve the problems of low accuracy and high cost in long-distance depth estimation in the prior art.

[0004] The present invention achieves the above object through the following technical solutions:

[0005] A depth estimation method based on a binocular polarization camera and an atmospheric scattering model according to the present invention includes:

[0006] Obtain the information of the binocular polarization camera;

[0007] Calculate a first near-field dense depth map according to the information of the binocular polarization camera, and calculate a first far-field sparse depth map based on the atmospheric scattering model;

[0008] Align and normalize the first near-field dense depth map and the first far-field sparse depth map to obtain a second near-field dense depth map and a second far-field sparse depth map;

[0009] Verify the second far-field sparse depth map of the atmospheric scattering model according to the second near-field dense depth map to mark the reliable depth points in the second far-field sparse depth map, and generate a confidence mask;

[0010] Input the second close-range dense depth map, the second far-scene sparse depth map, and the confidence mask into a depth estimation network model to output a full-scene dense depth map. The depth estimation network model is used to first extract the features of the second close-range dense depth map and the second far-scene sparse depth map, and is also used to calculate attention weights in the high-confidence regions of the second far-scene sparse depth map based on the Transformer network and the confidence mask, and mark the reliable regions in the second far-scene sparse depth map to obtain the features of the far-scene dense depth map. It is further used to perform multi-scale fusion on the features of the second close-range dense depth map and the features of the far-scene dense depth map to obtain a depth map.

[0011] Further, obtain the information of the binocular polarization camera, including:

[0012] Obtain the camera baseline length, camera focal length, and disparity of the binocular polarization camera, where the binocular polarization camera is a short-baseline binocular polarization camera;

[0013] Obtain the polarization angle of the binocular polarization camera, the total light intensity of the scene, and the polarization image when the polarization direction is 0°.

[0014] Further, calculate the first close-range dense depth map according to the information of the binocular polarization camera, and calculate the first far-scene sparse depth map based on the atmospheric scattering model, including:

[0015] Calculate the first close-range dense depth map and the first far-scene sparse depth map based on the information of the binocular polarization camera and the atmospheric scattering model, including:

[0016] The formula for calculating the first close-range dense depth map is as follows:

[0017]

[0018] where B is the camera baseline length, f is the camera focal length, d is the disparity, is the first close-range dense depth map;

[0019] The formula for calculating the first far-scene sparse depth map is as follows:

[0020]

[0021]

[0022]

[0023] Among them, β is the atmospheric scattering coefficient, T(x) is the transmission function, A is the global atmospheric light intensity, I(x) is the pixel intensity received by the camera, J(x) is the true scene intensity without fog, α is the polarization angle, DoP is the degree of polarization, S0 represents the total light intensity of the scene, and I0 represents the polarization image when the polarization direction is 0°.

[0024] Further, aligning and normalizing the first close-range dense depth map and the first far-scene sparse depth map includes:

[0025] Performing pixel alignment on the first close-range dense depth map and the first far-scene sparse depth map;

[0026] Performing normalization processing on the depth values of the first close-range dense depth map and the first far-scene sparse depth map.

[0027] Further, the depth estimation network model is also used to enhance the details of the full-scene dense depth map based on U-Net.

[0028] Further, inputting the second close-range dense depth map, the second far-scene sparse depth map, and the confidence mask into the depth estimation network model to output a full-scene dense depth map includes:

[0029] Constructing a multi-modal feature extraction module, a sparse depth completion module, a fusion module, and a detail enhancement module;

[0030] The multi-modal feature extraction module is used to extract the features of the second close-range dense depth map and the second far-scene sparse depth map using a convolutional neural network;

[0031] The sparse depth completion module is used to calculate attention weights in the high-confidence regions of the second far-scene sparse depth map based on the Transformer network and the confidence mask, and mark the reliable regions in the second far-scene sparse depth map to obtain the features of the far-scene dense depth map;

[0032] The fusion module is used to project the features of the far-scene dense depth map and the features of the second close-range dense depth map into a unified dimension, then calculate the attention weights of the features of the far-scene dense depth map and the features of the second close-range dense depth map through a multi-head self-attention mechanism, splice and globally average the outputs of all attention heads to obtain sparse depth weights and dense depth weights, and then fuse the features of the far-scene dense depth map and the features of the second close-range dense depth map through a multi-scale fusion operation according to the sparse depth weights and dense depth weights to obtain a depth map;

[0033] The detail enhancement module is used to enhance the details of the depth map using U-Net to obtain a full-scene dense depth map.

[0034] Furthermore, the optimization objective of the depth estimation network model is to minimize the loss function, and the loss function is as follows:

[0035]

[0036]

[0037]

[0038]

[0039] Among them, is the predicted depth, is the dense depth ground truth, is the depth regression loss, is the sparse supervision loss, is the structure preservation loss, is the confidence mask, I is the original RGB image captured by the binocular polarization camera, N is the total number of pixels of the predicted depth and the ground truth depth at its corresponding position, i is the position index of the predicted depth and the ground truth depth, M is the total number of pixels of the predicted depth and the sparse depth at its corresponding position, j is the position index of the predicted depth and the sparse depth, ▽xZpred represents the gradient of Zpred in the x direction, ▽yZpred represents the gradient of Zpred in the y direction, ▽xI represents the gradient of I in the x direction, and ▽yI represents the gradient of I in the y direction.

[0040] The beneficial effects of the present invention are as follows:

[0041] 1) The technical solution of the present invention innovatively combines a short-baseline binocular polarization camera with an atmospheric scattering model, enhancing the robustness of depth estimation in complex environments. Through a progressive extrapolation strategy of driving long-distance estimation by short-distance depth, a new type of depth estimation framework is provided.

[0042] 2) The present invention realizes the optimization of depth continuity in the full scene through the multi-model fusion of geometric and physical depths. It improves the adaptability and scalability of complex scenes, especially performs well in environments with haze and complex lighting, can improve the accuracy of motion estimation of unmanned platforms, and reduce hardware costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flowchart of the depth estimation method based on a binocular polarization camera and an atmospheric scattering model in this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only a part rather than all of the embodiments of the present invention. Components of the embodiments of the present invention usually described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations.

[0045] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0046] It should be noted that like reference numerals and letters denote like items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0047] In the description of the present invention, it should be understood that the orientation or positional relationships indicated by the terms "upper", "lower", "inner", "outer", "left", "right", etc. are based on the orientation or positional relationships shown in the accompanying drawings, or the orientation or positional relationships when the product of the invention is normally placed, or the orientation or positional relationships commonly understood by those skilled in the art. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation to the present invention.

[0048] In addition, the terms "first", "second", etc. are only used for descriptive distinction and should not be construed as indicating or implying relative importance.

[0049] In the description of the present invention, it should also be noted that unless otherwise clearly defined and limited, terms such as "set", "connect" should be understood in a broad sense. For example, "connect" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0050] The following will specifically describe the embodiments of the present invention with reference to the accompanying drawings.

[0051] As Figure 1 shown, a depth estimation method based on a binocular polarization camera and an atmospheric scattering model includes:

[0052] S1: Obtain the information of the binocular polarization camera;

[0053] S2: Calculate the first close-range dense depth map according to the information of the binocular polarization camera, and calculate the first far-scene sparse depth map based on the atmospheric scattering model;

[0054] S3: Align and normalize the first close-range dense depth map and the first far-scene sparse depth map to obtain the second close-range dense depth map and the second far-scene sparse depth map;

[0055] S4: Verify the second far-scene sparse depth map of the atmospheric scattering model according to the second close-range dense depth map to mark the reliable depth points in the second far-scene sparse depth map, and generate a confidence mask;

[0056] S5: Input the second close-range dense depth map, the second far-scene sparse depth map and the confidence mask into the depth estimation network model, and output the full-scene dense depth map. The depth estimation network model is used to first extract the features of the second close-range dense depth map and the second far-scene sparse depth map, and is also used to calculate the attention weights in the high-confidence region of the second far-scene sparse depth map based on the Transformer network and the confidence mask, and mark the reliable regions in the second far-scene sparse depth map to obtain the features of the far-scene dense depth map, and is also used to perform multi-scale fusion on the features of the second close-range dense depth map and the features of the far-scene dense depth map to obtain the depth map.

[0057] In some embodiments, perform close-range depth estimation first:

[0058] Use a binocular polarization camera with a short baseline (2 cm) to obtain dense depth information within the range of 1-2 meters through the following steps:

[0059] Use the Semi-Global Matching (SGM) method to calculate the disparity map. Combine the camera intrinsic and extrinsic parameters, and use formula (1) to convert the disparity to depth.

[0060] (1)

[0061] Where B is the camera baseline length, f is the camera focal length, d is the disparity, and Z is the required scene depth, thus obtaining the first close-range dense depth map.

[0062] In some embodiments, then perform atmospheric scattering model parameter estimation, including:

[0063] Calibrate the key parameters of the atmospheric scattering model based on the close-range depth data:

[0064] Based on the polarization information, the atmospheric light intensity can be calculated by the following formula:

[0065] (2)

[0066] where α is the polarization angle, d is the degree of polarization, S0 represents the total light intensity of the scene, and I0 represents the polarization image when the polarization direction is 0°.

[0067] According to the atmospheric physical scattering model, the transmission function T(x) is calculated through the global atmospheric light intensity A and the image intensity I(x):

[0068] (3)

[0069] where is the pixel intensity received by the camera, is the true scene intensity in the case of no fog, is the transmittance, which decreases with the increase of the distance z, A is the global atmospheric light intensity, and β is the atmospheric scattering coefficient.

[0070] Assuming that the atmospheric scattering coefficient β is a constant value, the first far-scene sparse depth map can be calculated :[[]]END]]

[0071] (4)

[0072] Thus, the first far-scene sparse depth map is obtained.

[0073] In some embodiments, depth compensation and optimization are then performed, including:

[0074] (1) Depth map alignment and normalization: Pixel alignment is performed on the first close-range dense depth map and the first far-scene sparse depth map, the resolution is unified, the depth values are normalized, and the scale differences (such as depth range differences) are eliminated to obtain the second close-range dense depth map and the second far-scene sparse depth map.

[0075] (2) Sparse depth confidence calculation:

[0076] Using the second close-range dense depth map Verify the second far-scene sparse depth map of the atmospheric scattering model . Mark the reliable depth points in the second far-scene sparse depth map to generate a confidence mask.

[0077] (5)

[0078] where is the tolerance error threshold.

[0079] (3) Network structure

[0080] Input: The second close-range dense depth map , Second far - scene sparse depth map , RGB original image I, confidence mask .

[0081] Output: Full - scene dense depth map .

[0082] 1) Multimodal feature extraction module:

[0083] Extract the features of the second close - range dense depth map and the second far - scene sparse depth map using a convolutional neural network.

[0084] 2) Sparse depth completion module:

[0085] Calculate the attention weights in the high - confidence regions of the second far - scene sparse depth map based on the Transformer network and the confidence mask, and mark the reliable regions in the second far - scene sparse depth map to obtain the features of the far - scene dense depth map. Only calculate the attention weights in the high - confidence regions, mark the reliable regions (high - precision points) in the sparse depth map, and suppress the interference of noise or low - quality points.

[0086]

[0087] Where represents the total number of pixels in the high - confidence region, Q, K, and V are three core vectors of the sparse depth map, used to calculate the attention weights, is the scaling factor.

[0088] 3) Fusion module:

[0089] Project the features of the far - scene dense depth map and the features of the second close - range dense depth map into a unified dimension, then calculate the attention weights of the features of the far - scene dense depth map and the features of the second close - range dense depth map through the multi - head self - attention mechanism, splice and globally average the outputs of all attention heads to obtain the sparse depth weight and the dense depth weight, and then fuse the features of the far - scene dense depth map and the features of the second close - range dense depth map through a multi - scale fusion operation according to the sparse depth weight and the dense depth weight to obtain the depth map.

[0090] Generate dynamic weights by calculating the correlation between the sparse depth (far - field) and dense depth (near - field) features and :

[0091] 1. Feature projection:

[0092] Project the sparse depth feature and the dense depth feature into a unified dimension:

[0093] ;

[0094] ;

[0095] ;

[0096] , and are learnable weight matrices. Q is used to query the correlation between sparse depth and dense depth, and K, V generate key-value pairs based on dense depth.

[0097] 2. Multi-Head Self-Attention Calculation:

[0098] Calculate the attention weights of sparse depth to dense depth through the multi-head self-attention mechanism:

[0099]

[0100] Where: h: attention head number (a total of H heads), : scaling factor, : learnable weight of the h-th head.

[0101] 3. Global Weight Aggregation:

[0102] Concatenate the outputs of all attention heads and globally average them to obtain the weights of sparse and dense:

[0103]

[0104]

[0105] According to the importance of local features, adaptively allocate the weights of dense and sparse depth, so that the network depends on dense depth in the near field and sparse depth in the far field.

[0106] The final fused feature is:

[0107] (4)Loss Function

[0108] 1) Depth Regression Loss:

[0109] (6)

[0110] 2) Sparse Supervision Loss:

[0111] (7)

[0112] 3) Structure Preservation Loss: Combine the gradients of the RGB original image to ensure that the generated depth map is consistent with the scene structure:

[0113] (8)

[0114] 4) Total loss

[0115] (9)

[0116] Among them, is the predicted depth, is the dense depth ground truth, is the depth regression loss, is the sparse supervision loss, is the structure preservation loss, , , are the weights of the depth regression loss, the weights of the sparse supervision loss, and the weights of the structure preservation loss in sequence, is the confidence mask, I is the original RGB image captured by the binocular polarization camera, N is the total number of pixels of the predicted depth and the ground truth depth corresponding to its position, i is the position index of the predicted depth and the ground truth depth, M is the total number of pixels of the predicted depth and the sparse depth corresponding to its position, j is the position index of the predicted depth and the sparse depth, ▽xZpred represents the gradient of Zpred in the x direction, ▽yZpred represents the gradient of Zpred in the y direction, ▽xI represents the gradient of I in the x direction, and ▽yI represents the gradient of I in the y direction, is the second farthest scene sparse depth map when the position index of the predicted depth and the sparse depth is j.

[0117] The beneficial effects of the present invention are as follows:

[0118] 1) The technical solution of the present invention innovatively combines a short-baseline binocular polarization camera with an atmospheric scattering model, enhancing the robustness of depth estimation in complex environments. Through a progressive extrapolation strategy that drives long-distance estimation with short-distance depth, a new type of depth estimation framework is provided.

[0119] 2) The present invention realizes the optimization of depth continuity in the entire scene through the multi-model fusion of geometric and physical depths. It improves the adaptability and scalability of complex scenes, especially performs well in environments with haze and complex lighting, can improve the accuracy of motion estimation of unmanned platforms, and reduce hardware costs.

[0120] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A depth estimation method based on a binocular polarization camera and an atmospheric scattering model, characterized in that: include: Get the information of binocular polarization camera; Calculate a first close-range dense depth map based on information from the binocular polarization camera, and calculate a first far-field sparse depth map based on an atmospheric scattering model; Aligning and normalizing the first close-range dense depth map and the first far-scene sparse depth map to obtain a second close-range dense depth map and a second far-scene sparse depth map; Verifying a second far scene sparse depth map of an atmospheric scattering model according to the second near-range dense depth map to mark credible depth points in the second far scene sparse depth map, and generating a confidence mask; The second close-range dense depth map, the second far-scene sparse depth map, and the confidence mask are input into the depth estimation network model to output a full-scene dense depth map. The depth estimation network model is used to first extract features of the second close-range dense depth map and the second far-scene sparse depth map, and is also used to calculate attention weights in high-confidence areas of the second far-scene sparse depth map based on the Transformer network and the confidence mask, and mark credible areas in the second far-scene sparse depth map to obtain features of the far-scene dense depth map. It is also used to perform multi-scale fusion of the features of the second close-range dense depth map and the features of the far-scene dense depth map to obtain a depth map.

2. The depth estimation method based on a binocular polarization camera and an atmospheric scattering model according to claim 1, characterized in that: Get binocular polarization camera information, including: Obtaining a camera baseline length, camera focal length, and parallax of a binocular polarization camera, wherein the binocular polarization camera is a short-baseline binocular polarization camera; Obtain the polarization angle of the binocular polarization camera, the total light intensity of the scene, and the polarization image when the polarization direction is 0°.

3. The depth estimation method based on a binocular polarization camera and an atmospheric scattering model according to claim 2, characterized in that: Calculating a first close-range dense depth map based on information from the binocular polarization camera, and calculating a first far-field sparse depth map based on an atmospheric scattering model, including: The formula for calculating the first close-range dense depth map is as follows: ， Where B is the camera baseline length, f is the camera focal length, and d is the parallax. is the first close-range dense depth map; The formula for calculating the first far scene sparse depth map is as follows: ， ， ， Where β is the atmospheric scattering coefficient, T(x) is the transfer function, A is the global atmospheric light intensity, I(x) is the pixel intensity received by the camera, J(x) is the actual scene intensity in the absence of fog, α is the polarization angle, DoP is the degree of polarization, S0 represents the total light intensity of the scene, and I0 represents the polarization image when the polarization direction is 0°.

4. The depth estimation method based on a binocular polarization camera and an atmospheric scattering model according to claim 1, characterized in that: Aligning and normalizing the first close-range dense depth map and the first far-scene sparse depth map, including: Performing pixel alignment on the first close-range dense depth map and the first far-scene sparse depth map; The depth values of the first close-range dense depth map and the first far-scene sparse depth map are normalized.

5. The depth estimation method based on a binocular polarization camera and an atmospheric scattering model according to claim 1, characterized in that: The depth estimation network model is also used to perform detail enhancement on the full-scene dense depth map based on U-Net.

6. The depth estimation method based on a binocular polarization camera and an atmospheric scattering model according to claim 1, characterized in that: Inputting the second close-range dense depth map, the second far-scene sparse depth map, and the confidence mask into a depth estimation network model, and outputting a full-scene dense depth map, including: Construct a multimodal feature extraction module, a sparse depth completion module, a fusion module, and a detail enhancement module; The multimodal feature extraction module is used to extract features of the second close-range dense depth map and the second far-scene sparse depth map using a convolutional neural network; The sparse depth completion module is used to calculate attention weights in high-confidence areas of the second far scene sparse depth map based on the Transformer network and the confidence mask, and mark credible areas in the second far scene sparse depth map to obtain features of the far scene dense depth map; The fusion module is used to project the features of the distant scene dense depth map and the features of the second close-range dense depth map to a unified dimension, and then calculate the attention weights of the features of the distant scene dense depth map and the features of the second close-range dense depth map through a multi-head self-attention mechanism, splice and globally average the outputs of all attention heads to obtain sparse depth weights and dense depth weights, and then fuse the features of the distant scene dense depth map and the features of the second close-range dense depth map through a multi-scale fusion operation according to the sparse depth weights and dense depth weights to obtain a depth map; The detail enhancement module is used to perform detail enhancement on the depth map using U-Net to obtain a dense depth map of the entire scene.

7. The depth estimation method based on a binocular polarization camera and an atmospheric scattering model according to claim 1, characterized in that: The optimization goal of the depth estimation network model is to minimize the loss function, which is: ， ， ， ， in, is the predicted depth, is the dense depth truth, is the deep regression loss, is the sparse supervision loss, is the structure preservation loss, 、 、 The weights of the deep regression loss, sparse supervision loss, and structure retention loss are respectively: is the confidence mask, I is the original RGB image taken by the binocular polarization camera, N is the total number of pixels of the predicted depth and the true depth corresponding to its position, i is the position index of the predicted depth and the true depth, M is the total number of pixels of the predicted depth and the sparse depth corresponding to its position, j is the position index of the predicted depth and the sparse depth, ▽xZpred represents the gradient of Zpred in the x direction, ▽yZpred represents the gradient of Zpred in the y direction, ▽xI represents the gradient of I in the x direction, ▽yI represents the gradient of I in the y direction, The second far scene sparse depth map when the position index of the predicted depth and sparse depth is j.

Images