Depth estimation method and system based on continuous fusion of binocular intensity and polarization

Abstract

The present disclosure relates to a depth estimation method and system based on binocular intensity and polarization continuous fusion, the method comprising: acquiring a Stokes total intensity component and four polarization angle images, performing stereo matching in a parallax domain based on the Stokes total intensity component to obtain an intensity depth map; performing stereo matching on the four polarization angle images respectively to obtain four groups of polarization disparity estimation; fusing the four groups of polarization disparity estimation by using a truncated mean fusion strategy to obtain a polarization depth map; and performing pixel-level decision fusion on the intensity depth map and the polarization depth map according to the Stokes total intensity component and the four polarization angle images to obtain a final depth result. The present disclosure can effectively handle the matching ambiguity and information loss problems in complex scenes such as specular reflection, weak texture and highlight saturation, and significantly improves the robustness and estimation accuracy of three-dimensional reconstruction.

Description

Technical Field

[0001] This disclosure relates to the field of 3D reconstruction technology in computer vision, and in particular to a depth estimation method and system based on the continuous fusion of binocular intensity and polarization. Background Technology

[0002] 3D reconstruction technology serves as a bridge between the physical world and the digital space, holding significant application value in fields such as autonomous driving, machine vision, and intelligent manufacturing. Based on imaging mechanisms, existing technologies are mainly divided into two categories: active and passive. Among them, binocular stereo vision has become a current research hotspot due to its simple system structure, lack of need for active light source projection, and good concealment.

[0003] Traditional binocular stereo matching methods primarily rely on the grayscale information of local windows for cost calculation and aggregation, as well as the assumption of photometric consistency. While performing reasonably well in standard scenes, they are prone to matching ambiguity in areas with weak texture and specular reflection, exhibiting two main problems: First, both traditional algorithms and deep learning models rely heavily on the assumption of constant brightness in the left and right views. When facing non-Lambertian surfaces such as metal and glass, the intensity of reflected light from the object's surface fluctuates drastically with changes in the viewing angle (i.e., specular reflection), directly causing the matching cost function to fail, resulting in large-area depth holes or incorrect estimates. Second, in flat regions or regions with repetitive textures lacking significant intensity gradients, relying solely on intensity information is insufficient to construct a discriminative matching cost, leading to a "multi-peak" or flat distribution in depth estimation, making it difficult to recover high-precision geometric details.

[0004] Therefore, traditional binocular vision, which relies solely on intensity information, is insufficient to meet the reconstruction needs of complex optical scenes. Polarization imaging, as a physical observation method, can resolve the polarization state of light and reveal the normal and reflection characteristics of an object's surface, possessing inherent advantages in suppressing specular interference and enhancing geometric details. Therefore, how to effectively integrate polarization physical cues with binocular geometric constraints, and overcome their respective limitations, to obtain high-precision, high-completeness depth estimation under complex reflection conditions, is a pressing problem in this field. Summary of the Invention

[0005] To address the issues of gaps in depth estimation caused by matching ambiguities in weakly textured and specular regions in traditional binocular stereo matching due to the failure of the photometric consistency assumption, this disclosure proposes a depth estimation method based on the continuous fusion of binocular intensity and polarization to solve these problems.

[0006] According to one aspect of this disclosure, a depth estimation method based on the continuous fusion of binocular intensity and polarization is provided, comprising: S10. Acquire the Stokes total intensity component and four polarization angle images, which are synchronously acquired by a calibrated focal plane polarization binocular camera. S20. Based on the Stokes total intensity component, perform stereo matching in the disparity domain to obtain an intensity depth map; S30. Perform stereo matching on the four polarization angle images respectively to obtain four sets of polarization disparity estimates. Use the truncated mean fusion strategy to fuse the four sets of polarization disparity estimates to obtain a polarization depth map. S40. Based on the Stokes total intensity component and the four polarization angle images, perform pixel-level decision fusion on the intensity depth map and the polarization depth map to obtain the final depth result.

[0007] Preferably, the four sets of polarization disparity estimates are fused using a truncated mean fusion strategy to obtain a polarization depth map, including: Obtain the disparity values ​​of the four polarization angle channels at the same pixel location, and rearrange them into an ordered sequence according to their numerical values; Based on the ordered sequence, polarization fusion disparity is calculated by truncating the mean. Based on the calibration parameters of the focal plane polarization binocular camera, the polarization fusion parallax is converted into a polarization depth map.

[0008] Preferably, the polarization depth map is represented as follows: , In the formula, For the camera's focal length, B The baseline distance of the binocular camera. This is for polarization fusion parallax.

[0009] Preferably, the pixel-level decision fusion includes: Based on the four polarization angle images, the linear polarization degree of the pixels is calculated, and a continuous weighting function is determined based on the linear polarization degree. Determine the intensity availability mask based on the Stokes total intensity component; Based on the numerical characteristics of the four polarization angle images and the Stokes total intensity component, saturation regions are identified; The fusion rules are determined based on the continuous weight function, intensity availability mask, and saturation region identification results, and the intensity depth map and polarization depth map are fused according to the fusion rules.

[0010] Preferably, the continuous weighting function is expressed as: , In the formula, For pixels xLinear polarization degree at that point The modal switching threshold, As a scale factor, This is the lower bound of the weight.

[0011] Preferably, the intensity availability mask is represented as: , In the formula, Represents pixels x Stokes total intensity component at the location, This represents the intensity gradient energy within a local window. and These are the brightness and texture thresholds, respectively.

[0012] Preferably, the final depth result is expressed as: , In the formula, For pixels x The polarization depth estimation result is obtained by multi-polarization angle stereo matching and truncated mean fusion. For pixels x The continuous weights constructed based on the degree of linear polarization. For pixels x The intensity depth estimation results are obtained by stereo matching based on the Stokes total intensity component. For strength availability mask, For saturation region masking, and These represent the strength and polarization intensity validity masks determined based on the left-right consistency test and the uniqueness constraint of the matching cost, respectively.

[0013] According to one aspect of this disclosure, a depth estimation system based on binocular intensity and polarization continuity fusion is provided, comprising: The data acquisition module acquires the Stokes total intensity component and four polarization angle images, which are synchronously acquired by a calibrated focal plane polarization binocular camera. The intensity depth estimation module performs stereo matching in the disparity domain based on the Stokes total intensity component to obtain an intensity depth map; The polarization depth estimation module performs stereo matching on the four polarization angle images to obtain four sets of polarization disparity estimates. The four sets of polarization disparity estimates are then fused using a truncated mean fusion strategy to obtain a polarization depth map. The continuous fusion module performs pixel-level decision fusion of the intensity depth map and the polarization depth map based on the Stokes total intensity component and the four polarization angle images to obtain the final depth result.

[0014] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: execute the aforementioned depth estimation method based on binocular intensity and polarization continuity fusion.

[0015] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program / instructions and a bit stream thereon, wherein the computer program / instructions, when executed by a processor, implement the aforementioned depth estimation method based on binocular intensity and polarization continuity fusion to generate the bit stream.

[0016] Compared to the prior art, the beneficial effects of this disclosure are as follows: 1) This disclosure proposes a stereo matching method based on polarization information enhancement. By introducing multi-angle polarization observations from a focal plane polarization binocular camera as an additional geometric constraint, it provides more physical criteria for stereo matching under non-Lambertian reflection conditions, and significantly improves the robustness of depth estimation in complex reflection scenes.

[0017] 2) This disclosure proposes a physical-driven continuous weight fusion model that smoothly fuses intensity depth and polarization depth, significantly improving the accuracy of depth estimation in highly reflective regions.

[0018] 3) This disclosure proposes a depth estimation strategy that fuses multi-angle polarization and intensity information. It makes full use of the complementary characteristics of multi-angle polarization cues to make up for the lack of matching uncertainty of single light intensity cues in low-texture areas, thereby eliminating texture blind spots. It significantly improves the compactness and spatial continuity of depth maps in extreme scenes such as strong metallic reflection and weak plaster texture, while maintaining excellent geometric consistency.

[0019] 4) This disclosure proposes an adaptive weight allocation mechanism that can effectively suppress error propagation even if there is noise interference in the polarization channel, significantly improve depth coverage and edge sharpness, and achieve high-precision depth estimation for objects with weak texture.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0021] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0023] Figure 1A flowchart of a depth estimation method based on binocular intensity and polarization continuity fusion in an embodiment of this disclosure is shown; Figure 2 A schematic diagram showing the visualization results of the degree of linear polarization in a strongly reflective workpiece scene acquired by the camera in an embodiment of this disclosure is shown. Figure 3 The diagram illustrates the depth results of three methods on a metallic reflective sample in embodiments of this disclosure. Figure 4 The diagram illustrates the depth results of three methods on plaster samples in embodiments of this disclosure. Figure 5 Histograms of the plane fitting error distributions for three methods in embodiments of this disclosure are shown. Figure 6 A block diagram of a depth estimation system based on continuous fusion of binocular intensity and polarization is shown in an embodiment of this disclosure. Detailed Implementation

[0024] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0025] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0026] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0027] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this disclosure, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0029] Based on the above ideas, this disclosure proposes a depth estimation method based on the continuous fusion of binocular intensity and polarization. Figure 1 A flowchart is shown for a depth estimation method based on the continuous fusion of binocular intensity and polarization. The method includes: S10. Acquire the Stokes total intensity component and four polarization angle images, which are synchronously acquired by a calibrated focal plane polarization binocular camera. S20. Based on the Stokes total intensity component, perform stereo matching in the disparity domain to obtain an intensity depth map; S30. Perform stereo matching on the four polarization angle images respectively to obtain four sets of polarization disparity estimates. Use the truncated mean fusion strategy to fuse the four sets of polarization disparity estimates to obtain a polarization depth map. S40. Based on the Stokes total intensity component and the four polarization angle images, perform pixel-level decision fusion on the intensity depth map and the polarization depth map to obtain the final depth result.

[0030] The above embodiments acquire stereo images of the focal plane (DoFP) polarization simultaneously and extract the Stokes total intensity component and four-angle polarization information. Stereo matching is then performed on the four polarization angle images to obtain four sets of polarization disparity estimates. These estimates are then fused using a truncated mean fusion strategy to generate a polarization depth map. Based on this, a continuous weighting function is constructed using the linear polarization degree information derived from the Stokes total intensity component. Combined with intensity availability gating and saturation region discrimination mechanisms, adaptive fusion and smooth transition between the intensity depth map and the polarization depth map are achieved at the pixel level. This results in a highly complete and consistent depth estimation result, effectively overcoming matching ambiguities and information loss problems in complex optical scenes such as specular reflection, weak texture, and high light saturation, significantly improving the robustness and accuracy of 3D reconstruction.

[0031] This disclosure further extends the above-described method with detailed possible implementations, specifically including: S10. Acquire the Stokes total intensity component and four polarization angle images, which are synchronously acquired by a calibrated focal plane polarization binocular camera.

[0032] In one embodiment, the binocular system consists of two synchronously triggered focal plane polarization cameras, each capable of acquiring images in four polarization directions (0°, 45°, 90°, and 135°) in a single exposure. The binocular cameras are calibrated using the Zhang Zhengyou calibration method, and their intrinsic parameter matrices (focal length, principal point, distortion coefficients) and extrinsic parameter matrices (rotation matrix, translation vector) are accurately obtained using a standard checkerboard target, thus completing the geometric calibration of the binocular system.

[0033] S20. Based on the Stokes total intensity component, stereo matching is performed in the disparity domain to obtain an intensity depth map.

[0034] In this embodiment, the pixel coordinates in the left view are set as follows: , Its corresponding point in the right view is: , In the formula, d For parallax, u For pixels x Image coordinates in the horizontal direction, v For pixels x Image coordinates in the vertical direction. Parallax and scene depth. Z The two satisfy an inverse proportional relationship, expressed as: , In the formula, f For camera focal length, B Here is the binocular baseline length. To obtain additional physical observation information about scene reflection characteristics, a focal plane polarization camera (DoFP) is used in this embodiment. This camera can acquire multi-angle polarization intensity images at the same time while maintaining binocular geometric consistency. Let the pixel position be: , In the formula, This represents the pixel domain of the left view (left camera is the main viewpoint).

[0035] The set of polarization angles obtained by the polarization camera at this pixel position is The corresponding intensity image It can be represented as: , Specifically, in the Stokes imaging model that only considers the linear polarization component, at the pixel level... x The polarization state of light can be represented as a Stokes vector by a four-channel polarization intensity image. Wherein, Represents pixels x The total intensity component at that location, and and The component information describing the linear polarization state in different directions is represented as: , In stereo matching, it is assumed that the same physical point has similar brightness or local structure in the left and right views. Traditional matching costs are typically constructed based on pixel intensity or local structure information. For a given pixel... x and its candidate parallax d The most basic form of matching cost is: , In the formula, and These represent the intensity images of the left and right views, respectively. This embodiment primarily utilizes the total intensity component. Matching costs are constructed using this method. However, matching costs based on absolute intensity differences tend to degrade in the presence of variations in illumination, exposure differences, or non-Lambertian reflections.

[0036] To improve the robustness of the matching cost to changes in illumination, this embodiment uses the Census transform to describe the local structure and its Hamming distance as the matching cost. The matching cost function is defined as follows: , In the formula, and Representing pixels in the left and right views respectively x The Census descriptor for the corresponding disparity point. Since the Census transform depends only on the relative grayscale relationships between neighboring pixels, and not on absolute intensity values, it exhibits strong robustness to illumination changes and overall brightness shifts. Semi-global matching (SGM) approximates the global energy function by performing one-dimensional dynamic programming on the matching cost across multiple path directions. Its optimization objective function can be expressed as: , In the formula, Represents pixels x In parallax The matching cost item below, Represents the pixel neighborhood. Adjacent pixels x and The disparity smoothing constraint between them. The smoothing term usually adopts a piecewise penalty model, expressed as: , In the formula, Adjacent pixels x and The difference in parallax between them This is a penalty term for small parallax variations, used to encourage continuous variation in parallax within local areas. This is a penalty term for large parallax jumps, and its value is greater than... It is used to suppress drastic parallax changes between adjacent pixels.

[0037] Specifically, the model encourages continuous spatial variation of disparity while allowing jumps at object boundaries. Since directly solving the two-dimensional global optimization problem is computationally expensive, SGM decomposes the energy minimization process into one-dimensional path optimization along multiple directions. Along the path... r The recursive cost is defined as: , In the formula, For matching cost function, Along the path direction r In pixels x-r At this location, the corresponding parallax is d The cumulative value of the path over time k This is a disparity index variable used to iterate through pixels. x-r Consider all possible disparity states to compute the minimum path cost.

[0038] Finally, the path costs from multiple directions are summed and expressed as: , The disparity corresponding to the minimum cost is selected as the final estimate using a "winner-takes-all" strategy, expressed as: , In this embodiment, the aforementioned optimal disparity It is not the final goal, but rather an intermediate variable under the geometric constraints of binocular imaging. Based on the binocular imaging model, it can be further mapped to pixel-level depth estimation results. , is represented as: .

[0039] S30. Perform stereo matching on the four polarization angle images respectively to obtain four sets of polarization disparity estimates. Use the truncated mean fusion strategy to fuse the four sets of polarization disparity estimates to obtain a polarization depth map.

[0040] In this embodiment, the image acquired by the left camera is used as the reference pixel coordinate. After completing binocular stereo correction, the polar lines in the left and right views are aligned with the horizontal direction, for any polarization angle. , with left view pixels x For reference, obtain the corresponding pixel points along the epipolar direction in the right view. xd This constructs the left and right view pixel pairs corresponding to the polarization angle, represented as: , In the formula, For the left camera at a polarization angle of Pixels acquired at that time x The intensity value at that location, For the right camera at the same polarization angle Next pixel xd The corresponding intensity value obtained at that location, d This represents the parallax value of the left and right views along the epipolar direction.

[0041] Performing stereo matching independently on each of the above image pairs yields four sets of disparity estimates for polarization angles, expressed as follows: , Due to the effects of specular reflection saturation, imaging noise, etc., the parallax set The local error often exhibits a non-Gaussian distribution.

[0042] Furthermore, the disparity values ​​of the four polarization angle channels at the same pixel position are rearranged into an ordered sequence according to their numerical values; based on the ordered sequence, the polarization fusion disparity is calculated by truncating the mean; based on the calibration parameters of the focal plane polarization binocular camera, the polarization fusion disparity is converted into a polarization depth map.

[0043] To obtain a more accurate ensemble estimate, this embodiment employs a fusion strategy based on the truncated mean of statistics. For the same pixel location... x First, rearrange the effective disparity values ​​into an ordered sequence according to their numerical values: , In the formula, Indicates the sorted order of the first... k The polarization fusion disparity is defined as the mean of the intermediate components when all four polarization channel estimates are valid, and is expressed as: , This strategy works by removing the extreme values ​​at both ends of the sequence. and This strategy suppresses aberrant parallax caused by local specular reflection, high light saturation, or matching ambiguity, while preserving consistent estimates among most polarization channels. When the number of effective parallax channels decreases due to factors such as overexposure or shadows, the strategy adaptively degenerates into an arithmetic mean form to maintain the spatial continuity of the parallax field. This is still an intermediate result in the disparity domain. Based on the binocular geometry, it can be further mapped to a depth estimate, resulting in a polarization depth map as follows: , In the formula, For the camera's focal length, B The baseline distance of the binocular camera. This is for polarization fusion parallax.

[0044] S40. Based on the Stokes total intensity component and the four polarization angle images, perform pixel-level decision fusion on the intensity depth map and the polarization depth map to obtain the final depth result.

[0045] The pixel-level decision fusion includes: calculating the linear polarization degree of the pixels based on the four polarization angle images, and determining a continuous weight function based on the linear polarization degree; determining an intensity availability mask based on the Stokes total intensity component; identifying saturation regions based on the numerical characteristics of the four polarization angle images and the Stokes total intensity component; determining a fusion rule based on the continuous weight function, the intensity availability mask, and the saturation region identification result, and fusing the intensity depth map and the polarization depth map according to the fusion rule.

[0046] In this embodiment, when the incident light intensity exceeds the potential well capacity of the sensor, the pixel responses of different polarization angle channels will be simultaneously truncated to the saturation value. ,Right now: , In the formula, , , and These represent the pixel intensity values ​​acquired by the split-focus plane polarization camera at polarization angles of 0°, 45°, 90°, and 135°, respectively.

[0047] This will result in the Stokes component. and This leads to: , In the formula, Represents the pixel coordinates of the image.

[0048] It is important to emphasize that, under imaging conditions where saturation does not occur, the degree of linear polarization (DoLP) remains a physically valid and effective reliability indicator. The degradation phenomenon in this embodiment specifically refers to the truncation of illumination intensity information caused by sensor saturation. In such regions, the intensity component... The local gradient and texture structure are also disrupted, causing intensity parallax based on the photometric consistency assumption to be lost. Essentially ineffective. Conversely, polarization fusion parallax obtained by multi-polarization angle constraints... It still relies on geometric consistency and maintaining high stability across channels. Therefore, if the fusion weights are constructed solely based on DoLP, the depth estimation will incorrectly degenerate to the invalid intensity channel in saturated highlight regions. To avoid this logical conflict, intensity validity constraints must be explicitly introduced during the fusion process.

[0049] set up For intensity components The calculated disparity results are used as the intensity baseline for traditional stereo matching; To leverage the discriminative power of polarization information in non-Lambertian regions, the polarization parallax obtained by multi-polarization angle fusion is utilized. In this embodiment, a continuous weighting function is constructed using the Sigmoid function. , is represented as: , In the formula, For pixels x Linear polarization degree at that point The modal switching threshold, The scale factor. Linear polarization degree. Describes pixels x The relative proportion of the linearly polarized component in the total intensity is related to the surface's reflection mechanism. This function ensures that the weights rapidly approach 1 in regions with significant DoLP (typically corresponding to non-Lambertian surfaces), thus fully utilizing polarization information; while in low DoLP regions, the weights smoothly decay. To prevent the complete suppression of effective polarization channel information (such as texture edges) in extremely low polarization regions and to enhance the numerical stability of the fused field, a minimum polarization weight constraint is introduced. .

[0050] To accurately eliminate failure areas caused by sensor dynamic range cutoff (truncation / saturation), and to further eliminate interference from non-saturation failures such as low illumination or weak textures, this embodiment defines an intensity availability mask. Represented as: , In the formula, Represents pixels x Stokes total intensity component at the location, This represents the intensity gradient energy within a local window. and These are the brightness and texture thresholds, respectively. When At this point, the intensity depth is considered unreliable and excluded from subsequent fusion calculations. Taking into account the continuous weighting function, intensity availability mask, and saturation region identification results, the final depth result... Determined by the following piecewise function, expressed as: , In the formula, For pixels x The polarization depth estimation result is obtained by multi-polarization angle stereo matching and truncated mean fusion. For pixels x The continuous weights constructed based on the degree of linear polarization. For pixels x The intensity depth estimation results are obtained by stereo matching based on the Stokes total intensity component. For strength availability mask, For saturation region masking, and These represent the strength and polarization intensity validity masks determined based on the left-right consistency test and the uniqueness constraint of the matching cost, respectively.

[0051] To evaluate the performance of the depth estimation method based on the continuous fusion of binocular intensity and polarization, this embodiment constructs a polarization binocular stereo vision system using two identical polarization cameras. The polarization cameras are equipped with Sony IMX250MYR CMOS focal plane array (DoFP) polarization sensors, which integrate a micro-polarizer array, enabling simultaneous acquisition of intensity images in four polarization directions (0°, 45°, 90°, and 135°) in a single exposure. The sensor resolution is 2448×2048 pixels, with a pixel size of 3.45µm×3.45µm. The Zhang Zhengyou calibration method was used for intrinsic and extrinsic parameter calibration. The calibration process used a standard checkerboard target to obtain the intrinsic parameter matrices, distortion coefficients, and rotation and translation relationships between the left and right cameras. All experimental scenes were acquired under controlled lighting conditions, using an LED surface light source as the main light source to reduce interference from ambient stray light on polarization measurements.

[0052] To fully verify the applicability and robustness of the method in this embodiment under different reflection mechanisms, two types of typical samples are included: (1) Metal reflection samples have complex structures, exhibiting strong specular reflection characteristics as a whole, accompanied by obvious highlights and saturation areas, and containing complex reflection areas with irregular textures to test the reconstruction ability of non-Lambertian surfaces. (2) Plaster samples are obtained through surface treatment to achieve two surface states: smooth and rough. The smooth plaster surface approximates a Lambertian reflector, while the rough plaster surface has more significant diffuse reflection characteristics, used to test the ability to eliminate matching ambiguities in weakly textured areas.

[0053] This paper compares and analyzes the binocular depth estimation method based on intensity information (hereinafter referred to as the intensity baseline method), the binocular depth estimation method based on multi-polarization angle information fusion (hereinafter referred to as the multi-angle polarization method), and the depth estimation method based on continuous fusion of binocular intensity and polarization in this embodiment (hereinafter referred to as the intensity-polarization depth continuous fusion method).

[0054] First, we will focus on metal reflective samples. Figure 2This diagram illustrates the visualization results of the linear polarization degree (DoLP) in a scene of a highly reflective workpiece acquired by a camera. Figure 2 (a) in the image is the original intensity image; Figure 2 (c) in the diagram represents the original linear polarization degree distribution. Figure 2 (e) in the image represents the DoLP visualization result after contrast enhancement; Figure 2 (b) Figure 2 (d) and Figure 2 (f) in the middle are respectively Figure 2 (a) Figure 2 (c) and Figure 2 (e) in the image corresponds to a magnified view. From the original intensity image ( Figure 2 (a) and Figure 2 As can be observed in (b) of the diagram, the selected area exhibits a distinct high-brightness characteristic. The brightness in this local area approaches the upper limit of the dynamic range acceptable to the sensor, resulting in brightness saturation in this region. For this region dominated by strong specular reflection, the distribution of DoLP in this area exhibits a phenomenon contrary to its physical characteristics.

[0055] According to Fresnel's principle of reflection, specular reflection from a metal surface should typically have a high degree of polarization, but... Figure 2 (c) and Figure 2 In (d), the DoLP value of the highlight area is extremely low, appearing as a large dark area. A deeper analysis of its physical mechanism reveals that this pseudo-low value phenomenon is mainly caused by the dynamic range truncation of the sensor. In the saturation region, in the four angle images acquired by the polarization camera, the high-intensity component is forcibly truncated to the maximum gray value, which directly leads to the low DoLP value in the Stokes vector. and The differential amplitude is severely compressed. Meanwhile, the denominator of DoLP's calculation (total light intensity) is... The DoLP value remains at its maximum due to saturation. The combined effect of a smaller numerator and a very large denominator causes the calculated DoLP value to approach zero. If the reliability of polarization information is judged solely based on the magnitude of the DoLP value, this region will be misjudged as lacking effective polarization information, leading to the loss of effective geometric clues in subsequent stereo matching.

[0056] However, Figure 2 (e) and Figure 2 In (f), after histogram equalization enhancement of the DoLP data, it can be clearly observed that although the absolute value of DoLP in this region is extremely low, its spatial distribution is not random noise, but rather exhibits highly continuous geometric texture features. This phenomenon indicates that under extreme imaging conditions such as illumination saturation, the degradation of DoLP values ​​is not equivalent to the complete failure of polarization geometric constraints. Despite the full intensity The matching fails, accompanied by distortion of the DoLP value, but the relative relationships hidden between the multi-angle polarization channels are not completely lost. This confirms that in the strong reflection region, a single DoLP index is insufficient to accurately characterize the effectiveness of polarization information, and a reliability modeling method based on multi-angle redundancy must be introduced to fully explore the geometric clues remaining in the saturated data.

[0057] Figure 3 The results of three methods on metallic reflective samples are shown. Figure 3 (a) in the image is the original intensity map. Figure 3 (b) Figure 3 (d) and Figure 3 Figure (g) in the figure represents the depth maps for single-intensity, multi-angle polarization, and intensity-polarization fusion methods, respectively. Figure 3 (c) Figure 3 (f) and Figure 3 (h) represents the corresponding magnified local view. Table 1 quantitatively presents the evaluation results of the three methods, including the effective depth, void ratio, and left-right consistency index of different methods.

[0058] Table 1. Quantitative evaluation results of the three methods

[0059] like Figure 3 (b) and Figure 3 As shown in (c), the stereo matching results based on single intensity information exhibit significant depth loss in the highlight regions of the metal surface (shown in the red box). Physically, the strong specular reflection from the metal surface causes the incident light flux to exceed the camera's sensor dynamic range limit, leading to local brightness saturation, commonly known as "overexposure." This truncation directly disrupts the image's grayscale distribution, resulting in the complete loss of local texture information, with the intensity gradient in the overexposed region approaching zero. For stereo matching algorithms that rely on the photometric consistency assumption (such as SGM), this means the matching cost function completely degenerates in the saturated region, failing to find the correct disparity extrema, thus generating large-area matching holes and outliers. The quantitative data in Table 1 demonstrates this failure: the effective depth coverage of the intensity baseline method is only 88.9%, with a hole rate as high as 11.1%. This indicates that under complex lighting conditions dominated by non-Lambertian reflection, single intensity information is extremely fragile, unable to maintain reliable geometric constraints, and struggles to meet the requirements for high-completeness depth estimation.

[0060] Figure 3 (d) and Figure 3Figure (f) shows the depth estimation results after introducing multi-angle polarization geometric cues. Compared with the intensity baseline method, the most significant improvement of this method lies in its ability to fill specular holes. From the depth map, regions that were previously ineffective due to saturation have been restored with dense depth information. Its effectiveness stems from the unique multi-channel redundancy characteristics of polarization. Even when the total intensity channel is saturated, the saturation between individual polarization sub-channels is usually not completely synchronized, but rather exhibits asymmetric or partially truncated characteristics. The multi-angle polarization method utilizes this inter-channel information complementarity to construct a superior geometric consistency constraint, thereby achieving effective matching in strong specular regions. Quantitative analysis shows that the effective coverage of this method jumps to 99.5%, and the left-right consistency detection pass rate also improves from 92.1% to 95.3%, confirming the enormous potential of polarization information in overcoming strong light interference and restoring geometric structures.

[0061] Figure 3 (g) and Figure 3 Figure (h) shows the final result of the intensity-polarization depth continuous fusion method proposed in this embodiment. This method not only inherits the high density advantage of the polarization method in the highlight region, but also achieves further optimization in detail representation and smoothness. In principle, this method is not a simple linear addition, but achieves the complementary advantages of the intensity channel and the polarization channel based on the confidence gating mechanism. In the unsaturated region, which is also the quasi-specular reflection region, the high signal-to-noise ratio of the intensity information is fully utilized to ensure the smoothness and delicacy of the surface; in the strong specular reflection region, the polarization depth is switched to use its ability to still obtain information in the highlight region to repair holes. This mechanism ensures a smooth transition from depth obtained based on the intensity method to polarization depth, avoiding artifacts or noise that may be introduced by a single mode. The data in Table 1 show that the performance of this method has reached the optimal level: the effective coverage is as high as 99.9%, the hole rate limit is compressed to 0.1%, and the integrity of scene information is maximized while maintaining a high consistency accuracy of 95.3%.

[0062] In summary, Figure 3 The analysis in Table 1 reveals inherent limitations in single-modal depth estimation in non-Lambertian scenes such as metallic and strongly specular scenes: intensity information is constrained by radiation saturation, and polarization information is constrained by the signal-to-noise ratio. The method in this embodiment establishes an intensity-polarization continuum fusion mechanism, achieving complementarity between photometric and geometric cues at the physical level. This method not only maintains reliable geometric consistency in the saturation region but also maximizes the effective coverage of the depth map, providing a robust solution for achieving high-completeness and high-precision 3D perception under complex lighting conditions.

[0063] Secondly, plaster geometries with typical Lambertian reflectance characteristics were used as test objects. Plaster samples have typical near-Lambertian reflectance characteristics, and their surface albedo is uniform and lacks texture features. Because the surface albedo of plaster is uniform and lacks sufficient gray-level gradient variation, traditional stereo matching methods based on photometric consistency often fall into matching ambiguity due to the multi-peak or flat cost function. Figure 4 The image visually demonstrates the depth reconstruction effects of different methods on plaster samples. Figure 4 (a) in the image is the input image; Figure 4 (b) Figure 4 (d) and Figure 4 (g) in the figure represents the error visualizations for single-intensity, multi-angle polarization, and intensity-polarization fusion methods, respectively. Figure 4 (c) Figure 4 (f) and Figure 4 The (h) in the table represents the corresponding magnified detail image. Table 2 shows the corresponding quantitative performance metrics, including the effective depth, void ratio, and left-right consistency metrics for different methods.

[0064] Table 2 Corresponding Quantitative Performance Indicators

[0065] Figure 4 (b) and Figure 4 (c) in the figure shows the depth results based on a single intensity cue. In planar regions with gentle luminance gradients (as shown in the blue box), the depth map exhibits significant discontinuities, accompanied by a large amount of salt-and-pepper noise and local holes. The reason for this is that the smooth surface of plaster and the similarity cause the local gray-level variance of the image to approach zero, making the correlation-based matching cost function flat within the disparity search range, thus causing uncertainty in the solution space. This is further confirmed by the quantitative data in Table 2. The effective depth coverage of the intensity baseline method is only 92.6%, which means that 7.4% of the region (mainly the weak texture center area) is rejected due to low confidence. Although its left-right consistency detection pass rate reaches 94.7%, this metric only counts those pixels with strong texture and successful matching, thus masking the fact that a large number of weak texture areas fail to be reconstructed. Therefore, a single intensity cue cannot meet the high integrity reconstruction requirements of weak texture surfaces.

[0066] Figure 4 (d) and Figure 4Figure (f) shows the depth results after introducing multi-angle polarization cues. The most significant change is the visible improvement in the density of the depth map; previously fragmented, weakly textured areas are effectively filled, and surface smoothness is greatly improved. Quantitative data shows that the effective coverage of this method jumps to 99.6%, and the hole rate drops sharply to 0.4%, proving that the geometric cues of multi-polarization channel information successfully eliminate most of the texture blind spots. However, it is worth noting that despite the significant improvement in coverage, the left-right consistency pass rate of this method slightly decreases from 94.7% to 93.8%. This anomaly reveals the limitations of simple polarization matching. Although polarization information performs well on smooth surfaces, this deviation introduces a small number of noise points that do not meet geometric consistency requirements, causing edge pixels to fail the left-right consistency check. In summary, while relying solely on polarization depth solves the hole problem, it comes at a slight cost to the accuracy of geometric edges.

[0067] Figure 4 (g) and Figure 4 Figure (h) shows the final result of the intensity-polarization depth continuous fusion method proposed in this embodiment. This strategy is not a simple superposition, but fully considers the complementarity of intensity and polarization. In flat, weakly textured regions, the algorithm assigns higher weight to the polarization term, preserving... Figure 4 The details in (f) are improved, achieving an effective depth coverage of 99.9%, and the hole rate is further reduced to 0.1%. In the object edges and texture-rich areas, the high gradient advantage of intensity information is used to correct the polarization error. This method successfully solves the edge noise problem and significantly improves the left-right consistency pass rate to 96.2%, which is better than any single method.

[0068] In summary, in diffuse, weakly textured scenes, the continuous fusion method proposed in this embodiment overcomes the limitations of texture dependence on single intensity information and edge noise from single polarization information. While achieving near-full coverage (99.9%) of depth information, it ensures the highest geometric consistency accuracy (96.2%). To further verify the flatness and accuracy of this method in weakly textured local regions, plane fitting error analysis will be introduced for a more detailed evaluation.

[0069] Figure 5 Histograms of the plane fitting error distributions for the three methods are shown. Figure 5 (a) in the image is the original image; Figure 5 In the diagram, (b)-(d) represent the error histograms for single-intensity, multi-angle polarization, and intensity-polarization fusion methods, respectively. Figure 5As shown in (b), the depth error reconstructed using only intensity information exhibits a wide distribution. Although the mean error is close to zero, its probability density distribution curve is relatively flat with a large standard deviation. This indicates that in weakly textured regions, due to the lack of significant feature textures, the stereo matching algorithm struggles to find a unique corresponding point on the epipolar line, resulting in large random fluctuations in the calculated depth value near the true value. This discrete noise manifests as roughness and unevenness of the reconstructed plane in 3D space. (Comparison) Figure 5 As shown in (c), the error distribution curve narrows significantly and the peak density increases markedly after introducing multi-angle polarization information. The physical mechanism is that although the light intensity on the gypsum surface is uniform, the cues from the multi-polarization channels can still provide reliable constraints, thereby effectively correcting the matching ambiguity caused by the lack of texture in a single intensity channel, thus improving the precision of the reconstruction results.

[0070] Figure 5 Figure (d) shows the results of the intensity-polarization continuous weighted fusion algorithm proposed in this embodiment. Compared with the other two methods, the peak value is the highest and sharpest, and the probability density curve forms an extremely high and narrow peak at the zero error position, indicating that the vast majority of reconstructed points fall precisely on the fitting plane, and the error concentration is the highest. Compared with the previous two methods, the fusion method significantly reduces the long tails on both sides of the histogram, which shows that the method successfully identifies and removes large error outliers in intensity matching, while filling in details with high-confidence polarization information. The RMSE of the intensity information-based method is 0.104 mm, and the Std is 0.061 mm. In the current scene, although the texture of the plaster surface is weak, the SGM algorithm can still obtain a relatively smooth depth map through global cost aggregation, showing good continuity. The RMSE of the multi-angle polarization information fusion method increases to 0.116 mm, which is about 11.5% lower than the accuracy of the light intensity baseline; at the same time, its maximum standard deviation (Std) is 0.074 mm. This indicates that under the specific experimental conditions, the light flux attenuation problem introduced by the micro-polarizer array in the polarization camera leads to significant high-frequency random noise in the polarization information, resulting in a more discrete spatial distribution of the point cloud. Relying solely on polarization cues actually reduces the reconstruction quality.

[0071] Although the multi-angle polarization information fusion method performs far worse than the intensity information-based method, the intensity-polarization continuous weighted fusion depth method proposed in this embodiment can fully fuse the first two methods, achieving the lowest RMSE of 0.100 mm, which is 3.8% higher than the intensity baseline. This result proves the effectiveness of the fusion algorithm in this embodiment. The algorithm does not simply perform linear weighting on the two methods, but rather uses pixel-level confidence assessment to successfully identify and suppress large error noise in the polarization branch. Although the overall polarization performance is poor, it keenly captures the sparse high-confidence regions and uses these local effective information to correct the small systematic bias in intensity matching, thereby maintaining overall smoothness while further reducing the absolute error MAE to 0.077 mm. Table 3 shows the quantitative evaluation results of the three methods on weakly textured plaster samples, reflecting their generalization ability and consistency performance in weakly textured environments.

[0072] Table 3. Quantitative evaluation results of three methods for weakly textured plaster samples.

[0073] Based on the above experimental results, the intensity-polarization depth continuous fusion method proposed in this embodiment effectively solves the problem of intensity information loss in weakly textured diffuse reflection scenes. This method fully leverages the complementary characteristics of multi-angle polarization cues, compensating for the matching uncertainty of a single intensity cue in low-texture regions, thereby eliminating texture blind spots. Experiments show that even with noise interference introduced by the polarization channel, the method's built-in adaptive weight allocation mechanism can still effectively suppress error propagation. Ultimately, this method significantly improves depth coverage and edge sharpness while controlling the plane fitting error within the sub-millimeter range, achieving high-precision depth estimation for weakly textured objects.

[0074] As another aspect of this disclosure, a depth estimation system 100 based on the continuous fusion of binocular intensity and polarization is also provided, such as... Figure 6 As shown, it includes: Data acquisition module 1 acquires the Stokes total intensity component and four polarization angle images, which are synchronously acquired by a calibrated focal plane polarization binocular camera. Intensity depth estimation module 2 performs stereo matching in the disparity domain based on the Stokes total intensity component to obtain an intensity depth map; The polarization depth estimation module 3 performs stereo matching on the four polarization angle images to obtain four sets of polarization disparity estimates. The four sets of polarization disparity estimates are then fused using a truncated mean fusion strategy to obtain a polarization depth map. The continuous fusion module 4 performs pixel-level decision fusion of the intensity depth map and the polarization depth map based on the Stokes total intensity component and the four polarization angle images to obtain the final depth result.

[0075] Without causing contradictions, the above-described modules in the system of the present disclosure embodiments can implement any of the above-described methods.

[0076] Based on the description of the above embodiments, it can be seen that the embodiments of this disclosure can achieve the following technical effects: 1) This disclosure proposes a stereo matching method based on polarization information enhancement. By introducing multi-angle polarization observations from a focal plane polarization binocular camera as an additional geometric constraint, it provides more physical criteria for stereo matching under non-Lambertian reflection conditions, significantly improving the robustness of depth estimation in complex reflection scenarios.

[0077] 2) This disclosure proposes a physical-driven continuous weight fusion model that smoothly fuses intensity depth and polarization depth, significantly improving the accuracy of depth estimation in highly reflective regions.

[0078] 3) This disclosure proposes a depth estimation strategy that fuses multi-angle polarization and intensity information. It makes full use of the complementary characteristics of multi-angle polarization cues to make up for the lack of matching uncertainty of a single light intensity cue in low-texture areas, thereby eliminating texture blind spots. It significantly improves the compactness and spatial continuity of depth maps in extreme scenes such as strong metallic reflection and weak plaster texture, while maintaining excellent geometric consistency.

[0079] 4) This disclosure proposes an adaptive weight allocation mechanism that can effectively suppress error propagation even if there is noise interference in the polarization channel, significantly improve depth coverage and edge sharpness, and achieve high-precision depth estimation for objects with weak texture.

[0080] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to use the aforementioned depth estimation method based on binocular intensity and polarization continuity fusion. The electronic device can be provided as a terminal, a server, or other type of device.

[0081] This disclosure also proposes a computer-readable storage medium storing a computer program / instructions and a bitstream thereon. When executed by a processor, the computer program / instructions generate the bitstream using the depth estimation method based on the continuous fusion of binocular intensity and polarization described above. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0082] Those skilled in the art will understand that, in the above-described depth estimation method and system based on continuous fusion of binocular intensity and polarization in specific embodiments, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0084] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A depth estimation method based on the continuous fusion of binocular intensity and polarization, characterized in that, include: S10. Acquire the Stokes total intensity component and four polarization angle images, which are synchronously acquired by a calibrated focal plane polarization binocular camera. S20. Based on the Stokes total intensity component, perform stereo matching in the disparity domain to obtain an intensity depth map; S30. Perform stereo matching on the four polarization angle images respectively to obtain four sets of polarization disparity estimates. Use the truncated mean fusion strategy to fuse the four sets of polarization disparity estimates to obtain a polarization depth map. S40. Based on the Stokes total intensity component and the four polarization angle images, perform pixel-level decision fusion on the intensity depth map and the polarization depth map to obtain the final depth result.

2. The method according to claim 1, characterized in that, The four sets of polarization disparity estimates are fused using a truncated mean fusion strategy to obtain a polarization depth map, including: Obtain the disparity values ​​of the four polarization angle channels at the same pixel location, and rearrange them into an ordered sequence according to their numerical values; Based on the ordered sequence, polarization fusion disparity is calculated by truncating the mean. Based on the calibration parameters of the focal plane polarization binocular camera, the polarization fusion parallax is converted into a polarization depth map.

3. The method according to claim 2, characterized in that, The polarization depth map is represented as follows: ， In the formula, For the camera's focal length, B The baseline distance of the binocular camera. This is for polarization fusion parallax.

4. The method according to claim 1, characterized in that, The pixel-level decision fusion includes: Based on the four polarization angle images, the linear polarization degree of the pixels is calculated, and a continuous weighting function is determined based on the linear polarization degree. Determine the intensity availability mask based on the Stokes total intensity component; Based on the numerical characteristics of the four polarization angle images and the Stokes total intensity component, saturation regions are identified; The fusion rules are determined based on the continuous weight function, intensity availability mask, and saturation region identification results, and the intensity depth map and polarization depth map are fused according to the fusion rules.

5. The method according to claim 4, characterized in that, The continuous weighting function is expressed as follows: ， In the formula, For pixels x Linear polarization degree at that point The modal switching threshold, As a scale factor, This is the lower bound of the weight.

6. The method according to claim 4, characterized in that, The intensity availability mask is represented as: ， In the formula, Represents pixels x Stokes total intensity component at the location, This represents the intensity gradient energy within a local window. and These are the brightness and texture thresholds, respectively.

7. The method according to claim 4, characterized in that, The final depth result is expressed as follows: ， In the formula, For pixels x The polarization depth estimation result is obtained by multi-polarization angle stereo matching and truncated mean fusion. For pixels x The continuous weights constructed based on the degree of linear polarization. For pixels x The intensity depth estimation results are obtained by stereo matching based on the Stokes total intensity component. For strength availability mask, For saturation region masking, and These represent the strength and polarization intensity validity masks determined based on the left-right consistency test and the uniqueness constraint of the matching cost, respectively.

8. A depth estimation system based on the continuous fusion of binocular intensity and polarization, characterized in that, include: The data acquisition module acquires the Stokes total intensity component and four polarization angle images, which are synchronously acquired by a calibrated focal plane polarization binocular camera. The intensity depth estimation module performs stereo matching in the disparity domain based on the Stokes total intensity component to obtain an intensity depth map; The polarization depth estimation module performs stereo matching on the four polarization angle images to obtain four sets of polarization disparity estimates. The four sets of polarization disparity estimates are then fused using a truncated mean fusion strategy to obtain a polarization depth map. The continuous fusion module performs pixel-level decision fusion of the intensity depth map and the polarization depth map based on the Stokes total intensity component and the four polarization angle images to obtain the final depth result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the depth estimation method based on continuous fusion of binocular intensity and polarization as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program / instructions and a bit stream thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the depth estimation method based on the continuous fusion of binocular intensity and polarization as described in any one of claims 1-7 to generate the bit stream.

Images