Polarization imaging method, system and device based on a three-camera system
By using a multi-view polarization data acquisition and feature optimization method with a three-camera system, the resolution degradation and error problems of polarization imaging in dynamic scenes are solved, high-precision polarization image reconstruction is achieved, and the robustness and adaptability of the system are enhanced.
Patent Information
- Application Number
- CN202611020079.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-09
- Publication Date
- 2026-11-17
AI Technical Summary
Existing polarization imaging technology is difficult to adapt to dynamic scenes. Polarization mosaic leads to a decrease in spatial resolution and interpolation errors. Multi-frame acquisition strategies suffer from time synchronization errors and lack multi-view information fusion.
A three-camera system is used to acquire polarization data from multiple perspectives. A unified feature representation is constructed by cross-view alignment and physical prior information. The feature is then optimized by combining the confidence prediction module to generate high-quality polarization images.
It achieves high-precision polarization reconstruction in dynamic scenes, avoids interpolation errors, and improves the robustness and adaptability of the system in complex scenarios.
Smart Images

Figure CN123123553A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of polarization imaging technology, and more specifically, to a polarization imaging method, system, and device based on a three-camera system. Background Technology
[0002] Polarization imaging is an imaging technique that can acquire information about the polarization state of light waves. Compared to traditional RGB imaging, it can provide information such as the surface normal, reflection properties, and material properties of an object, making it valuable for applications in 3D reconstruction, reflection separation, and industrial inspection. Currently, mainstream polarization imaging methods primarily rely on polarization mosaic cameras or rotating polarizers to acquire information in different polarization directions.
[0003] Polarization mosaic cameras acquire multi-directional polarization information in a single exposure by arranging a filter array with different polarization angles on a single sensor. However, this approach leads to a decrease in spatial resolution, as each polarization direction occupies only a portion of the pixels, requiring a de-mosaic algorithm to recover the complete image. This process inevitably introduces interpolation errors, disrupting the consistency of the original pixels and causing error accumulation in subsequent tasks such as polarization super-resolution or physical parameter estimation. Another approach employs rotating polarizers or multi-frame acquisition strategies, acquiring different polarization information through multiple exposures. While this avoids the mosaic problem, it is difficult to apply to dynamic scenes and suffers from time synchronization errors, limiting its feasibility in practical applications. Summary of the Invention
[0004] This application provides a polarization imaging method, system, and device based on a three-camera system, which at least solves the technical problems in related technologies, such as the difficulty of polarization imaging methods adapting to dynamic scenes, the reduction in spatial resolution caused by polarization mosaic, and interpolation errors.
[0005] According to one aspect of the embodiments of this application, a polarization imaging method based on a three-camera system is provided, comprising: Multi-view polarization data is acquired, and the multi-view polarization data is aligned across viewpoints to obtain a polarization image aligned with a reference viewpoint. Physical prior information is constructed by inputting the reference viewpoint image, the aligned polarization image, and the physical prior information into the feature encoding module to perform multi-source information joint modeling and generate a unified feature representation. A pixel-level confidence map is generated based on the confidence prediction module, and confidence-guided feature optimization is performed on the unified feature representation based on the pixel-level confidence map. The optimized unified feature representation is input into the decoding module, the polarization parameters under the target view are output, and a polarization image under the target view is generated based on the polarization parameters.
[0006] In the embodiments of this application, the acquisition of multi-view polarization data includes: A polarization acquisition system is constructed, comprising a first camera, a second camera, and a third camera; wherein the first camera and the third camera are respectively equipped with polarizers with preset polarization directions, and the second camera is located between the first camera and the third camera to acquire unpolarized images as a reference viewpoint; The polarization acquisition system controls each camera to synchronously acquire data of the same target scene, thereby obtaining multi-view polarization data from different spatial locations.
[0007] In embodiments of this application, cross-viewpoint alignment is performed on the multi-viewpoint polarization data to obtain a polarization image aligned with a reference viewpoint, including: The multi-view polarization data is processed based on a stereo matching algorithm to estimate the disparity information of the polarization image of the non-reference viewpoint relative to the reference viewpoint. Based on the parallax information, a reprojection transformation is performed on the polarization image of the non-reference viewpoint to obtain a polarization image aligned with the reference viewpoint.
[0008] In the embodiments of this application, the construction of physical prior information includes: The polarization parameters are calculated based on the aligned polarization image to obtain the initial polarization angle and initial polarization degree, which are used as polarization priors. The scene depth is calculated based on the disparity information of the multi-view polarization data, and the surface normal information is derived from the scene depth as a geometric prior. The physical prior information is obtained based on the polarization prior and the geometric prior.
[0009] In the embodiments of this application, the reference viewpoint image, the aligned polarization image, and the physical prior information are input into the feature encoding module for multi-source information joint modeling to generate a unified feature representation, including: The reference viewpoint image, the aligned polarization image, and the physical prior information are input into the feature encoding module to extract the corresponding texture features, geometric features, and polarization features; the physical prior information includes geometric priors and polarization priors. The texture features, geometric features, and polarization features are integrated through a cross-modal fusion mechanism. With the polarization features as the dominant feature, the texture features and geometric features are adaptively weighted using an attention mechanism to generate the unified feature representation.
[0010] In the embodiments of this application, a pixel-level confidence map is generated based on the confidence prediction module, and confidence-guided feature optimization is performed on the unified feature representation based on the pixel-level confidence map, including: The confidence prediction module evaluates the confidence of the initial polarization estimate and generates a confidence map that reflects the credibility of each pixel region. Based on the pixel-level confidence map, the unified feature representation is optimized with confidence guidance to suppress the propagation of erroneous information in low-confidence regions while enhancing the feature contribution from reliable regions.
[0011] In the embodiments of this application, the optimized unified feature representation is input into the decoding module, and the polarization parameters under the target viewpoint are output, including: The optimized unified feature representation is input into the decoding module, and the polarization degree and polarization angle under the target view are output. Among them, the decomposition representation method is used to model the polarization angle.
[0012] According to another aspect of the embodiments of this application, a polarization imaging system is provided, comprising: A multi-view acquisition module is used to acquire multi-view polarization data and perform cross-view alignment on the multi-view polarization data to obtain a polarization image aligned with a reference view. The multimodal fusion module is used to construct physical prior information. It inputs the reference view image, the aligned polarization image, and the physical prior information into the feature encoding module to perform joint modeling of multi-source information and generate a unified feature representation. The feature optimization module is used to generate a pixel-level confidence map based on the confidence prediction module, and to perform confidence-guided feature optimization on the unified feature representation based on the pixel-level confidence map. The polarization reconstruction module is used to input the optimized unified feature representation into the decoding module, output polarization parameters under the target viewpoint, and generate a polarization image under the target viewpoint based on the polarization parameters.
[0013] In embodiments of this application, the following are included: The physical prior construction unit is used to analyze and calculate polarization parameters based on the aligned polarization image to obtain the initial polarization angle and initial polarization degree, which serve as polarization priors. The scene depth is calculated based on the disparity information of the multi-view polarization data, and the surface normal information is derived from the scene depth as a geometric prior. The physical prior information is obtained based on the polarization prior and the geometric prior.
[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described polarization imaging method based on a three-camera system through the computer program.
[0015] The technical solutions provided in this application embodiment may include the following beneficial effects: This application proposes a single-shot polarization imaging method based on a three-camera system. This method acquires multi-view polarization data through multi-camera collaboration, avoiding polarization mosaic structures, eliminating interpolation errors at the source, and achieving pixel-level information preservation. At the algorithm level, this method constructs physical prior information, which, along with a reference viewpoint and aligned polarization images, is input into a feature encoding module for multi-source information joint modeling, generating a unified feature representation. Furthermore, it combines a pixel-level confidence map with confidence-guided feature optimization of the unified features. This multi-view fusion and physical prior-guided reconstruction mechanism not only effectively improves the robustness of the model in complex scenarios such as occlusion boundaries and strong reflections but also significantly enhances the accuracy and stability of polarization image reconstruction. This application constructs an efficient single-shot polarization imaging framework, significantly improving the system's adaptability and practical value in real-world dynamic scenes while ensuring imaging efficiency. Attached Figure Description
[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a polarization imaging method based on a three-camera system according to an embodiment of this application; Figure 2 This is a flowchart of another polarization imaging method based on a three-camera system according to an embodiment of this application; Figure 3 This is a schematic diagram of an overall technical solution architecture for polarization imaging according to an embodiment of this application; Figure 4 This is a schematic diagram of a polarization imaging system based on a three-camera system according to an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0018] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0019] Existing polarization imaging technologies are mainly limited by three major bottlenecks: first, the polarization mosaic sampling structure of the focal plane sensor inevitably leads to a decrease in spatial resolution and interpolation errors; second, the multi-frame time-division acquisition mechanism is difficult to adapt to dynamic scenes and is prone to motion artifacts; and third, existing methods are mostly limited to single-view modeling and lack effective fusion and utilization of multi-view spatial information. To address these technical pain points, this application proposes a rapid polarization imaging method based on three-camera collaboration. By simultaneously acquiring data from multiple cameras and jointly modeling across different viewpoints, it avoids mosaic errors at the source, preserves pixel-level information, and significantly improves the system's polarization information reconstruction quality and environmental adaptability in dynamic scenes.
[0020] The polarization imaging method based on a three-camera system according to embodiments of this application will be described in detail below with reference to the accompanying drawings. Figure 1 As shown, the method mainly includes the following steps: S101 acquires multi-view polarization data, performs cross-view alignment on the multi-view polarization data, and obtains a polarization image aligned with the reference viewpoint.
[0021] In the embodiments of this application, acquiring multi-view polarization data includes: constructing a polarization acquisition system, which includes a first camera, a second camera, and a third camera; wherein the first camera and the third camera are respectively equipped with polarizers with preset polarization directions, and the second camera is located between the first camera and the third camera to acquire non-polarized images as a reference viewpoint; controlling each camera in the polarization acquisition system to synchronously acquire the same target scene to obtain multi-view polarization data from different spatial locations.
[0022] In an exemplary scenario, a polarization acquisition system consisting of three cameras is first constructed. The left and right cameras are equipped with polarizers with fixed polarization directions (e.g., 0° and 45°), while the middle camera serves as a reference viewpoint, acquiring unpolarized or adjustable polarized images. The three cameras simultaneously acquire polarization observation data from the same scene via synchronous triggering, obtaining data from different spatial locations. Camera calibration is then used to obtain the intrinsic and extrinsic parameters of each camera, establishing a unified spatial coordinate relationship and providing a foundation for subsequent cross-viewpoint alignment.
[0023] Furthermore, cross-viewpoint alignment is performed on the multi-viewpoint polarization data to obtain a polarization image aligned with the reference viewpoint, including: processing the multi-viewpoint polarization data based on a stereo matching algorithm to estimate the disparity information of the polarization image of the non-reference viewpoint relative to the reference viewpoint; and performing a reprojection transformation on the polarization image of the non-reference viewpoint based on the disparity information to obtain a polarization image aligned with the reference viewpoint.
[0024] Specifically, to achieve effective fusion of multi-view polarization data, this implementation first performs viewpoint alignment processing on the side-view images to resolve spatial misalignment and occlusion errors caused by differences in camera positions. This includes the following steps: First, the disparity mapping relationship from the left and right viewpoints to the reference viewpoint is estimated using a stereo matching method, where the reference viewpoint is typically the view from the center camera. This disparity reflects the pixel position offset of the same scene point between the side viewpoint and the reference viewpoint images.
[0025] Subsequently, based on the estimated parallax mapping relationship, the side-view polarization images are reprojected and transformed to map each side-view polarization image onto the reference view coordinate system, thereby obtaining a polarization image representation that is spatially aligned with the reference view.
[0026] The above process transforms polarization observations from different perspectives into a unified image coordinate system, establishes pixel-level correspondences across perspectives, eliminates misalignment and distortion caused by perspective differences, and thus provides spatially consistent high-quality input for subsequent polarization parameter recovery based on multi-view fusion.
[0027] S102 constructs physical prior information, inputs the reference viewpoint image, the aligned polarization image, and the physical prior information into the feature encoding module to perform multi-source information joint modeling and generate a unified feature representation.
[0028] In the embodiments of this application, the construction of physical prior information includes: calculating polarization parameters based on the aligned polarization image to obtain the initial polarization angle and initial polarization degree, as polarization prior; calculating scene depth based on the disparity information of multi-view polarization data, and deriving surface normal information based on scene depth, as geometric prior; and obtaining physical prior information based on polarization prior and geometric prior.
[0029] After completing cross-view alignment, further physical prior information to guide reconstruction is constructed, including polarization prior and geometric prior. The polarization prior is constructed as follows: by substituting the aligned polarization intensity into a polarization physics model (e.g., a system model based on Stokes parameters) for analytical calculation, initial estimates of the polarization angle and degree of polarization are obtained. Although this initial estimate contains some errors due to noise, occlusion, and other factors, it originates from the physical imaging model and can provide effective polarization physical constraints for subsequent optimization processes, thereby guiding the network to recover polarization information more accurately and stably within a solution space that conforms to physical laws.
[0030] Simultaneously, scene depth information is obtained through parallax calculation, and surface normal information is further derived as a geometric prior. Since there is a physical correlation between polarization state and surface normal, introducing this geometric prior can enhance the model's ability to model real physical laws.
[0031] Furthermore, the reference viewpoint image, the aligned polarization image, and the physical prior information are input into the feature encoding module to perform multi-source information joint modeling and generate a unified feature representation. This includes: first, the reference viewpoint image, the aligned polarization image, and the physical prior information are input into the feature encoding module to extract the corresponding texture features, geometric features, and polarization features.
[0032] Specifically, the reference viewpoint image, aligned polarization image, surface normals, and initial polarization estimate are used as multi-source inputs to the feature encoding module for joint modeling. Different modal information is extracted through independent encoding paths to obtain texture features, geometric features, and polarization features. To prevent the smoothing of polarization information from being destroyed during conventional spatial convolution, a pixel-by-pixel encoding strategy is adopted in the polarization feature extraction branch to preserve pixel-level polarization attributes. Furthermore, to improve the modeling capability for viewpoint-related polarization changes, viewpoint orientation information is introduced into the feature representation in a modulation manner.
[0033] Furthermore, texture features, geometric features, and polarization features are integrated through a cross-modal fusion mechanism, with polarization features as the dominant feature. An attention mechanism is used to adaptively weight texture features and geometric features to generate a unified feature representation.
[0034] Specifically, this implementation integrates the extracted texture features, geometric features, and polarization features through a cross-modal fusion mechanism. During the fusion process, polarization features are used as the primary clue, and an attention-based fusion strategy is introduced. This allows the polarization features to adaptively and selectively extract clues highly correlated with polarization information from the texture and geometric features, while suppressing irrelevant or interfering information, thereby forming a multimodal unified feature representation that is complementary in information and consistent in structure.
[0035] In the fusion process, texture features and geometric features are first concatenated along the channel dimension to construct structural context features. These context features are then used to generate key and value vectors for the cross-modal attention mechanism, while polarization features are mapped to query vectors. These query, key, and value vectors are fed into a normalization layer for cross-channel mean and variance adjustment. A multi-head attention mechanism then adaptively calculates the pixel-level association weights between polarization features and the structural context. After nonlinear spatial transformation through layer normalization, a feedforward network is used to perform channel interaction and eliminate feature redundancy between modalities, ultimately outputting a unified fused feature that combines physical polarization attributes with high-fidelity geometric texture structure.
[0036] This mechanism ensures that the fused features retain polarization physical properties while effectively enhancing the model's robustness to changes in scene structure, materials, and lighting.
[0037] S103 generates a pixel-level confidence map based on the confidence prediction module, and performs confidence-guided feature optimization on the unified feature representation based on the pixel-level confidence map.
[0038] In the embodiments of this application, a pixel-level confidence map is generated based on the confidence prediction module, and confidence-guided feature optimization is performed on the unified feature representation based on the pixel-level confidence map, including: The confidence prediction module evaluates the confidence of the initial polarization estimate and generates a confidence map that reflects the credibility of each pixel region. Based on the pixel-level confidence map, the unified feature representation is optimized with confidence guidance to suppress the propagation of erroneous information in low-confidence regions and enhance the feature contribution from reliable regions.
[0039] Specifically, this implementation introduces a confidence estimation mechanism to model the reliability of the initial polarization prior in different regions. A pixel-level confidence map is generated through a confidence prediction module to quantitatively evaluate the reliability of the initial polarization estimate.
[0040] This confidence prediction module employs a lightweight CNN architecture consisting of several shallow convolutional layers and activation functions cascaded alternately. The module receives the input reference image, surface normals, and analytically calculated pseudo-polarization angles and degrees. Within the network, it deeply mines and establishes strong correlation constraints between polarization physical properties and macroscopic spatial geometry, while simultaneously capturing the correspondence between images from multiple perspectives. Finally, the features extracted by the network are strictly constrained to a probability range of zero to one by a sigmoid activation function, thereby accurately mapping at the pixel scale and outputting the final pixel-level confidence map, used to evaluate the reliability of the initial polarization estimate in different regions.
[0041] Based on this confidence map, a confidence-guided feature optimization mechanism is constructed. During the fusion and propagation process, features in low-confidence regions are suppressed to reduce the impact of alignment errors, occlusion, and noise. Simultaneously, the contribution weight of features in high-confidence regions is enhanced in the final representation. This mechanism enables the model to adaptively distinguish and process polarization information from regions of different confidence levels, significantly improving robustness and reconstruction accuracy in complex scenarios such as occlusion boundaries and high-reflection areas.
[0042] S104 inputs the optimized unified feature representation into the decoding module, outputs the polarization parameters under the target viewpoint, and generates a polarization image under the target viewpoint based on the polarization parameters.
[0043] Based on the fused and optimized features, the polarization parameters, including the degree of polarization and the polarization angle, are output from the target's perspective through the decoding module. To ensure the continuity and physical consistency of angle prediction, a decomposition representation method is used for modeling, thereby avoiding angle ambiguity. The final output results show significant improvements in spatial structure, physical consistency, and detail representation.
[0044] The decoding module reconstructs the complete polarization information of each pixel from the target's perspective, including parameters such as polarization angle and degree of polarization, based on the optimized unified feature representation. Then, a polarization imaging physical model maps the polarization parameters to the corresponding polarization image representation, thereby generating a polarization image with accurate polarization attributes, spatial consistency, and clear details from the target's perspective. This process achieves high-quality polarization information reconstruction within a single acquisition framework, providing directly usable imaging results for subsequent polarization vision applications.
[0045] The physically constrained polarization reconstruction network involved in this application is trained using a data-driven approach to achieve the mapping from multi-view observations to the target viewpoint polarization parameters. To ensure the model's generalization ability and physical consistency, two types of training data sources are constructed simultaneously: synthetic data and real-world acquired data.
[0046] Regarding synthetic data, a multi-view polarization imaging dataset was constructed based on a physically based rendering engine. Specifically, 3D scene data was rendered to generate image observations from different viewpoints and polarization directions. By diversifying the settings for scene content and camera parameters (including focal length and baseline), training samples with rich geometric structures and lighting variations were constructed. During data generation, corresponding real polarization parameters (including polarization angle and degree of polarization) were simultaneously obtained as supervisory signals for network training. This type of data can provide physical consistency constraints under ideal conditions, which helps the network learn the basic laws of polarization imaging.
[0047] Regarding real-world data, this application constructs a synchronous polarization acquisition system consisting of three cameras. The system employs a narrow baseline configuration, with the left and right cameras each equipped with linear polarizers of fixed polarization directions (e.g., 0° and 45°), and the middle camera serving as a reference viewpoint. During data acquisition, a rotatable polarizer is introduced in front of the middle camera, and images from multiple polarization directions (e.g., 0°, 45°, 90°, 135°) are acquired sequentially using time multiplexing, thereby calculating high-precision polarization parameters as real-world labels. During the actual inference phase, the polarizer on the middle camera is removed, making it the input as an unpolarized reference image, achieving polarization information recovery under single-exposure conditions. To eliminate response differences between different cameras, the system undergoes radiometric consistency calibration before data acquisition. Specifically, this includes aligning the white balance and exposure of each camera under uniform illumination based on a standard color calibration board, and calculating the color correction matrix and intensity scaling factor to ensure photometric consistency across multiple viewpoints. This process effectively reduces polarization estimation errors introduced by hardware differences.
[0048] In the embodiments of this application, an end-to-end deep neural network model for joint modeling of multi-source information and generation of polarization parameters is trained, including a feature encoding module responsible for extracting and fusing multimodal features from multi-view images, geometric priors, and initial polarization estimates; a confidence prediction module responsible for evaluating the confidence of the initial estimate and generating a pixel-level confidence map; a feature optimization module that adaptively weights and optimizes the features using the confidence map; and a decoding module responsible for restoring the optimized unified feature representation and mapping it to polarization parameters under the target viewpoint.
[0049] The specific network training process includes: using multi-view observation images, cross-view alignment results, geometric prior information, and initial polarization estimates as network inputs, and using the true polarization parameters from the target viewpoint as supervision signals to perform end-to-end training. The network structure employs a combination of encoding and decoding, achieving joint modeling of multi-modal information through multi-layer feature extraction and fusion. During training, the difference between the predicted polarization parameters and the true values is constrained by a loss function, and gradient-based optimization algorithms are used to iteratively update the network parameters. Simultaneously, data augmentation strategies (such as viewpoint perturbation, illumination changes, and noise simulation) can be introduced to enhance the model's adaptability to complex scenes.
[0050] By inputting multi-view observation data and using the true polarization parameters from the target viewpoint as the supervision signal, the difference between the predicted and true values is calculated using a loss function. Subsequently, the parameters of each module in the above network model are iteratively updated through a gradient-based optimization algorithm until the model can accurately recover high-quality polarization parameters from multi-source inputs.
[0051] In one exemplary embodiment, a polarization image acquisition system is composed of three industrial cameras. Each camera is fitted with a polarizer at a specific polarization angle (e.g., 0°, 45°), and the relative positions of the cameras are fixed by rigid supports to ensure basic overlap of the fields of view. During acquisition, a hardware synchronization triggering mechanism controls the simultaneous exposure of the three cameras to obtain multi-view images at the same moment, ensuring temporal consistency and avoiding temporal misalignment caused by dynamic scenes. After acquisition, the obtained three-view images are input into the polarization reconstruction model of this method. This model, through multi-view alignment, physical prior modeling, multi-source information fusion, and confidence-guided optimization processing, ultimately outputs an accurate and complete polarization image from the target viewpoint.
[0052] To facilitate understanding of the polarization imaging method in the embodiments of this application, the following description is provided in conjunction with the appendix. Figure 2 Further description, such as Figure 2 As shown, the overall technical solution mainly includes six core steps: multi-view polarization data acquisition, cross-view alignment and initial mapping, physical prior construction, multi-modal feature encoding and fusion, confidence-guided optimization, and polarization parameter reconstruction. Together, they constitute a physically constrained polarization reconstruction framework.
[0053] To facilitate understanding of the polarization imaging method in the embodiments of this application, the following description is provided in conjunction with the appendix. Figure 3 Further description, such as Figure 3 As shown in the attached figure, a polarization image reconstruction network architecture based on multimodal feature fusion and confidence guidance is presented. The overall process consists of four main stages: image acquisition and preprocessing, pseudo-polarization prior, multimodal feature fusion and confidence estimation, and parameter reconstruction. First, reference viewpoint images and multi-view polarization images are simultaneously acquired using three cameras. After disparity estimation and normal calculation, disparity-aligned images and surface normals are obtained, constructing a physical prior. Subsequently, the texture encoder, geometric encoder, and polarization encoder extract texture features, geometric features, and polarization features, respectively. The polarization features are encoded pixel-wise and combined with viewpoint direction modulation. Next, multimodal feature fusion is achieved through a multi-head attention mechanism and a gated fusion module to form a unified feature set. Based on this, a confidence prediction module generates a pixel-level confidence map to guide feature optimization and suppress interference from low-confidence regions. Finally, the polarization angle (AoP) and polarization degree (DoP) under the target viewpoint are reconstructed through AoP and DoP branch networks, achieving high-precision and robust polarization information recovery.
[0054] The polarization imaging method of this application has the following technical effects: (a) It utilizes multi-camera collaboration to directly acquire complete images of different polarization directions, avoiding the interpolation process necessary for traditional polarization mosaic structures from the source, and achieving complete preservation of pixel-level polarization information; (b) Through multi-view alignment and fusion mechanisms, it effectively improves reconstruction accuracy and cross-view spatial consistency, especially performing better on complex surfaces and occluded areas; (c) Combining physical priors and confidence-guided optimization mechanisms, it significantly enhances the robustness and adaptability of the model in dynamic scenes, enabling high-precision polarization reconstruction to be completed in a single acquisition, greatly improving the practicality of the system in real-world applications while ensuring imaging efficiency.
[0055] According to another aspect of the embodiments of this application, a polarization imaging system for implementing the above-described polarization imaging method based on a three-camera system is also provided. For example... Figure 4 As shown, the system includes: The multi-view acquisition module 401 is used to acquire multi-view polarization data, perform cross-view alignment on the multi-view polarization data, and obtain a polarization image aligned with the reference view. The multimodal fusion module 402 is used to construct physical prior information. It inputs the reference view image, the aligned polarization image, and the physical prior information into the feature encoding module to perform joint modeling of multi-source information and generate a unified feature representation. The feature optimization module 403 is used to generate a pixel-level confidence map based on the confidence prediction module, and to perform confidence-guided feature optimization on the unified feature representation based on the pixel-level confidence map; The polarization reconstruction module 404 is used to input the optimized unified feature representation into the decoding module, output the polarization parameters under the target viewpoint, and generate a polarization image under the target viewpoint based on the polarization parameters.
[0056] In embodiments of this application, the following are included: The physical prior construction unit is used to analyze and calculate polarization parameters based on the aligned polarization image to obtain the initial polarization angle and initial polarization degree, which serve as the polarization prior; it calculates the scene depth based on the disparity information of the multi-view polarization data and derives the surface normal information based on the scene depth, which serves as the geometric prior; and it obtains the physical prior information based on the polarization prior and the geometric prior.
[0057] It should be noted that the polarization imaging system provided in the above embodiments, when executing the polarization imaging method based on a three-camera system, is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the polarization imaging system provided in the above embodiments and the polarization imaging method embodiments based on a three-camera system belong to the same concept, and its implementation process is detailed in the method embodiments, which will not be repeated here.
[0058] According to another aspect of the embodiments of this application, an electronic device corresponding to the polarization imaging method based on a three-camera system provided in the foregoing embodiments is also provided, so as to execute the polarization imaging method based on a three-camera system described above.
[0059] Please refer to Figure 5 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 5 As shown, the electronic device includes: a processor 500, a memory 501, a bus 502, and a communication interface 503. The processor 500, the communication interface 503, and the memory 501 are connected via the bus 502. The memory 501 stores a computer program that can run on the processor 500. When the processor 500 runs the computer program, it executes the polarization imaging method based on a three-camera system provided in any of the foregoing embodiments of this application.
[0060] The memory 501 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 503 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.
[0061] Bus 502 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. The memory 501 is used to store programs. After receiving an execution instruction, the processor 500 executes the program. The polarization imaging method based on a three-camera system disclosed in any of the aforementioned embodiments of this application can be applied to the processor 500, or implemented by the processor 500.
[0062] The processor 500 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 500 or by instructions in software form. The processor 500 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 501. The processor 500 reads the information in memory 501 and, in conjunction with its hardware, completes the steps of the above method.
[0063] The electronic device provided in this application embodiment and the polarization imaging method based on a three-camera system provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.
[0064] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0065] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.
Claims
1. A polarization imaging method based on a three-camera system, characterized in that, include: Multi-view polarization data is acquired, and the multi-view polarization data is aligned across viewpoints to obtain a polarization image aligned with a reference viewpoint. Physical prior information is constructed by inputting the reference viewpoint image, the aligned polarization image, and the physical prior information into the feature encoding module to perform multi-source information joint modeling and generate a unified feature representation. A pixel-level confidence map is generated based on the confidence prediction module, and confidence-guided feature optimization is performed on the unified feature representation based on the pixel-level confidence map. The optimized unified feature representation is input into the decoding module, the polarization parameters under the target view are output, and a polarization image under the target view is generated based on the polarization parameters.
2. The method according to claim 1, characterized in that, Acquire multi-view polarization data, including: A polarization acquisition system is constructed, comprising a first camera, a second camera, and a third camera; wherein the first camera and the third camera are respectively equipped with polarizers with preset polarization directions, and the second camera is located between the first camera and the third camera to acquire unpolarized images as a reference viewpoint; The polarization acquisition system controls each camera to synchronously acquire data of the same target scene, thereby obtaining multi-view polarization data from different spatial locations.
3. The method according to claim 1, characterized in that, Performing cross-viewpoint alignment on the multi-viewpoint polarization data to obtain a polarization image aligned with a reference viewpoint includes: The multi-view polarization data is processed based on a stereo matching algorithm to estimate the disparity information of the polarization image of the non-reference viewpoint relative to the reference viewpoint. Based on the parallax information, a reprojection transformation is performed on the polarization image of the non-reference viewpoint to obtain a polarization image aligned with the reference viewpoint.
4. The method according to claim 1, characterized in that, Constructing physical prior information includes: The polarization parameters are calculated based on the aligned polarization image to obtain the initial polarization angle and initial polarization degree, which are used as polarization priors. The scene depth is calculated based on the disparity information of the multi-view polarization data, and the surface normal information is derived from the scene depth as a geometric prior. The physical prior information is obtained based on the polarization prior and the geometric prior.
5. The method according to claim 1, characterized in that, The reference viewpoint image, the aligned polarization image, and the aforementioned physical prior information are input into the feature encoding module for multi-source information joint modeling to generate a unified feature representation, including: The reference viewpoint image, the aligned polarization image, and the physical prior information are input into the feature encoding module to extract the corresponding texture features, geometric features, and polarization features; the physical prior information includes geometric priors and polarization priors. The texture features, geometric features, and polarization features are integrated through a cross-modal fusion mechanism. With the polarization features as the dominant feature, the texture features and geometric features are adaptively weighted using an attention mechanism to generate the unified feature representation.
6. The method according to claim 1, characterized in that, A pixel-level confidence map is generated based on the confidence prediction module, and confidence-guided feature optimization is performed on the unified feature representation based on the pixel-level confidence map, including: The confidence prediction module evaluates the confidence of the initial polarization estimate and generates a confidence map that reflects the credibility of each pixel region. Based on the pixel-level confidence map, the unified feature representation is optimized with confidence guidance to suppress the propagation of erroneous information in low-confidence regions while enhancing the feature contribution from reliable regions.
7. The method according to claim 1, characterized in that, The optimized unified feature representation is input into the decoding module, and the output is the polarization parameters from the target's perspective, including: The optimized unified feature representation is input into the decoding module, and the polarization degree and polarization angle under the target view are output. Among them, the decomposition representation method is used to model the polarization angle.
8. A polarization imaging system, characterized in that, include: A multi-view acquisition module is used to acquire multi-view polarization data and perform cross-view alignment on the multi-view polarization data to obtain a polarization image aligned with a reference view. The multimodal fusion module is used to construct physical prior information. It inputs the reference view image, the aligned polarization image, and the physical prior information into the feature encoding module to perform joint modeling of multi-source information and generate a unified feature representation. The feature optimization module is used to generate a pixel-level confidence map based on the confidence prediction module, and to perform confidence-guided feature optimization on the unified feature representation based on the pixel-level confidence map. The polarization reconstruction module is used to input the optimized unified feature representation into the decoding module, output polarization parameters under the target viewpoint, and generate a polarization image under the target viewpoint based on the polarization parameters.
9. The system according to claim 8, characterized in that, include: The physical prior construction unit is used to analyze and calculate polarization parameters based on the aligned polarization image to obtain the initial polarization angle and initial polarization degree, which serve as polarization priors. The scene depth is calculated based on the disparity information of the multi-view polarization data, and the surface normal information is derived from the scene depth as a geometric prior. The physical prior information is obtained based on the polarization prior and the geometric prior.
10. An electronic device, characterized in that, It includes a processor and a memory storing program instructions, the processor being configured to execute the polarization imaging method based on a triple-camera system as described in any one of claims 1 to 7 when executing the program instructions.
