Mueller matrix determination method, apparatus, device, storage medium, and program product

Abstract

This application provides a method, apparatus, device, storage medium, and program product for determining a Mueller matrix. The method includes: determining an initial Mueller matrix of a target object based on initial physical parameters; the initial physical parameters are obtained from a polarization image of the target object; determining the Stokes vector of the target object based on the polarization image; simplifying the initial Mueller matrix based on the Stokes vector to obtain a first Mueller matrix of the target object; the dimension of the first Mueller matrix is ​​smaller than the dimension of the initial Mueller matrix; using the first Mueller matrix as a physical constraint of the initial physical parameters to determine the target physical parameters of the target object; and determining the target Mueller matrix of the target object based on the target physical parameters. This application can quickly and accurately obtain the Mueller matrix of an object with fewer polarization images, improving the efficiency of Mueller matrix capture.

Description

Technical Field

[0001] This application relates to the field of polarization imaging technology, specifically to a method, apparatus, device, storage medium, and program product for determining the Mueller matrix. Background Technology

[0002] The Mueller matrix provides a feasible path for analyzing the intrinsic properties of an object, such as its optical and structural characteristics. However, obtaining the Mueller matrix in practical applications is quite difficult: although four Stokes parameters can describe the polarization state of light, 16 elements are needed to characterize the change in polarization state after passing through the object. Theoretically, at least four pairs of independent incident and outgoing vectors are required to solve for a Mueller matrix. Traditional Stokes vector capture relies on acquiring intensity images in four polarization directions. For the Mueller matrix, a complete polarization system, including a polarizer and analyzer, is required, involving switching angles and acquiring a large number of images. Even with a single-image focal plane polarization imaging system, only four polarization images (0, 45, 90, and 135) can be acquired instantaneously. Capturing the Mueller matrix still requires changing the polarization state of the incident light at least four times, resulting in low overall efficiency. Summary of the Invention

[0003] At least one embodiment of this application provides a method, apparatus, device, storage medium, and program product for determining a Mueller matrix, which addresses the problem of low capture efficiency of Mueller matrices in the prior art.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a method for determining the Mueller matrix, including:

[0006] The initial Mueller matrix of the target object is determined based on the initial physical parameters; the initial physical parameters are obtained from the polarization image of the target object, which includes images of the target object corresponding to different polarization directions.

[0007] Based on the polarization image, determine the Stokes vector of the target object;

[0008] Based on the Stokes vector, the initial Mueller matrix is ​​simplified to obtain the first Mueller matrix of the target object; the dimension of the first Mueller matrix is ​​smaller than the dimension of the initial Mueller matrix.

[0009] The first Mueller matrix is ​​used as a physical constraint for the initial physical parameters to determine the target physical parameters of the target object;

[0010] Based on the target physical parameters, determine the target Mueller matrix of the target object.

[0011] Optionally, based on the Stokes vector, the initial Mueller matrix is ​​simplified through the network's reflection and channel separation structure to obtain the first Mueller matrix of the target object, including:

[0012] Based on the Stokes vector, the color channels, specular reflection parameters, and diffuse reflection parameters of the Mueller matrix are simplified through the reflection and channel separation structure of the polarization Mueller matrix network to obtain the first Mueller matrix.

[0013] Optionally, based on the initial physical parameters, the initial Mueller matrix of the target object is determined, including:

[0014] The initial physical parameters are used as input to the bidirectional reflection distribution function model to obtain the initial Mueller matrix.

[0015] Optionally, the initial physical parameters may include at least one of the following: initial roughness, initial albedo, and initial incident light direction.

[0016] Optionally, the polarization Mueller matrix network includes: an encoder and a decoder;

[0017] The encoder includes: a first channel for compressing the target physical parameters and a second channel for compressing the initial Mueller matrix;

[0018] The decoder includes a spatial adaptive discrimination module and a jump connection.

[0019] Optionally, the polarization direction of the polarization image includes:

[0020] Based on one or more of the following: 0 degrees, 45 degrees, 90 degrees, and 135 degrees, using the same polarizer.

[0021] Secondly, embodiments of this application provide a Mueller matrix determination apparatus, comprising:

[0022] The first determining module is used to determine the initial Mueller matrix of the target object based on the initial physical parameters; the initial physical parameters are obtained based on the polarization image of the target object, and the polarization image includes images of the target object corresponding to different polarization directions;

[0023] The second determining module is used to determine the Stokes vector of the target object based on the polarization image;

[0024] A simplification module is used to simplify the initial Mueller matrix based on the Stokes vector to obtain a first Mueller matrix of the target object; the dimension of the first Mueller matrix is ​​smaller than the dimension of the initial Mueller matrix.

[0025] The third determining module is used to determine the target physical parameters of the target object by using the first Mueller matrix as a physical constraint on the initial physical parameters.

[0026] The fourth determining module is used to determine the target Mueller matrix of the target object based on the target physical parameters.

[0027] Thirdly, embodiments of this application provide a Mueller matrix determination device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method described above.

[0028] Fourthly, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the method described above.

[0029] Compared with existing technologies, the Mueller matrix determination method, apparatus, device, storage medium, and program product provided in this application simplify the initial Mueller matrix corresponding to the polarization image using the Stokes vector of the polarization image, thereby determining the first Mueller matrix of the target object. Then, by using the first Mueller matrix as a physical constraint for the initial physical parameters, the target physical parameters of the target object are determined, thus determining the Mueller matrix of the target object. The solution in this application can quickly and accurately obtain the Mueller matrix of the target object with fewer polarization images, improving the efficiency of Mueller matrix capture. Attached Figure Description

[0030] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0031] Figure 1 This is a schematic diagram illustrating the steps of the Mueller matrix determination method according to an embodiment of this application;

[0032] Figure 2 This is a schematic diagram of the module of the Mueller matrix determination device according to an embodiment of this application;

[0033] Figure 3 This is a schematic diagram of the structure of the Mueller matrix determination device according to an embodiment of this application. Detailed Implementation

[0034] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, without limiting the number of objects; for example, the first object can be one or more. Furthermore, "or" in this application indicates at least one of the connected objects. For example, "A or B" covers three scenarios: Scenario 1: including A but not B; Scenario 2: including B but not A; Scenario 3: including both A and B. The character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0035] The term "instruction" in this application can be either a direct instruction (or explicit instruction) or an indirect instruction (or implicit instruction). A direct instruction can be understood as one in which the sender explicitly informs the receiver of specific information, the operation to be performed, or the requested result, etc.; an indirect instruction can be understood as one in which the receiver determines the corresponding information based on the instruction sent by the sender, or makes a judgment and determines the operation to be performed or the requested result, etc., based on the judgment result.

[0036] To enable those skilled in the art to better understand the embodiments of this application, the following description is provided first:

[0037] Polarization, as an intrinsic property of light, contains rich information independent of intensity and phase, and has demonstrated unique value in optical measurement, computational imaging, and computer vision. In visual research, a common paradigm involves illuminating an object's surface with polarized light and capturing the changes in polarization state after reflection or scattering. Changes in the incident and emitted polarization states can reveal additional physical information about the target object.

[0038] The Mueller matrix, as a tool for comprehensively describing the polarization state changes between incident and outgoing light, carries richer additional information about the object compared to traditional imaging methods. It can directly interpret not only explicit polarization parameters such as retardation, extinction, and depolarization, but also coupled deeper information such as the object's material and structure, which is difficult to extract directly. Therefore, Mueller matrix imaging has broad application potential in fields such as biology and medicine, for example, in cell diagnosis and identification, and disease detection. However, despite its promising application prospects, the relatively complex and difficult acquisition method of Mueller matrix imaging has, to some extent, limited the further development of this technology.

[0039] In polarization imaging, traditional polarization optical elements rely on the material's intrinsic properties to modulate the polarization state of light, making it difficult to simultaneously detect multiple polarization information in simple imaging systems. Although micro-patterning and liquid crystal technologies can achieve spatial polarization control, they are still limited by low spatial resolution and system complexity. Due to the inherent limitations of the aforementioned polarization modulation techniques, current mainstream methods for obtaining the Mueller matrix are still mainly based on time-division measurement. This involves controlling polarized light to illuminate the sample in a specific sequence and simultaneously acquiring the polarization response of reflected / transmitted light, finally reconstructing the complete Mueller matrix through calculation. This multi-stage measurement method requires acquiring a large number of image sequences, is time-consuming, and the imaging system contains multiple optical elements, resulting in a complex structure.

[0040] To overcome the limitations of time-division measurement, researchers have made improvements from both hardware and algorithmic perspectives. On the hardware side, they have introduced focal plane segmentation sensors, employed structured illumination techniques, or used hyperplane imaging to optimize the imaging performance and applicability of the Mueller matrix. At the algorithmic level, some researchers have attempted to introduce deep learning models to assist in solving the Mueller matrix, but these efforts have been limited to matrix parameter estimation and have failed to achieve the direct acquisition of the complete Mueller matrix.

[0041] Looking at these technological approaches, their core can be summarized as data-driven direct measurement. The polarization bidirectional reflectance distribution function (pBRDF), originating from computer graphics, offers a new solution from a model-driven theoretical framework. Its core value lies in quantitatively describing the complete process of the interaction between polarized light and the object surface at the macroscopic physical level. Although early models were based on idealized assumptions, a parameter-complete pBRDF model can directly construct its direct mapping from object properties to the linear Mueller matrix, providing theoretical support for achieving direct and rapid imaging of the Mueller matrix.

[0042] Generally, any polarization state of light can be completely described by a set of 4×1 Stokes vectors. Therefore, incident light and outgoing light each correspond to a set of Stokes vectors, and the transformation between them can be uniformly represented using a 4×4 Mueller matrix. The introduction of the Mueller matrix quantifies the change process of light polarization state and has a more crucial characteristic: its elements are determined only by the physical properties of the reflecting object itself, and are independent of the incident polarization state. Therefore, by extracting and analyzing information from the Mueller matrix, we can accurately deduce the optical parameters, composition, and structure of an object.

[0043] The Mueller matrix is ​​difficult to interpret because the elements of a 4×4 Mueller matrix are not independent but coupled with polarization properties such as biaxial attenuation, phase delay, and depolarization properties, as well as multiple pieces of information about the target object itself, such as its geometry and material. Without a prior model or a targeted decomposition algorithm, it is difficult to extract the required physical information from the matrix elements, resulting in a problem of abundant data but a lack of effective information.

[0044] The polarization-based bidirectional reflectance distribution function (pBRDF) model provides modeling assistance for solving these two problems. This model, based on the fundamental processes of light rays on an object's surface, is constructed in matrix form. Given specific parameters, it can fully describe the polarization state change of polarized light rays from incident to outgoing, directly obtaining the Mueller matrix. Many scholars have improved this model, enhancing its accuracy and applicability to various optical interactions. However, the modeling representation of the Mueller matrix highly depends on the precise selection of parameters such as the normal vector, incident / outgoing directions, and reflection characteristics. These physical quantities are often difficult to obtain directly in practice, becoming the primary bottleneck. More importantly, although existing models theoretically follow physical laws, their derivations are usually based on idealized assumptions. The complexity of real-world scenarios, such as surface microstructure and material inhomogeneity, far exceeds the scope that models can characterize, leading to discrepancies between theoretical predictions and actual observations. Therefore, there is an urgent need for a tool that can bypass explicit parameter estimation and directly learn complex mapping relationships from data; neural networks, with their powerful nonlinear fitting and representation capabilities, offer a solution.

[0045] In polarization-optics-driven visual prediction tasks, neural networks have rapidly become a mainstream research area, achieving excellent results in shape measurements such as normal vector prediction and depth prediction; reflection estimation such as albedo, roughness prediction, and reflection components; and inverse rendering combining both. With the support of neural networks and sufficient training data, multi-dimensional physical parameter information of an object can be obtained in one go with only sparse polarization input. Furthermore, leveraging the superior implicit representation and global optimization capabilities of neural networks, the output of traditional polarization reflection models is treated merely as a "physical prior." The network corrects residuals and compensates for non-ideal factors, thereby significantly improving the accuracy and robustness of the Mueller matrix without the need for additional complex data collection.

[0046] Since the polarization reflection model was first proposed, researchers have continuously expanded its reflection components, incorporating physical processes such as diffuse reflection, single scattering, and subsurface scattering, thus broadening the model's applicability. The polarization model proposed by Baek et al. is considered a fundamental framework in recent years. Under the dual components of specular and diffuse reflection, it obtains the initial normal vector through structured light projection, assigns initial values ​​to other parameters, and then undergoes complex optimization under the assumption of a coaxial light source and camera to finally reconstruct the complete pBRDF. Hwang et al. followed the same approach, using a portable device and computer vision methods to obtain the initial normal vector, still requiring nearly 10 hours of iterative optimization without actively controlling the incident light. Ichikawa et al. used a photometric stereo method to obtain the initial normal vector with a known light source direction. Kondo et al. further relaxed the coaxial constraint, employing a precise acquisition system and rotating polarizers, requiring nearly 100,000 images for each material to complete the polarization reflection feature fitting. Other studies also require extensive image acquisition and optimization. Besides directly fitting a complete polarization model, another research focus is on "indirectly supplementing appearance or reflection parameters using polarization as a clue." Ghoshet et al. calculated the reflection properties of an object using circularly polarized spherical illumination under the generated required light conditions; simultaneously, by combining shadow and polarization information, they achieved reflection parameter estimation and image rendering while reconstructing the shape. Deschaintre et al. first used flash lighting combined with deep learning to achieve shape and appearance acquisition, and subsequent studies have followed this approach, using neural networks to extract object polarization information to predict reflection parameters. [Authors' names omitted] introduced the radiation neural field method into the polarization domain, capturing multi-view polarized images to achieve high-precision rendering and reconstruction of objects. Compared to the large sampling and long optimization time of traditional methods, once the neural network is trained, it can instantly output a complete polarization model and Mueller matrix in unconstrained scenes with a single set of polarized images as input during the inference stage, truly balancing accuracy, speed, and ease of use.

[0047] The Stokes vector is typically used to represent the polarization pattern of light. It generally has four elements. The first three elements represent the intensity of the unpolarized light s, the linear polarization parameter, and the fourth element represents the circular polarization parameter. In this application, the influence of circular polarization is ignored, and only linear polarization is considered. Based on this, the polarization information of the light ray can be represented by this three-dimensional vector, including Dolp and Aop:

[0048] ;

[0049] Generally, the Stokes vector of light cannot be directly measured. Instead, it is calculated by changing the angle of the linear polarizer in front of the acquisition device. Existing polarization cameras can simultaneously acquire polarized images in four directions in a single shot, thus directly calculating the Stokes vector. The formula is as follows:

[0050] ;

[0051] Correspondingly, the process of polarization state change of light can be represented as:

[0052] ;

[0053] in, matrix The Mueller matrix was used to realize the incident polarization of light. to outgoing polarized light The change in polarization state between them. Generally, a complete Mueller matrix has 16 elements, so at least 4 pairs of incident and outgoing Stokes vectors are needed to solve the Mueller matrix. Considering that circular polarization is ignored, the Mueller matrix is ​​reduced to 9 elements. Therefore, solving the matrix with 16 equations can guarantee accuracy.

[0054] For polarized reflection models, they are generally divided into specular reflection and diffuse reflection components based on the type of reflected light. Similarly, based on the composition of the reflected light, they can be divided into intensity and polarization components, as shown below:

[0055] ;

[0056] in, and As an intensity parameter, it reflects the change in intensity of light after reflection; and As a polarization parameter, it is composed of the Mueller matrix and reflects the change in the polarization state of light after reflection. Based on the Baek pBRDF model, the specular reflection part is shown below:

[0057] ;

[0058] The intensity component is based on the Cook-Torrance BRDF model, including... For specular coefficient, and These are the GGX distribution function and the Smith function, respectively, which involve the roughness of the object. , It is the normal vector. For the polarized portion, the direction of the outgoing light ray is... For the rotating Mueller matrix, This is the Fresnel reflection matrix describing the pure bidirectional attenuation effect of polarized light. The diffuse reflection part is shown below:

[0059] ;

[0060] The strength component includes diffuse albedo and the direction of incident light For the polarization part, it includes the rotating Mueller matrix. Fresnel transmission matrix and depolarizer Combining the parameters required by the two parts, we can summarize the following: Following the assumptions of previous scholars, this paper further reduces the number of required parameters, keeping the specular reflection coefficient a constant of 1 and the refractive index a constant of 1.5. Thus, based on the provided parameters, a physically constrained Mueller matrix fitting of the object can be achieved using the pBRDF model.

[0061] As described in the background section, the prior art has low capture efficiency for the Mueller matrix. To solve the above problems, this application provides a Mueller matrix determination method, apparatus, device, storage medium, and program product, which can reduce or avoid the occurrence of the above situations, improve the capture efficiency of the Mueller matrix, and enhance the user experience.

[0062] This application provides a method and apparatus for determining the Mueller matrix. The method and apparatus are based on the same concept, and since the principles underlying the problem are similar, their implementations can be referred to interchangeably; repeated details will not be repeated.

[0063] like Figure 1 As shown in the figure, this application provides a method for determining the Mueller matrix, including the following steps:

[0064] Step 101: Determine the initial Mueller matrix of the target object based on the initial physical parameters; the initial physical parameters are obtained from the polarization images of the target object, and the polarization images include images of the target object corresponding to different polarization directions;

[0065] Step 102: Determine the Stokes vector of the target object based on the polarization image;

[0066] Step 103: Based on the Stokes vector, simplify the initial Mueller matrix to obtain the first Mueller matrix of the target object; the dimension of the first Mueller matrix is ​​smaller than the dimension of the initial Mueller matrix;

[0067] Step 104: Use the first Mueller matrix as a physical constraint for the initial physical parameters to determine the target physical parameters of the target object;

[0068] Step 105: Determine the target Mueller matrix of the target object based on the target physical parameters.

[0069] Optionally, the polarization image is an image obtained by taking a picture with a polarization camera, showing four different polarization states / polarization directions.

[0070] It should be noted that the polarization direction is changed by altering the polarization direction of the polarizer in the polarization camera.

[0071] The Mueller matrix determination method of this application simplifies the initial Mueller matrix corresponding to the polarization image using the Stokes vector of the polarization image, thereby determining the first Mueller matrix of the target object. Then, by using the first Mueller matrix as a physical constraint for the initial physical parameters, the target physical parameters of the target object are determined, thus determining the Mueller matrix of the target object. The solution of this application embodiment can quickly and accurately obtain the Mueller matrix of the target object with fewer polarization images, improving the efficiency of Mueller matrix capture.

[0072] Optionally, the initial parameters of the target object are estimated by performing initialization parameter estimation on the obtained polarization image of the target object, including:

[0073] The polarization image and mask are used as input to the Polarization Initialization Parameters Net (PIPNet). The target object is estimated through the Polarization Initialization Parameters Net to obtain the initial physical parameters of the target object.

[0074] The initial physical parameters include at least one of the following: initial normal vector, initial roughness, initial albedo, and initial incident light direction.

[0075] Optionally, the polarization direction of the polarization image includes:

[0076] Based on one or more of the following: 0 degrees, 45 degrees, 90 degrees, and 135 degrees, using the same polarizer.

[0077] In this embodiment of the application, the polarization direction of the polarization image includes:

[0078] Based on the same polarizer at 0 degrees, 45 degrees, 90 degrees and 135 degrees.

[0079] It should be noted that the initial input of the network is basically the same as that of common polarization-based networks, except that... Linear polarization images in four directions, denoted as Using these images, the Stokes vector of reflection from the object's surface can be directly calculated, and this process can be completed in a single exposure by a split-plane polarization camera. Furthermore, to obtain accurate Mueller matrix information for supervised training, while keeping the light source-object-camera pose unchanged, the polarization state of the incident light source is changed to four independent angles, and the corresponding outgoing polarization images are acquired for each. By solving a system of linear equations, the complete 3×3 Mueller matrix information of the object can be obtained. It is worth noting that during the network's inference phase, only one of the above four sets of outgoing polarization images is needed as input; it is not necessary to acquire all the images.

[0080] The goal of the initial parameter measurement module is to directly obtain the object parameter information for constructing the pBRDF model from four polarization images. The set of model parameters to be determined is as follows: ,in and It was fixed as a constant in the experiment; Depending on the camera model, if an orthographic model is used, then If a perspective model is used, the result is calculated based on the camera calibration. Therefore, the network only needs to output the following four items: the object's normal vector. diffuse reflectance albedo roughness and the direction of incident light ;

[0081] This module utilizes a U-Net-inspired neural network, whose encoder-decoder architecture has proven effective in various polarization vision tasks. The network outputs a normal vector and an incident direction vector, both based on camera coordinates, while the diffuse albedo reflects object color information. Therefore, the dimensions of the outputs for these three parts are all [missing information - likely a specific value]. All three components share the same encoder-decoder backbone, with each branch trained independently in parallel. For roughness, an average pooling layer and a fully connected layer are directly added to the end of the encoder to output the shared roughness value of the entire image at once. This part of the network can be represented as:

[0082] ;

[0083] In this context, a single dot on the parameter indicates that it has been optimized once, and two dots indicate that it has been optimized twice.

[0084] In the task of predicting normal vectors, using cosine similarity as the loss function is a common choice. For the prediction of other parameters, the L1 distance between the predicted and actual results is selected. Therefore, the overall loss function of the network is:

[0085] ;

[0086] in, The corresponding weights are used to balance the various loss functions.

[0087] Optionally, determining the initial Mueller matrix of the target object based on the initial physical parameters includes:

[0088] The initial physical parameters are used as input to the two-way reflection distribution function model pBRDF to obtain the initial Mueller matrix.

[0089] Optionally, based on the Stokes vector, the initial Mueller matrix is ​​simplified through the network's reflection and channel separation structure to obtain the first Mueller matrix of the target object, including:

[0090] Based on the Stokes vector, the color channels, specular reflection parameters, and diffuse reflection parameters of the Mueller matrix are simplified through the reflection and channel separation structure of the polarization Mueller matrix network to obtain the first Mueller matrix.

[0091] In this stage, a Mueller matrix refinement strategy using physical pBRDF constraints and residual optimization is employed; the initial parameters are... By directly inputting the data into the pBRDF model, it can be calculated and generated in one go. The initial Mueller matrix of the object; thanks to the model, specular reflection and diffuse reflection can be separated simultaneously in the calculation; however, it is difficult to achieve high accuracy by relying solely on the physical model: firstly, due to certain errors in the model itself, as the model makes assumptions about the polarized light reflection process; secondly, due to errors in the parameter estimation of the first stage; and finally, due to a complete 3-channel... The Mueller matrix has as many as 27 dimensions and combines both specular and diffuse reflection types. For these highly coupled reflection features, direct end-to-end optimization increases the difficulty of multi-channel feature extraction and fitting optimization.

[0092] The Mueller matrix based on physics can be written in the form of formula x, where For scalar strength weights, and matrix The core matrix that determines the polarization state of light is the intensity matrix. Experience shows that the error in the intensity weights is much smaller than the error in the polarization matrix, allowing the network to learn only the elements that need correction in the specular and diffuse reflection matrices. A complete Mueller matrix, expanded along the RGB three channels, has 27 dimensions, resulting in a large and redundant number of parameters. Therefore, we can make the following assumption: the polarization part of the Mueller matrix is ​​independent of the color channels, thus directly reducing the optimization space to 9 dimensions (single channel). Even so, 9 dimensions are still cumbersome, leading to low training efficiency. The optimization dimensionality can be further reduced by combining model-based specular and diffuse Mueller matrix relationships. For the diffuse reflection matrix, only the matrix... For the mirror reflection matrix, only optimization is needed. ,in For the intensity term, optimization is required in both parts of the network. This channel separation strategy compresses the dimensions to be optimized to less than 9, significantly improving optimization efficiency while fully preserving the physical meaning of the polarization model for specular and diffuse reflection. For specular reflection, the network can be represented as:

[0093] ;

[0094] in, This is the specular reflection polarization parameter of the Mueller matrix, with the subscript 5 indicating that the number of matrix channels is 5;

[0095] Similarly, for diffuse reflection, the network can be represented as:

[0096] ;

[0097] in, This is the diffuse reflection polarization parameter.

[0098] After obtaining the results, the matrix is ​​first expanded into a 9-channel matrix, and then merged into a complete Mueller matrix.

[0099] ;

[0100] For the strength term, we directly use the traditional model to give it. And direct prediction No further optimization is needed. The specular reflection portion does not contain color information; simply copying the 9 channels will yield a 27-dimensional image. The complete matrix is ​​used, while the diffuse part directly uses the optimized version. The network is expanded using the RGB three-channel architecture. The main body of the network still employs an encoder-decoder structure. The diffuse reflection network is constructed in residual mode, embedding a multi-channel attention mechanism module before convolutional encoding of the matrix. This module first performs global attention extraction on the multi-channel features before feature encoding to enhance the extraction effect. The specular reflection branch adds a reflection region detection and specialization module, first locating the bright specular reflection region and then enhancing the features of that region, allowing the network to maintain physical meaning while balancing detail and efficiency.

[0101] Regarding the choice of loss function, the L1 distance between the predicted Mueller matrix and the true Mueller matrix is ​​used:

[0102] .

[0103] Further, using the first Mueller matrix as a physical constraint on the initial physical parameters, the target physical parameters of the target object are determined, including:

[0104] The high-precision Mueller matrix output from the pre-processor network is used as an explicit geometric constraint to further refine the normal vector of the target object. Although PIPNet has obtained initial normal vector information using a single set of polarization images, the implicit polarization information represented in the polarization images is difficult to extract the prediction results directly with the desired accuracy. Inspired by the ideas of "diverse inputs" and "multi-dimensional constraints", the Mueller matrix optimized by the Polarization Mueller Matrix Network (PMMNet) is regarded as a strong prior with rich physical meaning, and is used together with the initial normal vector as input to construct the Polarization Normal Refinement Net (PNRNet) to achieve a second improvement in accuracy.

[0105] The entire network structure remains the same. The encoder receives two branches simultaneously—one with the initial normal vector predicted by PIPNet as input, and the other with the 27-channel Mueller matrix output by PMMNet as input. These are convolved to obtain high-dimensional normal vector features and polarization features, respectively. The two features are then fused according to their respective channel dimensions, while maintaining the original self-attention processing module, enabling the network to explicitly correlate local normals and global polarization constraints. The decoder uses SPADE upsampling blocks and skip connections to gradually restore high-resolution details. Finally, all pixel normal vectors are normalized, outputting normal vector parameters with a second-order precision improvement.

[0106] In general, the PNRNet network can be represented as:

[0107] ;

[0108] Since PIPNet already provides the initial normal vectors, the cosine similarity loss function is no longer used; instead, it directly uses... Distance measures the pixel-by-pixel deviation between the predicted and actual values:

[0109] .

[0110] In this embodiment, the entire network is implemented using PyTorch with the Adam optimizer and an NVIDIA RTX 4080 Super graphics card. The training process is divided into three stages, with weights stored independently. For PIPNet and PNRNet, the batch size is set to 12, and the training lasts for 40 epochs. The learning rate is set to 0.0002 for the first 20 epochs, and then linearly decreases for the last 20 epochs. In PMMNet, the batch size is 10, and the training lasts for 20 epochs, with the learning rate reduced in the last 10 epochs. A staged training approach is adopted, and the training results are output after each stage to begin the next stage. Due to the limited training scale, the original training set is divided into a training subset and a validation subset in an 8:2 ratio. Evaluation is performed on the validation set after each epoch, and the weight with the best performance on the validation set is taken as the best result for that stage of training.

[0111] When preparing virtual data samples for the first-stage network (PIPNet), the classic pBRDF model was selected and data was generated in batches on Mitsuba3. Over 200 3D object models of varying shapes were selected and uniformly scaled to appropriate sizes. Each model was rendered with 4-5 sets of scenes, resulting in a total of 1000 samples. For each sample, the camera pose was fixed on the model, with only the point light source orientation (zenith angle: 0-30°, azimuth angle: 0-360°) randomly varied, maintaining a consistent distance from the light source to the object's center. Two sets of RGB values ​​for surface albedo were randomly sampled and mapped onto the object's surface using a checkerboard texture. Roughness was randomly sampled between 0.01 and 0.5. Finally, the object was randomly rotated, and the model scaling was fine-tuned to ensure that the lighting covered the object's surface as fully as possible.

[0112] For the training of the second-stage (PMMNet) and third-stage (PNRNet) networks, the data rendered using the classic pBRDF model from the first stage contains errors compared to the polarization images and Mueller matrices of real objects, directly undermining the credibility of the optimization objective. To address this, a polarization dataset constructed using KAIST was employed: this dataset uses measured polarization images of real materials as its basis, calculates the Mueller matrix, and is well integrated into Mitsuba. This application selected 10 rendering materials and 100 models to generate 1000 datasets. To obtain the complete Mueller matrix for each dataset, a polarizer of a linear polarizer material was placed in front of the light source in the rendering scene and rotated sequentially. The corresponding outgoing polarization maps are rendered from these four angles, and then the Mueller matrix information is obtained by solving the simultaneous equations. Since the KAIST dataset does not directly provide the diffuse albedo of the material, the prediction results based on the classical model are used directly. For the remaining branches in the first stage, the two types of data are mixed and retrained to avoid performance degradation caused by data differences. After retraining PIPNet using the mixed data, PMMNet and PNRNet are trained using only the KAIST data.

[0113] The polarization and reflection characteristics of real-world objects are typically more complex and variable than those in simulation environments. Therefore, using real data for model training and validation is essential. While some object-level real-world polarization datasets exist, very few contain complete Mueller matrix information. The KAIST dataset mentioned earlier parameterizes spherical sample data, but rendering results based on this model inevitably introduce approximation errors. To address this, a directly usable dataset containing real-world object Mueller matrices was constructed. The dataset was acquired by fixing the object on a turntable and keeping the camera pose constant, systematically changing the zenith and azimuth angles of the illumination, and simultaneously controlling the turntable rotation to change the object's pose relative to the camera, thereby acquiring polarization information under multiple angles and illumination conditions. The dataset contains 13 different objects, each with four pose changes. For each object, 80 images were acquired under a single incident polarization angle, and 80 Mueller matrices were calculated after changing the four polarization angles. Furthermore, the object normal vector map corresponding to each sample was generated simultaneously, and the geometric parameters of the incident and outgoing beams were recorded.

[0114] To verify the effectiveness of the proposed network in predicting the Mueller matrix, existing mainstream polarization bidirectional reflectance distribution models were selected for comparison. Evaluation metrics were designed from three dimensions: matrix accuracy, functional verification, and physical information. These included the Peak Signal-to-Noise Ratio (PSNR) between the predicted and true Mueller matrices, directly reflecting the matrix reconstruction accuracy; the PSNR and Structural Similarity Index Measure (SSIM) between the reconstructed polarization images and the true images, indirectly verifying the matrix function; and finally, the overall error between the object's linear polarization degree and angle calculated based on the matrix and the true values, which, as core features of the linear polarization state, were used to evaluate the accuracy of the Mueller matrix in carrying physical information. A comprehensive quantitative comparison was conducted between the proposed method and existing pBRDF models on both virtual and real datasets, and the results are shown in Table 1.

[0115] Table 1

[0116]

[0117] Experimental results show that the proposed scheme achieves optimal performance across all metrics and datasets. In terms of metrics that best reflect the core task—the quality of Mueller matrix prediction—this method significantly outperforms other models on virtual data, especially real datasets, directly demonstrating its ability to reconstruct more accurate Mueller matrices. Furthermore, this method exhibits a clear advantage in polarization information reconstruction, particularly in polarization angle restoration, where its MAE value is significantly lower than other methods on both datasets. This is because the predicted polarization information is directly calculated based on the Mueller matrix, thus maintaining the physical consistency and accuracy of other polarization parameters while preserving matrix accuracy.

[0118] It is worth noting that the polarization reflection models compared all rely on accurate reflection information of the object during the optimization process, and assign initial parameters to the model for iterative processing. Their optimization objective is usually only the linear polarization degree of the object. This optimization strategy has certain limitations: while pursuing the optimization of a single index, the model struggles to consider the accuracy of other polarization information, resulting in acceptable accuracy in linear polarization degree, but significantly larger errors in linear polarization angle. In contrast, our method uses the complete Mueller matrix as the optimization objective, thus achieving better balance and accuracy in the overall polarization information restoration. It also lays a more comprehensive polarization information foundation for subsequent Mueller matrix information analysis and downstream application verification.

[0119] PNRNet uses polarization 3D reconstruction as a typical application scenario to further verify the effectiveness and accuracy reliability of the Mueller matrix information. To systematically verify the effectiveness of the generated Mueller matrix and its physical constraint value for improving normal vector accuracy, this application's embodiments design a progressive comparison scheme: First, the initial normal vector result of the first stage is used as a benchmark, representing the unoptimized starting point, aiming to directly demonstrate the fundamental role of refinement based on Mueller matrix physical constraints in reducing normal vector error; second, an optimization method based on a traditional physical model is introduced for comparison, directly substituting the model parameters predicted in the first stage into the generated Mueller matrix as the physical constraint for normal vector refinement; finally, the complete method of this application is presented, utilizing the upstream optimized high-precision Mueller matrix to achieve complementary advantages between the physical model and data-driven approaches. An angle error index is selected to directly quantify the deviation between the predicted and true normal vector values, while a vector accuracy index is introduced to measure the proportion of normal vectors with different accuracy requirements. Experimental results are shown in Table 2:

[0120] Table 2

[0121]

[0122] Experimental results show that, based on the initial normal vector baseline, the normal vector accuracy optimization driven by the traditional model is effective. It can significantly reduce the overall error of the normal vector and improve the accuracy. However, due to the dual limitations of the accuracy of the traditional polarization model itself and the accuracy of the prediction parameters, there is still room for improvement in the normal vector error. The method in this application achieves the best performance. The high-precision Mueller matrix optimized by PMMNet combined with the matrix physical constraint refinement strategy of PNRNet achieves the secondary accuracy optimization of the normal vector.

[0123] To further verify the physical accuracy of the generated Mueller matrix, pixel-level spatial correlation analysis was performed on the errors of the Mueller matrix used for training and the errors of the normal vectors obtained based on it, and tests were conducted on both virtual and real datasets. For more robust evaluation, a binning method with a bin number of 20 was used to calculate the Spearman rank correlation coefficient between the two. This method effectively reduces the impact of pixel-level noise by binning and averaging local regions, thus more reliably demonstrating the monotonic relationship between the two. Compared to the linear Pearson correlation coefficient, the Spearman coefficient is more suitable for analyzing complex error correlations in image data. The full channel and intensity channel of the Mueller matrix were selected and truncated respectively. Diattenuation channel ) and Depolarization channel ( The four channel combinations with relatively clear physical meanings were used for verification. The experimental results are shown in Table 3. On both datasets, the complete method of this application outperforms the benchmark method on all evaluation channels, especially in the matrix index across all channels. This demonstrates a clear statistical correlation between the accuracy of the predicted Mueller matrix and the accuracy of the normal vector. Model-based Mueller matrices, due to the dual limitations of model and initial parameter accuracy, have a weak correlation with object geometry; while the optimized method of this application not only improves matrix accuracy but also strengthens its physical representation ability, making it a more reliable physical prior.

[0124] Table 3

[0125]

[0126] The various methods of the embodiments of this application have been described above. Apparatus for implementing the above methods will now be provided.

[0127] like Figure 2 As shown in the figure, this application embodiment also provides a Mueller matrix determining device 200, including:

[0128] The first determining module 201 is used to determine the initial Mueller matrix of the target object based on the initial physical parameters; the initial physical parameters are obtained based on the polarization image of the target object, and the polarization image includes images of the target object corresponding to different polarization directions;

[0129] The second determining module 201 is used to determine the Stokes vector of the target object based on the polarization image;

[0130] The simplification module 203 is used to simplify the initial Mueller matrix according to the Stokes vector to obtain the first Mueller matrix of the target object; the dimension of the first Mueller matrix is ​​smaller than the dimension of the initial Mueller matrix.

[0131] The third determining module 204 is used to determine the target physical parameters of the target object by using the first Mueller matrix as a physical constraint on the initial physical parameters.

[0132] The fourth determining module 205 is used to determine the target Mueller matrix of the target object based on the target physical parameters.

[0133] The Mueller matrix determination device of this application embodiment simplifies the initial Mueller matrix corresponding to the polarization image using the Stokes vector of the polarization image, thereby determining the first Mueller matrix of the target object. Then, by using the first Mueller matrix as a physical constraint for the initial physical parameters, the target physical parameters of the target object are determined, thus determining the Mueller matrix of the target object. The solution of this application embodiment can quickly and accurately obtain the Mueller matrix of the target object with fewer polarization images, improving the Mueller matrix capture efficiency.

[0134] In another embodiment of this application, the Mueller matrix determines the nodes, such as... Figure 3 As shown, it includes a transceiver 310, a processor 300, a memory 320, and a program or instructions stored in the memory 320 and executable on the processor 300; when the processor 300 executes the program or instructions, it implements the various processes of the above-described Mueller matrix determination method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0135] The transceiver 310 is used to receive and send data under the control of the processor 300.

[0136] Among them, Figure 3 In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 300) and memory (memory 320). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 310 may be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. The processor 300 is responsible for managing the bus architecture and general processing, and the memory 320 may store data used by the processor 300 during operation.

[0137] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described Mueller matrix determination method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0138] This application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the above-described Mueller matrix determination method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0139] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0141] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for determining the Mueller matrix, characterized in that, include: Determine the initial Mueller matrix of the target object based on the initial physical parameters; The initial physical parameters are obtained based on the polarization image of the target object, which includes images of the target object corresponding to different polarization directions. Based on the polarization image, determine the Stokes vector of the target object; Based on the Stokes vector, the initial Mueller matrix is ​​simplified to obtain the first Mueller matrix of the target object; The dimension of the first Mueller matrix is ​​smaller than the dimension of the initial Mueller matrix; The first Mueller matrix is ​​used as a physical constraint for the initial physical parameters to determine the target physical parameters of the target object; Based on the target physical parameters, determine the target Mueller matrix of the target object.

2. The method according to claim 1, characterized in that, Based on the Stokes vector, the initial Mueller matrix is ​​simplified using the network's reflection and channel separation structure to obtain the first Mueller matrix of the target object, including: Based on the Stokes vector, the color channels, specular reflection parameters, and diffuse reflection parameters of the Mueller matrix are simplified through the reflection and channel separation structure of the polarization Mueller matrix network to obtain the first Mueller matrix.

3. The method according to claim 1, characterized in that, Based on the initial physical parameters, the initial Mueller matrix of the target object is determined, including: The initial physical parameters are used as input to the bidirectional reflection distribution function model to obtain the initial Mueller matrix.

4. The method according to claim 1, characterized in that, The initial physical parameters also include at least one of the following: initial roughness, initial albedo, and initial incident light direction.

5. The method according to claim 2, characterized in that, The polarization Mueller matrix network includes: an encoder and a decoder; The encoder includes: a first channel for compressing the target physical parameters and a second channel for compressing the initial Mueller matrix; The decoder includes a spatial adaptive discrimination module and a jump connection.

6. The method according to claim 1, characterized in that, The polarization direction of the polarization image includes: Based on one or more of the following: 0 degrees, 45 degrees, 90 degrees, and 135 degrees, using the same polarizer.

7. A Mueller matrix determining device, characterized in that, include: The first determining module is used to determine the initial Mueller matrix of the target object based on the initial physical parameters; The initial physical parameters are obtained based on the polarization image of the target object, which includes images of the target object corresponding to different polarization directions. The second determining module is used to determine the Stokes vector of the target object based on the polarization image; A simplification module is used to simplify the initial Mueller matrix based on the Stokes vector to obtain a first Mueller matrix of the target object; the dimension of the first Mueller matrix is ​​smaller than the dimension of the initial Mueller matrix. The third determining module is used to determine the target physical parameters of the target object by using the first Mueller matrix as a physical constraint on the initial physical parameters. The fourth determining module is used to determine the target Mueller matrix of the target object based on the target physical parameters.

8. A Mueller matrix determination device, characterized in that, include: Transceiver, processor, memory, and programs or instructions stored in the memory and executable on the processor; When the processor executes the program or instructions, it implements the steps of the Mueller matrix determination method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the Mueller matrix determination method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the Mueller matrix determination method as described in any one of claims 1 to 6.

Images