Polarization image generation method and device based on RGB image and diffusion prior

Abstract

The invention discloses a polarization image generation method and device based on an RGB image and diffusion prior, and the method comprises the steps: obtaining and normalizing the pixel value of the RGB image, and obtaining an original image parameter; the original image parameters are input into a pre-trained polarization image generation model, diffusion prior information corresponding to the RGB image is output, and the diffusion prior information comprises a linear polarization angle and a linear polarization degree of coding; the pre-trained polarization image generation model is obtained by finely adjusting the pre-training weight of the large diffusion model; according to the coded linear polarization angle and the linear polarization degree, calculating light passing intensities of different preset polarization angles corresponding to the original image; and carrying out visualization processing on the light passing intensities of different preset polarization angles to obtain a plurality of polarization images. Therefore, by adopting the embodiment of the invention, diffusion prior information can be output through the model, and the information can generate a polarization image with a vivid effect and physical accuracy only through a single RGB image without depending on three-dimensional assets.

Description

Technical Field

[0001] This application relates to the technical fields of computer vision and image processing, and particularly relates to a method and device for generating polarization images based on RGB images and diffusion priors. Background Art

[0002] Polarization images have a wide range of applications in the fields of computer vision and image processing, especially in enhancing vision tasks based on RGB images, such as recovering three-dimensional shapes from polarization information, image dehazing, and reflection removal. However, obtaining high-quality polarization images usually requires expensive polarization camera equipment, which is unaffordable for most users. In addition, diverse large-scale polarization image datasets with ground truth labels (such as depth, surface normals, and semantic segmentation masks) are scarce, and these datasets are crucial for improving the performance of corresponding learning-based computer vision tasks.

[0003] Currently, there are mainly two methods for existing polarization image simulators: one is based on the collected measured polarization bidirectional reflectance distribution function (pBRDF), and the other is based on a parameterized polarization reflection model. Simulators based on measured pBRDF are too limited in representing diverse real-world scenarios due to the expensive and time-consuming data collection process and are difficult to cover a wide range of application scenarios. On the other hand, simulators based on parameterized pBRDF models (such as Mitsuba) still have a gap in synthesizing the polarization characteristics of real-world scenarios due to the simplification of their reflection assumptions. Summary of the Invention

[0004] Embodiments of this application provide a method and device for generating polarization images based on RGB images and diffusion priors. To provide a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary part is not a general review, nor is it intended to identify key / important constituent elements or delineate the protection scope of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0005] In a first aspect, embodiments of this application provide a method for generating polarization images based on RGB images and diffusion priors, the method including:

[0006] Obtain and normalize the pixel values of the RGB image to obtain original image parameters;

[0007] Input the original image parameters into a pre-trained polarization image generation model to output diffusion prior information corresponding to the RGB image, where the diffusion prior information includes the encoded linear polarization angle and linear polarization degree; the pre-trained polarization image generation model is obtained by fine-tuning the pre-trained weights of a large diffusion model;

[0008] Calculate the light passing intensities corresponding to the original image for different preset polarization angles according to the encoded linear polarization angle and linear polarization degree;

[0009] Visualize the light passing intensities for different preset polarization angles to obtain multiple polarization images.

[0010] Optionally, the calculation formula for the light passing intensities for different preset polarization angles is:

[0011]

[0012] where I Θ is the light passing intensity for different preset polarization angles, I RGB represents the incident light intensity in front of the polarizer, P is the encoded linear polarization degree, which is used to represent the intensity ratio of linearly polarized light to the total incident light, and Φ is the encoded linear polarization angle, which is used to represent the oscillation direction of the polarization component.

[0013] Optionally, generate a pre-trained polarization image generation model according to the following steps, including:

[0014] Obtain historical polarization images, which are polarization images covering different shapes, materials, and reflection characteristics taken by a Lucid Triton RGB polarization camera;

[0015] Perform demosaicing on the historical polarization images using the bilinear interpolation algorithm;

[0016] Use the demosaiced polarization images and preset open-source polarization images as multiple polarization images to be analyzed;

[0017] Extract polarization attributes from each polarization image to be analyzed to obtain the true polarization image data of each polarization image to be analyzed. The polarization attribute extraction includes calculating and normalizing the linear polarization angle and linear polarization degree of each polarization image to be analyzed;

[0018] Obtain a pre-trained large diffusion model; among them, the pre-trained large diffusion model is Stable Diffusion v1.5;

[0019] Fine-tune the pre-trained weights of the large diffusion model according to the true polarization image data of each polarization image to be analyzed to learn the conditional distribution of generating polarization images from RGB images, and obtain a pre-trained polarization image generation model.

[0020] Optionally, extract polarization attributes from each polarization image to be analyzed to obtain true polarization image data, including:

[0021] Calculate the linear polarization angle and linear polarization degree of each polarization image to be analyzed;

[0022] Represent the linear polarization angle of each polarization image to be analyzed in the form of sine and cosine to preserve the periodic and continuous characteristics of each polarization image to be analyzed, and obtain the sine-cosine form encoding of the linear polarization angle of each polarization image to be analyzed;

[0023] Map the degree of linear polarization of each polarization image to be analyzed to the range of [-1, 1] to obtain the normalized result of the degree of linear polarization of each polarization image to be analyzed;

[0024] Concatenate the sine-cosine form encoding and the normalized result to obtain the polarization information encoding of each polarization image to be analyzed, which serves as the true polarization image data.

[0025] Optionally, the calculation formulas for the linear polarization angle and the degree of linear polarization of each polarization image to be analyzed are as follows:

[0026]

[0027] where Φ is the linear polarization angle of each polarization image to be analyzed, P is the degree of linear polarization of each polarization image to be analyzed, and I 0° 、I 45° 、I 90° 、I 135° are four polarization images under preset different polarization angles.

[0028] Optionally, the large diffusion model includes a pre-trained VAE encoder, a denoising U-Net network, and a pre-trained VAE decoder;

[0029] According to the true polarization image data of each polarization image to be analyzed, fine-tune the pre-trained weights of the preset diffusion model, including:

[0030] Visualize the polarization information encoding of each polarization image to be analyzed to obtain the encoded AoLP and DoLP maps;

[0031] The pre-trained VAE encoder converts the encoded AoLP and DoLP maps into latent codes;

[0032] Add preset noise to the latent codes to obtain the polarization attributes in the latent space;

[0033] The denoising U-Net network is based on the preset hierarchical guidance features for the noisy latent codes

[0034] The denoising U-Net network denoises the polarization attributes in the latent space based on the hierarchical guidance features to obtain the final target polarization attributes;

[0035] Fix the pre-trained weights of the pre-trained VAE encoder and the pre-trained VAE decoder, and use the target polarization attributes to fine-tune the pre-trained weights of the denoising U-Net network.

[0036] Optionally, the denoising U-Net network includes an objective function;

[0037] Fine-tuning the pre-trained weights of the denoising U-Net network using the target polarization attribute includes:

[0038] Input the target polarization attribute into the denoising U-Net network and output the loss value;

[0039] Generate a pre-trained polarization image generation model when the loss value reaches the minimum.

[0040] Optionally, the objective function is:

[0041]

[0042] Where, represents optimizing the parameter θ to minimize the objective function, θ is the parameter of the denoising U-Net network, is the expected value, which is used to measure the average performance of the model in all possible situations, x ∼ E vae represents a sample x sampled from the latent space generated by the pre-trained VAE encoder, t is the time step, which is used to control the noise level during the denoising process, ∈ ∼ N(0,1) represents the noise ∈ sampled from the standard normal distribution N(0,1), ∈ t is the actual noise at time step t, μ θ (z t ,t,c,E img (I RGB )) is the prediction output of the denoising U-Net network μ θ for denoising the latent representation z t at time step t, z t is the latent representation at time step t, c is the additional text prompt embedding generated by CLIP, E img (I RGB ) is the feature extracted from the RGB image.

[0043] Optionally, the large diffusion model includes a pre-trained VAE encoder, a pre-trained VAE decoder. The pre-trained VAE encoder includes an RGB image feature extractor, and the RGB image feature extractor includes multiple convolutional layers and SiLU activation functions;

[0044] Input the original image parameters into the pre-trained polarization image generation model and output the diffusion prior information corresponding to the RGB image, including:

[0045] The RGB image feature extractor extracts pixel features from the original image parameters;

[0046] Initialize the pixel features as standard Gaussian noise and denoise them in combination with an image encoder to obtain a denoised polarization attribute map;

[0047] Input the polarization attribute map into a pre-trained VAE encoder for feature convolution and output a convolution map;

[0048] Fuse the polarization attribute map and the convolution map to obtain a fused polarization attribute map;

[0049] Clarify the fused polarization attribute map and input it into a pre-trained VAE decoder to output the diffusion prior information corresponding to the RGB image.

[0050] In a second aspect, an embodiment of the present application provides a polarization image generation device based on an RGB image and a diffusion prior. The device includes:

[0051] An acquisition module for acquiring and normalizing the pixel values of an RGB image to obtain original image parameters;

[0052] An input module for inputting the original image parameters into a pre-trained polarization image generation model to output the diffusion prior information corresponding to the RGB image. The diffusion prior information includes the encoded linear polarization angle and linear polarization degree. The pre-trained polarization image generation model is obtained by fine-tuning the pre-trained weights of a large diffusion model;

[0053] An output module for calculating the light passing intensities of the original image corresponding to preset different polarization angles according to the encoded linear polarization angle and linear polarization degree;

[0054] A visualization module for performing visualization processing on the light passing intensities of preset different polarization angles to obtain a plurality of polarization images.

[0055] The technical solutions provided by the embodiments of the present application may include the following beneficial effects:

[0056] In the embodiments of the present application, on the one hand, high-quality polarization images can be generated by using ordinary RGB images, significantly reducing the equipment cost. On the other hand, based on the latent diffusion model, polarization images with high realism and physical accuracy can be generated, overcoming the deficiencies of existing physical model-based polarization image generation methods in terms of detail restoration and authenticity. On the other hand, the model can output diffusion prior information, which can generate realistic and physically accurate polarization images through a single RGB image without relying on 3D assets, simplifying the data preparation and processing process. Compared with relying on manually rotating a polarizer or using a dedicated polarization camera to collect data, this method can quickly generate large-scale, high-quality polarization image datasets, greatly improving the efficiency and scale of data generation and providing solid data support for the development of polarization vision applications.

[0057] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this application. Description of the Drawings

[0058] The drawings herein are incorporated into and constitute a part of this specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

[0059] Figure 1 It is a schematic flowchart of a method for generating a polarization image based on an RGB image and a diffusion prior provided by an embodiment of this application;

[0060] Figure 2 It is a schematic flowchart of a process for generating a polarization image based on an RGB image and a diffusion prior provided by this application;

[0061] Figure 3 It is a schematic flowchart of a model training method provided by an embodiment of this application;

[0062] Figure 4 It is a schematic diagram of a data processing process provided by an embodiment of this application;

[0063] Figure 5A It is a schematic flowchart of a model generation process provided by this application;

[0064] Figure 5B It is a schematic flowchart of a model generation process provided by this application;

[0065] Figure 6 It is a schematic structural diagram of a device for generating a polarization image based on an RGB image and a diffusion prior provided by this application;

[0066] Figure 7 It is a schematic structural diagram of an electronic device provided by an embodiment of this application. Detailed Embodiments

[0067] The following description and the drawings fully illustrate the specific embodiments of this application, enabling those skilled in the art to practice them.

[0068] It should be clear that the described embodiments are only a part of the embodiments of this application, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts belong to the scope of protection of this application.

[0069] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0070] In the description of the present application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific circumstances. In addition, in the description of the present application, unless otherwise specified, "a plurality of" means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the front and back associated objects.

[0071] The present application provides a method and device for generating polarization images based on RGB images and diffusion priors to solve the problems existing in related technical problems. In the embodiments of the present application, on the one hand, high-quality polarization images can be generated by using ordinary RGB images, significantly reducing the equipment cost. On the other hand, based on the latent diffusion model, polarization images with high realism and physical accuracy can be generated, overcoming the deficiencies of the existing polarization image generation methods based on physical models in terms of detail restoration and authenticity. On the other hand, the model can output diffusion prior information, and this information can generate polarization images with realistic effects and physical accuracy through only a single RGB image without relying on three-dimensional assets, simplifying the data preparation and processing process. Compared with relying on manually rotating the polarizer or using a dedicated polarization camera to collect data, this method can quickly generate a large-scale, high-quality polarization image dataset, greatly improving the efficiency and scale of data generation, and providing solid data support for the development of polarization vision applications. The following will be described in detail with exemplary embodiments.

[0072] The following will be combined with the attached Figure 1 - attached Figure 5B , to introduce in detail the method for generating polarization images based on RGB images and diffusion priors provided by the embodiments of the present application. This method can be implemented depending on a computer program and can run on a polarization image generation device based on the von Neumann architecture and based on RGB images and diffusion priors. This computer program can be integrated in an application or run as an independent tool class application.

[0073] Please refer to Figure 1, which is a schematic flowchart of a polarization image generation method based on RGB images and diffusion priors provided by an embodiment of the present application. As Figure 1 shown, the method of the embodiment of the present application may include the following steps:

[0074] S101, Obtain and normalize the pixel values of the RGB image to obtain the original image parameters;

[0075] Among them, the RGB image: refers to an image composed of three color channels of red (Red), green (Green), and blue (Blue), which is one of the most common image representation methods and is widely used in digital image processing and display. The pixel value refers to the color value of each pixel in the image, usually in the range of 0 to 255 (for 8-bit images), where 0 represents the darkest (no light) of the color channel and 255 represents the brightest (full light). The original image parameters refer to the pixel values of the RGB image after normalization processing, and these parameters will be used as the input of the neural network model.

[0076] In some embodiments, an RGB image with a size of 256x256 pixels, and the RGB values of each pixel point include "red channel value: 200, green channel value: 150, blue channel value: 100", read these pixel values from the image file. In Python, image processing libraries such as PIL or OpenCV can be used to implement this. Normalize these pixel values to the range of 0 to 1. This can be achieved by dividing the value of each channel by 255. The list of normalized pixel values is used as the original image parameters and serves as the input of the neural network model.

[0077] S102, Input the original image parameters into a pre-trained polarization image generation model, and output the diffusion prior information corresponding to the RGB image. The diffusion prior information includes the encoded linear polarization angle and linear polarization degree; the pre-trained polarization image generation model is obtained by fine-tuning the pre-trained weights of a large diffusion model;

[0078] Among them, the pre-trained polarization image generation model is a deep learning model trained to generate polarization images. This model is obtained by fine-tuning the pre-trained weights of a large diffusion model to learn how to generate polarization images from RGB images. The diffusion prior information is a type of information output by the model, which contains prior knowledge on how to generate polarization images from RGB images. This information includes the encoded angle of linear polarization (AoLP) and degree of linear polarization (DoLP). The angle of linear polarization refers to the angle between the vibration direction of polarized light and the reference direction. The encoded AoLP means that this angle information has been converted into a form that the model can understand. The degree of linear polarization refers to the purity of the polarization state of light waves, that is, the proportion of light waves whose vibration direction of polarized light is consistent with the reference direction. The encoded DoLP means that this information has been converted into a form that the model can understand.

[0079] Among them, the large diffusion model includes a pre-trained VAE encoder and a pre-trained VAE decoder. The pre-trained VAE encoder includes an RGB image feature extractor, and the RGB image feature extractor includes multiple convolutional layers and SiLU activation functions.

[0080] In some embodiments of the present application, the specific process of inputting the original image parameters into the pre-trained polarization image generation model and outputting the diffusion prior information corresponding to the RGB image includes: the RGB image feature extractor extracts pixel features from the original image parameters; initializes the pixel features as standard Gaussian noise and performs denoising in combination with the image encoder to obtain a denoised polarization attribute map; inputs the polarization attribute map into the pre-trained VAE encoder for feature convolution and outputs a convolution map; fuses the polarization attribute map and the convolution map to obtain a fused polarization attribute map; clarifies the fused polarization attribute map and then inputs it into the pre-trained VAE decoder to output the diffusion prior information corresponding to the RGB image. The diffusion prior information is

[0081] |cos2Φ; sin2Φ; P|, where cos2Φ and sin2Φ are the encoded angles of linear polarization, and P is the encoded degree of linear polarization.

[0082] S103, calculate the light passing intensities of the original image corresponding to preset different polarization angles according to the encoded angle of linear polarization and degree of linear polarization;

[0083] In some embodiments of the present application, after obtaining the encoded angle of linear polarization and degree of linear polarization, the light passing intensities of the original image corresponding to preset different polarization angles can be calculated according to the encoded angle of linear polarization and degree of linear polarization.

[0084] Among them, the calculation formula for the light passing intensities of preset different polarization angles is:

[0085]

[0086] Among them, I Θ is the intensity of light passing through with preset different polarization angles, I RGB represents the intensity of incident light in front of the polarizer, P is the encoded linear polarization degree, which is used to represent the intensity ratio of linearly polarized light to the total incident light, and Φ is the encoded linear polarization angle, which is used to represent the oscillation direction of the polarization component.

[0087] S104, perform visualization processing on the intensity of light passing through with preset different polarization angles to obtain a plurality of polarization images.

[0088] Among them, presetting different polarization angles means a series of polarization directions predefined when generating polarization images. These angles usually include 0°, 45°, 90°, 135°, etc., and are used to simulate the passing conditions of light in different polarization directions. The intensity of light passing through refers to the intensity of light waves passing through a certain medium (such as air, glass or water) at a specific polarization angle, and this intensity is affected by the polarization characteristics of the medium and the polarization state of the light. Visualization processing is the process of converting data or information into graphics or images. Visualization processing refers to converting the calculated intensity of light passing through into observable polarization images.

[0089] In some embodiments of the present application, by mapping the intensity values to the range of 0 - 255 and converting them into 8-bit images, a set of visualized polarization images can be obtained, and each image corresponds to a specific polarization angle.

[0090] For example Figure 2 as shown Figure 2 is a schematic flowchart of the polarization image generation process based on RGB images and diffusion priors provided by the present application. Input an RGB image, use a convolutional neural network to extract features from the RGB image to obtain a feature map, sample an initial latent code z0 from the standard normal distribution N(0, 1), the denoising U-Net network receives the latent code and the feature map and performs denoising processing to generate a denoised latent code. This process simulates the process of recovering useful information from noise. By iteratively applying the denoising U-Net network, the noise in the latent code is gradually reduced until a clear representation of polarization attributes is obtained. The denoised latent code is converted into diffusion prior information through a decoder, and polarization images at different angles can be obtained through calculations using the diffusion prior information. These polarization images correspond to different polarization angles (such as 0°, 45°, 90°, 135°).

[0091] In the embodiments of the present application, on the one hand, high-quality polarization images can be generated by using ordinary RGB images, significantly reducing the equipment cost. On the other hand, based on the latent diffusion model, polarization images with high realism and physical accuracy can be generated, overcoming the deficiencies of existing physical model-based polarization image generation methods in terms of detail restoration and authenticity. On the other hand, the model can output diffusion prior information, which can generate polarization images with realistic effects and physical accuracy from only a single RGB image without relying on 3D assets, simplifying the data preparation and processing process. Compared with relying on manually rotating polarizers or using dedicated polarization cameras to collect data, this method can quickly generate large-scale, high-quality polarization image datasets, greatly improving the efficiency and scale of data generation, and providing solid data support for the development of polarization vision applications.

[0092] Please refer to Figure 3 , which is a schematic flowchart of a method for training an injury level analysis model provided by the embodiments of the present application. As Figure 3 shown, the method of the embodiments of the present application may include the following steps:

[0093] S201, obtaining historical polarization images, which are polarization images captured by a Lucid Triton RGB polarization camera covering different shapes, materials, and reflection characteristics;

[0094] Among them, historical polarization images refer to polarization image data captured and saved in the past, which are used to train or optimize polarization image generation models. The Lucid Triton RGB polarization camera is a camera device capable of capturing polarization information and is produced by Lucid. This camera can record both RGB color information and polarization information simultaneously, making the captured images contain both color and polarization states. Different shapes, materials, and reflection characteristics mean that when capturing historical polarization images, the selected objects should have diversity, including different geometric shapes, different material properties (such as metals, plastics, wood, etc.), and different surface reflection characteristics (such as specular reflection, diffuse reflection, etc.).

[0095] In some embodiments of the present application, a Lucid Triton RGB polarization camera is utilized, which is capable of capturing polarization images of objects with different shapes, materials, and reflection characteristics. This camera employs Sony's IMX250MZR CMOS polarization sensor, which has four different oriented polarization filters (0°, 90°, 45°, and 135°) on every four pixels and can output the intensity and polarization angle of each image pixel. By using the above camera to capture polarization images of a series of objects with different characteristics, these images will serve as historical polarization image data. These images not only include the images captured using the Lucid Triton RGB polarization camera but may also include preset open-source polarization images as multiple polarization images to be analyzed for subsequent model training and optimization.

[0096] S202, perform demosaicing on the historical polarization images using the bilinear interpolation algorithm;

[0097] Among them, the bilinear interpolation algorithm is an image processing technique used for estimating pixel values in an image. In demosaicing, the bilinear interpolation uses the values of known pixel points to estimate the values of unknown pixel points, thereby generating a higher-quality image. The historical polarization images refer to the polarization images captured and saved previously. Demosaicing is an image processing technique used to recover a full-color image from a mosaic image. A mosaic image refers to an image in which each pixel on the image sensor records only one color (red, green, or blue), and demosaicing combines this incomplete information to generate a complete RGB image.

[0098] In some embodiments of the present application, a series of polarization images of objects are captured using a Lucid Triton RGB polarization camera. These images may contain objects with different shapes, materials, and reflection characteristics. Due to the sensor design of the camera, the captured images are mosaic images, that is, each pixel contains only one color information (for example, green). The bilinear interpolation algorithm is used to perform demosaicing on these mosaic images. The bilinear interpolation algorithm estimates the values of unknown pixel points based on the values of known pixel points, thereby generating the demosaiced images. These images now contain complete RGB color information and can be used for subsequent polarization image generation model training.

[0099] For example Figure 4As shown, the selectable camera is Triton 5.0MP (Sony IMX250MYR). In terms of object selection, we selected approximately 100 objects representing different materials. During the shooting process, we aimed to include at least 2-3 different objects in each frame to maximize the useful information captured. Our dataset contains a total of 1,146 polarized images. The final four polarized images (corresponding to polarization angles of 0°, 45°, 90°, and 135°) were obtained by demosaicking the captured raw images using the bilinear interpolation algorithm. The training set was supplemented with some open-source datasets to increase the diversity of the training set data distribution.

[0100] S203, using the demosaicked polarized images and the preset open-source polarized images as multiple polarized images to be analyzed;

[0101] S204, performing polarization attribute extraction on each polarized image to be analyzed to obtain the true polarization image data of each polarized image to be analyzed, where the polarization attribute extraction includes calculating and normalizing the linear polarization angle and linear polarization degree of each polarized image to be analyzed;

[0102] In some embodiments of the present application, the specific process of performing polarization attribute extraction on each polarized image to be analyzed to obtain the true polarization image data of each polarized image to be analyzed includes: calculating the linear polarization angle and linear polarization degree of each polarized image to be analyzed; representing the linear polarization angle of each polarized image to be analyzed in the form of sine and cosine to retain the periodic and continuous characteristics of each polarized image to be analyzed, obtaining the sine-cosine form encoding of the linear polarization angle of each polarized image to be analyzed; mapping the linear polarization degree of each polarized image to be analyzed to the range of [-1,1] to obtain the normalized result of the linear polarization degree of each polarized image to be analyzed; splicing the sine-cosine form encoding and the normalized result to obtain the polarization information encoding of each polarized image to be analyzed as the true polarization image data.

[0103] Specifically, the calculation formulas for the linear polarization angle and linear polarization degree of each polarized image to be analyzed are:

[0104]

[0105]

[0106] where Φ is the linear polarization angle of each polarized image to be analyzed, P is the linear polarization degree of each polarized image to be analyzed, and I 0° 、I 45° 、I 90° 、I 135° are four polarized images at preset different polarization angles.

[0107] For example, since the polarization attributes calculated from the image are damaged because of the lack of physical polarization constraints during the denoising process. To generate physically reasonable polarization attributes, it is proposed to estimate the corresponding AoLP and DoLP maps and simulate polarization images at arbitrary polarization angles. Considering the π periodicity of AoLP, AoLP is encoded in the form of sine (cos2Φ, sin2Φ), which is also a continuous representation and convenient for network learning. Consistent with the numerical ranges of the encoded AoLP and VAE, DoLP is normalized in the range of [-1, 1], and then the encoded AoLP is concatenated with DoLP as the diffusion output: |cos2Φ; sin2Φ; P|.

[0108] S205, obtain a pre-trained large diffusion model; wherein, the pre-trained large diffusion model is StableDiffusion v1.5;

[0109] S206, fine-tune the pre-trained weights of the large diffusion model according to the true polarization image data of each polarization image to be analyzed, so as to learn the conditional distribution of generating polarization images from RGB images, and obtain a pre-trained polarization image generation model.

[0110] Wherein, the large diffusion model includes a pre-trained VAE encoder, a denoising U-Net network, and a pre-trained VAE decoder.

[0111] In some embodiments of the present application, the specific process of fine-tuning the pre-trained weights of the preset diffusion model according to the true polarization image data of each polarization image to be analyzed includes: visualizing the polarization information encoding of each polarization image to be analyzed to obtain the encoded AoLP and DoLP maps; the pre-trained VAE encoder converts the encoded AoLP and DoLP maps into latent codes; adding preset noise to the latent codes to obtain polarization attributes in the latent space; the denoising U-Net network denoises the noisy latent codes based on the preset hierarchical guiding features, and the denoising U-Net network denoises the polarization attributes in the latent space based on the hierarchical guiding features to obtain the final target polarization attributes; fixing the pre-trained weights of the pre-trained VAE encoder and the pre-trained VAE decoder, and fine-tuning the pre-trained weights of the denoising U-Net network with the target polarization attributes.

[0112] Wherein, the denoising U-Net network contains an objective function.

[0113] In some embodiments of the present application, the specific process of fine-tuning the pre-trained weights of the denoising U-Net network with the target polarization attributes includes: inputting the target polarization attributes into the denoising U-Net network and outputting a loss value; generating a pre-trained polarization image generation model when the loss value reaches the minimum.

[0114] Specifically, the objective function is:

[0115]

[0116] where represents optimizing the parameter θ to minimize the objective function, and θ is the parameter of the denoising U-Net network. is the expected value, which is used to measure the average performance of the model in all possible situations, and x ∼ E vae represents a sample x sampled from the latent space generated by the pre-trained VAE encoder. t is the time step, which is used to control the noise level in the denoising process. ∈ ∼ N(0, 1) represents the noise ∈ sampled from the standard normal distribution N(0, 1). ∈ t is the actual noise at the time step t, μ θ (z t , t, c, E img (I RGB )) is the predicted output of the denoising U-Net network μ θ for denoising the latent representation z t at the time step t. z t is the latent representation at the time step t, c is the additional text prompt embedding generated by CLIP, and E img (I RGB ) are the features extracted from the RGB image.

[0117] For example Figure 5A and 5B as shown, Figure 5A and 5B are the schematic flow diagrams of a model generation process provided by this application. The input is a set of polarization images taken at different polarization angles (0°, 45°, 90°, 135°). The angle of linear polarization (AoLP) and degree of linear polarization (DoLP) information are extracted from the polarization images. The polarization angle is encoded using cosine and sine functions to preserve its periodic characteristics, obtaining cos2Φ and sin2Φ. The encoded polarization angle information is concatenated with the degree of linear polarization information to form the polarization information encoding [cos2Φ; sin2Φ; P]. The polarization information encoding is input into the pre-trained VAE encoder E vae to obtain the latent representation zt. Noise ∈t is added to the latent representation zt to simulate the diffusion process. The noise is sampled from a standard normal distribution N(0, 1). The latent representation ct with added noise is input into the denoising U-Net μ θMeanwhile, the feature Eimg of the RGB image is also input into the denoising U-Net network as conditional information. The denoising U-Net network learns to recover the original polarization information from the noise, generates the denoised latent representation, and uses a loss function to calculate the difference between the denoised latent representation and the original latent representation. Through optimization algorithms such as backpropagation and gradient descent, the parameters of the denoising U-Net network are updated to minimize the loss function.

[0118] Furthermore, the generated polarization image not only has a high-quality visual effect but also can effectively support downstream tasks such as polarization-based normal estimation and 3D reconstruction. By applying the generated polarization information to existing shape recovery algorithms, the accuracy of downstream tasks is improved.

[0119] Specifically, the 3D reconstruction quality evaluation method includes the following specific steps:

[0120] E1. Mean Angle Error (MAE) of the normal vector: Calculate the average angle error between the estimated normal vector and the true normal vector in the reconstruction model. The smaller the MAE value, the higher the reconstruction quality.

[0121] E2. Chamfer Distance (CD) of the object model: Measure the Chamfer distance between the reconstructed 3D model and the true model. The smaller the CD, the higher the accuracy of the model reconstruction.

[0122] In the embodiments of the present application, on the one hand, by using ordinary RGB images, high-quality polarization images can be generated, significantly reducing the equipment cost. On the other hand, based on the latent diffusion model, polarization images with high realism and physical accuracy can be generated, overcoming the deficiencies of existing physical model-based polarization image generation methods in terms of detail restoration and authenticity. On the other hand, through the model, diffusion prior information can be output. Without relying on 3D assets, this information can generate polarization images with realistic effects and physical accuracy through a single RGB image, simplifying the data preparation and processing process. Compared with relying on manually rotating polarizers or using special polarization cameras to collect data, this method can quickly generate large-scale, high-quality polarization image datasets, greatly improving the efficiency and scale of data generation, and providing solid data support for the development of polarization vision applications.

[0123] The following is an embodiment of the device of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the device embodiment of the present application, please refer to the method embodiment of the present application.

[0124] Please refer to Figure 6, which shows a schematic structural diagram of a polarization image generation device provided by an exemplary embodiment of the present application. The polarization image generation device based on RGB images and diffusion priors can be implemented as all or part of an electronic device through software, hardware, or a combination of both. The device 1 includes an acquisition module 10, an input module 20, an output module 30, and a visualization module 40.

[0125] The acquisition module 10 is configured to acquire and normalize the pixel values of the RGB image to obtain the original image parameters;

[0126] The input module 20 is configured to input the original image parameters into a pre-trained polarization image generation model and output the diffusion prior information corresponding to the RGB image. The diffusion prior information includes the encoded linear polarization angle and the linear polarization degree. The pre-trained polarization image generation model is obtained by fine-tuning the pre-trained weights of a large diffusion model;

[0127] The output module 30 is configured to calculate the light transmission intensities corresponding to the preset different polarization angles of the original image according to the encoded linear polarization angle and the linear polarization degree;

[0128] The visualization module 40 is configured to perform visualization processing on the light transmission intensities corresponding to the preset different polarization angles to obtain a plurality of polarization images.

[0129] It should be noted that when the above-mentioned polarization image generation device based on RGB images and diffusion priors executes the polarization image generation method based on RGB images and diffusion priors, only the above-mentioned division of each functional module is used for illustration. In practical applications, the above functions can be assigned to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the above-mentioned polarization image generation device based on RGB images and diffusion priors and the embodiment of the polarization image generation method based on RGB images and diffusion priors belong to the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.

[0130] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0131] In the embodiments of the present application, on the one hand, high-quality polarization images can be generated by using ordinary RGB images, significantly reducing the equipment cost. On the other hand, based on the latent diffusion model, polarization images with high realism and physical accuracy can be generated, overcoming the deficiencies of existing physical model-based polarization image generation methods in terms of detail restoration and authenticity. On the other hand, the model can output diffusion prior information, which can generate polarization images with realistic effects and physical accuracy through a single RGB image without relying on 3D assets, simplifying the data preparation and processing process. Compared with relying on manually rotating the polarizer or using a dedicated polarization camera to collect data, this method can quickly generate a large-scale, high-quality polarization image dataset, greatly improving the efficiency and scale of data generation, and providing solid data support for the development of polarization vision applications.

[0132] The present application also provides a computer-readable medium, on which program instructions are stored, and when the program instructions are executed by a processor, the polarization image generation method based on RGB images and diffusion priors provided by the above-mentioned various method embodiments is implemented.

[0133] The present application also provides a computer program product containing instructions, which when running on a computer, causes the computer to execute the polarization image generation method based on RGB images and diffusion priors of the above-mentioned various method embodiments.

[0134] Please refer to Figure 7 , which is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 7 shown, the electronic device 1000 may include: at least one processor 1001, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002.

[0135] Among them, the communication bus 1002 is used to realize the connection and communication between these components.

[0136] Among them, the user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may further include a standard wired interface and a wireless interface.

[0137] Among them, the network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0138] Among them, the processor 1001 may include one or more processing cores. The processor 1001 connects various parts within the entire electronic device 1000 through various interfaces and lines. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and by calling data stored in the memory 1005, it performs various functions of the electronic device 1000 and processes data. Optionally, the processor 1001 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 1001 may integrate a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 1001 and may be implemented separately by a single chip.

[0139] Among them, the memory 1005 may include random access memory (RAM) and may also include read-only memory. Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1005 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store the data involved in the above-mentioned various method embodiments. Optionally, the memory 1005 may also be at least one storage system located far from the aforementioned processor 1001. As Figure 7 shown, the memory 1005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for generating polarization images based on RGB images and diffusion priors.

[0140] In Figure 7In the illustrated electronic device 1000, the user interface 1003 is mainly used to provide an interface for the user to input and obtain the data input by the user; and the processor 1001 can be used to call the polarization image generation application program stored in the memory 1005 based on the RGB image and diffusion prior, and specifically perform the following operations:

[0141] Obtain and normalize the pixel values of the RGB image to obtain the original image parameters;

[0142] Input the original image parameters into a pre-trained polarization image generation model, and output the diffusion prior information corresponding to the RGB image. The diffusion prior information includes the encoded linear polarization angle and linear polarization degree; the pre-trained polarization image generation model is obtained by fine-tuning the pre-trained weights of a large diffusion model;

[0143] Calculate the light passing intensities corresponding to the preset different polarization angles of the original image according to the encoded linear polarization angle and linear polarization degree;

[0144] Perform visualization processing on the light passing intensities corresponding to the preset different polarization angles to obtain multiple polarization images.

[0145] In one embodiment, when the processor 1001 executes to generate the pre-trained polarization image generation model, it specifically performs the following operations:

[0146] Obtain historical polarization images, which are polarization images covering different shapes, materials, and reflection characteristics captured by a Lucid Triton RGB polarization camera;

[0147] Perform demosaicing processing on the historical polarization images using the bilinear interpolation algorithm;

[0148] Use the demosaiced polarization images and the preset open-source polarization images as multiple polarization images to be analyzed;

[0149] Extract the polarization attributes of each polarization image to be analyzed to obtain the true polarization image data of each polarization image to be analyzed. The polarization attribute extraction includes calculating and normalizing the linear polarization angle and linear polarization degree of each polarization image to be analyzed;

[0150] Obtain a pre-trained large diffusion model; wherein, the pre-trained large diffusion model is Stable Diffusion v1.5;

[0151] Fine-tune the pre-trained weights of the large diffusion model according to the true polarization image data of each polarization image to be analyzed to learn the conditional distribution of generating polarization images from RGB images, and obtain the pre-trained polarization image generation model.

[0152] In one embodiment, when the processor 1001 executes the extraction of polarization attributes for each polarization image to be analyzed to obtain real polarization image data, the following operations are specifically performed:

[0153] Calculate the linear polarization angle and degree of linear polarization of each polarization image to be analyzed;

[0154] Represent the linear polarization angle of each polarization image to be analyzed in the form of sine and cosine to preserve the periodic and continuous characteristics of each polarization image to be analyzed, and obtain the sine-cosine form encoding of the linear polarization angle of each polarization image to be analyzed;

[0155] Map the degree of linear polarization of each polarization image to be analyzed to the range of [-1, 1] to obtain the normalized result of the degree of linear polarization of each polarization image to be analyzed;

[0156] Concatenate the sine-cosine form encoding and the normalized result to obtain the polarization information encoding of each polarization image to be analyzed as the real polarization image data.

[0157] In one embodiment, when the processor 1001 executes the fine-tuning of the pre-trained weights of the preset diffusion model according to the real polarization image data of each polarization image to be analyzed, the following operations are specifically performed:

[0158] Visualize the polarization information encoding of each polarization image to be analyzed to obtain the encoded AoLP and DoLP maps;

[0159] The pre-trained VAE encoder converts the encoded AoLP and DoLP maps into latent codes;

[0160] Add a preset noise to the latent codes to obtain the polarization attributes in the latent space;

[0161] The denoising U-Net network is based on the preset hierarchical guiding features for the noisy latent codes

[0162] The denoising U-Net network denoises the polarization attributes in the latent space based on the hierarchical guiding features to obtain the final target polarization attributes;

[0163] Fix the pre-trained weights of the pre-trained VAE encoder and the pre-trained VAE decoder, and fine-tune the pre-trained weights of the denoising U-Net network using the target polarization attributes.

[0164] In one embodiment, when the processor 1001 executes the fine-tuning of the pre-trained weights of the denoising U-Net network using the target polarization attributes, the following operations are specifically performed:

[0165] Input the target polarization attributes into the denoising U-Net network and output the loss value;

[0166] Generate a pre-trained polarization image generation model when the loss value reaches the minimum.

[0167] In one embodiment, when the processor 1001 executes the operation of inputting the original image parameters into the pre-trained polarization image generation model and outputting the diffusion prior information corresponding to the RGB image, it specifically performs the following operations:

[0168] The RGB image feature extractor extracts pixel features from the original image parameters;

[0169] Initialize the pixel features as standard Gaussian noise and perform denoising in combination with the image encoder to obtain a denoised polarization attribute map;

[0170] Input the polarization attribute map into the pre-trained VAE encoder for feature convolution and output a convolution map;

[0171] Fuse the polarization attribute map with the convolution map to obtain a fused polarization attribute map;

[0172] Clarify the fused polarization attribute map and then input it into the pre-trained VAE decoder to output the diffusion prior information corresponding to the RGB image.

[0173] In the embodiments of the present application, on the one hand, by using ordinary RGB images, high-quality polarization images can be generated, significantly reducing the device cost. On the other hand, based on the latent diffusion model, polarization images with high realism and physical accuracy can be generated, overcoming the deficiencies of existing physical model-based polarization image generation methods in terms of detail restoration and authenticity. On the other hand, through the model, diffusion prior information can be output. Without relying on 3D assets, this information can generate realistic and physically accurate polarization images through a single RGB image, simplifying the data preparation and processing process. Compared with relying on manually rotating polarizers or using dedicated polarization cameras to collect data, this method can quickly generate large-scale, high-quality polarization image datasets, greatly improving the efficiency and scale of data generation and providing solid data support for the development of polarization vision applications.

[0174] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The program for generating polarization images based on RGB images and diffusion priors can be stored in a computer-readable storage medium. When this program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium of the program for generating polarization images based on RGB images and diffusion priors can be a magnetic disk, an optical disk, a read-only memory, or a random access memory, etc.

[0175] The above disclosure is only for the preferred embodiments of the present application. Of course, the scope of rights of the present application cannot be limited thereby. Therefore, equivalent changes made according to the claims of the present application still fall within the scope covered by the present application.

Claims

1. A method for generating polarization images based on RGB images and diffusion priors, characterized in that, The method includes: Obtain and normalize the pixel values of the RGB image to obtain the original image parameters; Input the original image parameters into a pre-trained polarization image generation model, and output the diffusion prior information corresponding to the RGB image, where the diffusion prior information includes the encoded linear polarization angle and linear polarization degree; the pre-trained polarization image generation model is obtained by fine-tuning the pre-trained weights of the large diffusion model; Calculate the light passing intensities corresponding to the original image at preset different polarization angles according to the encoded linear polarization angle and linear polarization degree; Perform visualization processing on the light passing intensities at preset different polarization angles to obtain multiple polarization images.

2. The method according to claim 1, characterized in that The calculation formula for the light passing intensities at preset different polarization angles is: Among them, I Θ is the intensity of light passing through with preset different polarization angles, I RGB represents the intensity of incident light in front of the polarizer, P is the encoded degree of linear polarization, which is used to represent the intensity ratio of linearly polarized light to the total incident light, and Φ is the encoded linear polarization angle, which is used to represent the oscillation direction of the polarization component.

3. The method according to claim 1, characterized in that, Generate the pre-trained polarization image generation model according to the following steps, including: Obtain historical polarization images, where the historical polarization images are polarization images covering different shapes, materials, and reflection characteristics captured using a Lucid Triton RGB polarization camera; Perform demosaicing on the historical polarization images using the bilinear interpolation algorithm; Use the demosaiced polarization images and preset open-source polarization images as multiple polarization images to be analyzed; Extract polarization attributes from each polarization image to be analyzed to obtain the true polarization image data of each polarization image to be analyzed, where the polarization attribute extraction includes calculating and normalizing the linear polarization angle and linear polarization degree of each polarization image to be analyzed; Obtain a pre-trained large diffusion model; where the pre-trained large diffusion model is Stable Diffusion v1.5; Fine-tune the pre-trained weights of the large diffusion model according to the true polarization image data of each polarization image to be analyzed to learn the conditional distribution of generating polarization images from RGB images, and obtain the pre-trained polarization image generation model.

4. The method according to claim 3, wherein The extracting polarization attributes from each polarization image to be analyzed to obtain the true polarization image data includes: Calculate the linear polarization angle and linear polarization degree of each polarization image to be analyzed; Express the linear polarization angle of each polarization image to be analyzed in the form of sine and cosine to retain the periodic and continuous characteristics of each polarization image to be analyzed, and obtain the sine-cosine form encoding of the linear polarization angle of each polarization image to be analyzed; Map the linear polarization degree of each polarization image to be analyzed to the range of [-1, 1] to obtain the normalized result of the linear polarization degree of each polarization image to be analyzed; Concatenate the sine-cosine form encoding and the normalized result to obtain the polarization information encoding of each polarization image to be analyzed as the true polarization image data.

5. The method according to claim 4, wherein The calculation formulas for the linear polarization angle and linear polarization degree of each polarization image to be analyzed are: where Φ is the linear polarization angle of each polarization image to be analyzed, P is the degree of linear polarization of each polarization image to be analyzed, and I 0° 、I 45° 、I 90° 、I 135° are four polarization images at preset different polarization angles.

6. The method according to claim 4, wherein The large diffusion model includes a pre-trained VAE encoder, a denoising U-Net network, and a pre-trained VAE decoder; The fine-tuning the pre-trained weights of the preset diffusion model according to the true polarization image data of each polarization image to be analyzed includes: Visualize the polarization information encoding of each polarization image to be analyzed to obtain encoded AoLP and DoLP maps; The pre-trained VAE encoder converts the encoded AoLP and DoLP maps into latent codes; Add a preset noise to the latent codes to obtain the polarization attributes in the latent space; The denoising U-Net network is based on the preset hierarchical guiding features for the noisy latent codes The denoising U-Net network denoises the polarization attributes in the latent space based on the hierarchical guiding features to obtain the final target polarization attributes; Fix the pre-trained weights of the pre-trained VAE encoder and the pre-trained VAE decoder, and fine-tune the pre-trained weights of the denoising U-Net network using the target polarization attributes.

7. The method according to claim 6, wherein The denoising U-Net network includes an objective function; The fine-tuning of the pre-trained weights of the denoising U-Net network using the target polarization attributes includes: Input the target polarization attributes into the denoising U-Net network and output a loss value; Generate a pre-trained polarization image generation model when the loss value reaches the minimum.

8. The method according to claim 7, characterized in that, The objective function is: Among them, represents optimizing the parameter θ to minimize the objective function, where θ is the parameter of the denoising U-Net network, is the expected value, which is used to measure the average performance of the model in all possible cases, x ∼ E vae represents a sample x sampled from the latent space generated by the pre-trained VAE encoder. t is the time step used to control the noise level during the denoising process. ∈ ∼ N(0,1) represents the noise ∈ sampled from the standard normal distribution N(0,1). ∈ t is the actual noise at the time step t. μ θ (z t ,t,c,E img (I RGB )) is the denoising U-Net network μ θ The predicted output for denoising the latent representation z t at the time step t, z t is the latent representation at the time step t, c is the additional text prompt embedding generated by CLIP, and E img (I RGB ) are the features extracted from the RGB image.

9. The method according to claim 1, characterized in that, The large diffusion model includes a pre-trained VAE encoder and a pre-trained VAE decoder. The pre-trained VAE encoder includes an RGB image feature extractor, and the RGB image feature extractor includes multiple convolutional layers and SiLU activation functions; The inputting the original image parameters into the pre-trained polarization image generation model and outputting the diffusion prior information corresponding to the RGB image includes: The RGB image feature extractor extracts pixel features from the original image parameters; Initialize the pixel features as standard Gaussian noise and perform denoising in combination with an image encoder to obtain a denoised polarization attribute map; Input the polarization attribute map into the pre-trained VAE encoder for feature convolution and output a convolution map; Fuse the polarization attribute map and the convolution map to obtain a fused polarization attribute map; Clarify the fused polarization attribute map and then input it into the pre-trained VAE decoder to output the diffusion prior information corresponding to the RGB image.

10. A polarization image generation device based on RGB images and diffusion priors, characterized in that, The device includes: An acquisition module for acquiring and normalizing the pixel values of an RGB image to obtain original image parameters; An input module for inputting the original image parameters into a pre-trained polarization image generation model and outputting the diffusion prior information corresponding to the RGB image. The diffusion prior information includes the encoded angle of linear polarization and degree of linear polarization; the pre-trained polarization image generation model is obtained by fine-tuning the pre-trained weights of the large diffusion model; An output module for calculating the light passing intensities of the original image corresponding to preset different polarization angles according to the encoded angle of linear polarization and degree of linear polarization; A visualization module for visualizing the light passing intensities of preset different polarization angles to obtain a plurality of polarization images.

Images