Polarization-guided delay-reflected three-dimensional gaussian splash reconstruction method and apparatus

Abstract

The application relates to the technical field of three-dimensional reconstruction, and provides a polarization-guided delayed reflection three-dimensional Gaussian splash reconstruction method and device. The method first acquires multi-view polarization images, constructs a multi-view polarization image dataset, calculates corresponding multi-view color RGB images by using an image interpolation method, and determines camera poses and sparse point clouds; subsequently, linear polarization degrees and linear polarization angles are determined based on the multi-view polarization image dataset, uncorrected polarization surface normals are calculated based on the linear polarization degrees and the linear polarization angles, and RGB images are separated into specular reflection and diffuse reflection images based on a preset refractive index and the linear polarization degrees; finally, the above results are input into a polarization information guided delayed reflection module to realize high-quality three-dimensional reconstruction. The method significantly improves the specular reflection processing capability, the normal estimation accuracy and the overall reconstruction quality, and can be applied to high-precision map three-dimensional reconstruction, environment perception, cultural heritage protection and digital twinning and other fields.

Description

Technical Field

[0001] This application relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional Gaussian splash reconstruction method and apparatus based on polarization-guided delayed reflection. Background Technology

[0002] High-quality 3D reconstruction using multiple views refers to the process of recovering the 3D geometry, texture information, and even material properties of objects in a scene using images obtained from multiple different perspectives. Polarization imaging, 3D Gaussian Splatting (3DGS), and Deferred Reflection (DR) techniques are commonly used in 3D reconstruction. Polarization imaging, by capturing the intensity, color, and polarization state of light, effectively provides physical priors closely related to the surface normals and material properties of objects. 3DGS represents a 3D scene by optimizing a large number of Gaussian distributed parameters in 3D space and uses Gaussian projection and blending rendering to quickly generate high-quality views. DR is a staged rendering technique that first obtains basic scene information through initial rendering steps, then delays the calculation of high-frequency shading effects such as specular reflection to the pixel level for fine processing, thereby improving rendering accuracy and realism.

[0003] 3D reconstruction technology has been widely applied in various fields such as cultural relic reconstruction, virtual reality, digital twins, and smart cities. Traditional 3D reconstruction methods focus on the accuracy of geometric structures, using techniques such as multi-view matching and depth estimation to achieve a basic restoration of the 3D shape of a scene. Implicit representation methods based on deep learning, such as those based on Neural Radiance Field (NeRF), emphasize the rendering accuracy of the reconstruction, effectively improving reconstruction quality, but they face problems such as low rendering efficiency and long training times. Explicit methods such as 3DGS have achieved breakthroughs in rendering speed and detail preservation. Although these reconstruction methods have significantly improved reconstruction quality, modeling specular reflective surfaces remains a key challenge in high-quality reconstruction.

[0004] In recent years, 3D reconstruction technology has developed rapidly, with numerous reconstruction methods based on 3DGS being proposed. While existing 3DGS technology possesses real-time rendering capabilities, its reconstruction accuracy is insufficient when handling scenes containing specular reflections due to the tight coupling between specular reflection effects and geometric structures. Polarization information can provide physical priors related to object surface normals and material properties, and has been applied to NeRF 3D reconstruction methods in recent years. However, NeRF methods generally suffer from long training times and low rendering efficiency, making it difficult to achieve fast and real-time high-quality 3D reconstruction. Therefore, simply relying on the combination of NeRF methods and polarization information still has limitations. Furthermore, the explicit rasterization mechanism of 3DGS restricts the direct fusion of polarization information, making it impossible to adopt the inverse rendering-based fusion method found in NeRF.

[0005] Therefore, how to further improve the accuracy and efficiency of 3D reconstruction in reflective scenarios is a technical problem that needs to be solved. Summary of the Invention

[0006] In view of this, embodiments of this application provide a method and apparatus for three-dimensional Gaussian splash reconstruction based on polarization-guided delayed reflection, in order to solve the problem that the accuracy and efficiency of three-dimensional reconstruction in reflection scenarios in the prior art need to be improved.

[0007] A first aspect of this application provides a method for reconstructing three-dimensional Gaussian splash using polarization-guided delayed reflection, comprising:

[0008] A multi-view polarization image dataset is constructed based on multi-view polarization images of the target scene. The multi-view polarization images are converted into polarized red-green-blue (RGB) images using image interpolation and the corresponding multi-view color RGB images are calculated. The camera pose and sparse point cloud are determined based on the multi-view color RGB images. The target scene includes at least a reflection scene.

[0009] The linear polarization degree and linear polarization angle are determined based on a multi-view polarization image dataset, and the uncorrected polarization surface normal is calculated based on the linear polarization degree and linear polarization angle.

[0010] Obtain the preset refractive index, and separate the polarized RGB image based on the preset refractive index and linear polarization degree to obtain the specular reflection image and the diffuse reflection image;

[0011] The preset polarization-guided 3D Gaussian splash 3DGS and delayed reflection model are initialized using multi-view color RGB images, camera pose, and sparse point cloud to obtain the initialized 3DGS delayed reflection model.

[0012] The prior surface normal is determined using an initial 3DGS delayed reflection model, the target threshold is determined based on the linear polarization degree, the normal fuzzing resolution strategy is determined based on the uncorrected polarized surface normal, and the corrected polarized surface normal is determined based on the prior surface normal, the target threshold, and the normal fuzzing resolution strategy.

[0013] The specular reflection image, diffuse reflection image, and corrected polarized surface normal are input to initialize the 3DGS delayed reflection model. The specular reflection image, diffuse reflection image, and polarized surface normal are jointly guided and supervised for training. After the model iteratively converges, the 3D reconstructed image of the target scene is obtained.

[0014] A second aspect of this application provides a three-dimensional Gaussian splash reconstruction apparatus for polarization-guided delayed reflection, comprising:

[0015] The construction module is configured to build a multi-view polarization image dataset based on multi-view polarization images of the target scene, convert the multi-view polarization images into polarization RGB images using image interpolation and calculate the corresponding multi-view color RGB images, and determine the camera pose and sparse point cloud based on the multi-view color RGB images; wherein, the target scene includes at least a reflection scene.

[0016] The determination module is configured to determine the linear polarization degree and linear polarization angle based on a multi-view polarization image dataset, and to calculate the uncorrected polarization surface normal based on the linear polarization degree and linear polarization angle.

[0017] The separation module is configured to obtain a preset refractive index and separate the polarized RGB image based on the preset refractive index and linear polarization degree to obtain a specular reflection image and a diffuse reflection image;

[0018] The initialization module is configured to use multi-view color RGB images, camera pose, and sparse point cloud to initialize the preset polarization-guided 3D Gaussian splash 3DGS and delayed reflection model to obtain the initialized delayed reflection 3DGS model.

[0019] The correction module is configured to determine the prior surface normal using an initial delayed reflection 3DGS model, determine the target threshold based on the linear polarization degree, determine the normal fuzzing resolution strategy based on the uncorrected polarized surface normal, and determine the corrected polarized surface normal based on the prior surface normal, the target threshold, and the normal fuzzing resolution strategy.

[0020] The reconstruction module is configured to input specular reflection image, diffuse reflection image and corrected polarized surface normal into the 3DGS delayed reflection model, perform joint guided and supervised training on specular reflection image, diffuse reflection image and polarized surface normal, and obtain the 3D reconstructed image of the target scene after the model iterative convergence.

[0021] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0022] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0023] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment first acquires multi-view polarization images, constructs a multi-view polarization image dataset, calculates the corresponding multi-view color RGB images using image interpolation, and uses these multi-view color RGB images to determine the camera pose and sparse point cloud; subsequently, it determines the linear polarization degree and linear polarization angle based on the multi-view polarization image dataset, calculates the uncorrected polarized surface normal based on the linear polarization degree and linear polarization angle, and separates the RGB images into specular reflection and diffuse reflection images based on a preset refractive index and linear polarization degree; finally, it inputs the above results into a delayed reflection module guided by polarization information to achieve high-quality 3D reconstruction. This method significantly improves specular reflection processing capability, normal estimation accuracy, and overall reconstruction quality, and can be applied to high-precision mapping. Figure 3 It covers multiple fields such as reconstruction, environmental perception, cultural heritage protection, and digital twins. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic flowchart of a three-dimensional Gaussian splash reconstruction method based on polarization-guided delayed reflection provided in an embodiment of this application.

[0026] Figure 2 This is a schematic flowchart of another polarization-guided delayed reflection three-dimensional Gaussian splash reconstruction method provided in the embodiments of this application.

[0027] Figure 3 This is a schematic diagram of the specular reflection map and diffuse reflection map obtained by processing using the method provided in the embodiments of this application.

[0028] Figure 4 This is a visualization of the uncorrected polarized surface normal obtained by processing using the method provided in the embodiments of this application.

[0029] Figure 5 These are comparison images of the 3D reconstruction results of mirror reflection images.

[0030] Figure 6 This is a schematic diagram of a three-dimensional Gaussian splash reconstruction device for polarization-guided delayed reflection provided in an embodiment of this application.

[0031] Figure 7 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0032] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0033] The following will describe in detail, with reference to the accompanying drawings, a method and apparatus for three-dimensional Gaussian splash reconstruction based on polarization-guided delayed reflection according to embodiments of this application.

[0034] As mentioned above, how to further improve the accuracy and efficiency of 3D reconstruction in reflective scenarios is a technical problem that needs to be solved.

[0035] In view of this, this application provides a method for reconstructing 3D Gaussian splashes using polarization-guided delayed reflection. First, multi-view polarization images are acquired, and a multi-view polarization image dataset is constructed. Image interpolation is used to calculate the corresponding multi-view color RGB images, and the camera pose and sparse point cloud are determined using these multi-view color RGB images. Then, the linear polarization degree and linear polarization angle are determined based on the multi-view polarization image dataset. The uncorrected polarized surface normal is calculated based on the linear polarization degree and linear polarization angle. The RGB images are separated into specular and diffuse reflection images based on a preset refractive index and linear polarization degree. Finally, the above results are input into a polarization-guided delayed reflection module to achieve high-quality 3D reconstruction. This method significantly improves specular reflection processing capabilities, normal estimation accuracy, and overall reconstruction quality, and can be applied to high-precision 3D reconstruction. Figure 3 It covers multiple fields such as reconstruction, environmental perception, cultural heritage protection, and digital twins.

[0036] Figure 1 This is a schematic flowchart of a three-dimensional Gaussian splash reconstruction method based on polarization-guided delayed reflection provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0037] In step S101, a multi-view polarization image dataset is constructed based on the multi-view polarization images of the target scene. The multi-view polarization images are converted into polarization RGB images using image interpolation and the corresponding multi-view color RGB images are calculated. The camera pose and sparse point cloud are determined based on the multi-view color RGB images.

[0038] The target scene includes at least a reflection scene.

[0039] In step S102, the linear polarization degree and linear polarization angle are determined based on the multi-view polarization image dataset, and the uncorrected polarization surface normal is calculated based on the linear polarization degree and linear polarization angle.

[0040] In step S103, a preset refractive index is obtained, and the polarized RGB image is separated based on the preset refractive index and the degree of linear polarization to obtain a specular reflection image and a diffuse reflection image.

[0041] In step S104, the preset polarization-guided 3D Gaussian splash 3DGS and delayed reflection model are initialized using multi-view color RGB images, camera pose, and sparse point cloud to obtain the initialized 3DGS delayed reflection model.

[0042] In step S105, the prior surface normal is determined using the initialized 3DGS delayed reflection model, the target threshold is determined based on the linear polarization degree, the normal fuzzing resolution strategy is determined based on the uncorrected polarized surface normal, and the corrected polarized surface normal is determined based on the prior surface normal, the target threshold, and the normal fuzzing resolution strategy.

[0043] In step S106, the specular reflection image, diffuse reflection image, and corrected polarized surface normal are input into the 3DGS delayed reflection model to initialize it. The specular reflection image, diffuse reflection image, and polarized surface normal are jointly guided and supervised for training. After the model iterative convergence, the three-dimensional reconstructed image of the target scene is obtained.

[0044] In some embodiments of this application, the method may be executed by a server or by a terminal device with certain processing capabilities.

[0045] In some embodiments of this application, multi-view polarization images of a target scene can be acquired, the target scene including at least a reflective scene. A multi-view polarization image dataset can be constructed based on the acquired multi-view polarization images.

[0046] In some embodiments of this application, image interpolation can be used to convert a multi-view polarized image into a polarized RGB image and calculate the corresponding multi-view color RGB image. Alternatively, other methods can be used to convert a multi-view polarized image into a polarized RGB image and calculate the corresponding multi-view color RGB image. No limitation is imposed here.

[0047] In some embodiments of this application, the linear polarization degree and linear polarization angle can be determined based on a multi-view polarization image dataset, and the uncorrected polarization surface normal can be calculated based on the linear polarization degree and linear polarization angle.

[0048] In some examples, a multi-view polarization image dataset can be input into a polarization information data preprocessing module to calculate the degree of linear polarization (DoLP) and the angle of linear polarization (AoLP).

[0049] In other examples, the calculated DoLP and AoLP can be input into the rough polarized surface normal prior extraction submodule to calculate the uncorrected polarized surface normal.

[0050] In some embodiments of this application, a preset refractive index can be obtained, and the polarized RGB image can be separated based on the preset refractive index and linear polarization degree to obtain a specular reflection image and a diffuse reflection image. Simultaneously, a preset polarization-guided 3D Gaussian splash 3DGS and a delayed reflection model can be initialized using multi-view color RGB images, camera pose, and sparse point clouds to obtain an initialized 3DGS delayed reflection model. This initialized 3DGS delayed reflection model represents a 3DGS incorporating a delayed reflection module.

[0051] In some embodiments of this application, the prior surface normal can be determined using an initial 3DGS delayed reflection model, the target threshold can be determined based on the linear polarization degree, the normal fuzzing resolution strategy can be determined based on the uncorrected polarized surface normal, and the corrected polarized surface normal can be determined based on the prior surface normal, the target threshold, and the normal fuzzing resolution strategy.

[0052] Finally, the specular reflection image, diffuse reflection image, and corrected polarized surface normal are input into the 3DGS delayed reflection model for initialization. The specular reflection image, diffuse reflection image, and polarized surface normal are jointly guided and supervised for training. After the model converges iteratively, the 3D reconstructed image of the target scene is obtained.

[0053] According to the technical solution provided in the embodiments of this application, firstly, multi-view polarization images are acquired, a multi-view polarization image dataset is constructed, and the corresponding multi-view color RGB images are calculated using image interpolation. The camera pose and sparse point cloud are then determined using these multi-view color RGB images. Subsequently, the linear polarization degree and linear polarization angle are determined based on the multi-view polarization image dataset. Uncorrected polarized surface normals are calculated based on the linear polarization degree and linear polarization angle. The RGB images are separated into specular reflection and diffuse reflection images based on a preset refractive index and linear polarization degree. Finally, the above results are input into a delayed reflection module guided by polarization information to achieve high-quality 3D reconstruction. This method significantly improves specular reflection processing capabilities, normal estimation accuracy, and overall reconstruction quality, and can be applied to high-precision mapping. Figure 3 It covers multiple fields such as reconstruction, environmental perception, cultural heritage protection, and digital twins.

[0054] In some embodiments of this application, the multi-view polarization image dataset can be obtained by selecting frames at equal intervals from the multi-view polarization images; and the camera pose and sparse point cloud are obtained by performing geometric initialization processing on the multi-view color RGB images, with the geometric initialization processing implemented based on a motion structure recovery algorithm.

[0055] In other words, in some examples, a multi-view polarization image dataset can be constructed by selecting a portion of the multi-view polarization image through equally spaced frame extraction. The frame extraction interval can be set according to actual needs, for example, 10-20 Hz. In other examples, other methods can also be used to construct the multi-view polarization image dataset; no restrictions are placed here.

[0056] Furthermore, in some examples, multi-view color RGB images can be input into motion structure recovery algorithms for geometric initialization to obtain camera pose and sparse point cloud, or other methods can be used to determine camera pose and sparse point cloud, which is not limited here.

[0057] Taking multi-view polarization images as an example, which are polarization images simultaneously captured by a polarization camera using a linear polarizer in four different directions, these four directions are, for example, , , and The polarization images in the four directions are denoted as follows: , , and Let the Stokes vector of the incident light be... ;in, , , , , and These represent the intensity of right-handed and left-handed circularly polarized light, respectively. When the camera cannot measure... and It can make and All are equal to 0.

[0058] Image interpolation methods, such as Bayer mode image interpolation, can be used to obtain four-channel RGB images, denoted as... , , and Furthermore, the multi-view color RGB image is calculated as follows: .

[0059] By inputting the multi-view color RGB image into a motion structure recovery algorithm, such as the COLMAP algorithm, for geometric initialization, the camera pose and sparse point cloud can be obtained.

[0060] Taking multi-view polarization images as an example, which are four polarization images simultaneously captured by a polarization camera using a linear polarizer in four different directions, DoLP and AoLP can be calculated using the following formulas:

[0061] ;

[0062] .

[0063] In some embodiments of this application, calculating the uncorrected polarized surface normal based on the linear polarization degree and linear polarization angle may include first calculating the zenith angle based on the linear polarization degree. And determine the equivalent azimuth angle of the linear polarization angle. Then use the formula The uncorrected polarized surface normal was calculated. Because there is a periodicity between AoLP and the normal azimuth angle ( and The direction is ambiguous, therefore the polarization surface normal calculated using the above formula inevitably has some ambiguity. and The problem of unclear direction, therefore This represents the uncorrected polarized surface normal.

[0064] Among them, zenith angle and equivalent azimuth One method for determining this is to separate the polarized RGB image based on a preset refractive index and linear polarization degree to obtain a specular reflection image and a diffuse reflection image. Then, the DoLP of the diffuse reflection image is determined, which is the angle of incidence when light strikes the object surface. And determine the azimuth angle equivalent to AoLP as that equivalent azimuth angle. .

[0065] In some embodiments of this application, separating polarized RGB images based on a preset refractive index and linear polarization degree to obtain specular reflection and diffuse reflection images may include adjusting the linear polarizer at an angle... The Mueller matrix below is denoted as The Mueller matrices for Fresnel reflection and transmission are denoted as... and Then the Stokes vector of the polarization component of the mirror reflection Stokes vector of diffuse reflection polarization component They can be represented as follows: ; ;in, This is the Stokes vector of the depolarized scattered light inside the medium.

[0066] Reflected light can generally be divided into three components: polarized specular reflection, polarized diffuse reflection, and unpolarized diffuse reflection. Since the unpolarized diffuse reflection component is relatively small, it is ignored in the calculation, thus yielding... .

[0067] Based on this, the DoLP of polarized diffuse reflection can be calculated to obtain the zenith angle, which is the angle of incidence when light strikes the object's surface. Then, the refractive index of the object's surface is set to a preset refractive index, and the corresponding angle of refraction is calculated using Snell's law based on this preset refractive index. Finally, specular reflection and diffuse reflection images are calculated based on the zenith angle, the angle of refraction, and the multi-view color RGB image.

[0068] In some implementation methods, formulas can be used first. Calculate the zenith angle ;in, , , , , For linear polarization degree, The preset refractive index.

[0069] Simultaneously use formulas Calculate the angle of refraction.

[0070] Then determine the specular reflection image. for:

[0071] ;

[0072] ;

[0073] ;

[0074] in, Indicates the channel angle of the polarizer. The maximum intensity measured for specular reflection. The minimum intensity measured for specular reflection. The angle between the polarization direction and the x-axis direction. , .

[0075] And, determine the diffuse reflection image for:

[0076] ;

[0077] ;

[0078] ;

[0079] in, The maximum intensity measured for diffuse reflection. This is the minimum intensity for diffuse reflection measurement.

[0080] In some embodiments of this application, the corrected polarization surface normal can be determined based on the prior surface normal, the target threshold, and a normal fuzzing resolution strategy. Specific correction methods include:

[0081] First, the nearest neighbor candidate normal set of the prior surface normal is determined according to the normal fuzzy analysis strategy. Then use the formula. ,and Determine the corrected polarization surface normal ;in, For vector normalization operators, This indicates that cosine similarity is calculated between the 3DGS predicted normal and the candidate set. The calculation formula is as follows: , Let L2 norm be the vector. , It is a positive integer greater than 1. Represent a dimensional vector, This is a mask function based on linear polarization degree.

[0082] In other words, the normal direction predicted by the initialized 3DGS delayed reflection model can be used as a priori, and it can be matched and judged with four directions in the candidate set of polarization normal angles. The correct angle direction of the polarization normal can be determined by selecting the candidate direction with the smallest cosine similarity difference.

[0083] Therefore, when correcting the polarization surface normal, the multi-view color RGB image, camera pose, and sparse point cloud can be pre-initialized into the polarization-guided 3DGS and delayed reflection model to obtain the initialized 3DGS delayed reflection model.

[0084] Then, the nearest neighbor candidate normal set of the prior surface normal is determined according to the normal fuzzy analysis strategy. The fuzzy parsing strategy can be to construct a candidate set and take the nearest neighbor strategy, thus determining the set. It can be ,in, Indicates rotation about the z-axis Finally, use the formula. Calculate the corrected polarization surface normal .

[0085] In some embodiments of this application, during the iterative process of jointly guiding and supervising the training of the specular reflection image, diffuse reflection image, and corrected polarization surface normal into the 3DGS delayed reflection model, a loss function can be designed for the guidance and supervision loss of polarization information, which includes at least an image reconstruction loss term, a specular reflection supervision loss term, a diffuse reflection supervision loss term, and a surface normal supervision loss term.

[0086] Among them, the image reconstruction loss term for ; This represents the balance parameter for RGB image supervision. This represents the final image obtained by fusing specular and diffuse reflection using intensity maps. With image truth value Between loss, Loss is calculated and The mean absolute error is obtained. , For the number of pixels, To predict pixel values, These are the actual pixel values; for and The difference in structural similarity index loss between them , and These represent the mean of the predicted pixel value and the mean of the actual pixel value, respectively. and These represent the variances of the predicted pixel value and the actual pixel value, respectively. This represents the covariance between the predicted pixel value and the actual pixel value. and All are stability constants.

[0087] Specular reflection monitoring loss item for ; This represents the balance parameter for monitoring specular reflection images. This represents the rendered specular reflection image. and Between loss, for and The difference in structural similarity index loss between them.

[0088] Diffuse reflection supervision loss term for ; The balance parameter represents the supervision of diffuse image. Represents the rendered diffuse image and Between loss, for and The difference in structural similarity index loss between them.

[0089] Surface normal monitoring loss term for ; , and These are the components of the surface normal vector on the x, y, and z axes, respectively.

[0090] The final joint loss function of the model can be expressed as: ,in , , and These represent the supervised weight parameters for RGB image, specular reflection image, diffuse reflection image, and surface normal, respectively, used to combine the proportion of each loss value in the total loss.

[0091] This joint loss function can guide and supervise the 3DGS and delayed reflection modules to perform 3D reconstruction, generating a 3D reconstruction model with high precision for the reflection area.

[0092] Figure 2 This is a schematic flowchart of another polarization-guided delayed reflection three-dimensional Gaussian splash reconstruction method provided in an embodiment of this application. Figure 2 As shown, polarization image acquisition can be performed first; then, frames can be extracted from the acquired polarization image data at equal intervals to obtain a multi-view polarization image dataset.

[0093] On the one hand, Stokes vectors can be calculated on multi-view polarized image datasets, and multi-view polarized color (RGB) image datasets can be obtained through Bayer interpolation; then, light intensity calculation is performed on the multi-view polarized color images, and the polarized color RGB dataset is geometrically initialized using the COLMAP algorithm to obtain camera pose and sparse point cloud; next, the color RGB images, camera pose and sparse point cloud are used to initialize the three-dimensional Gaussian splash and delayed reflection model to obtain the initialized 3DGS delayed reflection model.

[0094] On the other hand, polarization information can be calculated using multi-view polarized image datasets to obtain DoLP and AoLP. By combining camera pose, sparse point cloud, DoLP, and AoLP, the multi-view polarized image dataset can be calculated to obtain the rough polarized surface normal, which is the uncorrected polarized surface normal. Furthermore, DoLP and AoLP can be combined with a given prior refractive index, i.e., a preset refractive index, to calculate multi-view polarized color images to separate specular reflection and diffuse reflection, thereby obtaining specular reflection images and diffuse reflection images.

[0095] Figure 3 This is a schematic diagram of the specular reflection map and diffuse reflection map obtained by processing using the method provided in the embodiments of this application. Figure 4 This is a visualization of the uncorrected polarized surface normal obtained by processing using the method provided in the embodiments of this application.

[0096] in, Figure 3 The four images from left to right are a color RGB image, an image combining specular and diffuse reflection, a polarized diffuse reflection image, and a polarized specular reflection image.

[0097] Uncorrected polarized surface normals can be corrected using an initialized 3DGS delayed reflection model. The correction can be achieved using 3D Gaussian prior, DoLP thresholding, and fuzzy judgment strategies.

[0098] Finally, information such as color RGB images, camera pose, sparse point clouds, corrected polarization surface normals, specular reflection images, and diffuse reflection images can be used to guide the initialization of the 3DGS delayed reflection model based on polarization information to train a high-quality 3D reconstruction model, thereby obtaining high-quality 3D new perspective synthesis results in real specular reflection scenes.

[0099] Figure 5 These are comparison images of the 3D reconstruction results of mirror reflection images. Figure 5 The three images from left to right are, respectively, the ground truth image, the image reconstructed using conventional 3DGS and delayed reflection methods, and the image reconstructed using the method provided in the embodiments of this application. Figure 5It can be seen that the reconstruction effect of the specular reflection portion of the image is better in the image reconstructed using the method provided in the embodiments of this application.

[0100] The technical solution provided in this application combines 3DGS with polarization information, using polarization information to guide and supervise the reconstruction process, thereby significantly improving the modeling accuracy of 3DGS in complex reflection scenes. Specifically, the technical solution provides explicit separation of specular and diffuse reflection components through a polarization physical model and introduces uncorrected polarized surface normals as geometric priors, effectively enhancing the model's constraint capability in the reflection region. Simultaneously, the ambiguity in the polarization normals is corrected using the surface normals initialized by 3DGS, forming a complementary fusion of polarization priors and Gaussian reconstruction results.

[0101] The technical solution provided in this application not only improves the geometric accuracy and material detail reproduction of the reflective surface, but also enhances the robustness and rendering effect in various complex indoor and outdoor reflective environments, achieving more realistic and stable 3D reconstruction results.

[0102] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0103] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0104] Figure 6 This is a schematic diagram of a three-dimensional Gaussian splash reconstruction device for polarization-guided delayed reflection provided in an embodiment of this application. Figure 6 As shown, the device includes...

[0105] The construction module 601 is configured to construct a multi-view polarization image dataset based on the multi-view polarization images of the target scene, convert the multi-view polarization images into polarization RGB images using image interpolation and calculate the corresponding multi-view color RGB images, and determine the camera pose and sparse point cloud based on the multi-view color RGB images; wherein, the target scene includes at least a reflection scene.

[0106] The determination module 602 is configured to determine the linear polarization degree and linear polarization angle based on a multi-view polarization image dataset, and to calculate the uncorrected polarization surface normal based on the linear polarization degree and linear polarization angle.

[0107] The separation module 603 is configured to obtain a preset refractive index and separate the polarized RGB image based on the preset refractive index and linear polarization degree to obtain a specular reflection image and a diffuse reflection image.

[0108] Initialization module 604 is configured to initialize a preset polarization-guided 3D Gaussian splash 3DGS and delayed reflection model using multi-view color RGB images, camera pose, and sparse point cloud, to obtain an initialized delayed reflection 3DGS model.

[0109] The correction module 605 is configured to determine the prior surface normal using an initial delayed reflection 3DGS model, determine the target threshold based on the linear polarization degree, determine the normal fuzzing resolution strategy based on the uncorrected polarized surface normal, and determine the corrected polarized surface normal based on the prior surface normal, the target threshold, and the normal fuzzing resolution strategy.

[0110] The reconstruction module 606 is configured to input the specular reflection image, diffuse reflection image and corrected polarization surface normal into the 3DGS delayed reflection model, perform joint guided and supervised training on the specular reflection image, diffuse reflection image and polarization surface normal, and obtain the three-dimensional reconstructed image of the target scene after the model iterative convergence.

[0111] According to the technical solution provided in the embodiments of this application, firstly, multi-view polarization images are acquired, a multi-view polarization image dataset is constructed, and the corresponding multi-view color RGB images are calculated using image interpolation. The camera pose and sparse point cloud are then determined using these multi-view color RGB images. Subsequently, the linear polarization degree and linear polarization angle are determined based on the multi-view polarization image dataset. Uncorrected polarized surface normals are calculated based on the linear polarization degree and linear polarization angle. The RGB images are separated into specular reflection and diffuse reflection images based on a preset refractive index and linear polarization degree. Finally, the above results are input into a delayed reflection module guided by polarization information to achieve high-quality 3D reconstruction. This method significantly improves specular reflection processing capabilities, normal estimation accuracy, and overall reconstruction quality, and can be applied to high-precision mapping. Figure 3 It covers multiple fields such as reconstruction, environmental perception, cultural heritage protection, and digital twins.

[0112] In some implementations, the multi-view polarization image dataset is obtained by selecting frames at equal intervals from the multi-view polarization images; the camera pose and sparse point cloud are obtained by performing geometric initialization processing on the multi-view color RGB images, and the geometric initialization processing is implemented based on the motion structure recovery algorithm.

[0113] In some implementations, the uncorrected polarization surface normal is calculated based on the linear polarization degree and the linear polarization angle, including: calculating the zenith angle based on the linear polarization degree. The equivalent azimuth angle for determining a linear polarization angle. Use formula The uncorrected polarized surface normal is calculated; where, This represents the uncorrected polarized surface normal.

[0114] In some implementations, the polarized RGB image is separated based on a preset refractive index and linear polarization degree to obtain a specular reflection image and a diffuse reflection image, including: using the formula Calculate the zenith angle ;in, , , , For linear polarization degree, Preset refractive index; use formula Calculate the angle of refraction Determine the specular reflection image for: ; ; ;in, Indicates the channel angle of the polarizer. The maximum intensity measured for specular reflection. The minimum intensity measured for specular reflection. For the incident light intensity, The angle between the polarization direction and the x-axis direction. , Determine the diffuse reflection image for: ; ; ;in, The maximum intensity measured for diffuse reflection. The minimum intensity for diffuse reflection measurement. The Stokes vector of the depolarized scattered light inside the medium. , Let Stokes vector be the incident light. This is the Stokes vector for mirror reflection.

[0115] In some implementations, determining the corrected polarization surface normal based on the prior surface normal, the target threshold, and a normal fuzzy resolution strategy includes: determining the nearest neighbor candidate normal set of the prior surface normal based on the normal fuzzy resolution strategy. Use formula ,and Determine the corrected polarization surface normal ;in, For vector normalization operators, This indicates that cosine similarity is calculated between the 3DGS predicted normal and the candidate set. The calculation formula is as follows: , Let L2 norm be the vector. This is a mask function based on linear polarization degree.

[0116] In some implementations, during model iteration, the loss function includes at least an image reconstruction loss term, a specular reflection supervision loss term, a diffuse reflection supervision loss term, and a surface normal supervision loss term.

[0117] In some implementations, the image reconstruction loss term for ;in, This represents the balance parameter for RGB image supervision. This represents the final image obtained by fusing specular and diffuse reflection using intensity maps. With image truth value Between loss, Loss is calculated and The mean absolute error is obtained. for and The difference in structural similarity index loss; the specular reflection supervision loss term. for ;in, This represents the balance parameter for monitoring specular reflection images. This represents the rendered specular reflection image. and Between loss, for and The difference in structural similarity index loss; diffuse reflection supervision loss term for ;in, The balance parameter represents the supervision of diffuse image. Represents the rendered diffuse image and Between loss, for and The difference in structural similarity index loss; surface normal supervision loss term. for ;in, , and These are the components of the surface normal vector on the x, y, and z axes, respectively.

[0118] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0119] Figure 7This is a schematic diagram of an electronic device provided in an embodiment of this application. For example... Figure 7 As shown, the electronic device 7 of this embodiment includes: a processor 701, a memory 702, and a computer program 703 stored in the memory 702 and executable on the processor 701.

[0120] The electronic device can be connected to a polarization camera 704 to receive multi-view polarization images of the target scene acquired by the polarization camera 704. The polarization camera 704 can be a single camera or a group of cameras. The electronic device can save the received multi-view polarization images in a memory 702 for subsequent processing by a processor 701. Alternatively, the electronic device can directly send the received multi-view polarization images to the processor 701 for real-time processing.

[0121] When processor 701 executes computer program 703, it implements the steps in the above-described method embodiments. Alternatively, when processor 701 executes computer program 703, it implements the functions of each module / unit in the above-described device embodiments.

[0122] Electronic device 7 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 7 may include, but is not limited to, processor 701 and memory 702. Those skilled in the art will understand that... Figure 7 This is merely an example of electronic device 7 and does not constitute a limitation on electronic device 7. It may include more or fewer components than shown, or different components.

[0123] The processor 701 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0124] The memory 702 can be an internal storage unit of the electronic device 7, such as a hard disk or RAM of the electronic device 7. The memory 702 can also be an external storage device of the electronic device 7, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 7. The memory 702 can also include both internal and external storage units of the electronic device 7. The memory 702 is used to store computer programs and other programs and data required by the electronic device.

[0125] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0126] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0127] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for reconstructing three-dimensional Gaussian splash using polarization-guided delayed reflection, characterized in that, include: A multi-view polarized image dataset is constructed based on multi-view polarized images of the target scene. The multi-view polarized images are converted into polarized RGB images using image interpolation and the corresponding multi-view color RGB images are calculated. The camera pose and sparse point cloud are determined based on the multi-view color RGB images. The target scene includes at least a reflection scene. Based on the multi-view polarization image dataset, the linear polarization degree and linear polarization angle are determined, and the uncorrected polarization surface normal is calculated based on the linear polarization degree and the linear polarization angle. A preset refractive index is obtained, and the polarized RGB image is separated based on the preset refractive index and the linear polarization degree to obtain a specular reflection image and a diffuse reflection image; The multi-view color RGB image, the camera pose and the sparse point cloud are used to initialize the preset polarization-guided 3D Gaussian splash 3DGS and delayed reflection model to obtain the initialized 3DGS delayed reflection model. The prior surface normal is determined using the initial 3DGS delayed reflection model, the target threshold is determined based on the linear polarization degree, the normal fuzzing resolution strategy is determined based on the uncorrected polarized surface normal, and the corrected polarized surface normal is determined based on the prior surface normal, the target threshold, and the normal fuzzing resolution strategy. The specular reflection image, the diffuse reflection image, and the corrected polarized surface normal are input into the initial 3DGS delayed reflection model. The specular reflection image, the diffuse reflection image, and the polarized surface normal are jointly guided and supervised for training. After the model iterative convergence, the three-dimensional reconstructed image of the target scene is obtained. The process of determining the corrected polarization surface normal based on the prior surface normal, the target threshold, and the normal fuzzy resolution strategy includes: The nearest neighbor candidate normal set of the prior surface normal is determined according to the normal fuzzy analysis strategy. ; Use formula ,and Determine the corrected polarization surface normal ;in, For vector normalization operators, This indicates that cosine similarity is calculated between the 3DGS predicted normal and the candidate set. The calculation formula is as follows: , Let L2 norm be the vector. This is a mask function based on linear polarization degree.

2. The method according to claim 1, characterized in that, The multi-view polarization image dataset is obtained by selecting frames at equal intervals from the multi-view polarization images; The camera pose and sparse point cloud are obtained by performing geometric initialization processing on the multi-view color RGB image, and the geometric initialization processing is implemented based on the motion structure recovery algorithm.

3. The method according to claim 1, characterized in that, The uncorrected polarization surface normal is calculated based on the linear polarization degree and the linear polarization angle, including: The zenith angle is calculated based on the linear polarization degree. ; Determine the equivalent azimuth angle of the linear polarization angle. ; Use formula The uncorrected polarization surface normal is calculated; wherein, The uncorrected polarization surface normal is given.

4. The method according to claim 1, characterized in that, Based on the preset refractive index and the linear polarization degree, the polarized RGB image is separated to obtain a specular reflection image and a diffuse reflection image, including: Use formula Calculate the zenith angle ;in, , , , , The linear polarization degree, The preset refractive index; Use formula Calculate the angle of refraction ; Determine the specular reflection image for: ； ； ； in, Indicates the channel angle of the polarizer. The maximum intensity measured for specular reflection. The minimum intensity measured for specular reflection. For the incident light intensity, The angle between the polarization direction and the x-axis direction. , ; Determine diffuse reflection image for: ； ； ； in, for, The maximum intensity measured for diffuse reflection. The minimum intensity for diffuse reflection measurement. The Stokes vector of the depolarized scattered light inside the medium. , Let be the incident vector. This is the specular reflection vector.

5. The method according to claim 1, characterized in that, During model iteration, the loss function includes at least an image reconstruction loss term, a specular reflection supervision loss term, a diffuse reflection supervision loss term, and a surface normal supervision loss term.

6. The method according to claim 5, characterized in that, The image reconstruction loss term for ;in, This represents the balance parameter for RGB image supervision. This represents the final image obtained by fusing specular and diffuse reflection using intensity maps. With image truth value Between Loss, the aforementioned Loss is calculated and The mean absolute error is obtained. for and The difference in structural similarity index loss between them; The specular reflection monitoring loss term for ;in, This represents the balance parameter for monitoring specular reflection images. This represents the rendered specular reflection image. and Between loss, for and The difference in structural similarity index loss between them; The diffuse reflection monitoring loss term for ;in, The balance parameter represents the supervision of diffuse image. Represents the rendered diffuse image and Between loss, for and The difference in structural similarity index loss between them; The surface normal monitoring loss term for ;in, , and These are the components of the surface normal vector on the x, y, and z axes, respectively.

7. A three-dimensional Gaussian splash reconstruction device based on polarization-guided delayed reflection, characterized in that, include: The construction module is configured to build a multi-view polarization image dataset based on multi-view polarization images of the target scene, convert the multi-view polarization images into polarization RGB images using image interpolation and calculate the corresponding multi-view color RGB images, and determine the camera pose and sparse point cloud based on the multi-view color RGB images; wherein, the target scene includes at least a reflection scene. The determination module is configured to determine the linear polarization degree and linear polarization angle based on the multi-view polarization image dataset, and calculate the uncorrected polarization surface normal based on the linear polarization degree and the linear polarization angle; The separation module is configured to obtain a preset refractive index and separate the polarized RGB image based on the preset refractive index and the linear polarization degree to obtain a specular reflection image and a diffuse reflection image; The initialization module is configured to use the multi-view color RGB image, the camera pose and the sparse point cloud to initialize the preset polarization-guided three-dimensional Gaussian splash 3DGS and delayed reflection model to obtain the initialized delayed reflection 3DGS model. The correction module is configured to determine the prior surface normal using the initial delayed reflection 3DGS model, determine the target threshold based on the linear polarization degree, determine the normal fuzzing resolution strategy based on the uncorrected polarized surface normal, and determine the corrected polarized surface normal according to the prior surface normal, the target threshold, and the normal fuzzing resolution strategy. The reconstruction module is configured to input the specular reflection image, the diffuse reflection image, and the corrected polarization surface normal into the initial 3DGS delayed reflection model, perform joint guided and supervised training on the specular reflection image, the diffuse reflection image, and the polarization surface normal, and obtain the three-dimensional reconstructed image of the target scene after the model iterative convergence. The process of determining the corrected polarization surface normal based on the prior surface normal, the target threshold, and the normal fuzzy resolution strategy includes: The nearest neighbor candidate normal set of the prior surface normal is determined according to the normal fuzzy analysis strategy. ; Use formula ,and Determine the corrected polarization surface normal ;in, For vector normalization operators, This indicates that cosine similarity is calculated between the 3DGS predicted normal and the candidate set. The calculation formula is as follows: , Let L2 norm be the vector. This is a mask function based on linear polarization degree.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The electronic device is connected to a polarization camera and receives multi-view polarization images of the target scene acquired by the polarization camera. When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images