Semantic-guided Gaussian point rendering method, system and device and medium

Abstract

The invention relates to the technical field of computer graphic processing, and discloses a semantic-guided Gaussian point rendering method, system and device and a medium, and the method comprises the steps: obtaining a multi-view image, obtaining a corresponding 2D image feature, carrying out the association with a 3D Gaussian point, and carrying out the distillation of the semantic feature of the 3D Gaussian point; extracting semantic features and position information of each Gaussian point after distillation, generating comprehensive appearance features, and mapping the appearance features to material attributes; performing consistency evaluation between the distorted image and the target diffuse reflection image, and constraining local consistency of a depth map and a normal map of the 3D Gaussian point at the same time, so as to optimize and adjust the 3D Gaussian point through consistency loss; and calculating a final pixel color in combination with the learnable environment map. According to the method, the rendering quality is improved, the material attribute of the object is represented more accurately through the semantic feature distillation and the implicit material dictionary, and the rendering problem of the reflecting surface is effectively solved.

Description

Technical Field

[0001] This invention relates to the field of computer graphics processing technology, and in particular to a semantically guided Gaussian point rendering method, system, device, and medium. Background Technology

[0002] With the development of computer graphics, Gaussian point rendering technology has gradually become a research hotspot. Traditional rendering methods have many shortcomings when dealing with complex scenes and reflective surfaces. For example, although Neural Render Fields (NeRFs) can achieve high-fidelity view composition, they have high computational costs and are difficult to implement in real-time rendering; although 3DGS has a fast rendering speed, it is not good at representing the textures and lighting of scenes with specular reflections, resulting in inconsistencies in multi-view rendering and affecting the accuracy of geometric reconstruction.

[0003] Existing technologies fail to fully consider the semantic information and material characteristics of objects during the rendering process, making it difficult to accurately distinguish the source of color, which limits rendering quality and efficiency. Summary of the Invention

[0004] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a semantically guided Gaussian point rendering method to solve the above problems.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] In a first aspect, the present invention provides a semantically guided Gaussian point rendering method, comprising:

[0007] Obtain multi-view images;

[0008] Based on the multi-view image, the corresponding 2D image features are obtained and associated with 3D Gaussian points. A semantic feature regularization loss function is constructed to distill the semantic features of the 3D Gaussian points.

[0009] The semantic features and positional information of each Gaussian point after distillation are extracted to generate a comprehensive appearance feature, and the appearance feature is mapped to the material property to obtain a material property map.

[0010] Based on the material information of the Gaussian points, a consistency evaluation is performed between the distorted image and the target diffuse reflection image, and the local consistency of the depth map and normal map of the 3D Gaussian points is constrained at the same time, so as to optimize and adjust the 3D Gaussian points through consistency loss.

[0011] Based on the material property map and normal map, combined with the learnable environment map, the final pixel color is calculated to achieve high-fidelity rendering.

[0012] As a preferred embodiment of the semantically guided Gaussian point rendering method described in this invention, the following steps are included: obtaining the corresponding 2D image features and associating them with 3D Gaussian points:

[0013] The multi-view images are input into a pre-trained 2D base model to obtain the corresponding 2D feature maps;

[0014] By using differentiable rendering technology, 3D Gaussian points are projected onto the image plane of the 2D feature map, the feature vector of each pixel is calculated, and the 2D image features are associated with the 3D Gaussian points.

[0015] The feature vector of each pixel is calculated by multiplying the Gaussian opacity by the weighted sum of the features.

[0016] As a preferred embodiment of the semantically guided Gaussian point rendering method of the present invention, the following steps are included: constructing a semantic feature regularization loss function to distill the semantic features of the 3D Gaussian points:

[0017] The initial high-dimensional features of the 2D feature map are compressed using a convolutional encoder, and then the dimensions are restored using a decoder.

[0018] A semantic feature regularization loss function is constructed, the difference between the decoded features and the initial high-dimensional features is calculated, and the difference between the two is reduced by the gradient descent algorithm to complete the semantic feature distillation of the 3D Gaussian points;

[0019] The beneficial effect of this preferred solution is that it can more accurately represent the material properties of objects through semantic feature distillation and implicit material dictionary, effectively solving the rendering problem of reflective surfaces.

[0020] As a preferred embodiment of the semantically guided Gaussian point rendering method described in this invention, obtaining the material property map includes:

[0021] The semantic features and positional information of each Gaussian point are extracted, and then concatenated and transformed to generate a comprehensive appearance feature. The semantic features are obtained from the distillation process, and the positional information is the coordinates of the Gaussian point in 3D space.

[0022] The appearance features are mapped to material properties through an implicit material dictionary to obtain a material property map, which includes a diffuse color map, a reflection intensity map, and a residual color component map.

[0023] As a preferred embodiment of the semantically guided Gaussian point rendering method described in this invention, the consistency evaluation between the distorted image and the target diffuse image includes:

[0024] Source frame pixels are mapped to reference frames using a homeomorphic matrix, employing a planar patch-based photometric multi-view approach. Figure 1Consistency constraints are applied, and the consistency between the distorted image and the target diffuse image is evaluated using the diffuse color map.

[0025] Building multi-view Figure 1 The consistency loss function, based on the calculated consistency difference, optimizes the properties of Gaussian points to reduce the consistency difference through backpropagation using gradient descent.

[0026] As a preferred embodiment of the semantically guided Gaussian point rendering method described in this invention, constraining the local consistency of the depth map and normal map of the 3D Gaussian points includes:

[0027] A depth map is calculated from a viewpoint using the 3D Gaussian points, and the depth map is converted into a gradient normal map using the finite difference method.

[0028] By minimizing the difference between the gradient normal of the rendered depth map and the rendered normal map, the local consistency of depth and normal is constrained, a normal-geometric consistency loss function is constructed, the cosine similarity difference and absolute error between the two are calculated, and the 3D Gaussian point is optimized through gradient descent backpropagation.

[0029] The advantages of this preferred solution are improved geometric accuracy and the introduction of geometrically consistent modeling, including multi-view... Figure 1 Consistency and normal-geometric consistency constraints ensure global geometric alignment, improve the accuracy of geometric reconstruction, and enable complete reconstruction of scene geometry with reflective surfaces.

[0030] As a preferred embodiment of the semantically guided Gaussian point rendering method described in this invention, high-fidelity rendering includes:

[0031] By constructing the diffuse color map, reflection intensity map, and surface discovery map obtained by optimizing the 3D Gaussian points, and combining them with the learnable environment map, the final pixel color is calculated through the deferred reflection channel to complete the high-fidelity rendering of deferred shading.

[0032] Secondly, this invention provides a semantically guided Gaussian point rendering system, comprising:

[0033] The acquisition module is used to acquire multi-view images;

[0034] The first processing module is used to obtain the corresponding 2D image features based on the multi-view image, associate them with 3D Gaussian points, construct a semantic feature regularization loss function, and distill the semantic features of the 3D Gaussian points.

[0035] The second processing module is used to extract the semantic features and positional information of each Gaussian point after distillation, generate comprehensive appearance features, and map the appearance features to material properties to obtain a material property map.

[0036] The third processing module is used to evaluate the consistency between the distorted image and the target diffuse reflection image based on the material information of the Gaussian point, and simultaneously constrain the local consistency of the depth map and normal map of the 3D Gaussian point, so as to optimize and adjust the 3D Gaussian point through consistency loss.

[0037] The output module is used to calculate the final pixel color based on the material property map and normal map, combined with the learnable environment map, in order to achieve high-fidelity rendering.

[0038] Thirdly, the present invention provides a computer device, comprising:

[0039] Memory and processor;

[0040] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the semantically guided Gaussian point rendering method.

[0041] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the semantically guided Gaussian point rendering method.

[0042] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention improves rendering quality by more accurately representing the material properties of objects through semantic feature distillation and implicit material dictionaries, effectively solving the rendering problem of reflective surfaces. Especially when dealing with scenes with significant reflective surfaces, the rendering quality is significantly better than traditional methods. It also improves geometric accuracy by introducing geometric consistency modeling, including multi-view... Figure 1 Consistency and normal-geometric consistency constraints ensure global geometric alignment, improve the accuracy of geometric reconstruction, and enable complete reconstruction of scene geometry with reflective surfaces; they also enhance multi-view capabilities. Figure 1 Consistency is improved by using diffuse color to maintain consistency across multiple views, avoiding appearance changes caused by reflective surfaces and enhancing the stability of the overall rendering effect; rendering efficiency is improved by optimizing the rendering process based on 3D Gaussian point rendering technology, achieving fast rendering while ensuring high-quality visual effects. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of the overall process of the semantically guided Gaussian point rendering method according to an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of the semantic feature extraction and distillation process of the semantically guided Gaussian point rendering method according to an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram illustrating the implicit material dictionary construction process of the semantically guided Gaussian point rendering method according to an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of the Gaussian geometric consistency modeling process of the semantically guided Gaussian point rendering method according to an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the deferred rendering process of the semantically guided Gaussian point rendering method according to an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram of the joint parameter optimization process of the semantically guided Gaussian point rendering method according to an embodiment of the present invention;

[0050] Figure 7 This is a schematic diagram comparing the rendering quality of the semantically guided Gaussian point rendering method according to an embodiment of the present invention;

[0051] Figure 8 This is a schematic diagram comparing the convergence speed of the semantically guided Gaussian point rendering method according to an embodiment of the present invention;

[0052] Figure 9 This is a schematic diagram comparing the geometric reconstruction quality of the semantically guided Gaussian point rendering method according to an embodiment of the present invention.

[0053] Figure 10 This is a schematic diagram of the geometric reconstruction result of the semantically guided Gaussian point rendering method according to an embodiment of the present invention. Detailed Implementation

[0054] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0055] Example 1, referring to Figures 1-6 As an embodiment of the present invention, a semantically guided Gaussian point rendering method is provided, comprising:

[0056] S101, Obtain multi-view image;

[0057] S102: Based on the multi-view image, obtain the corresponding 2D image features and associate them with 3D Gaussian points to construct a semantic feature regularization loss function and distill the semantic features of the 3D Gaussian points.

[0058] S103, extract the semantic features and positional information of each Gaussian point after distillation, generate comprehensive appearance features, and map the appearance features to material properties to obtain a material property map;

[0059] S104, based on the material information of Gaussian points, performs consistency evaluation between the distorted image and the target diffuse reflection image, and simultaneously constrains the local consistency of the depth map and normal map of the 3D Gaussian points, so as to optimize and adjust the 3D Gaussian points through consistency loss;

[0060] S105, based on material property maps and normal maps, combined with learnable environment maps, calculates the final pixel color to achieve high-fidelity rendering.

[0061] It should be noted that multi-view images are geometric images of an object to be reconstructed taken from different directions, and can be obtained through various camera devices (such as mobile phones and cameras).

[0062] In a preferred embodiment, obtaining the corresponding 2D image features and associating them with 3D Gaussian points includes:

[0063] The multi-view images are input into a pre-trained 2D base model to obtain the corresponding 2D feature maps;

[0064] By using differentiable rendering technology, 3D Gaussian points are projected onto the image plane of 2D feature maps, the feature vector of each pixel is calculated, and the 2D image features are associated with 3D Gaussian points.

[0065] The feature vector of each pixel is calculated by multiplying the Gaussian opacity by the weighted sum of the features.

[0066] In this embodiment, the basic model used in the 2D base model is the Language-driven Semantic Segmentation (LSeg) model. When an RGB image of (H, W, 3) is input into the model, the model will infer a feature map of (H, W, 512), where 3 represents the three channels of RGB and 512 represents the dimension of the feature.

[0067] It should be noted that the input multi-view image is fed into the pre-trained 2D base model to obtain the corresponding feature maps. These feature maps contain high-level semantic information of the image, which helps to understand the material and characteristics of objects in the scene.

[0068] It should be noted that in this invention, a 3D object is represented by Gaussian points. A Gaussian point can be simply understood as a semi-transparent ellipsoid in 3D space. Many ellipsoids are stacked to form the appearance of a real 3D object. Each ellipsoid has some predefined attributes. The position information is the position (x, y, z) of its center point in 3D space. The semantic feature is a 512-dimensional vector (or feature, an array string) stored on each ellipsoid.

[0069] Specifically, using differentiable rendering technology, 3D Gaussian points are projected onto a 2D image plane using rasterization. During the rendering process, the feature vector at each pixel is calculated. This vector integrates semantic information from different Gaussian points. The Gaussian points are semi-transparent, and the rendering feature of each pixel is a weighted sum of Gaussian opacity multiplied by its own features. By constructing a loss function with the rendering features and the base model features, the features stored on the Gaussian points can learn the features of the base model.

[0070] It should be noted that the feature map is 2D, while the Gaussian point is a 3D point. Therefore, the Gaussian point needs to be projected onto the 2D feature map to assign the corresponding feature to the Gaussian point. At the same time, since the Gaussian point can be projected onto multiple feature maps to obtain multiple features, the feature stored in the Gaussian point is the one that best reflects the average characteristics of these features. It needs to be optimized using gradient descent. Therefore, the "rendering" process of projecting the Gaussian point onto the 2D feature map needs to be differentiable. The Gaussian point is first randomly initialized, and then its position and size are optimized, which also includes operations such as splitting, merging, and deleting.

[0071] In an alternative implementation, the 2D base model can also be the DeepLabv3+ model, which captures multi-scale semantic information through atrous convolution and atrous spatial pyramid pooling (ASPP) structures, and supports pre-trained weight transfer.

[0072] In another alternative implementation, the 2D base model can also be an instance segmentation model of Faster R-CNN, which can not only identify categories but also generate a mask for each instance, making it suitable for sophisticated semantic analysis.

[0073] In a preferred embodiment, constructing a semantic feature regularization loss function and distilling the semantic features of 3D Gaussian points includes:

[0074] The initial high-dimensional features of the 2D feature map are compressed using a convolutional encoder, and then the dimensions are restored using a decoder.

[0075] A semantic feature regularization loss function is constructed to calculate the difference between the decoded features and the initial high-dimensional features. The difference between the two is reduced by the gradient descent algorithm to complete the semantic feature distillation of 3D Gaussian points.

[0076] Specifically, the features stored in the feature map output by the basic model are 512-dimensional. Since the dimension is too large and takes up too much storage space, the encoder is used to compress it to 64-dimensional storage. When used, it is restored to 512-dimensional by the corresponding decoder. The semantic feature regularization loss function is the cosine similarity and absolute value error between two features.

[0077] Feature compression and restoration: A convolutional encoder is used to compress the initial high-dimensional features, significantly reducing the feature dimension from Mhigh to Mlow. The parameters θencoder and θdecoder of the convolutional encoder and decoder are adjusted according to the actual application scenario to balance feature compression efficiency and restoration quality. Subsequently, the feature dimension is restored by the decoder for subsequent processing. A semantic feature regularization loss function is constructed to ensure that the distilled features are consistent with the model features. Semantic feature regularization is used to optimize the semantic features of Gaussian points.

[0078] In an alternative implementation, semantic feature distillation of 3D Gaussian points can also be achieved through...

[0079] It should be noted that semantic feature regularization enables 3D Gaussian points to obtain semantic features with correct meanings, so that the correct material properties can be queried later through semantic features.

[0080] In a preferred embodiment, obtaining the material property map includes:

[0081] The semantic features and positional information of each Gaussian point are extracted, and then concatenated and transformed to generate a comprehensive appearance feature. The semantic features are obtained from the distillation process, and the positional information is the coordinates of the Gaussian point in 3D space.

[0082] The appearance features are mapped to material properties through an implicit material dictionary to obtain a material property map, which includes a diffuse color map, a reflection intensity map, and a residual color component map.

[0083] Specifically, the semantic features and positional information of each Gaussian point are extracted, and after concatenating the semantic features and positional information, they are transformed into a feature through a neural network, which is equivalent to a new feature vector, i.e., appearance feature.

[0084] Then, material properties are mapped. Through the implicit material dictionary θmaterial, appearance features are mapped to specific material properties. These properties include diffuse color d, reflection intensity r, and residual color component Δc, which together determine the visual effect of the object's surface. The implicit material dictionary is a neural network that takes appearance features as input and outputs material properties with physical meaning. Appearance features are indices, and material properties are the values ​​retrieved. The query process is a black box modeled by the neural network.

[0085] In an alternative implementation, mapping appearance features to material properties can also be achieved through Conditional Generative Models (CGMs). These CGMs use appearance features as conditions to generate material property maps that conform to physical laws. For example, a Conditional Generative Adversarial Network (CGN) can generate material property maps (such as diffuse color maps) by taking appearance features (conditional vectors) as input to a generator. A discriminator can then distinguish between the generated maps and the real material maps, while maintaining consistency in the constraints. This approach can generate high-resolution, detailed material maps that support direct mapping from semantic features to complex textures.

[0086] In a preferred embodiment, assessing the consistency between the distorted image and the target diffuse image includes:

[0087] Source frame pixels are mapped to reference frames using a homeomorphic matrix, employing a planar patch-based photometric multi-view approach. Figure 1 Consistency constraints are applied, and the consistency between the distorted image and the target diffuse image is evaluated using diffuse color maps.

[0088] Building multi-view Figure 1 The consistency loss function, based on the calculated consistency difference, optimizes the properties of Gaussian points to reduce the consistency difference through backpropagation using gradient descent.

[0089] In this embodiment, pixel mapping utilizes a homeomorphic matrix Hsr to map pixels ps in the source frame to pixels pr in the reference frame. This process ensures the correspondence between pixels in different views. The homeomorphic matrix is ​​obtained from camera parameters and is a homogeneous (4*4) matrix. It transforms pixel coordinates and their depth values ​​into homogeneous (4*1) vectors. Through a single matrix multiplication—multiplying the homeomorphic matrix by the homogeneous pixel vector—the pixels are mapped to the reference frame via a homogeneous transformation. For consistency constraints, a photometric multiview based on planar patches is employed. Figure 1 Consistency constraints. A multi-view constraint is constructed by comparing the similarity between the distorted image IDs→r and the target diffuse image IDr. Figure 1The consistency loss function Lap uses normalized cross correlation (NCC) to measure consistency and allows gradients to propagate only through the rendered image, where a planar patch is an image block, for example, an image is cut into a 3x3 grid, and each small block is called a patch.

[0090] It should be noted that the color of an object's surface can be simplified to diffuse reflection (matte, consistent color when viewed from different angles) and specular reflection (highlight, inconsistent color when viewed from different angles). The diffuse color image is the component remaining after removing specular reflection. The consistency evaluation is achieved by mapping from source frame pixels to reference frame pixels using the homeomorphism matrix mentioned above, and calculating the color difference of corresponding pixels. The smaller the difference, the stronger the consistency.

[0091] In a preferred embodiment, constraining the local consistency of the depth map and normal map of 3D Gaussian points includes:

[0092] Calculate a depth map from a viewpoint using 3D Gaussian points, and then convert the depth map into a gradient normal map using the finite difference method.

[0093] By minimizing the difference between the gradient normal of the rendered depth map and the rendered normal map, the local consistency of depth and normal is constrained. A normal-geometric consistency loss function is constructed, and the cosine similarity difference and absolute error between the two are calculated. The 3D Gaussian points are optimized through gradient descent backpropagation.

[0094] In this embodiment, normal-geometry consistency is achieved by minimizing the gradient normals of the rendered depth map. The difference between the rendered normal map Nd and the local consistency of the normals is constrained by depth and normals. A normal-geometric consistency loss function Lnormal is constructed to ensure the accuracy and consistency of surface normals.

[0095] Among them, the weights of the loss function, in the total loss function, include the semantic feature regularization loss (Lsemantic) and the multi-view loss. Figure 1 The weights of consistency loss Lmvs and normal-geometry consistency loss Lnormal are set according to the specific scene and requirements to achieve the best rendering effect.

[0096] In one alternative implementation, each Gaussian point has its own normal. The depth map can be converted into a gradient normal map by projecting it onto the image to obtain the normal map, and then the cosine similarity difference and absolute error between the two can be calculated.

[0097] In a preferred embodiment, high-fidelity rendering includes:

[0098] By constructing a diffuse color map, a reflection intensity map, and a surface discovery map obtained by optimizing 3D Gaussian points, and combining it with a learnable environment map, the final pixel color is calculated through the deferred reflection channel to achieve high-fidelity rendering with deferred shading.

[0099] In this embodiment, the environment map is queried, combining the diffuse color map D, the surface normal map N, the reflection intensity map R, and the learnable environment map E. The final pixel color Cd is calculated through the deferred reflection channel. During the rendering process, the corresponding reflection color is queried from the environment map according to the calculated reflection direction. The final pixel color Cd is composed of the diffuse color and the reflection color, and the specific formula is expressed as follows:

[0100]

[0101] Where v represents the view direction.

[0102] Specifically, the learnable environment map is obtained by a neural network and Gaussian points. A reflection color map is calculated using the normal map and the environment map. The reflection intensity map can be regarded as a weight map. The reflection color and diffuse color are weighted and summed by reflection intensity to obtain the final pixel color. The resolution of the learnable environment map E is selected according to the complexity of the scene and the rendering accuracy requirements to ensure the realism of the reflection effect.

[0103] It should be noted that, through the above steps and parameter settings, this invention can achieve highly realistic novel view synthesis and geometric reconstruction, especially excelling in processing objects with reflective surfaces, effectively improving rendering quality and geometric accuracy, while also enhancing multi-view capabilities. Figure 1 Consistency and rendering efficiency.

[0104] This invention improves rendering quality by using semantic feature distillation and an implicit material dictionary to more accurately represent the material properties of objects, effectively solving the rendering problem of reflective surfaces. Especially when dealing with scenes with significant reflective surfaces, the rendering quality is significantly better than traditional methods. It also enhances geometric accuracy by introducing geometric consistency modeling, including multi-view... Figure 1 Consistency and normal-geometric consistency constraints ensure global geometric alignment, improve the accuracy of geometric reconstruction, and enable complete reconstruction of scene geometry with reflective surfaces; they also enhance multi-view capabilities. Figure 1 Consistency is improved by using diffuse color to maintain consistency across multiple views, avoiding appearance changes caused by reflective surfaces and enhancing the stability of the overall rendering effect; rendering efficiency is improved by optimizing the rendering process based on 3D Gaussian point rendering technology, achieving fast rendering while ensuring high-quality visual effects.

[0105] The above is an illustrative scheme of a semantically guided Gaussian point rendering method according to this embodiment. It should be noted that the technical solution of this semantically guided Gaussian point rendering system and the technical solution of the semantically guided Gaussian point rendering method described above belong to the same concept. For details not described in detail in the technical solution of the semantically guided Gaussian point rendering system in this embodiment, please refer to the description of the technical solution of the semantically guided Gaussian point rendering method described above.

[0106] Example 2: This example provides a semantically guided Gaussian point rendering system, including:

[0107] The acquisition module is used to acquire multi-view images;

[0108] The first processing module is used to obtain the corresponding 2D image features based on the multi-view image, associate them with 3D Gaussian points, construct a semantic feature regularization loss function, and distill the semantic features of the 3D Gaussian points.

[0109] The second processing module is used to extract the semantic features and location information of each Gaussian point after distillation, generate comprehensive appearance features, and map the appearance features to material properties to obtain a material property map.

[0110] The third processing module is used to evaluate the consistency between the distorted image and the target diffuse reflection image based on the material information of the Gaussian points, and at the same time constrain the local consistency of the depth map and normal map of the 3D Gaussian points, so as to optimize and adjust the 3D Gaussian points through consistency loss.

[0111] The output module is used to calculate the final pixel color based on the material property map and normal map, combined with the learnable environment map, to achieve high-fidelity rendering.

[0112] This embodiment also provides a computer device suitable for semantically guided Gaussian point rendering, including:

[0113] The system includes a memory and a processor. The memory stores computer-executable instructions, and the processor executes these instructions to implement the semantically guided Gaussian point rendering method as described in the above embodiments.

[0114] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements the semantically guided Gaussian point rendering method proposed in the above embodiments.

[0115] The storage medium proposed in this embodiment and the semantically guided Gaussian point rendering method proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0116] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0117] Example 3, referring to Figures 7-10 As an embodiment of the present invention, a semantically guided Gaussian point rendering method is provided. To verify its beneficial effects, it is scientifically demonstrated through economic benefit calculations and simulation experiments.

[0118] Reflectance-based Neural Radiance Fields (Ref-NeRF) is an improved neural radiance field method specifically designed to enhance the realism and accuracy of specular surfaces in view synthesis. While traditional NeRF techniques can effectively represent fine geometries with smooth variations, they have limitations when handling smooth surfaces and specular reflections, and are prone to artifacts such as translucency or haziness. Key content includes:

[0119] 1) Using reflected light as the input, Ref-NeRF uses reflected light instead of outward radiation as the way of view parameterization because surface orientation does not affect the reflected light, which allows the multilayer perceptron (MLP) to interpolate better on it.

[0120] 2) Normal vector estimation: Since the reflected light depends on the accurate normal vector, and the geometry generated by the original NeRF is not ideal, resulting in excessive noise in the normal vector, Ref-NeRF adds a new regularizer to the volume density to excite the volume density to concentrate on the surface, thereby obtaining a more accurate normal vector.

[0121] 3) The structured view depends on appearance. To further improve rendering quality, Ref-NeRF uses integrated directional coding, which allows the model to represent the roughness of materials and introduces the concepts of diffuse and specular colors to better simulate the behavior of different materials.

[0122] 4) To enhance the realism of the rendering, Ref-NeRF also significantly improves the quality of the normal vectors through a new volume density adjuster and concentrates the volume density around the surface, enabling the model to calculate accurate reflection vectors and present realistic specular reflections.

[0123] Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images (NeRO) is an advanced technique for reconstructing the geometry and bidirectional reflectance distribution function (BRDF) of reflective objects from multiview images. Presented at SIGGRAPH 2023, this work aims to accurately recover the true 3D shape and surface properties of smooth or specular reflective objects from photographs taken from a series of different viewpoints. NeRO combines neural networks and traditional computer vision techniques to achieve high-quality reconstruction of reflective objects. Its main features include:

[0124] 1) Simultaneous reconstruction of geometry and material: Unlike traditional methods that only focus on geometric reconstruction, NeRO can simultaneously estimate the three-dimensional shape of an object and the optical properties of its surface (i.e., BRDF), which is especially important for reflective objects, as their appearance is strongly dependent on the viewing angle and lighting conditions.

[0125] 2) Multi-view image utilization: By using input images from multiple perspectives, NeRO can capture the details of objects more comprehensively and can handle complex reflection phenomena, such as specular highlights and shadows.

[0126] 3) Deep learning framework: The method uses a deep learning model to learn the mapping relationship from images to geometric shapes and material properties. This gives the method strong generalization ability and adaptability, and it can cope with various reflective materials and scene settings.

[0127] 3D Gaussian Splatting is a technique for real-time radiance field rendering. It represents the scene using a 3D Gaussian distribution and renders it using rasterization. This method combines the speed advantages of traditional explicit representations (such as meshes and point clouds) with the quality advantages of implicit representations (such as NeRF), providing a novel perspective compositing solution. Here are some key aspects of this method:

[0128] 1) 3D Gaussian Representation: Each 3D Gaussian can be viewed as an ellipsoid with position, color, transparency, and covariance matrix. These properties define the shape, size, and appearance of the Gaussian in space.

[0129] 2) Splatting technique: Projecting 3D Gaussians onto a 2D image plane to create a "splattering" effect. This process involves calculating the projection of each Gaussian onto the screen and blending them based on their transparency and color.

[0130] 3) Differentiable rendering: The parameters of a 3D Gaussian can be optimized through backpropagation to minimize reconstruction errors. This allows for efficient learning and adjustment of the scene.

[0131] The Gaussian Shader extends 3D Gaussian Splatting (3DGS) in neural rendering, enabling it to efficiently render scenes containing reflective surfaces while maintaining real-time rendering performance. Traditional 3DGS has limitations in handling view-dependent phenomena such as specular highlights and reflections, while the Gaussian Shader significantly enhances the realism of the rendering by introducing simplified shading functions and improved geometric property handling.

[0132] 1) Simplified Shading Functions: Instead of directly applying complex physically based rendering equations, Gaussian Shader introduces a simplified shading model for each 3D Gaussian to compensate for complex lighting effects that the model fails to fully capture. This decomposition allows the model to learn and represent reflection properties more explicitly.

[0133] 2) Differentiable ambient lighting: To simulate the capture of ambient light by reflective surfaces, GaussianShader integrates a differentiable ambient lighting representation, typically a learnable environment map (such as a cubemap). This allows the model to jointly optimize the scene representation and ambient lighting during training, thereby better capturing reflections.

[0134] 3) Extended Gaussian properties: In addition to the original position, covariance and opacity, GaussianShader adds extra "shading properties" to each 3D Gaussian to store and compute the parameters required for the simplified shading function, such as diffuse color, specular parameters, etc.

[0135] 3D Gaussian Splatting with DeferredReflection is an innovative extension to the 3D Gaussian Splatting (3DGS) method, designed to address its challenges in rendering view-dependent effects such as specular highlights and specular reflections. This method introduces deferred shading and incorporates a dedicated normal propagation mechanism during training, significantly improving 3DGS's ability to model reflective surfaces while maintaining its core real-time rendering advantage.

[0136] 1) Multi-stage rendering, Gaussian rendering process: In this stage, 3D Gaussian points are projected and rasterized into screen space, generating a series of geometric buffers to store basic information related to Gaussians for each pixel.

[0137] 2) Pixel Shading Process: In this stage, for each pixel in screen space, shading calculations are performed using information stored in G-buffers. The method uses normal information to calculate the reflection direction and queries a learnable environment map to obtain the ambient light color in that reflection direction, i.e., the specular reflection color. The final pixel color is obtained by mixing the base color and the specular reflection color according to the reflection intensity. By delaying the shading calculation to the pixel level, this method can more accurately control the generation of specular reflections, avoiding the difficulty of directly simulating high-frequency reflections using SH in the Gaussian Splatting stage.

[0138] 3) Normal Propagation and Optimization. Accurate normals are crucial for achieving high-quality specular reflections. Directly optimizing normals is challenging due to the discreteness and potential noise inherent in 3D Gaussians. This method introduces a normal propagation training algorithm: in the early stages of training, the model may primarily focus on view-independent color fitting.

[0139] As training progresses, the normal estimation becomes relatively more accurate for Gaussians with higher reflection intensity and whose color changes are more affected by the viewing angle. This method utilizes pixel-level reflection gradients calculated by deferred shading to propagate this relatively accurate normal information to neighboring Gaussians. By encouraging normal consistency among neighboring Gaussians, the normal estimation of the entire reflective surface gradually improves and stabilizes.

[0140] Planar-based Gaussian Splatting Reconstruction (PGSR) is an open-source project for efficient and high-fidelity surface reconstruction from multi-view RGB images. Proposed by researchers from Zhejiang University and other institutions, it aims to improve the accuracy of 3D object surface reconstruction techniques. Key features include:

[0141] 1) Planar-based Gaussian Splatting Representation: PGSR utilizes planar-based Gaussian splatting to achieve accurate reconstruction of object surfaces. This method does not rely on any geometric priors (such as depth or normals of pre-trained models), but instead approximates the local properties of the object surface through a Gaussian distribution.

[0142] 2) Unbiased Depth Rendering Method: PGSR proposes a novel unbiased depth rendering method that allows each pixel to be rendered with reliable planar parameters. This helps to incorporate various geometric constraints, thereby improving reconstruction accuracy.

[0143] Single-view and multi-view regularization methods: In order to optimize the planar parameters of each pixel and achieve high-precision global geometric consistency, PGSR introduces single-view and multi-view regularization methods.

[0144] 3) Exposure compensation: Simple exposure compensation strategies are used to further improve reconstruction quality.

[0145] 4) High performance and efficiency: Compared with other state-of-the-art reconstruction methods based on NeRF, PGSR not only achieves the best reconstruction accuracy to date, but also has a training time that is nearly 100 times faster, requiring only one hour of training on a single GPU.

[0146] The comparative experiment was conducted under the following conditions:

[0147] Datasets: The Shiny Blender and Glossy Synthetic datasets, which are widely used for reflective object datasets, were employed.

[0148] The evaluation indicators are as follows:

[0149] Rendering quality is assessed using Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Patch Similarity (LPIPS).

[0150] Geometric reconstruction quality is assessed using the chamfer distance.

[0151] Implementation process: Training was conducted on a single Nvidia RTX 3090 equipped with 24GB of memory, with a fixed training step count of 30,000 steps.

[0152] The beneficial effects of the experiment are as follows:

[0153] Rendering quality comparison (Shiny Blender dataset), such as Figure 7 As shown, the rendering performance comparison results of this invention on the Shiny Blender dataset demonstrate that it outperforms existing state-of-the-art methods such as Ref-NeRF, NeRO, original 3DGS, GShader, and 3DGS-DR. It exhibits particularly superior performance when handling scenes with highly reflective surfaces.

[0154] Convergence speed comparison, such as Figure 8 As shown, the invention compares the convergence speed with 3DGS-DR. Thanks to its advanced semantic understanding of the scene, the invention demonstrates a significantly accelerated convergence speed.

[0155] Geometric reconstruction quality comparison (Glossy Synthetic dataset), such as Figure 9 As shown, the reconstruction performance of this invention on the GlossySynthetic dataset is compared. The results show that, compared with the state-of-the-art Gaussian point (GS) based surface reconstruction method PGSR, this invention shows a significant improvement when processing reflective object datasets.

[0156] like Figure 10 As shown in the geometric reconstruction diagram, the present invention successfully restored the complete scene geometry, while PGSR has difficulty in processing large-area reflective surface scenes, resulting in the inability to reconstruct accurate geometric shapes.

[0157] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A semantically guided Gaussian point rendering method, characterized in that, include: Obtain multi-view images; Based on the multi-view image, the corresponding 2D image features are obtained and associated with 3D Gaussian points. A semantic feature regularization loss function is constructed to distill the semantic features of the 3D Gaussian points. The semantic features and positional information of each Gaussian point after distillation are extracted to generate a comprehensive appearance feature, and the appearance feature is mapped to the material property to obtain a material property map. Based on the material information of the Gaussian points, a consistency evaluation is performed between the distorted image and the target diffuse reflection image, and the local consistency of the depth map and normal map of the 3D Gaussian points is constrained at the same time, so as to optimize and adjust the 3D Gaussian points through consistency loss. Based on the material property map and normal map, combined with the learnable environment map, the final pixel color is calculated to achieve high-fidelity rendering.

2. The semantically guided Gaussian point rendering method as described in claim 1, characterized in that, Obtaining the corresponding 2D image features and associating them with 3D Gaussian points includes: The multi-view images are input into a pre-trained 2D base model to obtain the corresponding 2D feature maps; By using differentiable rendering technology, 3D Gaussian points are projected onto the image plane of the 2D feature map, the feature vector of each pixel is calculated, and the 2D image features are associated with the 3D Gaussian points. The feature vector of each pixel is calculated by multiplying the Gaussian opacity by the weighted sum of the features.

3. The semantically guided Gaussian point rendering method as described in claim 2, characterized in that, Constructing a semantic feature regularization loss function to distill the semantic features of the 3D Gaussian points includes: The initial high-dimensional features of the 2D feature map are compressed using a convolutional encoder, and then the dimensions are restored using a decoder. A semantic feature regularization loss function is constructed, the difference between the decoded features and the initial high-dimensional features is calculated, and the difference between the two is reduced by the gradient descent algorithm to complete the semantic feature distillation of the 3D Gaussian points.

4. The semantically guided Gaussian point rendering method as described in claim 1, characterized in that, The obtained material property map includes: The semantic features and positional information of each Gaussian point are extracted, and then concatenated and transformed to generate a comprehensive appearance feature. The semantic features are obtained from the distillation process, and the positional information is the coordinates of the Gaussian point in 3D space. The appearance features are mapped to material properties through an implicit material dictionary to obtain a material property map, which includes a diffuse color map, a reflection intensity map, and a residual color component map.

5. The semantically guided Gaussian point rendering method as described in claim 4, characterized in that, The consistency assessment between the distorted image and the target diffuse image includes: The source frame pixels are mapped to the reference frame using a homeomorphic matrix, and a photometric multi-view consistency constraint based on planar patching is adopted. The consistency between the distorted image and the target diffuse image is evaluated using the diffuse color map. A multi-view consistency loss function is constructed. Based on the calculated consistency difference, the properties of Gaussian points are optimized to reduce the consistency difference through backpropagation of gradient descent.

6. The semantically guided Gaussian point rendering method as described in claim 5, characterized in that, Constraining the local consistency of the depth map and normal map of the 3D Gaussian points includes: A depth map is calculated from a viewpoint using the 3D Gaussian points, and the depth map is converted into a gradient normal map using the finite difference method. By minimizing the difference between the gradient normal of the rendered depth map and the rendered normal map, the local consistency of depth and normal is constrained. A normal-geometric consistency loss function is constructed, and the cosine similarity difference and absolute error between the two are calculated. The 3D Gaussian points are then optimized through gradient descent backpropagation.

7. The semantically guided Gaussian point rendering method as described in claim 6, characterized in that, Achieving high-fidelity rendering includes: By constructing the diffuse color map, reflection intensity map, and surface discovery map obtained by optimizing the 3D Gaussian points, and combining them with the learnable environment map, the final pixel color is calculated through the deferred reflection channel to complete the high-fidelity rendering of deferred shading.

8. A semantically guided Gaussian point rendering system, employing the semantically guided Gaussian point rendering method as described in any one of claims 1 to 7, characterized in that, include, The acquisition module is used to acquire multi-view images; The first processing module is used to obtain the corresponding 2D image features based on the multi-view image, associate them with 3D Gaussian points, construct a semantic feature regularization loss function, and distill the semantic features of the 3D Gaussian points. The second processing module is used to extract the semantic features and positional information of each Gaussian point after distillation, generate comprehensive appearance features, and map the appearance features to material properties to obtain a material property map. The third processing module is used to evaluate the consistency between the distorted image and the target diffuse reflection image based on the material information of the Gaussian point, and simultaneously constrain the local consistency of the depth map and normal map of the 3D Gaussian point, so as to optimize and adjust the 3D Gaussian point through consistency loss. The output module is used to calculate the final pixel color based on the material property map and normal map, combined with the learnable environment map, in order to achieve high-fidelity rendering.

9. A computer device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the semantically guided Gaussian point rendering method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of a semantically guided Gaussian point rendering method as described in any one of claims 1 to 7.

Images