A real-time three-dimensional rendering method, device and medium based on reflection perception

Abstract

The application relates to the technical field of three-dimensional scene reconstruction, and discloses a real-time three-dimensional rendering method and device based on reflection perception and a medium, the method comprising the following steps: recovering a point cloud from an input image, defining each point as a three-dimensional Gaussian primitive; generating a rendering normal vector map and a reflection intensity map; extracting a depth map from the input image, calculating a standard normal vector map based on the depth map; generating a mask based on the reflection intensity map to identify a highlight area; calculating the specular reflection color of each pixel based on the rendering normal vector map and an ambient light map; weighting and fusing the diffuse reflection color and the specular reflection color of each pixel to output a final pixel color; constructing a joint loss function, and optimizing and training a model by using a gradient descent algorithm, so that high-quality real-time three-dimensional scene reconstruction is finally realized. By introducing the reflection modeling and normal supervision mechanism, the rendering robustness and generalization ability of the system under a complex light and reflection environment are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional scene reconstruction technology, specifically to a real-time three-dimensional rendering method, device, and medium based on reflection perception. Background Technology

[0002] With the rapid development of multiple disciplines such as computer vision, computer graphics, robotics, and artificial intelligence, 3D reconstruction technology is gradually becoming a key foundation supporting numerous real-world applications. 3D reconstruction aims to recover the geometric structure and surface properties of objects or scenes from images, point clouds, or multimodal data, providing core support for subsequent tasks such as recognition, understanding, and interaction.

[0003] In the real world, reflective objects are widespread, such as glassware, stainless steel utensils, mirrored surfaces, and liquid surfaces. Specular reflection and highlight areas are particularly common in typical scenarios like chemical laboratories, home kitchens, museum exhibits, and automated industrial settings. These high-gloss materials exhibit strong specular reflection, highlight spots, and significant view-dependent appearance changes in images, severely interfering with the modeling process of traditional 3D reconstruction methods based on image consistency, feature matching, and volume integration. This often leads to geometric distortion, texture ghosting, and even reconstruction failure. For a long time, mainstream reconstruction methods have generally adopted the diffuse reflection assumption, which assumes that the reflection of an object's surface is consistent across different viewpoints. However, this assumption clearly no longer holds true when dealing with specular reflection scenarios, causing problems such as unstable apparent colors and difficulty in matching feature points, ultimately resulting in geometric drift, sparse point clouds, and discontinuous reconstructed structures.

[0004] Therefore, effectively separating specular reflection components and improving the geometric recovery capability of reflective areas have become key issues that urgently need to be addressed in the 3D reconstruction of reflective scenes. Currently, many high-quality reconstruction methods still perform well primarily on synthetic data or in controlled experimental environments, making it difficult to generalize to complex scenes containing real reflective materials. Conducting research on the 3D reconstruction of reflective scenes can effectively compensate for the limitations of existing technologies in real-world applications, and promote the leap from idealized conditions to realistic, complex environments in 3D reconstruction. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a real-time 3D rendering method, apparatus, and medium based on reflection perception. This invention attempts to introduce a physical reflection modeling mechanism and combine it with neural rendering and perception strategies to improve the algorithm's ability to model reflection scenes and expand the range of expression of complex materials by the visual system.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a real-time 3D rendering method based on reflection perception, wherein the construction and training process of the rendering model includes:

[0008] Recover the point cloud from the input image, defining each point as a 3D Gaussian primitive;

[0009] Extend the standard differential rasterization pipeline to simultaneously generate rendering normal maps and reflection intensity maps;

[0010] A depth map is extracted from the input image using a pre-trained depth estimation network, and a standard normal map is calculated based on the depth map to supervise the rendering of the normal map.

[0011] A mask is generated based on the reflection intensity map to identify the highlight areas, and a specular highlight loss is constructed to enhance the highlight areas.

[0012] Based on the rendering normal map and ambient light map, the specular reflection color of each pixel is calculated using the specular reflection law;

[0013] The three-dimensional Gaussian primitives are projected onto the two-dimensional imaging plane, and the diffuse color of each pixel and the specular color are weighted and fused according to the reflection intensity map to output the final pixel color.

[0014] A multi-stage optimization strategy is adopted to train the rendering model, including a normal field construction stage based on normal consistency loss and image reconstruction loss, a global optimization stage based on appearance consistency loss, and a stage for enhancing specular region details based on image reconstruction loss and specular specular loss.

[0015] In one embodiment, the step of recovering the point cloud from the input image, defining each point as a three-dimensional Gaussian unit, specifically includes:

[0016] Point clouds are extracted from the input image using motion structure recovery techniques, and the points in the point cloud are defined as three-dimensional Gaussian primitives with volumetric density:

[0017] ;

[0018] In the formula, Representing a three-dimensional Gaussian element, This represents the center position of each 3D Gaussian element in 3D space. The covariance matrix of the three-dimensional Gaussian elements defines the shape and orientation of the distribution of the three-dimensional Gaussian elements in space. Indicates the location of the query point;

[0019] Decompose the covariance matrix into a combination of rotation and scaling matrices:

[0020] ;

[0021] In the formula, Represents the rotation matrix. Represents the scaling matrix. This is a transpose.

[0022] In one embodiment, the process of generating the rendering normal vector map specifically includes:

[0023] The direction corresponding to the shortest principal axis in the covariance matrix of each 3D Gaussian element is defined as the normal vector of that 3D Gaussian element. During rendering, when a 3D Gaussian element is projected onto a 2D image plane, multiple Gaussians will affect the same pixel. The normal vectors of all 3D Gaussian elements overlapping with the current pixel are weighted and fused to obtain the rendering normal vector of the current pixel. This leads to the rendering normal vector map composed of the rendering normal vectors of all pixels:

[0024] ;

[0025] in, This represents the total number of 3D Gaussian pixels overlapping with the current pixel. For the index of the i-th three-dimensional Gaussian element, Let be the normal vector of the i-th 3D Gaussian element. for The corresponding weighting coefficients.

[0026] In one embodiment, the step of extracting a depth map from the input image using a pre-trained depth estimation network and calculating a standard normal map based on the depth map for supervising the rendering of the normal map specifically includes:

[0027] A pre-trained depth estimation network is used to extract depth maps from the input image. In conjunction with known camera parameters, each pixel is back-projected into a three-dimensional spatial coordinate point using a perspective projection model. , These are the x-coordinate, y-coordinate, and z-coordinate values, respectively.

[0028] Constructing surfaces using depth maps And the normal vector of a point Q on the surface is located as the normal direction of the tangent plane at that point:

[0029] ;

[0030] Calculate pixels in the image plane The gradient of the 3D coordinates of the adjacent pixels is used to construct a local tangent plane. Let Q be the projection of point Q onto the image plane; and use the normal of the local tangent plane as the pixel. The unit normal vector is used to obtain the complete pixel-level normal vector map:

[0031] ;

[0032] Normalize the pixel-level normal vector map to obtain the standard normal vector. The formula is:

[0033] ;

[0034] Represents the magnitude of a vector;

[0035] A normal vector supervision mechanism is introduced to drive the rendering model to learn local shape representations by minimizing the difference between the rendered normal map and the standard normal map.

[0036] ;

[0037] This indicates the loss of normal uniformity. This represents the rendering normal vector.

[0038] In one embodiment, the step of generating a mask based on the reflection intensity map to identify highlight regions and constructing a specular highlight loss to enhance the highlight regions specifically includes:

[0039] Introducing a mask Identify the high-brightness areas in the reflectance intensity diagram:

[0040] ;

[0041] In the formula, Represents the pixel position in the reflection intensity map The reflection intensity value at that location, The set threshold;

[0042] Apply the mask to both the input and rendered images to enhance the highlight areas:

[0043] ; ;

[0044] In the formula, Indicates the input image. This represents the enhanced input image. Indicates the rendered image. This represents the enhanced rendered image. This represents element-wise multiplication;

[0045] Build based on loss and structural similarity loss Specular highlight loss Supervision:

[0046] ;

[0047] These are the weighting coefficients.

[0048] In one embodiment, the step of calculating the specular reflection color of each pixel based on the rendering normal map and ambient light map using the specular reflection law specifically includes:

[0049] Construct a basic reflection model and, based on the specular reflection law, calculate the reflection of any pixel in the input image. Direction of reflected light :

[0050] ;

[0051] In the formula, The direction of the incident light. For pixels Surface normal vector;

[0052] Pixels are calculated using ambient ray mapping based on the direction of reflected light. Specular reflection color :

[0053] ;

[0054] In the formula, This describes the process of querying the ambient ray map using a bilinear interpolation strategy.

[0055] In one embodiment, the step of projecting three-dimensional Gaussian pixels onto a two-dimensional imaging plane, and weighting and fusing the diffuse color and specular color of each pixel according to the reflection intensity map to output the final pixel color, specifically includes:

[0056] The three-dimensional Gaussian elements are projected onto the two-dimensional imaging plane using an elliptic weighted average sputtering method:

[0057] ;

[0058] In the formula, The two-dimensional Gaussian is obtained by projecting a three-dimensional Gaussian element onto a two-dimensional imaging plane. and Let represent the two-dimensional mean and covariance matrix after the projection of the three-dimensional Gaussian elements, respectively;

[0059] Pixels are calculated using alpha mixing. diffuse color :

[0060] ;

[0061] ;

[0062] In the formula, Indicates the relationship with the current pixel The number of overlapping 3D Gaussian elements. and They represent the first A three-dimensional Gaussian color vector and opacity; Indicates the first The final opacity of a three-dimensional Gaussian element;

[0063] A delayed reflection strategy is employed, and the final pixel color is calculated through weighted fusion. To effectively integrate the contributions of diffuse reflection and specular reflection:

[0064] ;

[0065] in, Represents the reflection intensity map. The color is the color reflected by a specular surface.

[0066] In one embodiment, the multi-stage optimization strategy for training the rendering model includes a normal field construction stage based on normal consistency loss and image reconstruction loss, a global appearance consistency optimization stage based on image reconstruction loss, and a specular region detail enhancement stage based on image reconstruction loss and specular specular loss, specifically including:

[0067] Loss function during the normal field construction phase ; Image reconstruction loss: , For balance parameters; for loss, For structural similarity loss; Loss due to normal consistency: ; To render the normal vectors, It is the standard normal vector; The weight for the normal consistency loss;

[0068] Loss function in the global optimization phase of appearance consistency Image reconstruction loss ;

[0069] Loss function in the highlight detail enhancement stage : ; For specular highlight loss The weight.

[0070] In a second aspect, the present invention provides an apparatus comprising:

[0071] At least one processor;

[0072] At least one memory for storing at least one program;

[0073] When the at least one program is executed by the at least one processor, the method of any one embodiment of the first aspect can be implemented.

[0074] Thirdly, the present invention provides a storage medium storing processor-executable instructions, which, when executed by a processor, are used to implement the method of any embodiment of the first aspect.

[0075] Compared with the prior art, the beneficial technical effects of the present invention are:

[0076] This invention employs a depth-map-guided standard normal supervision mechanism to drive the model to learn more accurate local shape representations. This effectively guides the normal vectors generated during rendering to conform to real geometry at the pixel level, significantly improving local detail accuracy. Furthermore, this invention designs an efficient specular enhancement strategy, significantly improving the reconstruction accuracy of specular specular regions and effectively mitigating visual distortion caused by detail loss or blurring effects. It effectively improves the realism and visual quality of reflective areas without requiring explicit material modeling. Through a phased training strategy, this invention focuses on three aspects: geometric structure restoration, global appearance optimization, and specular detail enhancement. This not only achieves collaborative modeling of geometry and lighting but also improves rendering quality globally. Attached Figure Description

[0077] Figure 1 This is a flowchart of the method in an embodiment of the present invention.

[0078] Figure 2 This is an overall architecture diagram of an embodiment of the present invention. Detailed Implementation

[0079] The technical solutions in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the embodiments described below are merely some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of the present invention.

[0080] Please refer to Figure 1This invention discloses a real-time 3D rendering method based on reflection perception, which specifically includes the following steps:

[0081] Step S1: Recover the point cloud from the input image and define each point as a 3D Gaussian primitive.

[0082] In this embodiment, three-dimensional Gaussian primitives are used to model point clouds in space, combining the advantages of neural implicit fields and point-based rendering methods. This achieves the high-fidelity rendering quality of the former while maintaining the real-time rendering capability of the latter. Specifically, it uses a large number of three-dimensional anisotropic Gaussians to represent the radiation field and defines the points in the point cloud as three-dimensional Gaussian primitives with volume density.

[0083] In one embodiment, step S1 specifically includes the following steps:

[0084] Step S11: For a set of input images, extract sparse point clouds using motion structure reconstruction, and define the points in the point cloud as three-dimensional Gaussian elements with volumetric density, as shown in the formula:

[0085] ;

[0086] In the formula, This represents the center position (mean) of each 3D Gaussian element in 3D space. Let be the covariance matrix of the 3D Gaussian elements, defining the shape and orientation of the 3D Gaussian elements' distribution in space. Indicates the location of the query point.

[0087] Step S12: Decompose the covariance matrix into a combination of rotation and scaling matrices to ensure the positive semidefiniteness of the covariance matrix. The formula is:

[0088] ;

[0089] In the formula, Represents the rotation matrix. This represents the scaling matrix.

[0090] Step S2: Extend the standard differentiable rasterization process to simultaneously generate rendering normal vector maps and reflection intensity maps. This invention introduces a deferred rendering strategy based on the traditional three-dimensional Gaussian model, calculating the normal vector and reflection intensity simultaneously with the pixel diffuse color.

[0091] In one embodiment, the process of generating the rendering normal vector map specifically includes:

[0092] The direction corresponding to the shortest principal axis in the covariance matrix of each 3D Gaussian element is defined as the normal vector of that 3D Gaussian element. During rendering, when 3D Gaussian elements are projected onto a 2D image plane, multiple Gaussians may affect the same pixel. To obtain the final normal vector at that pixel, this invention performs a weighted fusion of the normal vectors of all overlapping 3D Gaussian elements to obtain the rendering normal vector. The formula is:

[0093] ;

[0094] In the formula, Indicates the relationship with the current pixel The total number of overlapping 3D Gaussian elements, These are the weighting coefficients corresponding to each 3D Gaussian element during the rasterization process, which are usually calculated based on the projection contribution of the 3D Gaussian element at that pixel; Let be the normal vector of the i-th three-dimensional Gaussian element.

[0095] Step S3: Extract a depth map from the input image using a pre-trained depth estimation network, and calculate a standard normal map based on the depth map to supervise the rendering of the normal map.

[0096] In one embodiment, step S3 specifically includes the following steps:

[0097] Step S31: Extract depth maps from the original image using a pre-trained depth estimation network (e.g., Depth Anything V2). In conjunction with known camera parameters, each pixel is back-projected into a three-dimensional spatial coordinate point using a perspective projection model. .

[0098] Step S32: Construct a surface using depth information Furthermore, the normal vector of a point Q on the surface is located as the normal direction of the tangent plane at that point:

[0099] .

[0100] Step S33: Calculate pixels in the image plane The gradient of the 3D coordinates of the adjacent pixels is used to construct a local tangent plane. Let Q be the projection of point Q onto the image plane; and use the normal of the local tangent plane as the pixel. The unit normal vector is used to obtain the complete pixel-level normal vector map:

[0101] .

[0102] Step S34: Normalize the pixel-level normal vector map to obtain the standard normal vector, using the following formula:

[0103] .

[0104] Step S35: Introduce a normal vector supervision mechanism. By minimizing the difference between the rendered normal map and the standard normal map, the model is driven to learn a more accurate local shape representation. The formula is:

[0105] ;

[0106] This represents the loss of normal uniformity.

[0107] Step S4: Generate a mask based on the reflection intensity map to identify the highlight area, and construct a specular highlight loss to enhance the highlight area.

[0108] In one embodiment, step S4 specifically includes:

[0109] Step S41: A significant correlation exists between the reflection intensity map and the highlighted areas in the scene, so a mask is introduced. This is used to identify areas of high brightness in the reflection intensity map, and the formula is:

[0110] ;

[0111] In the formula, Represents the pixel position in the image The reflection intensity value at that location, The threshold is set based on experimental experience and is used to distinguish between areas with strong reflection and areas with weak reflection.

[0112] Step S42: Apply the mask to both the input image and the rendered image to enhance the highlight areas. The formula is:

[0113] ;

[0114] In the formula, Indicates the input image. Indicates the rendered image. This indicates element-wise multiplication.

[0115] Step S43: Construct the loss function and use... loss and structural similarity loss To monitor it, the formula is:

[0116] .

[0117] Step S44: The detail enhancement effect highly depends on the completeness and accuracy of the reflection intensity map. Two strategies are used to improve the overall rendering quality:

[0118] Step S441: The first strategy is to introduce normal supervision in the early stage of training to stabilize the learning process of geometric structure. After the model converges to a certain extent, the specular highlight detail enhancement module is gradually introduced to achieve layer-by-layer optimization of high-frequency information.

[0119] Step S442, the second strategy adopts a staged training process: first, pre-training is performed using only normal supervision to obtain an initial model with good geometric perception capabilities, and then a specular enhancement mechanism is introduced on this basis to further optimize the detail and realism of the mirror area.

[0120] Step S5: Based on the rendering normal map and ambient light map, calculate the specular reflection color of each pixel using the specular reflection law.

[0121] In one embodiment, step S5 specifically includes:

[0122] Step S51: Construct a basic reflection model and calculate the reflection of any pixel in the image according to the specular reflection law. The direction of the reflected light is given by the formula:

[0123] ;

[0124] In the formula, The direction of the reflected light. The direction of the incident light. The surface normal vector of this pixel.

[0125] Step S52: Calculate pixels using ambient ray mapping based on the direction of reflected light. Specular reflection color The formula is:

[0126] ;

[0127] In the formula, This describes the process of querying the ambient ray map using a bilinear interpolation strategy.

[0128] Step S6: Project the three-dimensional Gaussian primitives onto the two-dimensional imaging plane, and perform weighted fusion of the diffuse reflection color and the specular reflection color of each pixel according to the reflection intensity map to output the final pixel color.

[0129] In one embodiment, step S6 specifically includes:

[0130] Step S61: Project the three-dimensional Gaussian elements from step S11 onto the two-dimensional imaging plane using the elliptical weighted average sputtering method, with the following formula:

[0131] ;

[0132] In the formula, and Let represent the two-dimensional mean (location) and covariance matrix after projection of the three-dimensional Gaussian distribution, respectively.

[0133] Step S62: Calculate pixels using alpha mixing. diffuse color The formula is:

[0134] ;

[0135] ;

[0136] In the formula, Indicates the relationship with the current pixel The number of overlapping 3D Gaussian elements. and They represent the first A three-dimensional Gaussian color vector and opacity. Indicates the first The final opacity of a 3D Gaussian primitive. The color vector is represented using spherical harmonic coefficients to model viewpoint-dependent color variations, while the opacity controls the contribution of this 3D Gaussian primitive to the final image rendering result.

[0137] Step S63: Using a delayed reflection strategy, calculate the final pixel color through weighted fusion. To effectively integrate the contributions of diffuse reflection and specular reflection, the formula is:

[0138] ;

[0139] In the formula, where, This represents a reflection intensity map, used to measure the weight of specular reflection at a pixel location.

[0140] Step S7: The rendering model is trained using a multi-stage optimization strategy, including a normal field construction stage based on normal consistency loss and image reconstruction loss, a global optimization stage based on appearance consistency loss based on image reconstruction loss, and a stage for enhancing the details of the specular region based on image reconstruction loss and specular specular loss, ultimately achieving high-quality, real-time 3D scene reconstruction.

[0141] In one embodiment, step S7 specifically includes:

[0142] Step S71: Depth Information-Guided Normal Field Construction Stage. This stage aims to eliminate surface ambiguity and establish a high-precision normal field through geometric constraints. This invention introduces a normal consistency loss based on the pixel-by-pixel normal field derived from the depth map to ensure that the rendered normal map maintains angular consistency with the true normal direction. Simultaneously, this normal consistency loss accelerates the convergence process of the normal map through a gradient backpropagation mechanism. The image reconstruction part combines... loss and structural similarity loss To jointly optimize the rendered image. Therefore, the loss function in the normal field construction stage. It can be represented as:

[0143] ;

[0144] ;

[0145] In the formula, It is a balance parameter. The weights for the normal consistency loss are... For image reconstruction loss, This is the loss due to the uniformity of the normal lines.

[0146] Step S72: Global Optimization Stage for Appearance Consistency. This stage aims to further improve the realism and consistency of image rendering. Based on the optimization in Step S71, the normal map has basically converged, and the reflection intensity map has also reached a stable state. At this point, the image reconstruction loss continues to be used as the optimization objective, but no additional geometric constraints are added, emphasizing the fitting and preservation of appearance details. The loss function for the global optimization stage of appearance consistency is as follows. The definition is as follows:

[0147] .

[0148] Step S73: Detail Enhancement Stage for Highlight Areas. To address issues such as blurring, distortion, and artifacts in specular highlight areas, a mask of the reflection intensity map is introduced. The pixel regions with high reflectance values ​​in the image are marked. Based on this, a weighted loss mechanism is used to optimize the highlight regions. The loss function and structural similarity loss constrain the rendering accuracy of this region. Simultaneously, leveraging existing gradient propagation paths, the sharpness of highlight edges is further enhanced, suppressing overexposure artifacts; the loss function in the highlight region detail enhancement stage... :

[0149] ;

[0150] In the formula, This represents the weight of specular highlight loss.

[0151] See the overall architecture of this invention. Figure 2 . Figure 2 SH represents the spherical harmonic function, used to model viewpoint-dependent color changes.

[0152] To verify the effectiveness of this invention, this implementation selected three advanced 3D reconstruction methods: 3DGS (3D Gaussian Splatting), GShader (Gaussian Shader), and 3DGS-DR (3D Gaussian Splatting with Deferred Reflection). The models were trained on a public dataset and a self-made dataset to evaluate the performance of different methods. The two datasets were the Ref-Real dataset and the Smart Chemistry Lab dataset. Their evaluation metrics were Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Patch Similarity (LPIPS). The experimental results are shown in Table 1.

[0153] Table 1 shows the experimental results of the method of the present invention and the selected comparative method on two 3D reconstruction datasets.

[0154]

[0155] Experimental results show that the method of this invention (Ours), due to its specialized modeling and processing of specular reflections in the scene, can more effectively restore the geometric details of the real scene. This invention can accurately recover clear and structured edge information, effectively avoiding common problems such as blurring and artifacts, significantly improving the reconstruction quality in reflective environments, and making the final rendering result more realistic and visually consistent. Furthermore, this invention achieves superior performance compared to existing methods on two different types of 3D reconstruction datasets, verifying its versatility and advancement under complex lighting and reflection conditions. In Table 1, upward arrows for each evaluation index indicate that larger values ​​result in better performance, while downward arrows indicate that smaller values ​​result in better performance. Ours (Origin) and Ours (Pre-mode) are two strategies of this invention. The Origin strategy introduces normal supervision in the early stages of training to stabilize the learning process of geometric structures. After the model converges to a certain extent, a specular highlight detail enhancement module is gradually introduced to achieve layer-by-layer optimization of high-frequency information. The pre-mode strategy adopts a staged training process: first, pre-training is performed using only normal supervision to obtain an initial model with good geometric perception capabilities; then, a specular enhancement mechanism is introduced on this basis to further optimize the detail and realism of the mirror area.

[0156] In one embodiment, the present invention provides an apparatus, further comprising:

[0157] It contains at least one high-performance processor core, which has powerful data processing and computing capabilities and can efficiently execute complex algorithms;

[0158] It is equipped with at least one high-speed, high-capacity memory, which is not only used to store massive amounts of data, but also designed to have enough space to store at least one written program;

[0159] When the written program is executed accurately by at least one of the aforementioned high-performance processors, it activates a series of complex computational processes, enabling the processor to intelligently implement the aforementioned real-time 3D rendering method based on reflection perception. By deeply mining the geometric features in the image and combining them with a strategy of enhancing highlight areas, the rendering model is continuously optimized, thereby achieving high-precision reconstruction of complex lighting and reflection environments, providing core support for subsequent tasks such as recognition and interaction.

[0160] The content of the above method embodiments is applicable to the device embodiments of the present invention. The specific functions implemented by the device embodiments of the present invention are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0161] In one embodiment, the present invention provides a storage medium storing processor-executable instructions, which, when executed by a processor, are used to implement the above-described real-time 3D rendering method based on reflection perception.

[0162] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0163] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0164] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A real-time 3D rendering method based on reflection perception, characterized in that, The construction and training process of the rendering model used includes: Recover the point cloud from the input image, defining each point as a 3D Gaussian primitive; Extend the standard differential rasterization pipeline to simultaneously generate rendering normal maps and reflection intensity maps; A depth map is extracted from the input image using a pre-trained depth estimation network, and a standard normal map is calculated based on the depth map to supervise the rendering of the normal map. A mask is generated based on the reflection intensity map to identify the highlight areas, and a specular highlight loss is constructed to enhance the highlight areas. Based on the rendering normal map and ambient light map, the specular reflection color of each pixel is calculated using the specular reflection law; The three-dimensional Gaussian primitives are projected onto the two-dimensional imaging plane, and the diffuse color of each pixel and the specular color are weighted and fused according to the reflection intensity map to output the final pixel color. A multi-stage optimization strategy is adopted to train the rendering model, including a normal field construction stage based on normal consistency loss and image reconstruction loss, a global optimization stage based on appearance consistency loss, and a stage for enhancing specular region details based on image reconstruction loss and specular specular loss.

2. The real-time 3D rendering method based on reflection perception according to claim 1, characterized in that, The process of recovering the point cloud from the input image, defining each point as a three-dimensional Gaussian unit, specifically includes: Point clouds are extracted from the input image using motion structure recovery techniques, and the points in the point cloud are defined as three-dimensional Gaussian primitives with volumetric density: ； In the formula, Representing a three-dimensional Gaussian element, This represents the center position of each 3D Gaussian element in 3D space. The covariance matrix of the three-dimensional Gaussian elements defines the shape and orientation of the distribution of the three-dimensional Gaussian elements in space. Indicates the location of the query point; Decompose the covariance matrix into a combination of rotation and scaling matrices: ； In the formula, Represents the rotation matrix. Represents the scaling matrix. This is a transpose.

3. The real-time 3D rendering method based on reflection perception according to claim 1, characterized in that, The process of generating the rendering normal vector map specifically includes: The direction corresponding to the shortest principal axis in the covariance matrix of each 3D Gaussian element is defined as the normal vector of that 3D Gaussian element. During rendering, when a 3D Gaussian element is projected onto a 2D image plane, multiple Gaussians will affect the same pixel. The normal vectors of all 3D Gaussian elements overlapping with the current pixel are weighted and fused to obtain the rendering normal vector of the current pixel. This leads to the rendering normal vector map composed of the rendering normal vectors of all pixels: ； in, This represents the total number of 3D Gaussian pixels overlapping with the current pixel. For the index of the i-th three-dimensional Gaussian element, Let be the normal vector of the i-th 3D Gaussian element. for The corresponding weighting coefficients.

4. The real-time 3D rendering method based on reflection perception according to claim 1, characterized in that, The process of extracting a depth map from the input image using a pre-trained depth estimation network and calculating a standard normal map based on the depth map to supervise the rendering of the normal map specifically includes: A pre-trained depth estimation network is used to extract depth maps from the input image. In conjunction with known camera parameters, each pixel is back-projected into a three-dimensional spatial coordinate point using a perspective projection model. , These are the x-coordinate, y-coordinate, and z-coordinate values, respectively. Constructing surfaces using depth maps And the normal vector of a point Q on the surface is located as the normal direction of the tangent plane at that point: ； Calculate pixels in the image plane The gradient of the 3D coordinates of the adjacent pixels is used to construct a local tangent plane. Let Q be the projection of point Q onto the image plane; and use the normal of the local tangent plane as the pixel. The unit normal vector is used to obtain the complete pixel-level normal vector map: ； Normalize the pixel-level normal vector map to obtain the standard normal vector. The formula is: ； Represents the magnitude of a vector; A normal vector supervision mechanism is introduced to drive the rendering model to learn local shape representations by minimizing the difference between the rendered normal map and the standard normal map. ； This indicates the loss of normal uniformity. This represents the rendering normal vector.

5. The real-time 3D rendering method based on reflection perception according to claim 1, characterized in that, The process of generating a mask based on the reflection intensity map to identify highlight regions and constructing a specular highlight loss to enhance and supervise the highlight regions specifically includes: Introducing a mask Identify the high-brightness areas in the reflectance intensity diagram: ； In the formula, Represents the pixel position in the reflection intensity map The reflection intensity value at that location, The set threshold; Apply the mask to both the input and rendered images to enhance the highlight areas: ； ； In the formula, Indicates the input image. This represents the enhanced input image. Indicates the rendered image. This represents the enhanced rendered image. This represents element-wise multiplication; Build based on loss and structural similarity loss Specular highlight loss Supervision: ； These are the weighting coefficients.

6. The real-time 3D rendering method based on reflection perception according to claim 1, characterized in that, The calculation of the specular reflection color of each pixel based on the rendering normal map and ambient light map, using the specular reflection law, specifically includes: Construct a basic reflection model and, based on the specular reflection law, calculate the reflection of any pixel in the input image. Direction of reflected light : ； In the formula, The direction of the incident light. For pixels Surface normal vector; Pixels are calculated using ambient ray mapping based on the direction of reflected light. Specular reflection color : ； In the formula, This describes the process of querying the ambient ray map using a bilinear interpolation strategy.

7. The real-time 3D rendering method based on reflection perception according to claim 1, characterized in that, The process of projecting three-dimensional Gaussian elements onto a two-dimensional imaging plane, and then weighting and fusing the diffuse and specular colors of each pixel according to the reflection intensity map to output the final pixel color, specifically includes: The three-dimensional Gaussian elements are projected onto the two-dimensional imaging plane using an elliptic weighted average sputtering method: ； In the formula, The two-dimensional Gaussian is obtained by projecting a three-dimensional Gaussian element onto a two-dimensional imaging plane. and Let represent the two-dimensional mean and covariance matrix after the projection of the three-dimensional Gaussian elements, respectively; Pixels are calculated using alpha mixing. diffuse color : ； ； In the formula, Indicates the relationship with the current pixel The number of overlapping 3D Gaussian elements. and They represent the first A three-dimensional Gaussian color vector and opacity; Indicates the first The final opacity of a three-dimensional Gaussian element; A delayed reflection strategy is employed, and the final pixel color is calculated through weighted fusion. To effectively integrate the contributions of diffuse reflection and specular reflection: ； in, Represents the reflection intensity map. The color is the color reflected by a specular surface.

8. The real-time 3D rendering method based on reflection perception according to claim 1, characterized in that, The multi-stage optimization strategy for training the rendering model includes a normal field construction stage based on normal consistency loss and image reconstruction loss, a global optimization stage for appearance consistency based on image reconstruction loss, and a stage for enhancing specular region details based on image reconstruction loss and specular specular loss. Specifically, it includes: Loss function during the normal field construction phase ; Image reconstruction loss: , For balance parameters; for loss, For structural similarity loss; Loss due to normal consistency: ; To render the normal vectors, It is the standard normal vector; The weight for the normal consistency loss; Loss function in the global optimization phase of appearance consistency Image reconstruction loss ; Loss function in the highlight detail enhancement stage : ; For specular highlight loss The weight.

9. An apparatus, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, it enables the implementation of the method as described in any one of claims 1 to 8.

10. A storage medium storing processor-executable instructions, characterized in that: The processor-executable instructions, when executed by the processor, are used to implement the method as described in any one of claims 1 to 8.

Images