Real-time three-dimensional rendering method and device based on reflection perception and medium

Abstract

The invention relates to the technical field of three-dimensional scene reconstruction, and discloses a real-time three-dimensional rendering method and device based on reflection perception and a medium, and the method comprises the steps: recovering a point cloud from an input image, and defining each point as a three-dimensional Gaussian primitive; generating a rendering normal vector diagram and a reflection intensity diagram; extracting a depth map from the input image, and calculating a standard normal vector diagram based on the depth map; generating a mask based on the reflected intensity map to identify a highlight area; calculating the specular reflection color of each pixel based on the rendering normal vector diagram and the ambient light map; performing weighted fusion on the diffuse reflection color and the specular reflection color of each pixel, and outputting a final pixel color; and constructing a joint loss function, performing optimization training on the model by using a gradient descent algorithm, and finally realizing high-quality and real-time three-dimensional scene reconstruction. According to the invention, by introducing a reflection modeling and normal supervision mechanism, the rendering robustness and generalization ability of the system in a complex illumination and reflection environment are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional scene reconstruction, and in particular to a real-time three-dimensional rendering method, device and medium based on reflection perception. Background Art

[0002] With the rapid development of multiple disciplines such as computer vision, computer graphics, robotics, and artificial intelligence, 3D reconstruction technology is becoming a key foundation for numerous real-world applications. 3D reconstruction aims to recover the geometric structure and surface properties of objects or scenes from images, point clouds, or multimodal data, providing core support for subsequent tasks such as recognition, understanding, and interaction.

[0003] In the real world, reflective objects are ubiquitous, such as glassware, stainless steel utensils, mirrored countertops, and liquid surfaces. Specular reflections and highlights are particularly common in typical scenes such as chemical laboratories, home kitchens, museum exhibits, and automated industries. These high-gloss materials exhibit strong specular reflections, high flares, and significant perspective-dependent appearance changes in images, severely interfering with the modeling process of traditional 3D reconstruction methods based on image consistency, feature matching, and volume integration, often leading to geometric distortion, texture ghosting, and even reconstruction failure. For a long time, mainstream reconstruction methods have generally adopted the diffuse reflection assumption, which assumes that the reflection of an object's surface is consistent at different viewpoints. However, this assumption clearly no longer holds true when faced with specular reflection scenes, leading to problems such as unstable apparent color and difficulty matching feature points, ultimately causing geometric drift, sparse point clouds, and discontinuous reconstructed structures.

[0004] Therefore, effectively separating specular components and improving the geometric recovery capabilities of reflective areas have become key issues that need to be addressed in the 3D reconstruction of reflective scenes. Currently, many high-quality reconstruction methods still perform well primarily on synthetic data or in controlled experimental environments, and are difficult to generalize to complex scenes containing real reflective materials. Research on 3D reconstruction of reflective scenes can effectively address the adaptability shortcomings of existing technologies in real-world applications and advance the transition of 3D reconstruction from idealized conditions to complex, realistic environments. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a real-time three-dimensional rendering method, device and medium based on reflection perception; the present invention attempts to introduce a physical reflection modeling mechanism, and combines neural rendering and perception strategies to enhance the algorithm's modeling capabilities for reflective scenes and expand the visual system's expression range for complex materials.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a real-time 3D rendering method based on reflection perception, wherein the construction and training process of the rendering model adopted includes: Recover a point cloud from the input image, defining each point as a 3D Gaussian basis. Extending the standard differentiable rasterization pipeline to simultaneously generate rendered normal maps and reflection intensity maps; Extracting a depth map from an input image through a pre-trained depth estimation network, and calculating a standard normal map based on the depth map for supervising the rendered normal map; Generate a mask based on the reflection intensity map to identify the highlight area, and construct a specular highlight loss to enhance supervision of the highlight area; Based on the rendered normal vector map and the ambient light map, the specular color of each pixel is calculated using the specular reflection law; Projecting the three-dimensional Gaussian primitives onto a two-dimensional imaging plane, weightedly fusing the diffuse reflection color of each pixel with the specular reflection color according to the reflection intensity map, and outputting the final pixel color; A multi-stage optimization strategy is adopted to train the rendering model, including the normal field construction stage based on normal consistency loss and image reconstruction loss, the appearance consistency global optimization stage based on image reconstruction loss, and the highlight area detail enhancement stage based on image reconstruction loss and specular highlight loss.

[0007] In one embodiment, recovering a point cloud from an input image and defining each point as a three-dimensional Gaussian primitive specifically includes: A point cloud is extracted from the input image using structure-from-motion techniques, and the points in the point cloud are defined as 3D Gaussian primitives with volume density: ; Where, represents a three-dimensional Gaussian primitive, represents the center position of each 3D Gaussian primitive in 3D space, is the covariance matrix of the three-dimensional Gaussian basis element, which defines the distribution shape and direction of the three-dimensional Gaussian basis element in space; represents the location of the query point; Decompose the covariance matrix into a combination of a rotation matrix and a scaling matrix: ; Where, represents the rotation matrix, represents the scaling matrix, is transposed.

[0008] In one embodiment, the process of generating the rendered normal vector map specifically includes: The direction corresponding to the shortest principal axis in the covariance matrix of each 3D Gaussian basis element is defined as the normal vector of the 3D Gaussian basis element. During the rendering process, when the 3D Gaussian basis element is projected onto the 2D image plane, multiple Gaussians will affect the same pixel. The normal vectors of all 3D Gaussian basis elements that overlap with the current pixel are weighted and fused to obtain the rendering normal vector of the current pixel. , and then get the rendering normal vector map composed of the rendering normal vectors of all pixels: ; in, Represents the total number of 3D Gaussian primitives overlapping the current pixel, is the index of the i-th three-dimensional Gaussian basis element, is the normal vector of the i-th three-dimensional Gaussian basis element, for The corresponding weight coefficient.

[0009] In one embodiment, extracting a depth map from an input image using a pre-trained depth estimation network and calculating a standard normal map based on the depth map for supervising the rendered normal map specifically includes: Use a pre-trained depth estimation network to extract a depth map from the input image , and combined with the known camera parameters, each pixel is back-projected into a three-dimensional space coordinate point through the perspective projection model , They are x-coordinate value, y-coordinate value, and z-coordinate value respectively; Constructing surfaces using depth maps , and position the normal vector of a point Q on the surface as the normal direction of the tangent plane at that point: ; By counting pixels in the image plane The gradient of the three-dimensional coordinates corresponding to the adjacent pixels of the pixel is used to construct the local tangent plane. is the projection of point Q on the image plane; and the normal of the local tangent plane is used as the pixel The unit normal vector of , thus obtaining a complete pixel-level normal vector map: ; Normalize the pixel-level normal vector map to obtain the standard normal vector , the formula is: ; represents the magnitude of a vector; A normal vector supervision mechanism is introduced to drive the rendering model to learn local shape representation by minimizing the difference between the rendered normal map and the standard normal map: ; represents the normal consistency loss; Represents the rendering normal vector.

[0010] In one embodiment, generating a mask based on the reflection intensity map to identify highlight areas, and constructing a specular highlight loss to enhance supervision of the highlight areas, specifically includes: Introducing a mask Identify areas of high brightness in the reflection intensity map: ; Where, Represents the pixel position in the reflection intensity map The reflection intensity value at is the set threshold; Apply the mask to the input image and the rendered image to enhance the highlight areas: ; ; Where, represents the input image, represents the enhanced input image, represents a rendered image, represents the enhanced rendered image, represents element-wise multiplication; Build based on loss and structural similarity loss Specular highlight loss To conduct supervision: ; is the weight coefficient.

[0011] In one embodiment, the step of calculating the specular color of each pixel based on the rendered normal map and the ambient light map using the specular reflection law specifically includes: Construct a basic reflection model and calculate the reflection of any pixel in the input image according to the law of mirror reflection The direction of reflected light : ; Where, is the incident light direction, Pixels The surface normal vector of Calculate pixels using the ambient light map based on the direction of reflected light Specular color : ; Where, This is the process of querying from the ambient light map using a bilinear interpolation strategy.

[0012] In one embodiment, projecting the three-dimensional Gaussian primitives onto a two-dimensional imaging plane, weightedly fusing the diffuse reflection color of each pixel with the specular reflection color according to the reflection intensity map, and outputting the final pixel color specifically includes: The elliptical weighted average sputtering method is used to project the 3D Gaussian primitives onto the 2D imaging plane: ; Where, is the two-dimensional Gaussian obtained by projecting the three-dimensional Gaussian basis element onto the two-dimensional imaging plane, and They represent the two-dimensional mean and covariance matrix after the three-dimensional Gaussian basis element projection respectively; Calculating pixels using alpha blending Diffuse color : ; ; Where, Represents the current pixel The number of overlapping 3D Gaussian primitives, and Respectively represent The color vector and opacity of the three-dimensional Gaussian primitives; Indicates the The final opacity of the 3D Gaussian primitives; Using a delayed reflection strategy, the final pixel color is calculated by weighted fusion , to effectively integrate the contributions of diffuse and specular reflections: ; in, represents the reflection intensity map, is the specular color.

[0013] In one embodiment, the rendering model is trained using a multi-stage optimization strategy, including a normal field construction stage based on normal consistency loss and image reconstruction loss, a global optimization stage of appearance consistency based on image reconstruction loss, and a highlight area detail enhancement stage based on image reconstruction loss and specular highlight loss, specifically including: Loss function in the normal field construction phase ; is the image reconstruction loss: , is the balance parameter; for loss, is the structural similarity loss; is the normal consistency loss: ; is the rendering normal vector, is the standard normal vector; is the weight of normal consistency loss; Loss function in the global optimization phase of appearance consistency Using image reconstruction loss ; Loss function for highlight area detail enhancement : ; Specular highlight loss The weight of .

[0014] In a second aspect, the present invention provides a device comprising: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the method of any one embodiment of the first aspect can be implemented.

[0015] In a third aspect, the present invention provides a storage medium storing processor-executable instructions, which, when executed by a processor, are used to implement the method of any one embodiment of the first aspect.

[0016] Compared with the prior art, the beneficial technical effects of the present invention are: The present invention uses a standard normal supervision mechanism based on depth map guidance to drive the model to learn more accurate local shape representations, effectively guiding the normal vectors generated during the rendering process to fit the real geometry at the pixel level, significantly improving the accuracy of local details. In addition, the present invention designs an efficient highlight enhancement strategy, which significantly improves the reconstruction accuracy of the specular highlight area, and effectively alleviates the visual distortion problems caused by missing details or blurring effects, and effectively improves the realism and visual quality of the reflection area without explicit material modeling. The present invention focuses on the three aspects of geometric structure recovery, global appearance optimization, and highlight detail enhancement in a phased training strategy, which not only realizes the collaborative modeling of geometry and lighting, but also improves the rendering quality on a global scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 4 is a flow chart of a method in an embodiment of the present invention.

[0018] Figure 22 is an overall architecture diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described below are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of the present invention.

[0020] Please refer to Figure 1 In an embodiment of the present invention, a real-time three-dimensional rendering method based on reflection perception is disclosed, which specifically includes the following steps: Step S1: Recover the point cloud from the input image and define each point as a three-dimensional Gaussian basis.

[0021] In this embodiment, three-dimensional Gaussian primitives are used to model point clouds in space, combining the advantages of neural implicit fields and point-based rendering methods to achieve high-fidelity rendering quality of the former while maintaining the real-time rendering capability of the latter. Specifically, it utilizes a large number of three-dimensional anisotropic Gaussians to represent the radiation field, and defines points in the point cloud as three-dimensional Gaussian primitives with volume density.

[0022] In one embodiment, step S1 specifically includes the following steps: Step S11: For a set of input images, use motion structure recovery to extract a sparse point cloud from them, and define the points in the point cloud as three-dimensional Gaussian primitives with volume density, as follows: ; Where, represents the center position (mean) of each 3D Gaussian primitive in 3D space, is the covariance matrix of the three-dimensional Gaussian basis element, which defines the distribution shape and direction of the three-dimensional Gaussian basis element in space. Represents the location of the query point.

[0023] Step S12: Decompose the covariance matrix into a combination of a rotation matrix and a scaling matrix to ensure the semi-positive definiteness of the covariance matrix. The formula is: ; Where, represents the rotation matrix, Represents the scaling matrix.

[0024] Step S2: Extending the standard differentiable rasterization process to simultaneously generate a rendered normal map and a reflection intensity map. Based on the traditional three-dimensional Gaussian model, the present invention introduces a deferred rendering strategy to calculate the normal vector and reflection intensity simultaneously with the pixel diffuse color.

[0025] In one embodiment, the process of generating a rendering normal map specifically includes: The direction corresponding to the shortest principal axis in the covariance matrix of each 3D Gaussian basis element is defined as the normal vector of the 3D Gaussian basis element. During the rendering process, when the 3D Gaussian basis element is projected onto the 2D image plane, multiple Gaussians may affect the same pixel. In order to obtain the final normal vector at the pixel, the present invention performs a weighted fusion of the normal vectors of all overlapping 3D Gaussian basis elements to obtain the rendering normal vector. , the formula is: ; Where, Represents the current pixel The total number of overlapping 3D Gaussian primitives, The weight coefficient corresponding to each 3D Gaussian primitive in the rasterization process is usually calculated based on the projection contribution of the 3D Gaussian primitive at the pixel; is the normal vector of the i-th three-dimensional Gaussian basis element.

[0026] Step S3: extracting a depth map from the input image through a pre-trained depth estimation network, and calculating a standard normal map based on the depth map for supervising the rendered normal map.

[0027] In one embodiment, step S3 specifically includes the following steps: Step S31: Use a pre-trained depth estimation network (such as Depth Anything V2) to extract a depth map from the original image. , and combined with the known camera parameters, each pixel is back-projected into a three-dimensional space coordinate point through the perspective projection model .

[0028] Step S32: Constructing a surface using depth information , and the normal vector of a point Q on the surface is positioned as the normal direction of the tangent plane at that point: .

[0029] Step S33, by calculating the pixels in the image plane The gradient of the three-dimensional coordinates corresponding to the adjacent pixels of the pixel is used to construct the local tangent plane. is the projection of point Q on the image plane; and the normal of the local tangent plane is used as the pixel The unit normal vector of , thus obtaining a complete pixel-level normal vector map: .

[0030] Step S34: normalize the pixel-level normal vector map to obtain a standard normal vector. The formula is: .

[0031] Step S35: Introduce a normal vector supervision mechanism to drive the model to learn a more accurate local shape representation by minimizing the difference between the rendered normal map and the standard normal map. The formula is: ; represents the normal consistency loss.

[0032] Step S4: Generate a mask based on the reflection intensity map to identify the highlight area, and construct a specular highlight loss to enhance the supervision of the highlight area.

[0033] In one embodiment, step S4 specifically includes: Step S41: There is a significant correlation between the reflection intensity map and the highlight area in the scene, so a mask is introduced. , used to identify the high brightness area in the reflection intensity map, the formula is: ; Where, Represents the pixel position in the image The reflection intensity value at It is a threshold value set based on experimental experience and is used to distinguish strong reflection areas from non-strong reflection areas.

[0034] Step S42: Apply the mask to the input image and the rendered image respectively to enhance the highlight area. The formula is: ; Where, represents the input image, represents a rendered image, Represents element-wise multiplication.

[0035] Step S43: Construct loss function and use loss and structural similarity loss To supervise it, the formula is: .

[0036] Step S44: The detail enhancement effect is highly dependent on the integrity and accuracy of the reflection intensity map. Two strategies are used to improve the overall rendering quality: Step S441: The first strategy is to introduce normal supervision at the beginning of training to stabilize the learning process of the geometric structure. After the model converges to a certain level, the specular highlight detail enhancement module is gradually introduced to achieve layer-by-layer optimization of high-frequency information.

[0037] Step S442, the second strategy adopts a staged training process: first, pre-training is performed using only normal supervision to obtain an initial model with good geometric perception capabilities, and then a highlight enhancement mechanism is introduced on this basis to further optimize the detail performance and realism of the mirror area.

[0038] Step S5: Based on the rendered normal map and the ambient light map, the specular reflection color of each pixel is calculated using the specular reflection law.

[0039] In one embodiment, step S5 specifically includes: Step S51: Construct a basic reflection model and calculate the reflection of any pixel in the image according to the law of mirror reflection. The direction of the reflected light is: ; Where, is the direction of reflected light, is the incident light direction, The surface normal for this pixel.

[0040] Step S52: Calculate the pixel value through the ambient light map according to the direction of the reflected light Specular color , the formula is: ; Where, This is the process of querying from the ambient light map using a bilinear interpolation strategy.

[0041] Step S6: Project the three-dimensional Gaussian primitives onto a two-dimensional imaging plane, perform weighted fusion of the diffuse reflection color of each pixel and the specular reflection color according to the reflection intensity map, and output a final pixel color.

[0042] In one embodiment, step S6 specifically includes: Step S61: Project the three-dimensional Gaussian basis element in step S11 onto the two-dimensional imaging plane using the elliptical weighted average sputtering method. The formula is: ; Where, and They represent the two-dimensional mean (position) and covariance matrix after projection of the three-dimensional Gaussian distribution.

[0043] Step S62: Calculate pixels using alpha blending Diffuse color , the formula is: ; ; Where, Represents the current pixel The number of overlapping 3D Gaussian primitives, and Respectively represent The color vector and opacity of the three-dimensional Gaussian primitives, Indicates the The final opacity of the three-dimensional Gaussian primitive. The color vector is represented by the coefficients of spherical harmonics to model the color changes related to the viewing angle, and the opacity controls the contribution of the three-dimensional Gaussian primitive to the final image rendering result.

[0044] Step S63: Use delayed reflection strategy to calculate the final pixel color through weighted fusion , to effectively integrate the contributions of diffuse and specular reflections, the formula is: ; In the formula, Represents a reflection intensity map that weights the specular reflection at the pixel location.

[0045] Step S7: A multi-stage optimization strategy is used to train the rendering model, including a normal field construction stage based on normal consistency loss and image reconstruction loss, a global optimization stage for appearance consistency based on image reconstruction loss, and a highlight area detail enhancement stage based on image reconstruction loss and specular highlight loss, ultimately achieving high-quality, real-time 3D scene reconstruction.

[0046] In one embodiment, step S7 specifically includes: Step S71, depth information guided normal field construction stage. This stage aims to eliminate surface ambiguity through geometric constraints and establish a high-precision normal field. The present invention introduces a normal consistency loss based on the pixel-by-pixel normal field derived from the depth map to supervise the rendered normal map to maintain the same angle with the true normal direction. At the same time, the normal consistency loss accelerates the convergence process of the normal map through the gradient back propagation mechanism. The image reconstruction part is combined with loss and structural similarity loss To jointly optimize the rendered image. Therefore, the loss function of the normal field construction stage is It can be expressed as: ; ; Where, is the equilibrium parameter, is the weight of the normal consistency loss, is the image reconstruction loss, is the normal consistency loss.

[0047] Step S72: Global optimization of appearance consistency. This stage aims to further improve the realism and consistency of image rendering. Based on the optimization in step S71, the normal map has basically converged and the reflection intensity map has also reached a stable state. At this point, the image reconstruction loss is still used as the optimization target, but no additional geometric constraints are added, emphasizing the fitting and preservation of appearance details. The loss function of the global optimization stage of appearance consistency is: The definition is as follows: .

[0048] Step S73, highlight area detail enhancement stage. In order to solve the problems of blur, distortion and artifacts in the specular highlight area, the mask of the reflection intensity map is introduced. , marking the pixel area with higher reflection value in the image. On this basis, the weighted loss mechanism is used to focus on optimizing the highlight area. The loss and structural similarity loss constrain the rendering accuracy of this area. At the same time, with the help of the existing gradient propagation path, the sharpness of the highlight edge is further enhanced and the generation of overexposure artifacts is suppressed; the loss function of the highlight area detail enhancement stage is: : ; Where, The weight of the specular highlight loss.

[0049] The overall structure of the present invention can be found in Figure 2 . Figure 2 SH represents spherical harmonics, which is used to model the color changes related to viewing angle.

[0050] To validate the effectiveness of this invention, we selected three advanced 3D reconstruction methods: 3DGS (3D Gaussian Splatting), GShader (Gaussian Shader), and 3DGS-DR (3D Gaussian Splatting with Deferred Reflection). We trained these methods on a public dataset and a custom dataset to evaluate the performance of the different models. The two datasets, the Ref-Real dataset and the Intelligent Chemical Laboratory dataset, were used as evaluation metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Patch Similarity (LPIPS). The experimental results are shown in Table 1.

[0051] Table 1. Experimental results of the method of the present invention and the selected comparative method on two 3D reconstruction datasets

[0052] Experimental results demonstrate that the proposed method (Ours) can more effectively restore geometric details in real scenes, thanks to its specialized modeling and processing of specular reflections. It accurately restores clear and structured edge information, effectively avoiding common issues such as blurring and artifacts, significantly improving reconstruction quality in reflective environments and making the final rendering more realistic and visually consistent. Furthermore, the proposed method outperforms existing methods on two different types of 3D reconstruction datasets, demonstrating its versatility and advanced capabilities under complex lighting and reflection conditions. In Table 1, upward arrows indicate better results for each evaluation metric, while downward arrows indicate better results for smaller values. Ours (Origin) and Ours (Pre-mode) are two strategies proposed in this paper. The Origin strategy introduces normal supervision at the beginning of training to stabilize the learning process of geometric structure. After the model converges to a certain level, it gradually introduces a specular highlight detail enhancement module to achieve layer-by-layer optimization of high-frequency information. The pre-mode strategy adopts a staged training process: first, pre-training is performed using only normal supervision to obtain an initial model with good geometric perception capabilities. Then, a highlight enhancement mechanism is introduced on this basis to further optimize the detail performance and realism of the mirror area.

[0053] In one embodiment, the present invention provides an apparatus further comprising: Contains at least one high-performance processor core with powerful data processing and computing capabilities, capable of efficiently executing complex algorithms; At least one high-speed, large-capacity memory is provided, which is not only used to store massive amounts of data, but also has enough space to store at least one programmed program; When the written program is accurately executed by at least one of these high-performance processors, it activates a series of complex computational processes, enabling the processor to intelligently implement the aforementioned real-time 3D rendering method based on reflection perception. By deeply mining the geometric features in the image and combining it with a strategy for enhancing highlight areas, the rendering model is continuously optimized, achieving high-precision reconstruction of complex lighting and reflective environments, providing core support for subsequent recognition, interaction, and other tasks.

[0054] The contents of the above method embodiments are all applicable to the device embodiments of the present invention. The functions specifically implemented by the device embodiments of the present invention are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0055] In one embodiment, the present invention provides a storage medium storing processor-executable instructions, wherein the processor-executable instructions are used to implement the above-mentioned real-time three-dimensional rendering method based on reflection perception when executed by the processor.

[0056] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0057] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The order of execution of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0058] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0059] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

Claims

1. A real-time 3D rendering method based on reflection perception, characterized in that: The construction and training process of the rendering model used includes: Recover a point cloud from the input image, defining each point as a 3D Gaussian basis. Extending the standard differentiable rasterization pipeline to simultaneously generate rendered normal maps and reflection intensity maps; Extracting a depth map from an input image through a pre-trained depth estimation network, and calculating a standard normal map based on the depth map for supervising the rendered normal map; Generate a mask based on the reflection intensity map to identify the highlight area, and construct a specular highlight loss to enhance supervision of the highlight area; Based on the rendered normal vector map and the ambient light map, the specular color of each pixel is calculated using the specular reflection law; Projecting the three-dimensional Gaussian primitives onto a two-dimensional imaging plane, weightedly fusing the diffuse reflection color of each pixel with the specular reflection color according to the reflection intensity map, and outputting the final pixel color; A multi-stage optimization strategy is adopted to train the rendering model, including the normal field construction stage based on normal consistency loss and image reconstruction loss, the appearance consistency global optimization stage based on image reconstruction loss, and the highlight area detail enhancement stage based on image reconstruction loss and specular highlight loss.

2. The real-time 3D rendering method based on reflection perception according to claim 1, characterized in that: The point cloud is recovered from the input image, and each point is defined as a three-dimensional Gaussian primitive, specifically including: A point cloud is extracted from the input image using structure-from-motion techniques, and the points in the point cloud are defined as 3D Gaussian primitives with volume density: ； Where, represents a three-dimensional Gaussian primitive, represents the center position of each 3D Gaussian primitive in 3D space, is the covariance matrix of the three-dimensional Gaussian basis element, which defines the distribution shape and direction of the three-dimensional Gaussian basis element in space; represents the location of the query point; Decompose the covariance matrix into a combination of a rotation matrix and a scaling matrix: ； Where, represents the rotation matrix, represents the scaling matrix, is transposed.

3. The real-time 3D rendering method based on reflection perception according to claim 1, characterized in that: The generation process of the rendering normal vector map specifically includes: The direction corresponding to the shortest principal axis in the covariance matrix of each 3D Gaussian basis element is defined as the normal vector of the 3D Gaussian basis element. During the rendering process, when the 3D Gaussian basis element is projected onto the 2D image plane, multiple Gaussians will affect the same pixel. The normal vectors of all 3D Gaussian basis elements that overlap with the current pixel are weighted and fused to obtain the rendering normal vector of the current pixel. , and then get the rendering normal vector map composed of the rendering normal vectors of all pixels: ； in, Represents the total number of 3D Gaussian primitives overlapping the current pixel, is the index of the i-th three-dimensional Gaussian basis element, is the normal vector of the i-th three-dimensional Gaussian basis element, for The corresponding weight coefficient.

4. The real-time 3D rendering method based on reflection perception according to claim 1, characterized in that: The method of extracting a depth map from an input image by using a pre-trained depth estimation network and calculating a standard normal map based on the depth map for supervising the rendered normal map specifically includes: Use a pre-trained depth estimation network to extract a depth map from the input image , and combined with the known camera parameters, each pixel is back-projected into a three-dimensional space coordinate point through the perspective projection model , They are x-coordinate value, y-coordinate value, and z-coordinate value respectively; Constructing surfaces using depth maps , and position the normal vector of a point Q on the surface as the normal direction of the tangent plane at that point: ； By counting pixels in the image plane The gradient of the three-dimensional coordinates corresponding to the adjacent pixels of the pixel is used to construct the local tangent plane. is the projection of point Q on the image plane; and the normal of the local tangent plane is used as the pixel The unit normal vector of , thus obtaining a complete pixel-level normal vector map: ； Normalize the pixel-level normal vector map to obtain the standard normal vector , the formula is: ； represents the magnitude of a vector; A normal vector supervision mechanism is introduced to drive the rendering model to learn local shape representation by minimizing the difference between the rendered normal map and the standard normal map: ； represents the normal consistency loss; Represents the rendering normal vector.

5. The real-time 3D rendering method based on reflection perception according to claim 1, characterized in that: Generating a mask based on the reflection intensity map to identify the highlight area, and constructing a specular highlight loss to enhance supervision of the highlight area, specifically includes: Introducing a mask Identify areas of high brightness in the reflection intensity map: ； Where, Represents the pixel position in the reflection intensity map The reflection intensity value at is the set threshold; Apply the mask to the input image and the rendered image to enhance the highlight areas: ； ； Where, represents the input image, represents the enhanced input image, represents a rendered image, represents the enhanced rendered image, represents element-wise multiplication; Build based on loss and structural similarity loss Specular highlight loss To conduct supervision: ； is the weight coefficient.

6. The real-time 3D rendering method based on reflection perception according to claim 1, characterized in that: The method of calculating the specular color of each pixel based on the rendering normal vector map and the ambient light map by the specular reflection law specifically includes: Construct a basic reflection model and calculate the reflection of any pixel in the input image according to the law of mirror reflection The direction of reflected light : ； Where, is the incident light direction, Pixels The surface normal vector of Calculate pixels using the ambient light map based on the direction of reflected light Specular color : ； Where, This is the process of querying from the ambient light map using a bilinear interpolation strategy.

7. The real-time 3D rendering method based on reflection perception according to claim 1, characterized in that: The projecting of the three-dimensional Gaussian primitives onto the two-dimensional imaging plane, weighted fusion of the diffuse reflection color of each pixel and the specular reflection color according to the reflection intensity map, and outputting the final pixel color specifically includes: The elliptical weighted average sputtering method is used to project the 3D Gaussian primitives onto the 2D imaging plane: ； Where, is the two-dimensional Gaussian obtained by projecting the three-dimensional Gaussian basis element onto the two-dimensional imaging plane, and They represent the two-dimensional mean and covariance matrix after the three-dimensional Gaussian basis element projection respectively; Calculating pixels using alpha blending Diffuse color : ； ； Where, Represents the current pixel The number of overlapping 3D Gaussian primitives, and Respectively represent The color vector and opacity of the three-dimensional Gaussian primitives; Indicates the The final opacity of the 3D Gaussian primitives; Using a delayed reflection strategy, the final pixel color is calculated by weighted fusion , to effectively integrate the contributions of diffuse and specular reflections: ； in, represents the reflection intensity map, The specular color.

8. The real-time 3D rendering method based on reflection perception according to claim 1, characterized in that: The multi-stage optimization strategy is used to train the rendering model, including a normal field construction stage based on normal consistency loss and image reconstruction loss, a global optimization stage of appearance consistency based on image reconstruction loss, and a highlight area detail enhancement stage based on image reconstruction loss and specular highlight loss. Specifically, the strategy includes: Loss function in the normal field construction phase ; is the image reconstruction loss: , is the balance parameter; for loss, is the structural similarity loss; is the normal consistency loss: ; is the rendering normal vector, is the standard normal vector; is the weight of normal consistency loss; Loss function in the global optimization phase of appearance consistency Using image reconstruction loss ; Loss function for highlight area detail enhancement : ; Specular highlight loss The weight of .

9. A device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the method according to any one of claims 1 to 8 can be implemented.

10. A storage medium storing instructions executable by a processor, characterized in that: The processor-executable instructions are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

Images