Real-time rendering method and device, electronic equipment and storage medium

Abstract

The invention provides a real-time rendering method and device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: constructing a three-dimensional Gaussian distribution model based on a key frame image of a large-scale scene; according to a camera center point and a preset initial radius corresponding to each key frame image, removing three-dimensional Gaussian distribution outside the preset initial radius in the three-dimensional Gaussian distribution model to obtain first three-dimensional Gaussian distribution; rendering each key frame image based on the first three-dimensional Gaussian distribution to obtain a rendered image of each key frame image, and determining a reference rejection radius according to each rendered image and each key frame image; based on each camera center point and the reference elimination radius, eliminating the three-dimensional Gaussian distribution outside the reference elimination radius to obtain second three-dimensional Gaussian distribution; and rendering the large-scale scene according to the second three-dimensional Gaussian distribution to obtain a rendered image of the large-scale scene. By adopting the method provided by the embodiment of the invention, the rendering efficiency and precision can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a real-time rendering method, apparatus, electronic device, and storage medium. Background Technology

[0002] In related technologies, 3D reconstruction of large-scale scenes, such as factories and cities, typically requires processing massive amounts of 3D (Three-Dimensional) data. For example, 3D Gaussian Splatting rendering is commonly used when processing such large volumes of 3D data. However, rendering large-scale scenes requires processing vast amounts of Gaussian point cloud data, which can easily lead to insufficient GPU memory, reduced rendering speed, and difficulty in ensuring real-time rendering. Summary of the Invention

[0003] This disclosure provides a real-time rendering method, apparatus, electronic device, and storage medium. The technical solution of this disclosure is as follows:

[0004] Firstly, this disclosure provides a real-time rendering method, including:

[0005] A three-dimensional Gaussian distribution model is constructed based on keyframe images of a large-scale scene; the keyframe images are captured by different cameras from different positions.

[0006] Based on the camera center point and preset initial radius corresponding to each keyframe image, the three-dimensional Gaussian distribution located outside the preset initial radius in the three-dimensional Gaussian distribution model is removed to obtain the first three-dimensional Gaussian distribution;

[0007] Each keyframe image is rendered based on the first three-dimensional Gaussian distribution to obtain a rendered image of each keyframe image, and a reference culling radius is determined based on each rendered image and each keyframe image.

[0008] Based on each camera center point and the reference culling radius, the three-dimensional Gaussian distribution located outside the reference culling radius is culled to obtain the second three-dimensional Gaussian distribution;

[0009] The large-scale scene is rendered according to the second three-dimensional Gaussian distribution to obtain a rendered image of the large-scale scene.

[0010] In one possible implementation, constructing a three-dimensional Gaussian distribution model based on keyframe images of a large-scale scene includes:

[0011] The keyframe images are processed using 3D reconstruction technology to reconstruct a 3D geometric representation of the large-scale scene; wherein, the 3D reconstruction technology includes structure-of-motion reconstruction technology or multi-view stereoscopic technology.

[0012] A set of three-dimensional Gaussian distributions is initialized based on the three-dimensional geometric representation of the large-scale scene;

[0013] A three-dimensional Gaussian distribution model of the large-scale scene is constructed based on the three-dimensional Gaussian distribution.

[0014] In one possible implementation, the step of removing three-dimensional Gaussian distributions located outside the preset initial radius in the three-dimensional Gaussian distribution model based on the camera center point corresponding to each keyframe image and a preset initial radius to obtain a first three-dimensional Gaussian distribution includes:

[0015] For each keyframe image, the camera center point corresponding to each keyframe image is taken as the center of a sphere, and the preset initial radius is taken as the filtering radius to remove the three-dimensional Gaussian distributions located outside the preset initial radius in the three-dimensional Gaussian distribution model.

[0016] The first three-dimensional Gaussian distribution is obtained by removing the three-dimensional Gaussian distributions located outside the preset initial radius.

[0017] In one possible implementation, before rendering each of the keyframe images based on the first three-dimensional Gaussian distribution to obtain a rendered image of each keyframe image, the method further includes:

[0018] Camera calibration is performed on each of the keyframe images to obtain the intrinsic and extrinsic parameters of each keyframe image;

[0019] Based on the intrinsic and extrinsic parameters of each keyframe image, the center point of the first three-dimensional Gaussian distribution is projected onto the image plane.

[0020] In one possible implementation, determining the reference culling radius based on each of the rendered images and each of the keyframe images includes:

[0021] For the i-th keyframe image, calculate the structural similarity loss between the i-th keyframe image and its rendered image; where i∈[1,n], and n is the number of keyframe images;

[0022] Determine whether the structural similarity loss is greater than a preset loss threshold;

[0023] If the structural similarity loss is greater than a preset loss threshold, the culling radius corresponding to the i-th keyframe image is adjusted, and the rendered image of the i-th keyframe image is re-rendered until the structural similarity loss between the i-th keyframe image and the rendered image of the i-th keyframe image is less than or equal to the preset loss threshold.

[0024] The removal radius when the structural similarity loss is less than or equal to a preset loss threshold is determined as the removal radius corresponding to the i-th keyframe image.

[0025] The reference rejection radius is calculated based on the rejection radius corresponding to each of the keyframe images.

[0026] In one possible implementation, rendering the large-scale scene based on the second three-dimensional Gaussian distribution to obtain a rendered image of the large-scale scene includes:

[0027] Based on the intrinsic and extrinsic parameters of each keyframe image, the center point of the second three-dimensional Gaussian distribution is projected onto the image plane;

[0028] The large-scale scene is rendered based on the parameters of the second three-dimensional Gaussian distribution projected onto the image plane to obtain a rendered image of the large-scale scene.

[0029] In one possible implementation, before determining the reference culling radius based on each of the rendered images and each of the keyframe images, the method further includes:

[0030] Calculate the color loss between the rendered image and the keyframe image;

[0031] The parameters of the corresponding three-dimensional Gaussian distribution are adjusted according to the color loss; wherein, the adjusted parameters of the three-dimensional Gaussian distribution are used to render the large-scale scene; the parameters of each three-dimensional Gaussian distribution include: center position, spherical harmonic coefficient, rotation factor, scaling factor, and opacity.

[0032] Secondly, this disclosure provides a real-time rendering apparatus, including:

[0033] A building module is used to construct a 3D Gaussian distribution model based on keyframe images of a large-scale scene; the keyframe images are captured by different cameras from different positions.

[0034] The first elimination module is used to eliminate the three-dimensional Gaussian distribution located outside the preset initial radius in the three-dimensional Gaussian distribution model based on the camera center point and preset initial radius corresponding to each keyframe image, so as to obtain the first three-dimensional Gaussian distribution.

[0035] The radius determination module is used to render each of the keyframe images based on the first three-dimensional Gaussian distribution to obtain a rendered image of each of the keyframe images, and to determine a reference culling radius based on each rendered image and each of the keyframe images.

[0036] The second elimination module is used to eliminate the three-dimensional Gaussian distribution located outside the reference elimination radius based on each camera center point and the reference elimination radius, so as to obtain the second three-dimensional Gaussian distribution.

[0037] The rendering module is used to render the large-scale scene according to the second three-dimensional Gaussian distribution to obtain a rendered image of the large-scale scene.

[0038] In one possible implementation, the building module is configured to:

[0039] The keyframe images are processed using 3D reconstruction technology to reconstruct a 3D geometric representation of the large-scale scene; wherein, the 3D reconstruction technology includes structure-of-motion reconstruction technology or multi-view stereoscopic technology.

[0040] A set of three-dimensional Gaussian distributions is initialized based on the three-dimensional geometric representation of the large-scale scene;

[0041] A three-dimensional Gaussian distribution model of the large-scale scene is constructed based on the three-dimensional Gaussian distribution.

[0042] In one possible implementation, the first rejection module is configured to:

[0043] For each keyframe image, the camera center point corresponding to each keyframe image is taken as the center of a sphere, and the preset initial radius is taken as the filtering radius to remove the three-dimensional Gaussian distributions located outside the preset initial radius in the three-dimensional Gaussian distribution model.

[0044] The first three-dimensional Gaussian distribution is obtained by removing the three-dimensional Gaussian distributions located outside the preset initial radius.

[0045] In one possible implementation, the real-time rendering apparatus further includes:

[0046] The calibration module is used to perform camera calibration on each of the keyframe images to obtain the intrinsic and extrinsic parameters of each keyframe image;

[0047] The projection module is used to project the center point of the first three-dimensional Gaussian distribution onto the image plane based on the intrinsic and extrinsic parameters of each keyframe image.

[0048] In one possible implementation, the radius determination module is configured to:

[0049] For the i-th keyframe image, calculate the structural similarity loss between the i-th keyframe image and its rendered image; where i∈[1,n], and n is the number of keyframe images;

[0050] Determine whether the structural similarity loss is greater than a preset loss threshold;

[0051] If the structural similarity loss is greater than a preset loss threshold, the culling radius corresponding to the i-th keyframe image is adjusted, and the rendered image of the i-th keyframe image is re-rendered until the structural similarity loss between the i-th keyframe image and the rendered image of the i-th keyframe image is less than or equal to the preset loss threshold.

[0052] The removal radius when the structural similarity loss is less than or equal to a preset loss threshold is determined as the removal radius corresponding to the i-th keyframe image.

[0053] The reference rejection radius is calculated based on the rejection radius corresponding to each of the keyframe images.

[0054] In one possible implementation, the rendering module is configured to:

[0055] Based on the intrinsic and extrinsic parameters of each keyframe image, the center point of the second three-dimensional Gaussian distribution is projected onto the image plane;

[0056] The large-scale scene is rendered based on the parameters of the second three-dimensional Gaussian distribution projected onto the image plane to obtain a rendered image of the large-scale scene.

[0057] In one possible implementation, the real-time rendering apparatus further includes:

[0058] A loss calculation module is used to calculate the color loss between the rendered image and the keyframe image;

[0059] The parameter adjustment module is used to adjust the parameters of the corresponding three-dimensional Gaussian distribution according to the color loss; wherein, the adjusted parameters of the three-dimensional Gaussian distribution are used to render the large-scale scene; the parameters of each three-dimensional Gaussian distribution include: center position, spherical harmonic coefficient, rotation factor, scaling factor and opacity.

[0060] Thirdly, this disclosure provides an electronic device, including:

[0061] processor;

[0062] Memory used to store the processor's executable instructions;

[0063] The processor is configured to execute the instructions to implement the method described in the first aspect.

[0064] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0065] Fifthly, this disclosure provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the method described in the first aspect.

[0066] The technical solution disclosed herein brings at least the following beneficial effects:

[0067] In the technical solution disclosed herein, a three-dimensional Gaussian distribution model is constructed based on keyframe images of a large-scale scene. These keyframe images are captured by different cameras from different positions. Based on the camera center point and a preset initial radius corresponding to each keyframe image, three-dimensional Gaussian distributions located outside the preset initial radius in the three-dimensional Gaussian distribution model are removed to obtain a first three-dimensional Gaussian distribution. Each keyframe image is rendered based on the first three-dimensional Gaussian distribution to obtain a rendered image of each keyframe image. A reference culling radius is determined based on each rendered image and each keyframe image. Based on each camera center point and the reference culling radius, three-dimensional Gaussian distributions located outside the reference culling radius are removed to obtain a second three-dimensional Gaussian distribution. The large-scale scene is rendered based on the second three-dimensional Gaussian distribution to obtain a rendered image of the large-scale scene. Thus, by constructing a three-dimensional Gaussian distribution model, removing irrelevant three-dimensional Gaussian distributions, and optimizing the culling radius for real-time rendering, the rendering efficiency and accuracy of large-scale scenes can be significantly improved. This enhances the responsiveness of user-interactive applications, improves user experience, and meets the requirements of real-time rendering.

[0068] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0069] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0070] Figure 1 A flowchart illustrating a real-time rendering method provided in an embodiment of this disclosure;

[0071] Figure 2 This is a schematic diagram of the structure of a real-time rendering device provided in an embodiment of this disclosure;

[0072] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0073] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0074] It should be noted that the terms "first," "second," etc., used in the embodiments of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed herein.

[0075] It should be noted that in the embodiments disclosed herein, there may be some existing solutions in the industry such as software, components, and models. These should be considered as exemplary, and their purpose is only to illustrate the feasibility of implementing the technical solutions disclosed herein. However, it does not mean that the applicant has used or necessarily used such solutions.

[0076] The technical solutions provided by the embodiments of this disclosure are described in detail below with reference to the accompanying drawings.

[0077] Figure 1 This is a flowchart illustrating a real-time rendering method provided in an embodiment of the present disclosure. This method can be applied to a server, such as a single server or a server cluster. Figure 1 As shown, the real-time rendering method may include the following steps:

[0078] S101, based on keyframe images of large-scale scenes, constructs a three-dimensional Gaussian distribution model.

[0079] The keyframe images were taken by different cameras from different locations.

[0080] In the embodiments of this disclosure, real image data of large-scale scenes can be collected. For example, photos can be taken at different locations within the large-scale scene using drones, vehicle-mounted cameras, or handheld devices. These photos will then serve as keyframe images of the large-scale scene. Scene information of the large-scale scene can then be extracted using these keyframe images taken from multiple different angles to construct a three-dimensional Gaussian distribution model. The three-dimensional Gaussian distribution model is a mathematical model that can be used to describe the spatial distribution of objects or feature points in a scene. The keyframe images can provide information about the scene from different perspectives, helping to construct a more comprehensive three-dimensional model of the large-scale scene.

[0081] S102, based on the camera center point and preset initial radius corresponding to each keyframe image, remove the three-dimensional Gaussian distribution located outside the preset initial radius in the three-dimensional Gaussian distribution model to obtain the first three-dimensional Gaussian distribution.

[0082] In the embodiments of this disclosure, an initial radius range, i.e., a preset initial radius, can be set based on the camera center point of each keyframe image. Three-dimensional Gaussian distributions located outside the preset initial radius range from the camera center are discarded, and the remaining three-dimensional Gaussian distribution is the first three-dimensional Gaussian distribution. This narrows the range of the three-dimensional Gaussian distribution model, retaining the portion more relevant to the current camera viewpoint, thus obtaining the first three-dimensional Gaussian distribution. In this way, scene information relevant to the current camera viewpoint can be initially filtered out, reducing the amount of data required for subsequent processing.

[0083] S103, render each keyframe image based on the first three-dimensional Gaussian distribution to obtain the rendered image of each keyframe image, and determine the reference culling radius based on each rendered image and each keyframe image.

[0084] In the embodiments of this disclosure, a first three-dimensional Gaussian distribution can be used to render the rendered image corresponding to each keyframe image. Then, by comparing the rendered image and the original keyframe image, the differences and matching degree between the two can be analyzed to determine a more suitable culling radius, namely the reference culling radius. The reference culling radius is adjusted based on the comparison between the actual rendering effect and the original image, which can more accurately filter out the three-dimensional Gaussian distributions with a high degree of matching with the scene.

[0085] S104, based on the center point of each camera and the reference culling radius, removes the three-dimensional Gaussian distribution located outside the reference culling radius to obtain the second three-dimensional Gaussian distribution.

[0086] In the embodiments of this disclosure, after determining the reference culling radius, the center point of each camera can be used as a reference (sphere center) to further remove 3D Gaussian distributions outside the range of the reference culling radius determined in the previous step. After this filtering and culling step, a more accurate 3D Gaussian distribution, i.e., a second 3D Gaussian distribution, can be obtained. In this way, the 3D Gaussian distribution model of large-scale scenes can be further optimized, making the 3D Gaussian distribution model more consistent with the characteristics of the actual scene and the information of the keyframe images.

[0087] S105: Render a large-scale scene according to the second three-dimensional Gaussian distribution to obtain a rendered image of the large-scale scene.

[0088] In the embodiments of this disclosure, after obtaining the second three-dimensional Gaussian distribution, the optimized second three-dimensional Gaussian distribution can be used to render the entire large-scale scene, generating the final rendered image of the large-scale scene. This rendered image can more accurately reflect the three-dimensional structure and appearance of the large-scale scene. At the same time, by combining information from multiple keyframe images and the optimized three-dimensional Gaussian model, the rendering result of the large-scale scene can be closer to the real scene.

[0089] In the technical solution disclosed herein, a three-dimensional Gaussian distribution model is constructed based on keyframe images of a large-scale scene. These keyframe images are captured by different cameras from different positions. Based on the camera center point and a preset initial radius corresponding to each keyframe image, three-dimensional Gaussian distributions located outside the preset initial radius in the three-dimensional Gaussian distribution model are removed to obtain a first three-dimensional Gaussian distribution. Each keyframe image is rendered based on the first three-dimensional Gaussian distribution to obtain a rendered image of each keyframe image. A reference culling radius is determined based on each rendered image and each keyframe image. Based on each camera center point and the reference culling radius, three-dimensional Gaussian distributions located outside the reference culling radius are removed to obtain a second three-dimensional Gaussian distribution. The large-scale scene is rendered based on the second three-dimensional Gaussian distribution to obtain a rendered image of the large-scale scene. Thus, by constructing a three-dimensional Gaussian distribution model, removing irrelevant three-dimensional Gaussian distributions, and optimizing the culling radius for real-time rendering, the rendering efficiency and accuracy of large-scale scenes can be significantly improved. This enhances the responsiveness of user-interactive applications, improves user experience, and meets the requirements of real-time rendering.

[0090] In some possible implementations, a three-dimensional Gaussian distribution model is constructed based on keyframe images of a large-scale scene, including:

[0091] 3D reconstruction technology is used to process keyframe images and reconstruct a 3D geometric representation of a large-scale scene; among which, 3D reconstruction technology includes motion reconstruction structure technology or multi-view stereo technology;

[0092] A set of three-dimensional Gaussian distributions is initialized based on the three-dimensional geometric representation of large-scale scenes;

[0093] Construct a 3D Gaussian distribution model for a large-scale scene based on the 3D Gaussian distribution.

[0094] In the embodiments of this disclosure, when constructing a 3D Gaussian distribution model based on keyframe images of a large-scale scene, the keyframe images can first be processed using 3D reconstruction technology to reconstruct the 3D geometric representation of the large-scale scene, that is, to reconstruct the 3D structure of the global scene from the image set. The 3D reconstruction technology can be, for example, SfM (Structure-from-Motion) or MVS (Multi-View Stereo). When using SfM, a series of acquired keyframe images can be input, and the camera's motion trajectory and the 3D structure of objects in the scene can be deduced by analyzing the motion trajectories of feature points in the keyframe images. For example, feature points such as corner points can be extracted from the keyframe images, and then, based on the matching relationship of feature points in different keyframe images, geometric constraints and optimization algorithms can be used to recover the 3D geometry of the scene and the camera's pose. When using MVS, keyframe images from multiple perspectives can be used to reconstruct the 3D geometric representation of the scene. By comparing the color and texture information of corresponding pixels in keyframe images from different perspectives, the depth value of the pixels can be calculated, thereby constructing a 3D model of the scene. For example, for an architectural scene, images taken from multiple different angles can be used with MVS technology to obtain a 3D surface model of the building, including the 3D position and shape of structures such as walls and roofs. This provides basic data for subsequent modeling.

[0095] Then, a set of 3D Gaussian distributions representing the scene can be initialized using the sparse point cloud obtained from 3D reconstruction technology. For example, some initial 3D Gaussian distributions can be determined based on the 3D geometric representation of the reconstructed large-scale scene. For instance, the center point of the Gaussian distribution can be selected according to certain rules, such as uniform distribution in the scene or based on the scene's geometric features (such as object boundaries, important structural regions, etc.). The center point, for example, the center (average) point μ, can be initialized using the SfM sparse point cloud. Simultaneously, an initial covariance matrix can be set for each Gaussian distribution. The covariance matrix determines the shape and range of the Gaussian distribution; for example, the initial value can be based on empirical estimates or simply set as the identity matrix, for subsequent further adjustment and optimization. This provides initial parameters for subsequent model optimization.

[0096] Subsequently, a 3D Gaussian distribution model of a large-scale scene can be constructed based on the 3D Gaussian distribution. For example, an initialized 3D Gaussian distribution can be used, and optimization algorithms (such as the expectation-maximization algorithm) can be employed to continuously adjust the parameters (mean and covariance matrix) of the Gaussian distribution. This allows the Gaussian distribution to better fit the 3D geometric representation of the large-scale scene, enabling it to more accurately describe the scene's shape, structure, and density distribution. Ultimately, a 3D Gaussian distribution model that effectively represents a large-scale scene can be constructed and used for various applications such as scene analysis, modeling, and rendering. In this way, effective modeling and representation of large-scale scenes can be achieved.

[0097] In some possible implementations, based on the camera center point corresponding to each keyframe image and a preset initial radius, the three-dimensional Gaussian distributions located outside the preset initial radius in the three-dimensional Gaussian distribution model are removed to obtain a first three-dimensional Gaussian distribution, including:

[0098] For each keyframe image, the camera center point corresponding to each keyframe image is taken as the center of the sphere, and the preset initial radius is taken as the filtering radius to remove the three-dimensional Gaussian distributions located outside the preset initial radius in the three-dimensional Gaussian distribution model.

[0099] The first three-dimensional Gaussian distribution is obtained by removing the three-dimensional Gaussian distributions located outside the preset initial radius.

[0100] In the embodiments of this disclosure, when eliminating 3D Gaussian distributions located outside the preset initial radius in the 3D Gaussian distribution model based on the camera center point corresponding to each keyframe image, for each keyframe image, its corresponding camera center point can be determined as the center of a sphere, and an initial radius (i.e., the preset initial radius) can be set as the filtering range. Then, in the 3D Gaussian distribution model, the distance from the center point of each 3D Gaussian distribution to the camera center point is calculated. If the distance is greater than the preset initial radius, the 3D Gaussian distribution is eliminated. After the above elimination operation, the remaining 3D Gaussian distributions are the first 3D Gaussian distribution. In this way, by eliminating Gaussian distributions exceeding the preset initial radius, the range of the 3D Gaussian distribution model is limited to the area covered by the keyframe image, reducing the complexity and computational load of the model. This allows the model to focus more on the scene parts of the keyframe image, improving the model's accuracy and precision in describing key areas, and providing a more effective data foundation for subsequent processing and analysis.

[0101] In some possible implementations, before rendering each keyframe image based on a first three-dimensional Gaussian distribution to obtain the rendered image of each keyframe image, the method further includes:

[0102] Camera calibration is performed on each keyframe image to obtain the intrinsic and extrinsic parameters of each keyframe image;

[0103] Based on the intrinsic and extrinsic parameters of each keyframe image, the center point of the first three-dimensional Gaussian distribution is projected onto the image plane.

[0104] In the embodiments of this disclosure, each keyframe image is rendered based on a first three-dimensional Gaussian distribution. Before obtaining the rendered image of each keyframe image, a calibration tool (such as a checkerboard calibration board) and a calibration algorithm (such as the Zhang Zhengyou calibration method) can be used to calibrate the camera for each keyframe image, obtaining the camera's intrinsic parameters (focal length, principal point coordinates, etc.) and extrinsic parameters (describing the camera's position and orientation in the world coordinate system). The intrinsic and extrinsic parameters can be used to determine the 6D pose of each keyframe image (6D pose is required in 3D reconstruction) to provide necessary parameters for subsequent 3D-to-2D projection, ensuring the accuracy of the projection. Then, using the camera's intrinsic and extrinsic parameters, the center point of the first three-dimensional Gaussian distribution can be projected from 3D space onto the 2D image plane using a projection formula, obtaining the coordinates of each Gaussian distribution center point on the image plane. This provides 2D position information for subsequent rendering, ensuring the spatial consistency between the rendered image and the original keyframe image. Each keyframe image is rendered based on a first three-dimensional Gaussian distribution to obtain a rendered image of each keyframe image. For example, the center point of the Gaussian distribution projected onto the image plane can be used to generate a rendered image similar to the original keyframe image by combining the parameters of the Gaussian distribution (such as the covariance matrix) on the image plane.

[0105] In some possible implementations, a reference culling radius is determined based on each rendered image and each keyframe image, including:

[0106] For the i-th keyframe image, calculate the structural similarity loss between the i-th keyframe image and its rendered image; where i∈[1,n], and n is the number of keyframe images;

[0107] Determine whether the structural similarity loss is greater than a preset loss threshold;

[0108] If the structural similarity loss is greater than the preset loss threshold, the culling radius corresponding to the i-th keyframe image is adjusted, and the rendered image of the i-th keyframe image is re-rendered until the structural similarity loss between the i-th keyframe image and the rendered image of the i-th keyframe image is less than or equal to the preset loss threshold.

[0109] The removal radius when the structural similarity loss is less than or equal to the preset loss threshold is determined as the removal radius corresponding to the i-th keyframe image.

[0110] Calculate the reference removal radius based on the removal radius corresponding to each keyframe image.

[0111] In the embodiments of this disclosure, when determining the reference culling radius based on each rendered image and each keyframe image, for the i-th keyframe image, the structural similarity (SSIM) loss between it and the corresponding rendered image can be calculated. SSIM loss is a metric for measuring the similarity between images, typically considering factors such as brightness, contrast, and structural information. SSIM loss quantifies the difference between the keyframe image and the rendered image, providing a basis for subsequent radius adjustment. Then, the calculated SSIM loss can be compared with a preset loss threshold to determine if it exceeds the threshold. If the SSIM loss exceeds the preset loss threshold, it indicates that the current culling radius leads to a large difference between the rendered image and the keyframe image, requiring adjustment of the culling radius. In this case, the culling radius corresponding to the i-th keyframe image can be adjusted, rendering can be performed again, and the SSIM loss can be recalculated until the loss is less than or equal to the preset loss threshold. Thus, by continuously adjusting the culling radius, a suitable radius value is found, ensuring that the structural similarity loss between the rendered image and the keyframe image reaches an acceptable range, thus ensuring the accuracy of the model.

[0112] Conversely, if the structural similarity loss is less than or equal to a preset loss threshold, the removal radius at this point can be determined as the removeable radius corresponding to the i-th keyframe image. This establishes an optimal removeable radius for each keyframe image, which is then used for subsequent reference removal radius calculations. Afterward, a reference removal radius can be calculated based on the removeable radius corresponding to each keyframe image. For example, the average, median, or other statistical methods can be used to calculate the reference removal radius. This yields a reference removal radius applicable to the entire scene, which can be used for subsequent model optimization and processing, improving the overall quality and consistency of the model.

[0113] In some possible implementations, a large-scale scene is rendered based on a second three-dimensional Gaussian distribution to obtain a rendered image of the large-scale scene, including:

[0114] Based on the intrinsic and extrinsic parameters of each keyframe image, the center point of the second three-dimensional Gaussian distribution is projected onto the image plane;

[0115] Based on the parameters of the second three-dimensional Gaussian distribution projected onto the image plane, a large-scale scene is rendered to obtain a rendered image of the large-scale scene.

[0116] In the embodiments of this disclosure, when rendering a large-scale scene based on a second three-dimensional Gaussian distribution to obtain a rendered image of the large-scale scene, the intrinsic and extrinsic parameters of each keyframe image can be used to project the center point of the second three-dimensional Gaussian distribution from three-dimensional space onto a two-dimensional image plane using a projection formula. This yields the coordinates of each center point of the second Gaussian distribution on the image plane, providing two-dimensional positional information for subsequent rendering and ensuring spatial consistency between the rendered image and the original keyframe image. Then, based on the position of the Gaussian distribution center point projected onto the image plane, combined with the parameters of the Gaussian distribution, rendering can be performed on the image plane to generate a rendered image of the large-scale scene.

[0117] The parameters of the three-dimensional Gaussian distribution include the center position, the spherical harmonic (SH) coefficients representing color, the rotation factor (rotation quaternion), the scaling factor, and the opacity. For example, the covariance matrix can be decomposed into a rotation matrix R and a scaling matrix S, which can be used for differentiable optimization, as follows:

[0118]

[0119] Where μ represents the center position, μ∈R 2 R 2 ∑ represents the two-dimensional real space; ∑ represents the covariance matrix.

[0120] In some possible implementations, before determining the reference culling radius based on each rendered image and each keyframe image, the following steps are also included:

[0121] Calculate the color loss between the rendered image and the keyframe image;

[0122] The parameters of the corresponding 3D Gaussian distribution are adjusted based on the color loss; the adjusted parameters of the 3D Gaussian distribution are used to render large-scale scenes; the parameters of each 3D Gaussian distribution include: center position, spherical harmonic coefficient, rotation factor, scaling factor, and opacity.

[0123] In embodiments of this disclosure, before determining the reference culling radius, the color loss between the rendered image and the keyframe image can be calculated, and the parameters of the corresponding three-dimensional Gaussian distribution can be adjusted based on this loss. For example, a color loss function (such as mean squared error (MSE), structural similarity loss (SSIM), etc.) can be used to quantify the color difference between the two images (rendered image and keyframe image) and evaluate the degree of color matching between the rendered image and the keyframe image. Then, the parameters of the three-dimensional Gaussian distribution can be adjusted based on the color loss to reduce the color loss, thereby improving the quality of the rendered image. For example, the center position of the Gaussian distribution can be adjusted to change the position of objects in the scene, thus affecting the rendering result; the spherical harmonic coefficients can be adjusted to change the color and lighting effects of the object surface; the rotation factor can be adjusted to change the orientation of the object, affecting the calculation of lighting and shadows; the scale factor can be adjusted to change the size of the object, affecting the depth and scale of the scene; and the opacity can be adjusted to change the transparency of the object, affecting the visual effect of the scene.

[0124] As an example, color loss is commonly used in gradient descent algorithms. The parameters of a Gaussian distribution can be updated by calculating the gradient of the loss function. For instance, the rendering process of the aforementioned 3D Gaussian distribution can be backpropagated to optimize the parameters of each 3D Gaussian distribution. A concrete example includes: Input color image: This is an actual captured color image (e.g., a keyframe image) that provides visual information about the scene. These images can be used to construct a sparse point cloud model of the scene using Structure-for-Motion (SfM) techniques and initialize the 3D Gaussian distribution. The input color image is compared with the re-rendered image to calculate the color loss and optimize the Gaussian parameters. And it includes: Rendered image: The image rendered using the 3D Gaussian distribution, which is splatted onto the image plane by approximating pixel coordinates by projecting the 3D Gaussian distribution along the depth dimension. Then, the final rendered color is calculated based on the color and opacity of each Gaussian distribution. This process involves blending color and opacity to generate a rendered image similar to the input color image.

[0125] Understandably, large-scale scenes can be re-rendered using adjusted 3D Gaussian distribution parameters to achieve rendering results closer to the keyframe images. For example, the adjusted 3D Gaussian distribution parameters can be input into the rendering engine to generate new rendered images. Color loss can be continuously reduced through iterative adjustments to the 3D Gaussian distribution parameters and re-rendering until satisfactory rendering quality is achieved or a preset number of iterations is reached. Based on this, a suitable reference culling radius can be determined based on the optimized rendering results to remove elements with minimal impact on rendering quality, thereby optimizing rendering efficiency.

[0126] To make the real-time rendering method provided in this disclosure clearer, a specific example is given below. The real-time rendering method provided in this disclosure includes the following steps:

[0127] Step 1: Construct a 3D GS model (3D Gaussian sputtering) based on multiple keyframe images of the scene. Details are as follows:

[0128] First, real-world image data of large-scale scenes can be collected. This can be done by using drones, vehicle-mounted cameras, or handheld devices to capture photos at different locations within the scene; these photos will then serve as keyframe images for the scene. Camera calibration is performed on the acquired images (keyframe images) to obtain the intrinsic and extrinsic parameters for each image. Intrinsic parameters include focal length and principal point coordinates, while extrinsic parameters describe the camera's position and orientation in the world coordinate system. These intrinsic and extrinsic parameters can be used to determine the 6D pose of each image, which is then used in 3D reconstruction.

[0129] Techniques such as SfM or MVS are used to reconstruct the 3D structure of the global scene from a collection of images. This step typically produces a 3D point cloud or mesh that represents the geometry of the scene.

[0130] The sparse point cloud obtained by SfM can also be used to initialize a set of 3D Gaussian distributions to represent the scene.

[0131] 3D Gaussian Graph (GS) can model a scene as a set of 3D Gaussian distributions, an explicit representation, as opposed to the implicit representation used in Neural Radiance Fields (NeRF). Each Gaussian distribution (a 3D Gaussian distribution) is characterized by its covariance matrix and center (mean) point μ. The mean μ of the 3D Gaussian is typically initialized using a SfM sparse point cloud. The covariance matrix can be decomposed into a rotation matrix R and a scaling matrix S, which can be further optimized for differentiability as follows:

[0132]

[0133] Where μ represents the center position, μ∈R 2 R 2 ∑ represents the two-dimensional real space; ∑ represents the covariance matrix.

[0134] Each three-dimensional Gaussian distribution has the following parameters: (1) the center location μ∈R 2 (2) The spherical harmonic (SH) coefficients of color c∈R k (k represents the degree of freedom); (3) Rotation factor (rotation quaternion) r∈R 4 ;R 4 Represents a four-dimensional real vector space; (4) Scale factor s∈R 3 R 3(5) Opacity α∈R, where R is the set of real numbers.

[0135] The covariance matrix describes the elliptic body configured by the scaling matrix S = diag([sx, sy, sz]) and the rotation matrix R = q²R([rw, n, ry, rz]), and can be calculated using the following formula:

[0136] Σ=RSS 晻 R

[0137] Rendering from a given camera view W involves the process of scattering Gaussian data onto the image plane, which can be approximated by projecting a 3D Gaussian vector along the depth dimension to pixel coordinates. Given a view transformation W (also known as camera pose), the covariance matrix in camera coordinates can be represented as:

[0138] Σ 2D =JWΣW 晻 J

[0139] Where J is the Jacobian matrix of the affine approximation of the projective transformation. Furthermore, the final rendered color can be formulated as a mixture of N ordered points α overlapping the pixel, as follows:

[0140]

[0141] Among them, c i α i The color and opacity of that point are represented by the learnable SH color coefficients and the opacity per point. This represents the product of the cumulative opacities over the first i-1 Gaussian distributions.

[0142] After the 3D GS is splatted onto the image plane (a method of scattering data points onto the image plane), the color loss between the input color image and the rendered image can be calculated and optimized for Gaussian parameters.

[0143] Color loss is typically used in gradient descent algorithms to update the parameters of a Gaussian distribution by calculating the gradient of the loss function. Specifically, the rendering process of the aforementioned 3D Gaussian distribution is backpropagated to each 3D Gaussian distribution for parameter optimization.

[0144] Input color images: These are actual captured color images that provide visual information about the scene. These images are used to construct a sparse point cloud model of the scene using Structure from Motion (SfM) technology and initialize a 3D Gaussian distribution. The input color images are compared with re-rendered images to calculate color loss and optimize the Gaussian parameters.

[0145] Image rendering: This is the image rendered using a 3D Gaussian distribution. The 3D Gaussian distribution is splatted onto the image plane by approximating pixel coordinates by projecting the 3D Gaussian along the depth dimension. Then, the final rendered color is calculated based on the color and opacity of each Gaussian distribution. This process involves blending color and opacity to generate a rendered image similar to the input color image.

[0146] Step 2: After constructing the 3D GS model of the scene, all parameters of the 3D GS model can be frozen. For the i-th keyframe, the corresponding camera center point O can be obtained. A 3D Gaussian distribution is then filtered within a radius r, with the camera center point O as the center. When rendering the 3D Gaussian model, 3D Gaussian point clouds with center positions μ outside this range can be removed, and the image from that viewpoint can be re-rendered. SSIM loss (structural similarity loss) is used to evaluate whether the re-rendered image meets the requirements (i.e., SSIM loss is less than or equal to a preset loss threshold; the smaller the SSIM loss, the higher the structural similarity between the two images). If the requirements are not met, the radius r is iteratively optimized until the re-rendered image meets the requirements.

[0147]

[0148] Among them, G t R represents the 3DGS model, R represents the 3DGS renderer, r() represents the removal of 3D Gaussian point clouds whose center position μ is outside the range with the camera center point O as the center and a radius of r (the initial value of r can be set to 0), I represents the real image, and ò is the preset loss threshold.

[0149] After completing the above training, we can obtain the r that meets the above requirements for the keyframe image. Then, we can perform the above steps on all keyframe images to obtain the set of removable radii r for all keyframe images. i ={r1, r2, ... r n Then, the average set of removable radii can be calculated based on the set of removable radii, which can be used as a reference removable radius:

[0150]

[0151] Where n represents the number of keyframe images.

[0152] Step 3: When rendering large-scale scenes, for each camera viewpoint, the center point O of the camera in that viewpoint can be used as the center of a sphere, and the 3D Gaussian point cloud outside the radius can be culled. Then, the image from that camera viewpoint is rendered in real time using a 3D Gaussian image renderer, thereby accelerating real-time rendering speed while ensuring rendering quality. In this way, high rendering quality can be maintained while improving rendering efficiency and enhancing the user experience.

[0153] The specific implementation and technical effects of each step in this embodiment are similar to those of the above method embodiments, and will not be repeated here.

[0154] Based on the same inventive concept, embodiments of this disclosure also provide a real-time rendering apparatus. For example... Figure 2 As shown, the real-time rendering device 200 includes:

[0155] Module 210 is used to construct a three-dimensional Gaussian distribution model based on keyframe images of a large-scale scene; the keyframe images are captured by different cameras from different positions.

[0156] The first elimination module 220 is used to eliminate the three-dimensional Gaussian distribution located outside the preset initial radius in the three-dimensional Gaussian distribution model according to the camera center point and preset initial radius corresponding to each key frame image, so as to obtain the first three-dimensional Gaussian distribution.

[0157] The radius determination module 230 is used to render each of the keyframe images based on the first three-dimensional Gaussian distribution to obtain a rendered image of each of the keyframe images, and to determine a reference culling radius based on each rendered image and each of the keyframe images.

[0158] The second elimination module 240 is used to eliminate the three-dimensional Gaussian distribution located outside the reference elimination radius based on each camera center point and the reference elimination radius, to obtain a second three-dimensional Gaussian distribution.

[0159] The rendering module 250 is used to render the large-scale scene according to the second three-dimensional Gaussian distribution to obtain a rendered image of the large-scale scene.

[0160] In one possible implementation, the building module 210 is configured to:

[0161] The keyframe images are processed using 3D reconstruction technology to reconstruct a 3D geometric representation of the large-scale scene; wherein, the 3D reconstruction technology includes structure-of-motion reconstruction technology or multi-view stereoscopic technology.

[0162] A set of three-dimensional Gaussian distributions is initialized based on the three-dimensional geometric representation of the large-scale scene;

[0163] A three-dimensional Gaussian distribution model of the large-scale scene is constructed based on the three-dimensional Gaussian distribution.

[0164] In one possible implementation, the first rejection module 220 is configured to:

[0165] For each keyframe image, the camera center point corresponding to each keyframe image is taken as the center of a sphere, and the preset initial radius is taken as the filtering radius to remove the three-dimensional Gaussian distributions located outside the preset initial radius in the three-dimensional Gaussian distribution model.

[0166] The first three-dimensional Gaussian distribution is obtained by removing the three-dimensional Gaussian distributions located outside the preset initial radius.

[0167] In one possible implementation, the real-time rendering apparatus 200 further includes:

[0168] The calibration module is used to perform camera calibration on each of the keyframe images to obtain the intrinsic and extrinsic parameters of each keyframe image;

[0169] The projection module is used to project the center point of the first three-dimensional Gaussian distribution onto the image plane based on the intrinsic and extrinsic parameters of each keyframe image.

[0170] In one possible implementation, the radius determination module 230 is configured to:

[0171] For the i-th keyframe image, calculate the structural similarity loss between the i-th keyframe image and its rendered image; where i∈[1,n], and n is the number of keyframe images;

[0172] Determine whether the structural similarity loss is greater than a preset loss threshold;

[0173] If the structural similarity loss is greater than a preset loss threshold, the culling radius corresponding to the i-th keyframe image is adjusted, and the rendered image of the i-th keyframe image is re-rendered until the structural similarity loss between the i-th keyframe image and the rendered image of the i-th keyframe image is less than or equal to the preset loss threshold.

[0174] The removal radius when the structural similarity loss is less than or equal to a preset loss threshold is determined as the removal radius corresponding to the i-th keyframe image.

[0175] The reference rejection radius is calculated based on the rejection radius corresponding to each of the keyframe images.

[0176] In one possible implementation, the rendering module 250 is configured to:

[0177] Based on the intrinsic and extrinsic parameters of each keyframe image, the center point of the second three-dimensional Gaussian distribution is projected onto the image plane;

[0178] The large-scale scene is rendered based on the parameters of the second three-dimensional Gaussian distribution projected onto the image plane to obtain a rendered image of the large-scale scene.

[0179] In one possible implementation, the real-time rendering apparatus 200 further includes:

[0180] A loss calculation module is used to calculate the color loss between the rendered image and the keyframe image;

[0181] The parameter adjustment module is used to adjust the parameters of the corresponding three-dimensional Gaussian distribution according to the color loss; wherein, the adjusted parameters of the three-dimensional Gaussian distribution are used to render the large-scale scene; the parameters of each three-dimensional Gaussian distribution include: center position, spherical harmonic coefficient, rotation factor, scaling factor and opacity.

[0182] The specific implementation and technical effects of the device provided in this disclosure are similar to those of the method embodiments described above, and will not be repeated here.

[0183] According to embodiments of this disclosure, this disclosure also discloses an electronic device, a computer-readable storage medium, and a computer program product.

[0184] Figure 3 A schematic block diagram of an example electronic device 200 that can be used to implement embodiments of the present disclosure is shown. The electronic device 200 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0185] like Figure 3 As shown, the electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. The RAM 303 may also store various programs and data required for the operation of the electronic device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0186] Multiple components in electronic device 300 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of displays, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0187] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as real-time rendering methods. For example, in some embodiments, the real-time rendering method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of the real-time rendering method described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform real-time rendering methods by any other suitable means (e.g., by means of firmware).

[0188] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0189] The program code of a computer program product used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0190] In the context of this disclosure, a computer-readable storage medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be a machine-readable signal medium or a machine-readable storage medium. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0191] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0192] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0193] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0194] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0195] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A real-time rendering method, characterized in that, include: A three-dimensional Gaussian distribution model is constructed based on keyframe images of a large-scale scene; the keyframe images are captured by different cameras from different positions. Based on the camera center point and preset initial radius corresponding to each keyframe image, the three-dimensional Gaussian distribution located outside the preset initial radius in the three-dimensional Gaussian distribution model is removed to obtain the first three-dimensional Gaussian distribution; Each keyframe image is rendered based on the first three-dimensional Gaussian distribution to obtain a rendered image of each keyframe image, and a reference culling radius is determined based on each rendered image and each keyframe image. Based on each camera center point and the reference culling radius, the three-dimensional Gaussian distribution located outside the reference culling radius is culled to obtain the second three-dimensional Gaussian distribution; The large-scale scene is rendered according to the second three-dimensional Gaussian distribution to obtain a rendered image of the large-scale scene.

2. The real-time rendering method according to claim 1, characterized in that, The construction of a 3D Gaussian distribution model based on keyframe images of a large-scale scene includes: The keyframe images are processed using 3D reconstruction technology to reconstruct a 3D geometric representation of the large-scale scene; wherein, the 3D reconstruction technology includes structure-of-motion reconstruction technology or multi-view stereoscopic technology. A set of three-dimensional Gaussian distributions is initialized based on the three-dimensional geometric representation of the large-scale scene; A three-dimensional Gaussian distribution model of the large-scale scene is constructed based on the three-dimensional Gaussian distribution.

3. The real-time rendering method according to claim 1, characterized in that, The step of removing three-dimensional Gaussian distributions located outside the preset initial radius in the three-dimensional Gaussian distribution model based on the camera center point and preset initial radius corresponding to each keyframe image to obtain the first three-dimensional Gaussian distribution includes: For each keyframe image, the camera center point corresponding to each keyframe image is taken as the center of a sphere, and the preset initial radius is taken as the filtering radius to remove the three-dimensional Gaussian distributions located outside the preset initial radius in the three-dimensional Gaussian distribution model. The first three-dimensional Gaussian distribution is obtained by removing the three-dimensional Gaussian distributions located outside the preset initial radius.

4. The real-time rendering method according to claim 2, characterized in that, Before rendering each keyframe image based on the first three-dimensional Gaussian distribution to obtain the rendered image of each keyframe image, the method further includes: Camera calibration is performed on each of the keyframe images to obtain the intrinsic and extrinsic parameters of each keyframe image; Based on the intrinsic and extrinsic parameters of each keyframe image, the center point of the first three-dimensional Gaussian distribution is projected onto the image plane.

5. The real-time rendering method according to claim 2, characterized in that, The step of determining the reference culling radius based on each rendered image and each keyframe image includes: For the i-th keyframe image, calculate the structural similarity loss between the i-th keyframe image and its rendered image; where i∈[1,n], and n is the number of keyframe images; Determine whether the structural similarity loss is greater than a preset loss threshold; If the structural similarity loss is greater than a preset loss threshold, the culling radius corresponding to the i-th keyframe image is adjusted, and the rendered image of the i-th keyframe image is re-rendered until the structural similarity loss between the i-th keyframe image and the rendered image of the i-th keyframe image is less than or equal to the preset loss threshold. The removal radius when the structural similarity loss is less than or equal to a preset loss threshold is determined as the removal radius corresponding to the i-th keyframe image. The reference rejection radius is calculated based on the rejection radius corresponding to each of the keyframe images.

6. The real-time rendering method according to claim 5, characterized in that, The process of rendering the large-scale scene based on the second three-dimensional Gaussian distribution to obtain a rendered image of the large-scale scene includes: Based on the intrinsic and extrinsic parameters of each keyframe image, the center point of the second three-dimensional Gaussian distribution is projected onto the image plane; The large-scale scene is rendered based on the parameters of the second three-dimensional Gaussian distribution projected onto the image plane to obtain a rendered image of the large-scale scene.

7. The real-time rendering method according to claim 5, characterized in that, Before determining the reference culling radius based on each rendered image and each keyframe image, the process further includes: Calculate the color loss between the rendered image and the keyframe image; The parameters of the corresponding three-dimensional Gaussian distribution are adjusted according to the color loss; wherein, the adjusted parameters of the three-dimensional Gaussian distribution are used to render the large-scale scene; the parameters of each three-dimensional Gaussian distribution include: center position, spherical harmonic coefficient, rotation factor, scaling factor, and opacity.

8. A real-time rendering apparatus, characterized in that, include: A building module is used to construct a 3D Gaussian distribution model based on keyframe images of a large-scale scene; the keyframe images are captured by different cameras from different positions. The first elimination module is used to eliminate the three-dimensional Gaussian distribution located outside the preset initial radius in the three-dimensional Gaussian distribution model based on the camera center point and preset initial radius corresponding to each keyframe image, so as to obtain the first three-dimensional Gaussian distribution. The radius determination module is used to render each of the keyframe images based on the first three-dimensional Gaussian distribution to obtain a rendered image of each of the keyframe images, and to determine a reference culling radius based on each rendered image and each of the keyframe images. The second elimination module is used to eliminate the three-dimensional Gaussian distribution located outside the reference elimination radius based on each camera center point and the reference elimination radius, so as to obtain the second three-dimensional Gaussian distribution. The rendering module is used to render the large-scale scene according to the second three-dimensional Gaussian distribution to obtain a rendered image of the large-scale scene.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-7.

Images