A rendering method, apparatus and computing device cluster

Abstract

A rendering method includes: rendering a virtual digital scene at a first moment to obtain a reference image; updating the appearance parameters of a first three-dimensional Gaussian representation of the virtual digital scene at a second moment, based on the reference image, to obtain a second three-dimensional Gaussian representation of the virtual digital scene at the first moment, where the second moment is earlier than the first moment; rendering the second three-dimensional Gaussian representation based on the user's viewpoint information to obtain a first rendered image from the user's viewpoint; and generating a final rendering result with the same viewpoint as the first rendered image based on the first rendered image. In this way, information from the reference image is aggregated into the three-dimensional Gaussian representation of the virtual digital scene, ensuring that the current three-dimensional Gaussian representation of the virtual digital scene accurately represents its current state. This allows for rendering from any viewpoint using three-dimensional Gaussian sputtering technology, eliminating repetitive or complex rendering calculations, reducing rendering resource waste, and improving rendering efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a rendering method, apparatus and computing device cluster. Background Technology

[0002] Cloud rendering is a technology that utilizes cloud computing resources to render images, animations, or videos. Leveraging the powerful processing capabilities of cloud computing, cloud rendering can efficiently handle complex graphics tasks, significantly reducing rendering time and improving rendering efficiency.

[0003] Traditional cloud rendering typically involves configuring dedicated cloud computing resources for each user individually to meet their rendering needs. However, in certain application scenarios, such as multiplayer cloud gaming, shared digital exhibition halls, and online conferences with multiple participants, multiple users may simultaneously view the same area within the same scene. In such cases, traditional cloud rendering methods lead to the repeated rendering of the same content, resulting in wasted resources. This waste increases with the number of users, reducing rendering efficiency. Summary of the Invention

[0004] This application provides a rendering method, apparatus, computing device cluster, computer storage medium, and computer product that can reduce the waste of rendering resources and improve rendering efficiency in scenarios involving multiple users.

[0005] In a first aspect, this application provides a rendering method, comprising: rendering a virtual digital scene at a first moment to obtain at least one reference image; updating the appearance parameters of a first three-dimensional Gaussian (3DGS) representation of the virtual digital scene at a second moment based on the reference image to obtain a second 3DGS representation of the virtual digital scene at the first moment, wherein the second moment is earlier than the first moment; rendering the second 3DGS representation based on user perspective information to obtain a first rendered image from the user's perspective; and generating a final rendering result consistent with the perspective of the first rendered image based on the first rendered image.

[0006] In this way, by aggregating the information from the reference image obtained by algorithms such as ray tracing at the current moment into the 3DGS representation of the virtual digital scene, the 3DGS representation of the virtual digital scene at the current moment can accurately express the current state of the virtual digital scene. This allows for rendering from any viewpoint using 3DGS Splatting in the future, eliminating repetitive or complex rendering calculations, reducing the waste of rendering resources, and improving rendering efficiency.

[0007] In one possible implementation, the appearance parameters of a first 3DGS representation of a virtual digital scene are updated based on a reference image. This includes: removing at least some texture details from the reference image by performing albedo decoupling to obtain a second rendered image; rendering a third rendered image from the first 3DGS representation, wherein the third rendered image and the second rendered image have the same viewpoint; and updating the appearance parameters of the first 3DGS representation based on the differences between the second and third rendered images. In this way, by removing high-frequency components from the reference image, the influence of these high-frequency components can be reduced, thereby reducing the optimization difficulty of the 3DGS representation and improving the optimization quality and efficiency.

[0008] In one possible implementation, rendering the second 3DGS representation based on the user's viewpoint information includes: rendering the second 3DGS representation using 3D Gaussian sputtering technology based on the user's viewpoint information to obtain a first rendered image from the user's viewpoint. In this way, the desired image can be rendered from the second 3DGS representation.

[0009] In one possible implementation, a final rendering result consistent with the viewpoint of the first rendered image is generated based on the first rendered image. This includes: obtaining the final rendering result from the user's viewpoint based on a first rasterized image and the first rendered image from the user's viewpoint; wherein the first rasterized image is used to provide detailed features of the image from the user's viewpoint, and the first rasterized image is obtained by rasterizing the virtual digital scene based on the user's viewpoint information. In this way, texture information from the virtual digital scene can be added to the first rendered image, improving the quality of the final rendering result.

[0010] In one possible implementation, before obtaining the final rendering result from the user's perspective based on the first rasterized image and the first rendered image from the user's perspective, the method further includes: performing albedo decoupling on the first rendered image to at least remove texture detail features from the first rendered image. This reduces the interference of high-frequency components in the first rendered image on the coupling, thereby improving the final rendering quality.

[0011] In one possible implementation, the method is applied to a system comprising a cloud computing platform and a client. The user's viewpoint information is uploaded to the cloud computing platform via the client; the reference image, the second 3DGS representation, and the first rendered image are all processed by the cloud computing platform; and the final rendering result is obtained via the client. This allows for image rendering through edge-cloud collaboration, and enables image rendering even when edge computing power is insufficient.

[0012] In one possible implementation, there are multiple clients, multiple users, multiple first-rendered images, and multiple final-rendered results. Specifically, a user's viewpoint information is uploaded to the cloud computing platform via a client; a first-rendered image is obtained by rendering a second 3DGS representation based on a user's viewpoint information through the cloud computing platform; and a final-rendered result is obtained by processing a first-rendered image from the cloud computing platform through a client.

[0013] In one possible implementation, the method is applied to a system comprising a cloud computing platform and a client. The user's viewpoint information is uploaded to the cloud computing platform via the client; the reference image and the second 3DGS representation are both processed by the cloud computing platform; and the first rendered image and the final rendered result are both processed by the client. This allows for image rendering through edge-cloud collaboration, fully utilizing the capabilities of the edge device.

[0014] In one possible implementation, there are multiple clients, multiple users, multiple first-rendered images, and multiple final-rendered results. Specifically, a user's viewpoint information is uploaded to the cloud computing platform via a client; a first-rendered image is obtained by a client rendering a second 3DGS representation from the cloud computing platform based on a user's viewpoint information; and a final-rendered result is obtained by a client processing the client-generated first-rendered image.

[0015] Secondly, this application provides a rendering apparatus, comprising: a first rendering module for rendering a virtual digital scene at a first moment to obtain at least one reference image; an optimization module for updating the appearance parameters of a first three-dimensional Gaussian (3DGS) representation of the virtual digital scene at a second moment based on the reference image to obtain a second 3DGS representation of the virtual digital scene at the first moment, wherein the second moment is earlier than the first moment; a second rendering module for rendering the second 3DGS representation based on user perspective information to obtain a first rendered image from the user's perspective; and a generation module for generating a final rendering result consistent with the perspective of the first rendered image based on the first rendered image.

[0016] In one possible implementation, when the optimization module updates the appearance parameters of the first 3DGS representation of the virtual digital scene based on the reference image, it specifically performs the following: by performing albedo decoupling on the reference image, at least the texture detail features in the reference image are removed to obtain a second rendered image; a third rendered image is rendered from the first 3DGS representation, wherein the third rendered image and the second rendered image have the same viewpoint; and the appearance parameters of the first 3DGS representation are updated based on the differences between the second rendered image and the third rendered image.

[0017] In one possible implementation, when the second rendering module renders the second 3DGS representation based on the user's viewpoint information, it specifically renders the second 3DGS representation based on the user's viewpoint information and using 3D Gaussian sputtering technology to obtain a first rendered image from the user's viewpoint.

[0018] In one possible implementation, when the generation module generates a final rendering result consistent with the viewpoint of the first rendered image based on the first rendered image, it is specifically used to: obtain the final rendering result from the user's viewpoint based on the first rasterized image and the first rendered image from the user's viewpoint; wherein, the first rasterized image is obtained by rasterizing the virtual digital scene based on the user's viewpoint information.

[0019] In one possible implementation, before obtaining the final rendering result from the user's perspective based on the first rasterized image and the first rendered image from the user's perspective, the generation module is also used to: remove at least the texture detail features in the first rendered image by performing albedo decoupling on the first rendered image.

[0020] In one possible implementation, the device is deployed in a system that includes a cloud computing platform and a client. Specifically, the first rendering module, the optimization module, and the second rendering module are deployed on the cloud computing platform, while the generation module is deployed on the client. The user's perspective information is uploaded to the cloud computing platform via the client.

[0021] In one possible implementation, there are multiple clients, multiple users, multiple first-rendered images, and multiple final-rendered results. Specifically, a user's viewpoint information is uploaded to the cloud computing platform via a client; a first-rendered image is obtained by a second rendering module on the cloud computing platform rendering a second 3DGS representation based on a user's viewpoint information; and a final-rendered result is obtained by a generation module on a client processing a first-rendered image from the cloud computing platform.

[0022] In one possible implementation, the device is deployed in a system that includes a cloud computing platform and a client. Specifically, the first rendering module and optimization module are deployed on the cloud computing platform, while the second rendering module and generation module are deployed on the client. The user's perspective information is uploaded to the cloud computing platform via the client.

[0023] In one possible implementation, there are multiple clients, multiple users, multiple first-rendered images, and multiple final-rendered results. Specifically, a user's viewpoint information is uploaded to a cloud computing platform via a client; a first-rendered image is obtained by a second rendering module on a client rendering a second 3DGS representation from the cloud computing platform based on the user's viewpoint information; and a final-rendered result is obtained by a generation module on a client processing the client-generated first-rendered image.

[0024] Thirdly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0025] Fourthly, this application provides a computer-readable storage medium including computer program instructions, which, when executed by a computing device, perform the method described in the first aspect or any possible implementation thereof; or, when executed by a cluster of computing devices, the cluster of computing devices performs the method described in the first aspect or any possible implementation thereof. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0026] Fifthly, this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the method described in the first aspect or any possible implementation thereof; or, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect or any possible implementation thereof. Exemplarily, a cluster of computing devices may include one or more computing devices.

[0027] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0028] Figure 1 This is a schematic diagram illustrating the implementation of a ray tracing algorithm provided in an embodiment of this application;

[0029] Figure 2 This is a schematic diagram illustrating the implementation of a 3DGS algorithm provided in an embodiment of this application;

[0030] Figure 3 This is a schematic diagram illustrating a rendering technique based on a combination of 3DGS and ray tracing, as provided in an embodiment of this application.

[0031] Figure 4 This is a schematic diagram illustrating another rendering technique based on the combination of 3DGS and ray tracing, provided in an embodiment of this application.

[0032] Figure 5 This is a schematic diagram illustrating another rendering technique based on the combination of 3DGS and ray tracing, provided in an embodiment of this application.

[0033] Figure 6 This is a schematic diagram illustrating another rendering technique based on the combination of 3DGS and ray tracing, provided in an embodiment of this application.

[0034] Figure 7 yes Figure 6 A schematic diagram of one arrangement of the parts shown;

[0035] Figure 8 Is Figure 7 The diagram shows the system architecture under the shown layout.

[0036] Figure 9 yes Figure 6 A schematic diagram of another arrangement of the parts shown;

[0037] Figure 10 This is a flowchart illustrating a rendering algorithm provided in an embodiment of this application;

[0038] Figure 11 This is a schematic diagram of the structure of a rendering device provided in an embodiment of this application;

[0039] Figure 12 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0040] Figure 13 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0041] Figure 14 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. Detailed Implementation

[0042] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0043] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0044] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0045] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0046] First, the relevant technical terms involved in the technical solution provided in this application will be introduced.

[0047] (1) Virtual digital scene

[0048] A virtual digital scene refers to a digital environment or space constructed using technologies such as computer graphics, digital modeling, and rendering. A virtual digital scene can be completely virtual or a simulation or reproduction of the real world. Examples of virtual digital scenes include: game scenes, virtual reality or augmented reality scenes, digital exhibition hall scenes, or conference scenes, etc.

[0049] (2) Scene Assets

[0050] Scene assets refer to all the resources and elements used in creating a virtual digital scene. Resources are the various files and data required to create a virtual digital scene, such as 3D models, meshes, materials, textures, lighting, particle systems, sounds, animations, scripts, etc. A mesh is the geometric part of a 3D model, defined by a series of vertices, edges, and faces. Elements are the concrete parts formed by combining and configuring the various resources used in creating a virtual digital scene, such as scene layouts, characters, props, environments, special effects, sound effects, etc.

[0051] (3) Rendering

[0052] Rendering refers to the process of transforming the resources and elements used in creating a virtual digital scene into a final image or video using computer graphics techniques. The goal of rendering a virtual digital scene is to either reproduce a real-world scene in a highly realistic way in the digital world, or to create a completely new virtual environment.

[0053] (4) Camera / Viewpoint / Sampling Camera

[0054] Camera or viewpoint: In a virtual digital scene, a camera or viewpoint refers to a position and direction within that scene from which the scene is observed. Camera parameters include position, orientation, and field of view.

[0055] Sampling camera: In a virtual digital scene, a sampling camera refers to a virtual camera that samples the scene from multiple different positions and directions during the rendering process in order to fully capture information about the visible surfaces in the scene (such as the surface of an object).

[0056] (5) Image frequency

[0057] Image frequency refers to the frequency in the frequency domain image obtained by performing a Fourier transform on an image. Image frequency can contain both low-frequency and high-frequency components. Low-frequency components are mainly concentrated in the central part of the frequency domain, representing large-scale features in the image. High-frequency components are mainly concentrated in the edge parts of the frequency domain, representing detailed features in the image. It should be understood that low and high frequencies are relative concepts. Low and high frequencies simply refer to the low and high frequencies in the frequency domain.

[0058] (6) Ray tracing algorithm

[0059] Ray tracing is a computer graphics technique used to generate highly realistic images (also known as "rendered images") by simulating light rays emitted from a camera onto objects in a scene, and then their interactions such as reflection, refraction, and scattering. It is primarily based on the Monte Carlo method, emitting virtual rays from the camera's perspective towards objects in the scene and tracing the reflection, refraction, and scattering of these rays upon encountering objects, as well as how they interact with other objects or light sources. Because ray tracing considers the interaction between light and objects, it can produce realistic shadows, reflections, refractions, and other effects.

[0060] The process of ray tracing algorithm is as follows Figure 1As shown, the main idea of ​​this method is to emit rays from the viewpoint to pixels on the imaging plane, find the intersection point of the nearest object that intersects with the ray, and if the surface at that point is a scattering surface, sample the light source and simultaneously sample the reflection direction based on the material properties, continuing to track in that direction; if the surface at that point is a specular or refracting surface, continue tracking another ray in the reflection or refraction direction, and so on recursively until the ray escapes the scene or reaches the set maximum recursion depth. To improve efficiency, the direction of the ray and the sampling of the light source are randomly selected based on a certain probability distribution during each recursive tracking. This randomness allows the algorithm to obtain results close to reality within a limited time. Due to the characteristics of random sampling, the results produced by a single ray often contain noise, resulting in uneven brightness and color in the image. To effectively reduce noise and improve rendering quality, it is usually necessary to emit a large number of rays to each pixel on the imaging plane to increase the number of samples. Therefore, obtaining high-quality results through ray tracing algorithms often requires a significant amount of time. The quality of the image rendered by the ray tracing algorithm depends on the number of rays sampled per pixel (SPP). A low SPP (Scanning Point Per Pixel) will result in rendering with a lot of noise. Low SPP means that fewer rays are sampled per pixel.

[0061] (7) 3D Gaussian sputtering (3DGS sputtering)

[0062] 3DGS is a novel view synthesis (NVS) technique that efficiently renders and generates new images from a given source image and its corresponding pose (i.e., camera position and orientation), as well as the target pose. It uses 3D Gaussian points to represent the scene and employs rasterization to render these Gaussian points into an image, achieving high-quality real-time scene rendering. Novel view synthesis refers to generating images from a new viewpoint given a set of source images and their corresponding camera poses through calculation and rendering.

[0063] The workflow of the 3DGS method is as follows: Figure 2As shown, the main idea of ​​this method is as follows: First, objects and surfaces in the scene are represented using 3D Gaussian points. Then, the density of the 3D Gaussian points is adjusted through adaptive density control to optimize the 3D Gaussian point representation and improve the quality of the synthesized image from the new perspective. Next, a differentiable tile rasterizer is used, and based on the camera pose, the 3D Gaussian point representation is projected onto a 2D plane. Depth sorting and visibility processing are then performed, finally generating the synthesized image from the new perspective. In other words, rasterization techniques can be used to render and generate the synthesized image from the new perspective. After obtaining the newly generated synthesized image, it can be compared with a reference image, gradients can be calculated, and backpropagation can be used to feed the gradients back to the 3D Gaussian points to optimize the 3D Gaussian point representation. Finally, after satisfying the preset convergence conditions, the final image can be obtained.

[0064] (8) Illumination probe method

[0065] Light probes are a technique for capturing and utilizing light information in the empty space of a scene. This method pre-computes and stores the lighting information of the surrounding environment by placing virtual "probes" at key locations. Then, during rendering, dynamic objects can sample lighting data based on their positions relative to the light probes. This allows dynamic objects to receive indirect light from the environment without generating reflected light in real time, thus simulating the effect of global illumination to some extent. In the light probe method, the position of the probes is crucial for capturing light; therefore, this method suffers from probe placement problems, such as significant redundancy and limitations in expressive power. Furthermore, light probes typically use simple and computationally efficient methods (such as spherical harmonic functions or small texture blocks) to represent and store lighting information, especially low-frequency diffuse lighting. Therefore, the light probe method is primarily applied to diffuse scenes.

[0066] (9) Rasterization technology

[0067] Rasterization is the process of converting geometric shapes in a 3D scene into a 2D image. There are various methods for rasterization, and common rasterization algorithms include: Bresenham algorithm, midpoint circle algorithm, polygon scanline algorithm, triangulation algorithm, depth buffer algorithm, anti-aliasing algorithm, and bounding box algorithm.

[0068] (10) Cloud computing platform

[0069] A cloud computing platform is an internet-based computing model that provides various computing resources and services via a network, including computing power, storage space, databases, and network functions. Cloud computing platforms use virtualization technology to aggregate and centrally manage computing resources, allowing users to access and use these resources on demand without owning and maintaining their own physical equipment and infrastructure. Cloud computing platforms typically consist of large data centers equipped with high-performance computing devices (such as servers), storage devices, and network equipment, as well as reliable power and network connections.

[0070] (11) Albedo Decoupling

[0071] Albedo decoupling refers to the process of separating texture, color, and other information from lighting effects in a rendered image. This can be achieved by processing the rendered image through the rendering engine. For example, the rendering engine can use the bidirectional reflectance distribution function (BRDF) pre-integration method to process the rendered image to realize albedo decoupling.

[0072] The technical solution provided in this application will be described below.

[0073] For example, Figure 3 This illustration shows a schematic diagram of a rendering technique based on a combination of 3DGS and ray tracing, according to an embodiment of this application. Figure 3 As shown, the rendering architecture includes: a scene asset rendering part, a 3DGS representation optimization part, and a 3DGS representation rendering part.

[0074] The scene asset rendering section is used to render the scene assets of the virtual digital scene at the current moment using ray tracing algorithms, rasterization algorithms, or hybrid pipeline algorithms to obtain at least one reference image. For example, "current moment" refers to a specific point in time during the rendering process. In some embodiments, "current moment" may also be referred to as "first moment".

[0075] The 3DGS representation optimization section uses the 3DGS algorithm to update the appearance parameters (such as color, opacity, or scale) of the virtual digital scene in the previous time step (also known as the "second time step") based on the reference image obtained in the scene asset rendering section. This updates the 3DGS representation of the virtual digital scene at the current time step (also known as the "current frame 3DGS representation" or "first 3DGS representation"). This allows information from the scene assets at the current time step to be aggregated into the 3DGS representation, thus reconstructing the scene at the current time step. The current frame 3DGS representation accurately expresses the current state of the virtual digital scene, enabling direct calculation of a rendered image matching the user's viewpoint using 3D Gaussian sputtering. This eliminates repetitive or complex rendering calculations, improving rendering efficiency. For example, when using ray tracing algorithms for rendering, it eliminates the need for ray tracing calculations, denoising, and post-processing. A 3DGS representation of a virtual digital scene can be obtained by preprocessing the mesh of the virtual digital scene. The 3DGS representation of the virtual digital scene represents a geometry that approximates the mesh geometry of the virtual digital scene. In some embodiments, during the initial rendering, the appearance parameters of the first 3DGS representation of the virtual digital scene can be, but are not limited to, default values.

[0076] 3DGS (3D View Rendering) is used to render the current frame's 3DGS representation of the virtual digital scene based on the user's viewpoint information, resulting in a rendered image that matches the user's viewpoint. Here, "user" refers to an object using the virtual digital scene. The user's viewpoint information refers to parameters such as the pose of the camera rendering for the user and the camera's physical properties (e.g., resolution, focal length, or field of view). In some embodiments, when there are multiple users, rendering the current frame's 3DGS representation of the virtual digital scene based on the viewpoint information of one user can yield a rendered image that matches that user's viewpoint.

[0077] Within this framework, by aggregating the information of the virtual digital scene at the current moment into its 3DGS representation, the state of the virtual digital scene at the current moment can be expressed using the 3DGS representation. Furthermore, the 3DGS representation allows for the rendering of images from any viewpoint, eliminating repetitive or complex rendering calculations, reducing the waste of rendering resources, and improving rendering efficiency.

[0078] exist Figure 3 Within the framework shown, the image rendered from the current frame's 3DGS representation of the virtual digital scene can be used as the final rendering result. However, to improve the quality of the final rendering result, such as... Figure 4As shown, the above framework can be further enhanced with: a scene asset rasterization processing section and a final rendering result generation section. The scene asset rasterization processing section is used to rasterize the scene assets based on the user's viewpoint information using rasterization technology, thereby obtaining a rasterized image that matches the user's viewpoint information. The rasterized image primarily provides detailed features of the image from the user's perspective, such as textures and shadows. In some embodiments, when there are multiple users, a single rasterized image can be obtained based on the viewpoint information of each user.

[0079] The final rendering result generation section couples the rendered images obtained from each user's viewpoint in the 3DGS rendering section with the rasterized images from the corresponding user's viewpoint to obtain the final rendering result required by each user. For example, coupling the rendered image from user 1's viewpoint obtained in the 3DGS rendering section with the rasterized image from user 1's viewpoint, such as through multiplication, yields the final rendering result from user 1's viewpoint. Similarly, coupling the rendered image from user 2's viewpoint obtained in the 3DGS rendering section with the rasterized image from user 2's viewpoint, such as through multiplication, yields the final rendering result from user 2's viewpoint.

[0080] exist Figure 4 Within the framework shown, by incorporating detailed features such as textures from the virtual digital scene into the rendered image obtained from 3DGS representation, the expression of detailed features in the rendered image obtained from 3DGS representation can be improved, thereby enhancing the rendering quality.

[0081] exist Figure 4 Within the framework shown, in order to reduce interference from detailed features in the rendered image obtained from the 3DGS representation and further improve rendering quality, such as... Figure 5 As shown, in the final rendering result generation part of the above framework, the rendered image obtained from the 3DGS representation can first be decoupled using albedo to remove detailed features from the rendered image obtained from the 3DGS representation. Then, the rendered image after albedo decoupling is coupled with the corresponding rasterized image.

[0082] In the above Figures 3 to 5 In the framework shown, to reduce the difficulty of 3DGS representation optimization in the 3DGS representation optimization part, the above framework can also be modified as follows: Figure 6 The frame shown. Figure 6The framework shown includes: a scene asset rendering section, a 3DGS representation optimization section, a scene asset rasterization processing section, a 3DGS representation rendering section, and a final rendering result generation section. In the scene asset rendering section, in addition to rendering a reference image, albedo decoupling can be performed on the reference image to remove at least some texture details, thus facilitating subsequent 3DGS representation optimization. In the 3DGS representation optimization section, the albedo-decoupled reference image can be used to optimize the appearance parameters of the previous frame's 3DGS representation. In the 3DGS representation rendering section, since the reference image's detail features were not used in the optimization stage, the rendered image in this section has relatively few detail features. Therefore, in the final rendering result generation section, the rasterized image obtained from the scene asset rasterization processing section can be coupled with the rendered image obtained from the 3DGS representation rendering section to obtain the final rendering result. Figure 6 For a detailed description of each part of the framework shown, please refer to the above. Figures 3 to 5 The corresponding descriptions in the framework shown will not be repeated here.

[0083] In the frameworks shown above, traditional rendering assets such as ray tracing are combined with 3DGS. Because the rendered scene has viewpoint consistency, the global viewpoint can be inferred from a relatively small number of rendered views, allowing the virtual digital scene to be represented in its current state using 3DGS representation. Furthermore, global information from the 3DGS representation can be rendered to each viewpoint using splatting, eliminating repetitive or complex rendering calculations, reducing rendering resource waste, and improving rendering efficiency. Additionally, since the framework uses the 3DGS representation corresponding to the virtual digital scene, probe placement is unnecessary. Moreover, the 3DGS representation has the ability to represent non-diffuse scenes, thus expanding its applicability to non-diffuse environments. Finally, the framework directly incorporates ray tracing results into the optimization process, eliminating the need to consider various rendering effects.

[0084] The components of the above framework can all be arranged on the cloud side, or some can be arranged on the cloud side and others on the edge side, depending on the actual situation, and no limitation is made here. In some embodiments, when using Figure 6 When the frame is shown, such as Figure 7 As shown, the scene asset rendering, 3DGS representation optimization, and 3DGS representation rendering components can be deployed on the cloud side, while the scene asset rasterization processing and final rendering result generation components can be deployed on the edge side. Figure 7In the arrangement shown, the cloud side can collect the perspective information of all users on the edge side in order to render the 3DGS representation of the virtual digital scene in the current frame, and then render the 3DGS representation to each perspective in real time and send it to the corresponding users on the edge side. Figure 7 The system architecture corresponding to the layout shown can be as follows: Figure 8 As shown. In Figure 8 In this context, the rendering server can be used to implement the aforementioned scene asset rendering portion, the optimization server can be used to implement the aforementioned 3DGS representation optimization portion, and the 3DGS rendering server can be used for the 3DGS representation rendering portion. In other embodiments, when the edge has sufficient computing power and network speed, the edge's conditions can be fully utilized. In this case, when using... Figure 6 When the frame is shown, such as Figure 9 As shown, the scene asset rendering and 3DGS representation optimization parts can be deployed on the cloud side, while the 3DGS representation rendering part, scene asset rasterization processing part, and final rendering result generation part can be deployed on the edge side. In this deployment, after calculating the 3DGS representation of the current frame's virtual digital scene, the cloud side can send the 3DGS representation of the current frame's virtual digital scene to the corresponding user on the edge side.

[0085] The following describes the specific implementation process of the above technical concept.

[0086] For example, Figure 10 This diagram illustrates a rendering method according to an embodiment of this application. It is understood that this method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 10 As shown, the rendering method includes:

[0087] S1001. Render the virtual digital scene at the first moment to obtain at least one reference image.

[0088] In this embodiment, the virtual digital scene at the first moment can be rendered using ray tracing algorithms, rasterization algorithms, or hybrid pipeline algorithms to obtain at least one reference image. For example, the virtual digital scene can be obtained by loading corresponding scene assets. In some embodiments, when using ray tracing algorithms for rendering, sampling cameras can be placed at suitable locations within the virtual digital scene, and then the sampling cameras can be used to render noisy, low-SPP ray tracing results.

[0089] S1002. Based on the reference image, update the appearance parameters of the first 3DGS representation of the virtual digital scene at the second time step to obtain the second 3DGS representation of the virtual digital scene at the first time step, wherein the second time step is earlier than the first time step.

[0090] In this embodiment, after obtaining the reference image, 3DGS Splatting technology can be used to render a rendered image (also called a "third rendered image") with the same viewpoint as each reference image from the 3DGS representation of the virtual digital scene at the previous time step (i.e., the second time step) (i.e., the first 3DGS representation). Then, based on the differences between the reference image and the third rendered image at the same viewpoint, the appearance parameters of the first 3DGS representation are updated to obtain the second 3DGS of the virtual digital scene at the first time step. For example, a loss function can be used to process the reference image and the third rendered image at the same viewpoint to calculate the loss; then, the gradient is calculated and backpropagated to feed the gradient back into the 3DGS representation to update the appearance parameters of the 3DGS representation. In this way, the information contained in the reference image can be aggregated into the 3DGS representation of the virtual digital scene, so that the obtained 3DGS representation can accurately express the current state of the virtual digital scene, and thus, 3DGS Splatting can be used to achieve rendering from any viewpoint in the future.

[0091] As one possible approach, to reduce the influence of high-frequency components in the reference image, improve the quality of 3DGS representation optimization, and reduce the difficulty of 3DGS representation optimization, albedo decoupling can be performed on the reference image first to remove detailed features (such as textures) from the reference image, thereby obtaining a low-frequency lighting image (also known as a "second rendered image"). Then, based on the differences between the second and third rendered images from the same viewpoint, the appearance parameters of the first 3DGS representation are optimized to obtain the second 3DGS of the virtual digital scene.

[0092] S1003. Based on the user's perspective information, render the second 3DGS representation to obtain the first rendered image from the user's perspective.

[0093] In this embodiment, after obtaining the second 3DGS representation, it can be rendered using 3DGS Splatting technology based on the user's viewpoint information to obtain a first rendered image from the user's viewpoint. In some embodiments, when there are multiple users, the second 3DGS representation can be rendered separately based on each user's viewpoint information to obtain a first rendered image from each user's viewpoint. In this case, there is one first rendered image from each user's viewpoint. For example, a rendered image from user 1's viewpoint can be rendered based on user 2's viewpoint information, and a rendered image from user 2's viewpoint can be rendered based on user 2's viewpoint information. Since the second 3DGS representation of the virtual digital scene can express the current state of the entire virtual digital scene, regardless of the number of users, rendering the second 3DGS representation of the virtual digital scene using 3DGS Splatting technology can obtain rendered images from each user's viewpoint, thus enabling rendering from any viewpoint.

[0094] S1004. Based on the first rendered image, obtain the final rendering result that is consistent with the viewpoint of the first rendered image.

[0095] In this embodiment, after rendering the first rendered image, a final rendered result (also referred to as the "final rendered image") consistent with the viewpoint of the first rendered image can be obtained from the first rendered image. For example, the first rendered image can be directly used as the desired final rendered result. As a possible implementation, to improve the quality of the final rendered result, the virtual digital scene can first be rasterized based on the user's viewpoint information to obtain a first rasterized image. This first rasterized image is used to provide detailed features of the image from the user's viewpoint, such as textures. Then, based on the first rasterized image and the first rendered image from the user's viewpoint, the final rendered result from the user's viewpoint is obtained. For example, the two images can be coupled (e.g., multiplied) to obtain the final rendered result. In this way, detailed features such as textures in the virtual digital scene can be added to the first rendered image, improving the quality of the final rendered result. In some embodiments, when there are multiple users, each user has a first rendered image from their viewpoint. In this case, a final rendered result can be obtained based on the first rendered image from each user's viewpoint. In some embodiments, in order to reduce the interference of detailed features in the first rendered image, before obtaining the final rendering result based on the first rendered image and the first rasterized image, the first rendered image can be decoupled by albedo to at least remove the texture detail features in the first rendered image.

[0096] Therefore, by aggregating the information contained in the reference image obtained by algorithms such as ray tracing at the current moment into the 3DGS representation of the virtual digital scene, the 3DGS representation of the virtual digital scene at the current moment can accurately express the current state of the virtual digital scene. This allows for rendering from any viewpoint using 3DGS Splatting, eliminating repetitive or complex rendering calculations, reducing the waste of rendering resources, and improving rendering efficiency.

[0097] In some embodiments, Figure 10 Each step in the method described herein can be executed by a cloud computing platform. After generating the final rendering result, the cloud computing platform can send it to a client associated with the platform. For example, the client associated with the cloud computing platform can be a desktop application, mobile application, web application, or web-based application.

[0098] When there are multiple users, a user's perspective information can be uploaded to the cloud computing platform by a single client. The final rendering result from a user's perspective can then be sent from the cloud computing platform to the client associated with that user's perspective. For example, client 1 can upload user 1's perspective information to the cloud computing platform, and client 2 can upload user 2's perspective information to the cloud computing platform. In this case, the cloud computing platform can send the final rendering result from user 1's perspective to client 1, and the final rendering result from user 2's perspective to client 2.

[0099] In some embodiments, Figure 10 S1001, S1002, and S1003 in the method described herein can be executed by a cloud computing platform, and S1004 can be executed by a client associated with the cloud computing platform. At this time, the user's viewpoint information can be uploaded from the client to the cloud computing platform. The cloud computing platform can then send the first rendered image, obtained from the 3DGS representation of the current frame, to the client so that the client can generate the final rendering result. For example, the 3DGS representation of the current frame can be understood as the aforementioned "second 3DGS representation".

[0100] When there are multiple users, a user's viewpoint information can be uploaded to the cloud computing platform by a single client. The number of clients and users can be matched one-to-one. At this point, the cloud computing platform can render a first rendered image from the SDGS representation of the current frame based on a single user's viewpoint information.

[0101] Furthermore, after the cloud computing platform renders the first rendered image from a user's perspective using the SDGS representation of the current frame, it can send this first rendered image to the client associated with that user's perspective. For example, client 1 can upload user 1's perspective information to the cloud computing platform, and client 2 can upload user 2's perspective information to the cloud computing platform. At this time, the cloud computing platform can send the first rendered image from user 1's perspective, rendered using the SDGS representation of the current frame, to client 1, and send the first rendered image from user 2's perspective, rendered using the SDGS representation of the current frame, to client 2.

[0102] After receiving the first rendered image from the cloud computing platform, a user's client can obtain a final rendered result with the same perspective as the first rendered image.

[0103] In some embodiments, Figure 10 In the method described, S1001 and S1002 can be executed by a cloud computing platform, while S1003 and S1004 can be executed by a client associated with the cloud computing platform. At this time, the user's perspective information can be uploaded from the client to the cloud computing platform. The cloud computing platform can then distribute its optimized second 3DGS representation of the virtual digital scene to the client, enabling the client to execute S1003 and S1004.

[0104] When there are multiple users, a user's perspective information can be uploaded to the cloud computing platform by one client. The number of clients and users can be matched one-to-one. At this point, the cloud computing platform can distribute its optimized current-frame 3DGS representation of the virtual digital scene to each client. For example, the current-frame 3DGS representation can be understood as the aforementioned "second 3DGS representation."

[0105] After receiving the current frame 3DGS representation from the cloud computing platform, each client can render the current frame 3DGS representation based on the user's perspective information on that client to obtain a first rendered image that matches the user's perspective on that client. For example, when client 1 is used by user 1, after receiving the current frame 3DGS representation, client 1 can render the current frame 3DGS representation based on user 1's perspective information to obtain a first rendered image from user 1's perspective.

[0106] Similarly, each client can also process its self-generated first rendered image to obtain the corresponding final rendered result.

[0107] Understandably, when Figure 10When S1004 of the described method is executed by a client associated with the cloud computing platform, the rasterization processing of the virtual digital scene can be performed by either the client or the cloud computing platform, depending on the actual situation; no limitation is made here. Specifically, when executed by the cloud computing platform, the platform can perform rasterization processing on the virtual digital scene based on the perspective information of each user to obtain rasterized images related to each user, and then distribute the obtained rasterized images to the corresponding clients.

[0108] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments or technical features involved in the embodiments described above can be combined according to the actual situation, and the combined solution is still within the protection scope of this application.

[0109] Based on the methods in the above embodiments, this application also provides a rendering apparatus.

[0110] For example, Figure 11 A schematic diagram of the structure of a rendering apparatus provided in an embodiment of this application is shown. Figure 11 As shown, the rendering apparatus 1100 includes: a first rendering module 1101, used to render a virtual digital scene at a first moment to obtain at least one reference image; an optimization module 1102, used to update the appearance parameters of a first three-dimensional Gaussian (3DGS) representation of the virtual digital scene at a second moment based on the reference image, to obtain a second 3DGS representation of the virtual digital scene at the first moment, wherein the second moment is earlier than the first moment; a second rendering module 1103, used to render the second 3DGS representation based on the user's viewpoint information to obtain a first rendered image from the user's viewpoint; and a generation module 1104, used to obtain a final rendering result consistent with the viewpoint of the first rendered image based on the first rendered image.

[0111] In some embodiments, when the optimization module 1102 updates the appearance parameters of the first 3DGS representation of the virtual digital scene based on the reference image, it is specifically used to: decouple the reference image by albedo to remove at least the texture detail features in the reference image to obtain a second rendered image; render a third rendered image from the first 3DGS representation, wherein the third rendered image and the second rendered image have the same viewpoint; and update the appearance parameters of the first 3DGS representation based on the difference between the second rendered image and the third rendered image.

[0112] In some embodiments, when the second rendering module 1103 renders the second 3DGS representation based on the user's perspective information, it is specifically used to: render the second 3DGS representation based on the user's perspective information and using 3D Gaussian sputtering technology to obtain a first rendered image from the user's perspective.

[0113] In some embodiments, when the generation module 1104 obtains a final rendering result consistent with the viewpoint of the first rendered image based on the first rendered image, it is specifically used to: obtain a final rendering result from the user's viewpoint based on the first rasterized image and the first rendered image from the user's viewpoint; wherein, the first rasterized image is used to provide detailed features of the image from the user's viewpoint, and the first rasterized image is obtained by rasterizing the virtual digital scene based on the user's viewpoint information.

[0114] In some embodiments, before obtaining the final rendering result from the user's perspective based on the first rasterized image and the first rendered image from the user's perspective, the generation module 1104 is further configured to: remove at least the texture detail features in the first rendered image by performing albedo decoupling on the first rendered image.

[0115] In some embodiments, the rendering apparatus 1100 is deployed in a system that includes a cloud computing platform and a client. Specifically, the first rendering module 1101, the optimization module 1102, and the second rendering module 1103 are deployed on the cloud computing platform, the generation module 1104 is deployed on the client, and the user's perspective information is uploaded to the cloud computing platform via the client.

[0116] In some embodiments, the number of clients, the number of users, the number of first rendered images, and the number of final rendered results are all multiple. Specifically, a user's viewpoint information is uploaded to the cloud computing platform via a client; a first rendered image is obtained by the second rendering module 1103 on the cloud computing platform rendering a second 3DGS representation based on a user's viewpoint information; and a final rendered result is obtained by the generation module 1104 on a client processing a first rendered image from the cloud computing platform.

[0117] In some embodiments, the rendering apparatus 1100 is deployed in a system that includes a cloud computing platform and a client. Specifically, the first rendering module 1101 and the optimization module 1102 are deployed on the cloud computing platform, while the second rendering module 1103 and the generation module 1104 are deployed on the client. The user's viewpoint information is uploaded to the cloud computing platform via the client.

[0118] In some embodiments, the number of clients, the number of users, the number of first rendered images, and the number of final rendered results are all multiple. Specifically, a user's viewpoint information is uploaded to the cloud computing platform via a client; a first rendered image is obtained by a second rendering module 1103 on a client rendering a second 3DGS representation from the cloud computing platform based on a user's viewpoint information; and a final rendered result is obtained by a generation module 1104 on a client processing the first rendered image generated by the client itself.

[0119] In some embodiments, Figure 11 The first rendering module 1101, optimization module 1102, second rendering module 1103, and generation module 1104 shown can all be implemented in software or in hardware. For example, the implementation of the first rendering module 1101 will be described below. Similarly, the implementation of the optimization module 1102, second rendering module 1103, and generation module 1104 can refer to the implementation of the first rendering module 1101.

[0120] As an example of a software functional unit, the first rendering module 1101 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, the first rendering module 1101 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0121] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0122] As an example of a hardware functional unit, the first rendering module 1101 may include at least one computing device, such as a server. Alternatively, the first rendering module 1101 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0123] The multiple computing devices included in the first rendering module 1101 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the first rendering module 1101 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the first rendering module 1101 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0124] It should be noted that, in other embodiments, the first rendering module 1101 can be used to execute any step in the rendering method described in the above embodiments, the optimization module 1102 can be used to execute any step in the rendering method described in the above embodiments, and the second rendering module 1103 can also be used to execute any step in the rendering method described in the above embodiments. Furthermore, the steps implemented by the first rendering module 1101, optimization module 1102, second rendering module 1103, and generation module 1104 can be specified as needed, and different steps in the rendering method described in the above embodiments can be implemented by the first rendering module 1101, optimization module 1102, second rendering module 1103, and generation module 1104 respectively. Figure 11 The rendering device 1100 shown has all the functions of the device.

[0125] This application also provides a computing device 1200. For example... Figure 12As shown, the computing device 1200 includes a bus 1202, a processor 1204, a memory 1206, and a communication interface 1208. The processor 1204, the memory 1206, and the communication interface 1208 communicate with each other via the bus 1202. The computing device 1200 can be a server or an electronic device. It should be understood that this application does not limit the number of processors and memories in the computing device 1200.

[0126] Bus 1202 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 12 The bus 1204 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1204 may include a path for transmitting information between various components of the computing device 1200 (e.g., memory 1206, processor 1204, communication interface 1208).

[0127] The processor 1204 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0128] The memory 1206 may include volatile memory, such as random access memory (RAM). The processor 1204 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0129] The memory 1206 stores executable program code, and the processor 1204 executes the executable program code to implement the aforementioned functions. Figure 11 The functions of the first rendering module 1101, optimization module 1102, second rendering module 1103, and generation module 1104 shown in the diagram are used to implement the rendering method described in the above embodiments. That is, the memory 1206 stores instructions for executing the rendering method described in the above embodiments.

[0130] Alternatively, the memory 1206 stores executable code, and the processor 1204 executes the executable code to implement the aforementioned functions respectively. Figure 11 The rendering apparatus 1100 shown in the figure functions to implement the rendering method described in the above embodiments. That is, the memory 1206 stores instructions for executing the rendering method described in the above embodiments.

[0131] The communication interface 1203 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1200 and other devices or communication networks.

[0132] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. This computing device can be a server, such as a central server, an edge server, or a local server in a local data center.

[0133] like Figure 13 As shown, the computing device cluster includes at least one computing device 1200. The memory 1206 of one or more computing devices 1200 in the computing device cluster may store the same instructions for executing the rendering method described in the above embodiments.

[0134] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster may also store partial instructions for executing the rendering method described in the above embodiments. In other words, a combination of one or more computing devices 1200 can jointly execute the instructions for executing the rendering method described in the above embodiments.

[0135] It should be noted that the memory 1206 in different computing devices 1200 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 11 The rendering apparatus 1100 shown contains some of the functions. That is, the instructions stored in the memory 1206 in the different computing devices 1200 can implement the functions of one or more of the first rendering module 1101, the optimization module 1102, the second rendering module 1103, and the generation module 1104.

[0136] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 14 One possible implementation is shown. For example... Figure 14As shown, the two computing devices 1200A and 1200B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1206 in computing device 1200A stores instructions for executing the functions of the first rendering module 1101 and the optimization module 1102. Simultaneously, the memory 1206 in computing device 1200B stores instructions for executing the functions of the second rendering module 1103 and the generation module 1104.

[0137] It should be understood that Figure 14 The functions of the computing device 1200A shown can also be performed by multiple computing devices 1200. Similarly, the functions of the computing device 1200B can also be performed by multiple computing devices 1200.

[0138] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 13 and Figure 14 The connection method of the computing device cluster is different in that the memory 1206 of one or more computing devices 1200 in the computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0139] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster may also store partial instructions for executing the aforementioned methods. In other words, a combination of one or more computing devices 1200 can jointly execute the instructions for executing the aforementioned methods.

[0140] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the methods in the above embodiments; or, when the computer program instructions are executed by a cluster of computing devices, the cluster of computing devices performs the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium that the computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0141] Based on the methods in the above embodiments, this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the methods in the above embodiments, or, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the methods in the above embodiments.

[0142] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0143] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0144] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, cloud computing platform, or data center to another website, computer, cloud computing platform, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a cloud computing platform or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0145] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A rendering method, characterized in that, The method includes: Render the virtual digital scene at the first moment to obtain at least one reference image; Based on the reference image, the appearance parameters of the first three-dimensional Gaussian (3DGS) representation of the virtual digital scene at the second time are updated to obtain the second 3DGS representation of the virtual digital scene at the first time, wherein the second time is earlier than the first time. Based on the user's perspective information, the second 3DGS representation is rendered to obtain a first rendered image from the user's perspective. Based on the first rendered image, a final rendering result with the same viewpoint as the first rendered image is obtained.

2. The method according to claim 1, characterized in that, The step of updating the appearance parameters of the first 3DGS representation of the virtual digital scene based on the reference image includes: By performing albedo decoupling on the reference image, at least the texture detail features in the reference image are removed to obtain the second rendered image; A third rendered image is obtained from the first 3DGS representation, wherein the third rendered image and the second rendered image have the same viewpoint; Based on the differences between the second rendered image and the third rendered image, the appearance parameters represented by the first 3DGS are updated.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining a final rendering result consistent with the viewpoint of the first rendered image based on the first rendered image includes: Based on the first rasterized image and the first rendered image from the user's perspective, the final rendering result from the user's perspective is obtained. The first rasterized image is used to provide at least the texture detail features of the image from the user's perspective. The first rasterized image is obtained by rasterizing the virtual digital scene based on the user's perspective information.

4. The method according to claim 3, characterized in that, Before obtaining the final rendering result from the user's perspective based on the first rasterized image and the first rendered image from the user's perspective, the process also includes: By performing albedo decoupling on the first rendered image, at least the texture detail features in the first rendered image are removed.

5. The method according to any one of claims 1-4, characterized in that, The method is applied to systems that include cloud computing platforms and clients; The user's perspective information is uploaded to the cloud computing platform via the client. The reference image, the second 3DGS representation, and the first rendered image are all obtained through processing by the cloud computing platform; The final rendering result is obtained through processing by the client.

6. The method according to claim 5, characterized in that, The number of clients, the number of users, the number of the first rendered images, and the number of the final rendered results are all multiple; The user's perspective information is uploaded to the cloud computing platform through one of the clients. The first rendered image is obtained by rendering the second 3DGS representation based on the user's perspective information through the cloud computing platform; A final rendering result is obtained by processing a first rendered image from the cloud computing platform through a client.

7. The method according to any one of claims 1-4, characterized in that, The method is applied to systems that include cloud computing platforms and clients; The user's perspective information is uploaded to the cloud computing platform via the client. Both the reference image and the second 3DGS representation are obtained through the cloud computing platform; Both the first rendered image and the final rendered result are obtained through processing by the client.

8. The method according to claim 7, characterized in that, The number of clients, the number of users, the number of the first rendered images, and the number of the final rendered results are all multiple; The user's perspective information is uploaded to the cloud computing platform through one of the clients. The first rendered image is obtained by rendering the second 3DGS representation from the cloud computing platform by a client based on the user's perspective information; One of the final rendering results is obtained by processing the first rendered image generated by the client itself through the client.

9. A rendering apparatus, characterized in that, include: The first rendering module is used to render the virtual digital scene at the first moment in order to obtain at least one reference image; An optimization module is used to update the appearance parameters of the first three-dimensional Gaussian (3DGS) representation of the virtual digital scene at a second time step, based on the reference image, so as to obtain the second 3DGS representation of the virtual digital scene at the first time step, wherein the second time step is earlier than the first time step; The second rendering module is used to render the second 3DGS representation based on the user's perspective information to obtain a first rendered image from the user's perspective. The generation module is used to obtain a final rendering result that is consistent with the perspective of the first rendered image, based on the first rendered image.

10. The apparatus according to claim 9, characterized in that, When the optimization module updates the appearance parameters of the first 3DGS representation of the virtual digital scene based on the reference image, it is specifically used for: By performing albedo decoupling on the reference image, at least the texture detail features in the reference image are removed to obtain the second rendered image; A third rendered image is obtained from the first 3DGS representation, wherein the third rendered image and the second rendered image have the same viewpoint; Based on the differences between the second rendered image and the third rendered image, the appearance parameters represented by the first 3DGS are updated.

11. The apparatus according to claim 9 or 10, characterized in that, When the generation module obtains a final rendering result consistent with the viewpoint of the first rendered image based on the first rendered image, it is specifically used for: Based on the first rasterized image and the first rendered image from the user's perspective, the final rendering result from the user's perspective is obtained. The first rasterized image is used to provide detailed features of the image from the user's perspective. The first rasterized image is obtained by rasterizing the virtual digital scene based on the user's perspective information.

12. The apparatus according to claim 11, characterized in that, Before obtaining the final rendering result from the user's perspective based on the first rasterized image and the first rendered image from the user's perspective, the generation module is further configured to: By performing albedo decoupling on the first rendered image, at least the texture detail features in the first rendered image are removed.

13. The apparatus according to any one of claims 9-12, characterized in that, The device is deployed in a system that includes a cloud computing platform and a client; The first rendering module, the optimization module, and the second rendering module are deployed on the cloud computing platform, the generation module is deployed on the client, and the user's perspective information is uploaded to the cloud computing platform through the client.

14. The apparatus according to claim 13, characterized in that, The number of clients, the number of users, the number of the first rendered images, and the number of the final rendered results are all multiple; The user's perspective information is uploaded to the cloud computing platform through one of the clients. The first rendered image is obtained by rendering the second 3DGS representation based on the user's perspective information using the second rendering module on the cloud computing platform; A final rendering result is obtained by processing a first rendered image from the cloud computing platform through a generation module on the client.

15. The apparatus according to any one of claims 9-12, characterized in that, The device is deployed in a system that includes a cloud computing platform and a client; The first rendering module and the optimization module are deployed on the cloud computing platform, the second rendering module and the generation module are deployed on the client, and the user's perspective information is uploaded to the cloud computing platform through the client.

16. The apparatus according to claim 15, characterized in that, The number of clients, the number of users, the number of the first rendered images, and the number of the final rendered results are all multiple; The user's perspective information is uploaded to the cloud computing platform through one of the clients. The first rendered image is obtained by rendering the second 3DGS representation from the cloud computing platform based on the user's perspective information by a second rendering module on the client. The final rendering result is obtained by processing the first rendered image generated by the client through a generation module on the client.

17. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-8.

18. A computer-readable storage medium, characterized in that, The method includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-8, wherein the cluster of computing devices includes at least one computing device.

19. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-8, wherein the computing device cluster includes at least one computing device.

Images