A rendering method and apparatus

Abstract

A rendering method and device, relating to the field of graphics rendering, the method comprising: obtaining sparse rendering data, the sparse rendering data comprising information of a small number of Gaussian spheres and neural texture data, the neural texture data being used to describe differences between the small number of Gaussian spheres and other Gaussian spheres; the number of the small number of Gaussian spheres being less than a first threshold; generating dense rendering data based on the sparse rendering data and a neural rendering model, the dense rendering data comprising information of a plurality of Gaussian spheres, the number of the plurality of Gaussian spheres being greater than a second threshold, the second threshold not being less than the first threshold; the second threshold being determined based on a number of Gaussian spheres required for a rendering image meeting a rendering quality requirement; and rendering using the dense rendering data to generate a second rendering image of a target object under a specified camera view angle. The method can only store sparse rendering data comprising information of a small number of Gaussian spheres when storing, thereby reducing model storage overhead of a three-dimensional scene.

Description

Technical Field

[0001] This application relates to the field of graphics rendering, and more particularly to a rendering method and apparatus. Background Technology

[0002] Traditional 3D graphics techniques typically rely on triangular meshes to depict object surfaces. While robust in most cases, they perform poorly when handling scenes with dense detail and dynamic changes. To overcome this limitation, researchers have explored new approaches based on volumetric representation, with 3D Gaussian Splatting being a key technique in its evolution. The core of 3D Gaussian Splatting lies in using a Gaussian distribution to define every tiny element in the scene—a Gaussian sphere.

[0003] In practical applications, millions of Gaussian spheres are placed in the scene, their parameters optimized to match real 3D data, ensuring high-quality images are generated quickly from any viewing angle. This technology has a wide range of applications, demonstrating exceptional value, especially in scenarios with extremely high requirements for real-time performance and interactivity. However, this solution requires storing a large number of Gaussian spheres in the scene, resulting in significant storage overhead. Summary of the Invention

[0004] This application provides a rendering method and apparatus for reducing model storage overhead in 3D scenes.

[0005] Firstly, this application provides a rendering method that can be executed by a rendering device (such as a terminal device, cloud rendering platform, or server device) or by a component of the rendering device (such as a processor). Taking the rendering device as an example, in this method, the rendering device acquires sparse rendering data, for example, the terminal device acquires it from a cloud rendering platform, or the rendering device acquires locally stored sparse rendering data, or the rendering device generates sparse rendering data. The sparse rendering data includes information about a small number of Gaussian spheres and neural texture data. The number of these small Gaussian spheres is less than a first threshold. These small Gaussian spheres are used to render a first rendered image of a target object from a specified camera viewpoint. The neural texture data is used to describe the differences between the small number of Gaussian spheres and other Gaussian spheres. Dense rendering data is generated based on the sparse rendering data and a neural rendering model. The dense rendering data includes information about multiple Gaussian spheres. The number of these multiple Gaussian spheres is greater than a second threshold. The second threshold is determined based on the number of Gaussian spheres required to meet the rendering quality requirements of the rendered image. For example, if the number of Gaussian spheres required to meet the rendering quality requirements is k, then the second threshold... The threshold can be k or a value greater than k, so that the rendered image based on the multiple Gaussian spheres can meet the rendering quality requirements. The second threshold is greater than or equal to the first threshold; that is, the first rendered image cannot meet the rendering quality requirements. The neural rendering model is used to generate a larger number of Gaussian spheres based on neural texture data and information from a small number of Gaussian spheres. In other words, the multiple Gaussian spheres included in the dense rendering data can be multiple Gaussian spheres generated iteratively by the neural rendering model and a small number of Gaussian spheres. The multiple Gaussian spheres included in the dense rendering data are used for rendering to generate a second rendered image of the target object under the specified camera viewpoint, and this second rendered image meets the rendering quality requirements.

[0006] The above design acquires sparse rendering data, which includes a small number of coarse-grained Gaussian spheres and neural texture data of the target object. A lightweight neural rendering model then generates a larger number of Gaussian spheres, and a rendered image meeting the required rendering quality is produced based on the dense rendering data. Since the sparse rendering data only includes a small number of Gaussian spheres, compared to existing methods that require storing all Gaussian spheres, this method significantly reduces the amount of data that needs to be stored, thus reducing model storage consumption.

[0007] In one possible design, the neural rendering model is used to filter Gaussian spheres based on the distance between the specified camera viewpoint and the Gaussian sphere; the distance between each Gaussian sphere generated by the neural rendering model and the specified camera viewpoint is no greater than a distance threshold.

[0008] Through the above design, the neural rendering model can adaptively generate Gaussian spheres that are closer to the target camera than the target sphere itself, based on the distance between the Gaussian sphere and the target camera. This means the distance to the target camera is within a distance threshold, allowing Gaussian spheres farther from the target camera to be deactivated in distant scenes, enabling rendering with a smaller number of Gaussian spheres and achieving efficient rendering in distant scenes. Compared to existing technologies that use the full number of Gaussian spheres for rendering regardless of the viewing distance, the adaptive rendering method provided in this application can further reduce storage consumption, improve rendering efficiency, and ensure rendering accuracy in close-range scenes.

[0009] In one possible design, the step of using the plurality of Gaussian spheres to render and generate a second rendered image of the target object from a specified camera viewpoint includes: determining a plurality of target Gaussian spheres from the plurality of Gaussian spheres, wherein the plurality of target Gaussian spheres include Gaussian spheres whose distance from the target camera viewpoint is not greater than a distance threshold; and rendering based on the plurality of target Gaussian spheres to generate the second rendered image.

[0010] Through the above design, Gaussian spheres that are far from the target camera are deactivated through post-processing, and a smaller number of Gaussian spheres are used for rendering, thus achieving efficient rendering in distant scenes.

[0011] In one possible design, the dense rendering data includes one or more layers of rendering data from the first output layer rendering data to the nth output layer rendering data generated by the neural rendering model based on the sparse rendering data; the first output layer rendering data is generated by the neural rendering model based on the sparse rendering data, the nth output layer rendering data is generated by the neural rendering model based on the (n-1)th output layer rendering data, the number of Gaussian spheres included in the i-th output layer rendering data is greater than the number of Gaussian spheres included in the (i-1)th output layer rendering data, and i takes any integer from 1 to n.

[0012] With the above design, since the nth rendering layer contains the most Gaussian spheres among the first to nth rendering layers, the rendering quality of the image rendered based on the nth rendering layer is the highest. Alternatively, rendering based on multiple rendering layers can also yield a high-resolution rendering image.

[0013] In one possible design, the method is applied to a terminal device; the acquisition of data from the first rendering layer includes: receiving the sparse rendering data sent by the server device, or receiving the sparse rendering data and neural rendering model parameters sent by the server device, wherein the neural rendering model parameters are used to construct the neural rendering model.

[0014] With the above design, the server device only sends the data of the first rendering layer to the terminal device. Since the first rendering layer includes a small number of Gaussian spheres, this method can significantly reduce the scale of data sent down from the server side.

[0015] In one possible design, the neural texture data includes each Gaussian sphere in the small number of Gaussian spheres, and the neural texture features represented by the corresponding raster in the trained neural texture space representation model; the neural texture space representation model includes at least one rasterized plane, each rasterized plane including multiple rasters, the multiple rasters being used to represent multiple neural texture features.

[0016] In one possible design, the small number of Gaussian spheres included in the sparse rendering data are generated based on multi-view images of the target object, which are obtained by acquiring or rendering multiple frames of the target object from multiple perspectives.

[0017] Secondly, this application provides a rendering method, which can be executed by a server or by a component of the server (such as a processor). Taking server execution as an example, in this method, the server generates sparse rendering data, which includes information about a small number of Gaussian spheres and neural texture data. The small number of Gaussian spheres is used to render a first rendered image of a target object from a specified camera viewpoint. The neural texture data is used to describe the differences between the small number of Gaussian spheres and multiple Gaussian spheres. The number of the small number of Gaussian spheres is less than a first threshold. The number of multiple Gaussian spheres is greater than a second threshold, and the second threshold is not less than the first threshold. The second threshold is determined based on the number of Gaussian spheres required to render an image that meets the rendering quality requirements. The sparse rendering data is then sent to a terminal device.

[0018] In one possible design, the neural texture data includes neural texture features represented by the corresponding grid in a trained neural texture spatial representation model for each of the few Gaussian spheres; the neural texture spatial representation model includes at least one rasterized plane, each rasterized plane including multiple grids, the multiple grids being used to represent multiple neural texture features.

[0019] In one possible design, the small number of Gaussian spheres included in the sparse rendering data are generated based on multi-view images of the target object, which are obtained by acquiring or rendering multiple frames of the target object from multiple perspectives.

[0020] Thirdly, this application also provides a rendering apparatus that has the function of implementing the behavior in the method example of the first aspect, or the rendering apparatus has the function of implementing the behavior in the method example of the second aspect. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the apparatus structure includes an acquisition module, a generation module, and a rendering module. Optionally, some or all of the modules can be the same module; for example, the generation module and the rendering module can also be the same module. These modules can perform the functions of the behavior in the method example of the first aspect, as detailed in the method example, and will not be repeated here. In another possible design, the apparatus structure includes a generation module and a sending module. Optionally, some or all of the modules can be the same module. These modules can perform the functions of the behavior in the method example of the second aspect, as detailed in the method example, and will not be repeated here.

[0021] Fourthly, this application also provides an electronic device including at least one processor and at least one memory. The one or more memories store one or more computer programs, each including instructions that, when executed by the one or more processors, cause the electronic device to perform any of the methods described in the first aspect above, or cause the electronic device to perform any of the methods described in the second aspect above.

[0022] Fifthly, this application also provides a chip including a processor and a power supply circuit. The power supply circuit is used to supply power to the processor, which is used to execute a computer program to implement any of the methods described in the first aspect above, or to implement any of the methods described in the second aspect above.

[0023] In a sixth aspect, this application also provides a device cluster including at least one electronic device, each electronic device including a processor and a memory; the processor of the at least one electronic device is configured to execute instructions stored in the memory of the at least one electronic device to cause the device cluster to perform any of the methods described in the first aspect above, or to cause the device cluster to perform any of the methods described in the second aspect above.

[0024] In a seventh aspect, this application provides a system comprising a server device and a terminal device, wherein the terminal device is configured to execute any of the methods provided in the first aspect above, and the server device is configured to execute any of the methods provided in the second aspect above.

[0025] Eighthly, this application provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform any of the methods provided in the first aspect above, or for causing a computer to perform any of the methods provided in the second aspect above.

[0026] Ninthly, this application provides a computer program product storing instructions that, when executed by a processor, implement any of the methods provided in the first aspect above, or implement any of the methods provided in the second aspect above.

[0027] The technical effects that can be achieved by any of the technical solutions in the second to ninth aspects mentioned above can be described with reference to the technical effects that can be achieved by the technical solution in the first aspect mentioned above, and the repeated parts will not be repeated. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of a system architecture provided in an embodiment of this application;

[0029] Figure 2 This is a schematic diagram of the process included in a rendering method provided in an embodiment of this application;

[0030] Figure 3A This is a schematic diagram of a multi-view image provided in an embodiment of this application;

[0031] Figure 3B This is a schematic diagram of another multi-view image provided in an embodiment of this application;

[0032] Figure 4 This is a schematic diagram illustrating the relationship between rendering layers provided in the embodiments of this application;

[0033] Figure 5 This is a schematic diagram illustrating a method for acquiring neural texture features according to an embodiment of this application;

[0034] Figure 6 This is a schematic diagram of a neural rendering model generating a Gaussian sphere according to an embodiment of this application;

[0035] Figure 7A These are schematic diagrams illustrating the rendering effects of different rendering layers provided in the embodiments of this application;

[0036] Figure 7B This is a comparative schematic diagram of the relationship between a neural texture map and a rendered image provided in an embodiment of this application;

[0037] Figure 7C This is a scene diagram showing different camera perspectives in the AR scene provided in the embodiments of this application;

[0038] Figure 8AThis is a schematic diagram of the structure of a rendering device provided in an embodiment of this application;

[0039] Figure 8B This is a schematic diagram of another rendering apparatus provided in an embodiment of this application;

[0040] Figure 9A This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0041] Figure 9B This is a schematic diagram of another computing device provided in an embodiment of this application. Detailed Implementation

[0042] First, some terms used in the embodiments of this application will be explained to facilitate understanding by those skilled in the art.

[0043] 1. 3D Scene Representation: A key technology in machine vision, it involves how to digitally describe and store real-world or imagined 3D objects and scenes in a computer. In other words, 3D scene representation uses computer graphics and 3D modeling techniques to mathematically represent the real world or creative ideas in a virtual environment. It includes information such as the shape, size, position, and spatial relationships of objects.

[0044] Specifically, 3D scene representation methods can include: (1) Voxel-based representation: Voxels are the basic units in 3D space, similar to pixels in 2D images. Voxel meshes can represent the shape and internal structure of 3D objects. (2) Mesh-based representation: Polygon meshes use vertices, edges, and faces to define the shape of 3D objects. It can flexibly represent various complex shapes. (3) Point cloud-based representation: Point clouds are collections of a large number of points in 3D space, which are usually obtained through scanning or measurement. Point cloud representation methods can capture the surface details of objects, but may be more complex in terms of processing and analysis. (4) Implicit representation: Instead of directly describing the surface or volume of objects, 3D scenes are implicitly represented through functions or neural networks. For example, Neural Radiance Fields (NeRF) is an implicit representation method that uses deep learning techniques to infer the propagation of light and the distribution of color in 3D scenes.

[0045] 3D scene representation has wide applications in many fields. For example, in robotics, it is used for tasks such as environmental perception, path planning, and object recognition. By building 3D scene models, robots can better understand their surroundings and make more accurate decisions. As another example, in augmented reality (AR) and virtual reality (VR) applications, 3D scene representation is key to achieving immersive experiences. By constructing realistic 3D scenes, users can interact and explore in virtual environments.

[0046] 2. Rendering: This is the process of converting a 3D scene representation into a 2D image. It simulates the process of a camera capturing an object, but uses computer graphics algorithms. Rendering techniques involve multiple aspects such as lighting, materials, textures, and shadows, aiming to generate realistic, high-quality images.

[0047] In one specific approach, rendering refers to the process of using software to render a 3D model of a scene to obtain a 2D image (i.e., a rendered image). The 3D model uses a strictly defined language or data structure to describe a three-dimensional object, including information such as geometry, viewpoint, texture, and lighting. The rendered image is a digital image or a bitmap image. Rendering can include offline rendering (pre-rendering) or real-time rendering (online rendering). Offline rendering generally takes longer to generate high-quality images and is typically used for real-world simulations with pre-defined scripts, such as in movies and advertisements. Real-time rendering is a fast-updating rendering method that updates the image in real time as the user interacts. Real-time rendering is commonly used in applications such as 3D games and virtual reality that lack pre-defined scripts and require rapid response and dynamic interaction.

[0048] 3. 3D Gaussian Splatting (3DGS): An advanced 3D scene representation and rendering technology with a wide range of applications, including but not limited to: Augmented Reality (AR) and Virtual Reality (VR), game development, biomedical modeling, animation and film production. 3DGS excels particularly in real-time scene reconstruction and new perspective compositing.

[0049] Specifically, the core of 3DGS lies in using a Gaussian distribution to define every tiny element in the scene—a Gaussian sphere. In essence, 3DGS transforms discrete data points or voxels into continuous surface or volume representations. Each data point or voxel can be considered a Gaussian sphere, its values ​​exhibiting a Gaussian distribution in space. By superimposing these Gaussian spheres, continuous surface or volume representations can be generated.

[0050] Gaussian spheres not only capture the geometric features of objects with greater detail but also accurately simulate the propagation and reflection of light, thus achieving a higher level of realistic rendering. In practical applications, millions of Gaussian spheres are placed in a scene, their parameters optimized to match real 3D data, ensuring high-quality images are generated quickly from any viewing angle. Therefore, 3DGS demonstrates exceptional value in scenarios with extremely high requirements for real-time performance and interactivity. For example, in the world of virtual reality and augmented reality, 3DGS can display high-fidelity virtual environments in real time, bringing users an unprecedented immersive experience.

[0051] 4. Gaussian sphere: A spherical region in three-dimensional space. The geometric properties of each Gaussian sphere can be determined by three parameters: center position, orientation, and variance. Each Gaussian sphere is also equipped with a set of spherical harmonic coefficients and opacity, which determine the color properties of the Gaussian sphere. They are like colored ellipsoids scattered in three-dimensional space. All Gaussian spheres together construct the visual image of the entire scene.

[0052] It should be noted that in practical applications, the Gaussian sphere may also have other names, such as Gaussian body, Gaussian kernel, Gaussian primitive, or Gaussian primitive.

[0053] No specific restrictions are imposed.

[0054] To address the problems raised in the background art, this application provides a rendering method. Taking a target object (such as a 3D model) as an example, this rendering method acquires sparse rendering data of the target object. The sparse rendering data includes information on a small number of Gaussian spheres, which can be used to generate a rendered image of the target object from a specified camera view (denoted as the first rendered image). The sparse rendering data also includes neural texture data, which describes the differences between the small number of Gaussian spheres and other Gaussian spheres. Dense rendering data is generated based on the sparse rendering data and the neural rendering model. The dense rendering data includes information on multiple Gaussian spheres, which are more numerous than the small number of Gaussian spheres. Rendering is performed using multiple Gaussian spheres to generate a rendered image of the target object from a specified camera view (denoted as the second rendered image). In this method, multiple Gaussian spheres for rendering are generated based on sparse rendering data and a lightweight neural rendering model. This allows for the storage of only information on a small number of Gaussian spheres. Compared to the original 3DGS scheme which stores a large number of Gaussian spheres, this application can effectively reduce storage consumption while maintaining rendering quality.

[0055] The rendering method provided in this application embodiment can have various applications. For example, the method can be applied to unilateral execution, specifically executed by a cloud rendering platform, terminal device, or rendering server. The rendering server executes the complete process steps of the rendering method provided in this application embodiment to obtain a rendered image. Optionally, the rendered image can be displayed on the rendering server, or the rendering server can send the rendered image to the terminal device for display. As another example, the method can be applied to bilateral interactive execution. Specifically, in this method, the rendering server generates data for a first rendering layer and sends the data of the first rendering layer and a neural rendering model to the terminal device. The terminal device generates a target rendering layer based on the first rendering layer and the neural rendering model, and uses the target rendering layer to render and obtain a rendered image. Optionally, the terminal device displays the rendered image.

[0056] The following explanation uses a system architecture applicable to a two-sided interactive execution scenario as an example. Figure 1 This is an exemplary architectural diagram of a system 100 provided in an embodiment of this application.

[0057] exist Figure 1 In this system, data acquisition device 160 is used to acquire images. In one application scenario, data acquisition device 160 acquires multi-view images of a target object. Multi-view images refer to images obtained by capturing the target object from different camera perspectives. The number of data acquisition devices 160 can be one or more, and is not specifically limited. After acquiring the multi-view images, optionally, data acquisition device 160 can store these multi-view images in database 130. It should be noted that in practical applications, the multi-view images stored in database 130 may not all originate from data acquisition device 160; they may also be received from other devices. In another optional approach, data acquisition device 160 can also directly send the multi-view images to the cloud side, such as a cloud computing system, cloud platform, or cloud server, etc., and is not specifically limited.

[0058] Figure 1 Taking cloud rendering platform 120 as an example, cloud rendering platform 120 is used to acquire multi-view images of the target object, and generate a three-dimensional scene representation of the target object under the specified camera view based on these multi-view images, namely a small number of Gaussian spheres.

[0059] Optionally, the cloud rendering platform 120 acquires multi-view images of the target object from a database 130 or a data acquisition device 160, and processes these multi-view images to generate a small amount of Gaussian sphere and neural texture data of the target object as sparse rendering data. Then, the cloud rendering platform 120 sends the sparse rendering data to the terminal device 110. In one embodiment, the cloud rendering platform 120 is equipped with a rendering application used to generate a small amount of Gaussian sphere information based on the multi-view images of the target object.

[0060] Optionally, the cloud rendering platform 120 can also send the trained neural rendering model or the trained parameters included in the neural rendering model to the terminal device 110. The terminal device 110 determines the target rendering layer based on the trained neural rendering model and the data of the first rendering layer of the target object.

[0061] Optionally, the trained neural rendering model can be obtained or received by the terminal device 110 from the cloud rendering platform 120, or it can be locally stored on the terminal device 110, or it can be obtained by the terminal device 110 from other locations such as an AI platform; the specific method is not limited. Alternatively, in the above embodiments, the trained neural rendering model can also be replaced by the trained parameters included in the neural rendering model, and the terminal device 110 constructs the trained neural rendering model based on the trained parameters included in the neural rendering model.

[0062] Specifically, the neural rendering model in this application embodiment can be a neural network model, which may include, but is not limited to, fully connected neural networks (FCNN), multilayer perceptrons (MLP), convolutional neural networks (CNN), recurrent neural networks (RNNS), etc. This application embodiment does not limit this. In some embodiments, the neural rendering model is a lightweight model with fewer parameters, thus reducing the computational load when executing the method of this application embodiment, lowering deployment costs, and making it suitable for rendering devices with greater computing power.

[0063] Terminal device 110 is also used to render using multiple Gaussian spheres included in dense rendering data to obtain a second rendered image of the target object from a specified camera viewpoint. In one embodiment, a rendering application client runs on terminal device 110. The rendering application client acts as a bridge between the user and the server. For example, the user inputs a command into the rendering application client, which may be to generate a rendered image of the target object from a specified camera viewpoint. The rendering application client translates the command into data or a request and sends it to cloud rendering platform 120. The rendering application in cloud rendering platform 120 acquires multi-view images of the target object, generates sparse rendering data of the target object based on the multi-view images, and sends the sparse rendering data to terminal device 110. The rendering application client generates dense rendering data based on the sparse rendering data and the rendering neural model, and renders it using multiple Gaussian spheres to generate a rendered image that meets the rendering quality requirements. The rendering application client presents the rendered image to the user. For example, the rendering application client may be a game application client, a VR application client, a movie special effects application client, or an animation application client, etc.

[0064] Optionally, the cloud rendering platform 120 may also use the rendering method provided in the embodiments of this application to generate a rendered image and send the rendered image to the terminal device 110, which will then present the rendered image to the user.

[0065] Terminal device 110 can be a terminal, such as a mobile phone, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, in-vehicle terminal, etc. Cloud rendering platform 120 may include devices or servers with data processing capabilities, such as cloud servers, network servers, application servers, rendering servers, and management servers. Optionally, the system architecture applicable to the embodiments of this application may not include cloud rendering platform 120; for example, in one possible design, the system includes a rendering server and terminal devices.

[0066] Next, let's apply it to... Figure 1 Taking the system shown as an example, the rendering method provided in this application embodiment will be described in detail.

[0067] Figure 2 This is a flowchart illustrating the rendering method provided in this application embodiment. The method can be executed by the cloud rendering platform 120 and the terminal device 110, or by components included in the cloud rendering platform 120 (such as a server or a processor included in the server) and components included in the terminal device 110 (such as a processor). Taking the cloud rendering platform 120 and the terminal device 110 as an example, the method may include some or all of the following steps: Figure 2 As shown, the method may include:

[0068] Step 201: The cloud rendering platform 120 acquires multi-view images of the target object.

[0069] The multi-view images of the target object include multiple frames of images of the target object, which show the target object from different camera perspectives.

[0070] In some embodiments, see Figure 3A As shown, multi-view images can be images obtained by capturing the target object from different camera perspectives using acquisition devices such as cameras.

[0071] In other embodiments, the multi-view images can be images reconstructed using techniques such as computer graphics. For example, for a 3D scene or 3D model in a 3D game (such as...). Figure 3B Images from different camera perspectives (as shown) were designed or reconstructed. It should be understood that... Figure 3B This is just a simple example and does not constitute a limitation on the solution.

[0072] Regardless of the method, each frame of a multi-view image includes part or all of the structure of the target object, which can be any physical or virtual 3D object, without any specific limitation.

[0073] Step 202: The cloud rendering platform 120 generates a small number of Gaussian spheres for the target object based on the multi-view images of the target object.

[0074] Taking multi-view images as an example, each frame of a multi-view image is a 2D image including part or all of the structure of the target object. Based on these multiple frames of 2D images including the target object, the cloud rendering platform 120 can use a 3D scene representation method to generate a 3DGS representation of the target object under a specified camera view, i.e., a Gaussian sphere. In some embodiments, the meta-rendering platform 120 directly generates a small number of Gaussian spheres, or the cloud rendering platform 120 generates multiple Gaussian spheres and then selects a small number of Gaussian spheres from the multiple Gaussian spheres.

[0075] Optionally, the aforementioned 3D scene representation method can be the original 3DGS method, or, i.e., using existing technology to generate a small number of Gaussian spheres for the target object. Alternatively, the 3D scene representation method can also use other methods, such as voxel-based methods, mesh-based methods, or point cloud-based methods to generate a small number of discrete data points or voxels, and then convert the small number of discrete data points or voxels into Gaussian spheres. Alternatively, the 3D scene representation method can also use methods that may be implemented in the future, without specific limitations.

[0076] The "small number of Gaussian spheres" here can be measured based on the total number of Gaussian spheres. For example, the number of Gaussian spheres may be less than a set number (such as a first threshold). The first threshold can be determined based on the number of Gaussian spheres required to render a rendered image that meets the required rendering quality (such as a second threshold). The first threshold can be any integer equal to or less than the second threshold. In other words, based on the small number of Gaussian spheres included in the first rendering layer, a first rendered image of the target object from a specified camera viewpoint can be rendered. This first rendered image may not meet the rendering quality requirements. For example, the first rendered image may have poor clarity, blurry details, or low realism. It should be noted that the first rendered image is used here to represent the function of the small number of Gaussian spheres, which is to construct the first rendered image. It is not necessary to use the first rendering layer to generate the first rendered image. In addition, those skilled in the art will understand that the small number of Gaussian spheres can not only render images from a specified camera viewpoint, but can also render images of the target object from other camera viewpoints, without any specific limitation.

[0077] In some embodiments, this application uses a small number of Gaussian spheres as the first rendering layer for the target object. The neural rendering model then iteratively generates more rendering layers based on the first rendering layer, see [link to relevant documentation]. Figure 4 As shown, the number of Gaussian spheres included in each rendering layer gradually increases. The small number of Gaussian spheres can also be reflected in the distribution density of the Gaussian spheres. Compared with other rendering layers, the first rendering layer includes a smaller number of more discrete Gaussian spheres, which is a relatively coarse-grained representation of the 3D scene.

[0078] Step 203: The cloud rendering platform 120 generates neural texture data of the target object.

[0079] In some embodiments, the neural texture data includes the neural texture information of each Gaussian sphere in some or all of the few Gaussian spheres. For ease of distinction, the few Gaussian spheres are referred to as the first rendering layer, and the Gaussian spheres included in the i-th rendering layer are referred to as the i-th Gaussian sphere. That is, all the few Gaussian spheres included in the first rendering layer are referred to as the first Gaussian spheres, and the Gaussian spheres included in the second rendering layer are referred to as the second Gaussian spheres. In step 203, the cloud rendering platform 120 can generate the neural texture information of each first Gaussian sphere. In one example, the neural texture information of the first Gaussian sphere is used to describe the differences between the first Gaussian sphere and multiple second Gaussian spheres. It should be noted that the neural texture information of the first Gaussian sphere is actually used to describe the differences between the first Gaussian sphere and other multiple Gaussian spheres. The multiple Gaussian spheres here are not the Gaussian spheres included in the few Gaussian spheres, but multiple Gaussian spheres that can be generated based on the first Gaussian sphere and the neural texture information of the first Gaussian sphere. Here, these other multiple Gaussian spheres are referred to as the second Gaussian spheres.

[0080] Optionally, the neural texture information of the first Gaussian sphere is determined based on a trained neural texture space representation model. This neural texture space representation model has multiple learnable neural texture features. The first Gaussian sphere is mapped (e.g., projected) onto this neural texture space representation model to obtain the corresponding neural texture features of the first Gaussian sphere in the neural texture space representation model. The neural texture space representation model may include triplanar neural texture representation, three-dimensional voxel texture representation, vector-matrix decomposition representation, CANDECOMP / PARAFAC representation, etc., and is not specifically limited. Triplanar neural texture representation is a texture representation method used in 3D modeling that combines the feature extraction capabilities of neural networks with the advantages of triplanar mapping techniques.

[0081] The following combination Figure 5 Taking the three-plane neural texture representation shown as an example, this section introduces how to determine the neural texture information of the first Gaussian sphere. The process may include:

[0082] Step 203a: Determine the three-plane representation of the neural texture corresponding to the first rendering layer.

[0083] Step 203b: Project each first Gaussian sphere onto the three-plane representation of the neural texture to obtain the neural texture information of each first Gaussian sphere.

[0084] In this embodiment, the three-plane neural texture representation is obtained as follows: First, a neural texture of a Gaussian sphere (not limited to a first Gaussian sphere) is generated using a neural network. The neural texture is a set of learnable high-dimensional feature maps that can be interpreted by a neural rendering model. Then, the generated neural texture is rasterized into three orthogonal view planes and placed in three axis-aligned feature planes, which correspond to the X, Y, and Z directions in 3D space, respectively. See also Figure 5 As shown, for any point in 3D space (such as a sampling point of the first Gaussian sphere), it is projected onto each plane, and the features are sampled to obtain the neural texture features of that point.

[0085] In short, the neural texture triplane comprises three rasterized planes, each containing multiple grids, each corresponding one-to-one with a neural texture feature. Projecting the sampling points of the first Gaussian sphere onto each rasterized plane, the neural texture feature corresponding to the grid containing each projection point is the neural texture feature of the first Gaussian sphere. The neural texture information of the first Gaussian sphere includes one or more neural texture features of the first Gaussian sphere. Optionally, multiple neural texture features of the first Gaussian sphere can be aggregated (stitched) using methods such as summation or concatenation, and the aggregated information can be used as the neural texture information of the Gaussian sphere.

[0086] Optionally, the sampling points of the first Gaussian sphere can be taken from points on or inside the first Gaussian sphere. Alternatively, see [link to relevant documentation]. Figure 5 The sampling points of the first Gaussian sphere can be sampling points determined based on the axis-aligned bounding box of the first Gaussian sphere, such as the vertices or center points of the axis-aligned bounding box of the first Gaussian sphere, etc., without any specific limitation. The number of sampling points can be one or more. For example, the sampling points of the first Gaussian sphere include the 8 vertices of the axis-aligned bounding box of the first Gaussian sphere, or one center point. Again, there is no specific limitation. The axis-aligned bounding box of each first Gaussian sphere is vertically projected onto each plane in the three-plane representation of the neural texture, and the neural texture information of each first Gaussian sphere is obtained through the method described above.

[0087] It should be understood that (first) Gaussian spheres are actually distributed in 3D space. Therefore, the neural texture triplane representation corresponding to the first rendering layer can refer to the smallest triplane in 3D space that can enclose all the first Gaussian spheres included in the first rendering layer. It is worth noting that the neural texture triplane representation in this embodiment is a trained representation model. The neural texture features included in the neural texture triplane representation can be used to characterize the differences between Gaussian spheres in different layers. The training of the neural texture triplane representation can be performed based on the rendering effect of the rendered image generated in step 206. For example, when the rendering effect of the rendered image reaches a set threshold, the training is considered complete. The rendering effect can be evaluated based on factors such as the accuracy and clarity of the rendered image.

[0088] In another example, the neural texture data of the target object may include neural texture information of a first Gaussian sphere and neural texture information of Gaussian spheres included in one or more other rendering layers, such as the neural texture information of a second Gaussian sphere, which is used to describe the differences between the second Gaussian sphere and multiple third Gaussian spheres. The neural texture information of a third Gaussian sphere is also used to describe the differences between the third Gaussian sphere and multiple fourth Gaussian spheres. This application does not limit this aspect.

[0089] Optionally, the neural texture information of some or all of the first Gaussian spheres included in the first rendering layer can be stored as a single frame of neural texture map, serving as the aforementioned neural texture data of a small number of Gaussian spheres. The resolution of this neural texture map can meet the resolution requirements of the rendered image quality. Storing or transmitting neural texture data in the form of neural texture maps is more effective in saving storage resources and data transmission bandwidth.

[0090] In addition to neural texture data, sparse rendering data also includes information about a small number of Gaussian spheres (i.e., a small number of first Gaussian spheres). The information for each Gaussian sphere may include, but is not limited to, the distance between the Gaussian sphere and the target camera, and one or more of the Gaussian sphere's properties. For example, in one approach, the information about the Gaussian sphere includes its properties. In another approach, the information about the Gaussian sphere includes both its properties and the distance between the Gaussian sphere and the target camera. The properties of the Gaussian sphere may include, but are not limited to, one or more of the following: size, color, opacity, transparency, and spatial position.

[0091] It should be understood that since the first rendering layer is used to render the target object as seen from a specified camera viewpoint, the target camera here is the "camera" within that specified "camera" viewpoint; in other words, the target camera is the camera located at that specified camera viewpoint position. This can be understood as the rendered image being the image obtained by the target camera capturing the target object from the specified camera viewpoint.

[0092] There are several ways to determine the distance between the Gaussian sphere and the target camera. For example, it can be determined by: establishing a 3D coordinate system, determining the spatial position of the target camera and the Gaussian sphere in the coordinate system, and determining the distance between them based on the coordinates of their spatial positions.

[0093] Step 204: The cloud rendering platform 120 sends the sparse rendering data to the terminal device 110.

[0094] Optionally, the cloud rendering platform 120 can also send the trained neural rendering model to the terminal device 110. This neural rendering model can be sent along with the data from the first rendering layer, or it can be sent separately, such as before or after the sparse rendering data.

[0095] In an alternative approach, the cloud rendering platform 120 sends the trained parameters of the trained neural rendering model to the terminal device 110. This replaces sending the neural rendering model, further reducing the amount of data transmitted between the cloud rendering platform 120 and the terminal device 110. The terminal device 110 can then construct the trained neural rendering model based on these trained parameters.

[0096] Step 205: Terminal device 110 generates dense rendering data based on sparse rendering data and the trained neural rendering model.

[0097] This neural rendering model generates a larger number of Gaussian spheres based on the input data of the i-th rendering layer, serving as the (i+1)-th rendering layer. The (i+1)-th rendering layer is the layer below the i-th rendering layer, where i is a positive integer. (The return value is missing from the original text.) Figure 4The number of Gaussian spheres included in the (i+1)th rendering layer is greater than the number of Gaussian spheres included in the ith rendering layer.

[0098] For example, the neural rendering model can generate a larger number of Gaussian spheres than the small number of Gaussian spheres included in the first rendering layer, and these newly generated Gaussian spheres serve as the second rendering layer. Then, the neural rendering model can generate a larger number of Gaussian spheres than the second rendering layer itself, based on the Gaussian spheres included in the second rendering layer, as the third rendering layer. And so on.

[0099] This step may include: inputting the data and neural texture data of the first rendering layer into the neural rendering model; the neural rendering model generating the second Gaussian sphere included in the second rendering layer; and the neural rendering model outputting the data of the second rendering layer. The data of the second rendering layer may include information about each second Gaussian sphere. The information about the second Gaussian sphere may include one or more of the following: the attributes of the second Gaussian sphere, the distance between the second Gaussian sphere and the target camera, and the neural texture information of the second Gaussian sphere. For example, the information about the second Gaussian sphere may include the distance between the second Gaussian sphere and the target camera. Another example is that the information about the second Gaussian sphere may include the attributes of the second Gaussian sphere. Yet another example is that the information about the second Gaussian sphere may include the distance between the second Gaussian sphere and the target camera, the attributes of the second Gaussian sphere, and the neural texture information of the second Gaussian sphere, etc. The meaning and determination method of the distance between the second Gaussian sphere and the target camera, the attributes of the second Gaussian sphere, and the neural texture information of the second Gaussian sphere can be found in the relevant introduction above, and will not be repeated here. Similar explanations will not be repeated below.

[0100] From a micro perspective, see Figure 6 Taking a Gaussian sphere as an example, this neural rendering model can generate multiple second Gaussian spheres based on the information of a Gaussian sphere and its neural texture information. These multiple second Gaussian spheres can be related to the first Gaussian sphere, such as being Gaussian spheres surrounding the first Gaussian sphere, or being Gaussian spheres within a set distance range from the first Gaussian sphere. In this way, the neural rendering model can generate a larger number of second Gaussian spheres based on the small number of first Gaussian spheres included in the first rendering layer.

[0101] Similarly, see Figure 7A As shown, the neural rendering model can also generate data for a third rendering layer based on the data from the second rendering layer. The third rendering layer includes a greater number of third Gaussian spheres than the second rendering layer. The data for the second rendering layer can be found in the description of the data for the first rendering layer, and will not be repeated here.

[0102] In some embodiments, the neural rendering data carried in the sparse rendering data includes the differences between Gaussian spheres included in any two adjacent rendering layers. In this step, information about a small number of Gaussian spheres included in the i-th rendering layer and the same set of neural rendering data can be directly input into the neural rendering model to generate the (i+1)-th rendering layer. That is, multiple rendering layers can share the neural texture data of the first rendering layer; that is, the neural texture data of each rendering layer can be the same. In this case, the neural texture data of the first rendering layer can include the differences between Gaussian spheres included in multiple rendering layers. This neural rendering data can be as follows: Figure 7B The neural texture map shown may, in one possible case, result in a rendered image that is missing parts of the neural texture map if some areas are missing.

[0103] Alternatively, in some embodiments, the neural rendering data carried in the sparse rendering data only includes the neural texture information of the first Gaussian sphere. That is, the neural texture data of each rendering layer can be different. In this case, when generating the data for the third rendering layer based on the data of the second rendering layer, the neural texture information of the second Gaussian sphere needs to be generated using the method described above. Afterward, the information of the second Gaussian sphere included in the second rendering layer and its neural texture information are input into the neural rendering model to generate the data for the third rendering layer. That is, each time a new rendering layer is generated, the neural texture data of the i-th rendering layer is updated using the method described above.

[0104] Optionally, the neural rendering model can also generate data for a fourth rendering layer based on the data from the third rendering layer, where the fourth rendering layer includes a greater number of fourth Gaussian spheres than the third rendering layer includes. And so on.

[0105] It is worth noting that, see Figure 7A As shown in the example above, regardless of which rendering layer is used, an image of the target object from the specified camera's perspective can be obtained. The difference lies in the rendering quality of the images obtained from different rendering layers. For example, as mentioned earlier, the number of Gaussian spheres included in each rendering layer, from largest to smallest, is: fourth rendering layer > third rendering layer > second rendering layer > first rendering layer. Therefore, the rendering quality of the images obtained from each rendering layer, from highest to lowest, is: fourth rendering layer > third rendering layer > second rendering layer > first rendering layer.

[0106] In this paper, the neural rendering model generates the (i+1)th rendering layer based on the data of the i-th rendering layer. This means generating the Gaussian sphere included in the (i+1)th rendering layer. It can also be described as the neural rendering model generating the (i+1)th rendering layer based on the i-th rendering layer.

[0107] The process of generating the Nth rendering layer from the first rendering layer can be called iteration, that is, the neural rendering model can iteratively generate the Nth rendering layer based on the first rendering layer.

[0108] Dense rendering data includes information about all Gaussian spheres included in one or more rendering layers from the first to the Nth rendering layer. For example, dense rendering data includes multiple consecutive or non-consecutive rendering layers. For instance, taking N=4 as an example, dense rendering data includes information about multiple Gaussian spheres included in the fourth rendering layer generated by the neural rendering model. Alternatively, dense rendering data includes information about Gaussian spheres included in the second, third, and fourth rendering layers. Or, dense rendering data includes information about Gaussian spheres included in the second and fourth rendering layers of the target rendering layer. Or, dense rendering data includes information about Gaussian spheres included in the first, second, third, and fourth rendering layers, etc.

[0109] The number of Gaussian spheres included in the dense rendering data is greater than or equal to a second threshold. The second threshold can be determined based on the number of Gaussian spheres required to render a rendering image that meets the rendering quality requirements. For example, the second threshold may be greater than or equal to the required number, but there is no specific limitation.

[0110] Step 206: The terminal device 110 uses dense rendering data to render and obtain a rendered image of the target object from the perspective of a specified camera (denoted as the second rendered image).

[0111] In one optional implementation, when the dense rendering data includes data from the Nth rendering layer, the terminal device 110 uses the Gaussian spheres included in the Nth rendering layer for rendering to obtain a second rendered image. When the dense rendering data includes data from multiple rendering layers, the terminal device 110 uses all the Gaussian spheres included in the multiple rendering layers for rendering to obtain a second rendered image. The Gaussian spheres included in the multiple rendering layers are spatially superimposed; some Gaussian spheres may overlap or not, which does not affect the final rendering effect.

[0112] Through the above design, the 3D data of the target object is decomposed into coarse-grained geometric information (i.e., the information of a small number of Gaussian spheres included in the first rendering layer) and high-resolution neural texture data. Finally, a lightweight neural rendering model is used to generate fine-grained geometric information, i.e., the information of multiple Gaussian spheres included in the dense rendering data. Based on the dense rendering data, a rendered image that meets the rendering quality is produced. Compared with existing methods, the cloud rendering platform 120 can store only a small number of Gaussian sphere information and neural texture data, without needing to store the full number of Gaussian spheres. This not only reduces model storage consumption but also effectively reduces the scale of data distributed by the cloud rendering platform 120.

[0113] It should be noted that, Figure 2 The example shown is that the rendering method is executed by the cloud rendering platform 120 and the terminal device 110. The rendering method provided in this embodiment can also be executed by one side, for example, entirely by the terminal device 110 or entirely by the cloud rendering platform 120. In an alternative solution, the cloud rendering platform 120 can also be replaced by one or more rendering servers, which is not limited in this embodiment.

[0114] Building upon the above solution, this application further provides an adaptive rendering method based on viewing distance. Viewing distance can refer to the distance between the target camera and the target object in a 3D scene, see [link to relevant documentation]. Figure 7C As shown, zooming in and out to observe 3D objects is a high-frequency operation in AR scenes. The embodiments of this application can adaptively select a Gaussian sphere in the scene for rendering based on the observation distance, thereby improving rendering efficiency and rendering effect and reducing storage consumption by selecting different Gaussian spheres for rendering at different observation distances.

[0115] The adaptive rendering method provided in this application embodiment can be implemented in two ways, which are described below:

[0116] In one implementation, post-processing is used: Utilizing the Gaussian spheres included in the target rendering layer generated in step 205, for a given camera position, the distance between the target camera and the Gaussian spheres is calculated, and a target Gaussian sphere is adaptively selected from the Gaussian spheres included in the target rendering layer for rendering. The target Gaussian sphere is one whose distance from the target camera is no greater than a distance threshold.

[0117] In another implementation, the aforementioned neural rendering model is used: this model adaptively generates Gaussian spheres based on the viewing distance. That is, the neural rendering module can filter Gaussian spheres based on the distance between the target camera (specified camera viewpoint) and the Gaussian sphere image, deactivating Gaussian spheres whose distance from the target camera exceeds a distance threshold. In other words, all Gaussian spheres generated by this neural rendering model are target Gaussian spheres, meaning the distance between the generated Gaussian spheres and the target camera is no greater than the distance threshold. Therefore, all Gaussian spheres included in the target rendering layer are target Gaussian spheres, and no post-processing filtering is required.

[0118] The above method allows for the deactivation of Gaussian spheres located far from the target camera when the viewing distance is long, enabling rendering with a smaller number of Gaussian spheres and achieving efficient rendering in long-distance scenes. Compared to existing technologies that use all Gaussian spheres for rendering regardless of the viewing distance, the adaptive rendering method provided in this application can further reduce storage consumption, improve rendering efficiency, and ensure rendering accuracy in close-range scenes while eliminating jagged noise rendered by Gaussian spheres located far from the target camera in long-distance scenes.

[0119] Based on the same inventive concept as the method embodiments, this application also provides a rendering apparatus for performing the above-described... Figure 2 Methods for the behavior of Zhongyun rendering platform 120 or terminal device 110. For example... Figure 8A As shown, in one example, the rendering apparatus 800 includes an acquisition module 801, a generation module 802, and a rendering module 803. Specifically, in the rendering apparatus 800, the modules are connected through a communication path.

[0120] The acquisition module is used to acquire sparse rendering data, which includes information about a small number of Gaussian spheres and neural texture data. The small number of Gaussian spheres are used to render a first rendered image of the target object from a specified camera viewpoint. The neural texture data is used to describe the differences between the small number of Gaussian spheres and other Gaussian spheres. The number of the small number of Gaussian spheres is less than a first threshold. See details... Figure 2 The description of step 204 in the method embodiment will not be repeated here.

[0121] A generation module is used to generate dense rendering data based on the sparse rendering data and the neural rendering model. The dense rendering data includes information about multiple Gaussian spheres, the number of which is greater than a second threshold, and the second threshold is not less than a first threshold. The second threshold is determined based on the number of Gaussian spheres required to render an image that meets the rendering quality requirements. The neural rendering model is used to generate a larger number of Gaussian spheres based on the neural texture data and the information from the small number of Gaussian spheres. See details... Figure 2The description of step 205 in the method embodiment will not be repeated here.

[0122] The rendering module is used to render using the multiple Gaussian spheres, generating a second rendered image of the target object from a specified camera viewpoint. See details... Figure 2 The description of step 206 in the method embodiment will not be repeated here.

[0123] In one possible implementation, the neural rendering model is used to filter Gaussian spheres based on the distance between the specified camera viewpoint and the Gaussian sphere; the distance between each Gaussian sphere generated by the neural rendering model and the specified camera viewpoint is no greater than a distance threshold.

[0124] In one possible implementation, when the rendering module uses the plurality of Gaussian spheres to render and generate a second rendered image of the target object from a specified camera viewpoint, it is specifically configured to: determine a plurality of target Gaussian spheres from the plurality of Gaussian spheres, wherein the plurality of target Gaussian spheres include Gaussian spheres whose distance from the target camera viewpoint is not greater than a distance threshold; and render based on the plurality of target Gaussian spheres to generate the second rendered image.

[0125] In one possible implementation, the dense rendering data includes one or more layers of rendering data from the first output layer rendering data to the nth output layer rendering data generated by the neural rendering model based on the sparse rendering data; the first output layer rendering data is generated by the neural rendering model based on the sparse rendering data, the nth output layer rendering data is generated by the neural rendering model based on the (n-1)th output layer rendering data, the number of Gaussian spheres included in the i-th output layer rendering data is greater than the number of Gaussian spheres included in the (i-1)th output layer rendering data, and i takes any integer from 1 to n.

[0126] In one possible implementation, the neural texture data includes each Gaussian sphere in the small number of Gaussian spheres, and the neural texture features represented by the corresponding raster in the trained neural texture space representation model; the neural texture space representation model includes at least one rasterized plane, each rasterized plane including multiple rasters, the multiple rasters being used to represent multiple neural texture features.

[0127] In one possible implementation, the small number of Gaussian spheres included in the sparse rendering data are generated based on multi-view images of the target object, which are obtained by acquiring or rendering multiple frames of the target object from multiple perspectives.

[0128] Based on the same inventive concept as the method embodiments, this application also provides a rendering apparatus for performing the above-described... Figure 2The method executed by the Zhongyun rendering platform 120. For example... Figure 8B As shown, in one example, the rendering apparatus 900 includes a generation module 901 and a sending module 902. Specifically, in the rendering apparatus 900, the modules are connected through a communication path.

[0129] Generation module 901 is used to generate sparse rendering data, which includes information about a small number of Gaussian spheres and neural texture data. The small number of Gaussian spheres is used to render a first rendered image of the target object from a specified camera viewpoint. The neural texture data is used to describe the differences between the small number of Gaussian spheres and multiple Gaussian spheres. The number of the small number of Gaussian spheres is less than a first threshold. The number of multiple Gaussian spheres is greater than a second threshold, and the second threshold is not less than the first threshold. The second threshold is determined based on the number of Gaussian spheres required to render an image that meets the rendering quality requirements. See details [link to relevant documentation]. Figure 2 The descriptions of steps 202 and 203 in the method embodiment will not be repeated here.

[0130] The sending module 902 is used to send the sparse rendering data to the terminal device. See details below. Figure 2 The description of step 204 in the method embodiment will not be repeated here.

[0131] In one possible implementation, the neural texture data includes neural texture features represented by the corresponding grid in a trained neural texture spatial representation model for each of the few Gaussian spheres; the neural texture spatial representation model includes at least one rasterized plane, each rasterized plane including multiple grids, the multiple grids being used to represent multiple neural texture features.

[0132] In one possible implementation, the small number of Gaussian spheres included in the sparse rendering data are generated based on multi-view images of the target object, which are obtained by acquiring or rendering multiple frames of the target object from multiple perspectives.

[0133] It should be noted that the module division in this embodiment is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. The functional modules in this embodiment can be integrated into one module, or each module can exist physically separately, or two or more modules can be integrated into one module. For example, the generation module 802 and the rendering module 803 can be integrated into one module. The integrated units described above can be implemented in hardware or as software functional units.

[0134] This application also provides a computing device 1000. For example... Figure 9AAs shown, the computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002. The computing device 1000 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1000.

[0135] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 9A The bus 1002 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1002 may include a path for transmitting information between various components of the computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0136] The processor 1004 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0137] The memory 1006 may include volatile memory, such as random access memory (RAM). The processor 1004 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0138] In one possible design, the memory 1006 stores executable program code, which the processor 1004 executes to implement the functions of the aforementioned acquisition module 801, generation module 802, and rendering module 803, thereby realizing the rendering method. That is, the memory 1006 stores instructions for the rendering device 800 to execute the rendering method provided in this application.

[0139] In another possible design, see Figure 9B As shown, the memory 1006 stores executable program code, and the processor 1004 executes the executable program code to implement the functions of the aforementioned generation module 901 and sending module 902, thereby realizing the rendering method. That is, the memory 1006 stores instructions for the rendering device 900 to execute the rendering method provided in this application.

[0140] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0141] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a desktop computer, a laptop computer, or a smartphone, or other terminal device.

[0142] Based on the above embodiments, this application also provides a computer program that, when run on a computer, causes the computer to perform... Figure 2 The rendering method provided in the illustrated embodiment.

[0143] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, causes the computer to perform... Figure 2 The illustrated embodiment provides a rendering method. The storage medium can be any available medium that a computer can access. For example, but not limited to, a computer-readable medium can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code having an instruction or data structure form and accessible by a computer.

[0144] Based on the above embodiments, this application also provides a chip, which is used to read a computer program stored in a memory and implement... Figure 2 The rendering method provided in the illustrated embodiment.

[0145] Based on the above embodiments, this application provides a chip system including a processor for supporting computer devices to implement... Figure 2 The illustrated embodiment provides a rendering method. In one possible design, the chip system further includes a memory for storing programs and data necessary for the computer device. The chip system may consist of chips or may include chips and other discrete components.

[0146] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0148] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0150] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of protection of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A rendering method, characterized in that, The method includes: Acquire sparse rendering data, which includes information about a small number of Gaussian spheres and neural texture data. The small number of Gaussian spheres are used to render a first rendered image of the target object from a specified camera viewpoint. The neural texture data is used to describe the differences between the small number of Gaussian spheres and other Gaussian spheres. The number of the small number of Gaussian spheres is less than a first threshold. Dense rendering data is generated based on the sparse rendering data and the neural rendering model. The dense rendering data includes information on multiple Gaussian spheres. The number of Gaussian spheres is greater than a second threshold, and the second threshold is not less than the first threshold. The second threshold is determined based on the number of Gaussian spheres required to render an image that meets the rendering quality requirements. The neural rendering model is used to generate a larger number of Gaussian spheres based on the neural texture data and the information on the small number of Gaussian spheres. The dense rendering data is used to render a second rendered image of the target object from a specified camera viewpoint.

2. The method as described in claim 1, characterized in that, The neural rendering model is used to filter Gaussian spheres based on the distance between the specified camera viewpoint and the Gaussian sphere; the distance between each Gaussian sphere generated by the neural rendering model and the specified camera viewpoint is no greater than a distance threshold.

3. The method as described in claim 1, characterized in that, The step of using the plurality of Gaussian spheres to render and generate a second rendered image of the target object from a specified camera viewpoint includes: Multiple target Gaussian spheres are determined from the plurality of Gaussian spheres, wherein the plurality of target Gaussian spheres include Gaussian spheres whose distance from the target camera's viewpoint is not greater than a distance threshold. The second rendered image is generated by rendering based on the plurality of target Gaussian spheres.

4. The method according to any one of claims 1-3, characterized in that, The dense rendering data includes one or more layers of rendering data from the first output layer rendering data to the nth output layer rendering data generated by the neural rendering model based on the sparse rendering data. The first output layer rendering data is generated by the neural rendering model based on the sparse rendering data, the nth output layer rendering data is generated by the neural rendering model based on the (n-1)th output layer rendering data, the number of Gaussian spheres included in the i-th output layer rendering data is greater than the number of Gaussian spheres included in the (i-1)th output layer rendering data, and i takes any integer from 1 to n.

5. The method according to any one of claims 1-4, characterized in that, The neural texture data includes each Gaussian sphere in the small number of Gaussian spheres, and the neural texture features represented by the corresponding raster in the trained neural texture space representation model; the neural texture space representation model includes at least one rasterized plane, each rasterized plane includes multiple rasters, and the multiple rasters are used to represent multiple neural texture features.

6. The method according to any one of claims 1-5, characterized in that, The sparse rendering data includes a small number of Gaussian spheres generated based on multi-view images of the target object, which are obtained by acquiring or rendering multiple frames of the target object from multiple perspectives.

7. A rendering method, characterized in that, The method includes: Sparse rendering data is generated, comprising information about a small number of Gaussian spheres and neural texture data. The small number of Gaussian spheres is used to render a first rendered image of the target object from a specified camera viewpoint. The neural texture data is used to describe the differences between the small number of Gaussian spheres and multiple Gaussian spheres. The number of the small number of Gaussian spheres is less than a first threshold. The number of multiple Gaussian spheres is greater than a second threshold, and the second threshold is not less than the first threshold. The second threshold is determined based on the number of Gaussian spheres required to render an image that meets the rendering quality requirements. The sparse rendering data is sent to the terminal device.

8. The method as described in claim 7, characterized in that, The neural texture data includes the neural texture features represented by the corresponding grid in the trained neural texture space representation model for each Gaussian sphere in the small number of Gaussian spheres; the neural texture space representation model includes at least one rasterized plane, each rasterized plane includes multiple grids, and the multiple grids are used to represent multiple neural texture features.

9. The method as described in claim 7 or 8, characterized in that, The sparse rendering data includes a small number of Gaussian spheres generated based on multi-view images of the target object, which are obtained by acquiring or rendering multiple frames of the target object from multiple perspectives.

10. A rendering apparatus, characterized in that, The device includes: An acquisition module is used to acquire sparse rendering data, which includes information on a small number of Gaussian spheres and neural texture data. The small number of Gaussian spheres are used to render a first rendered image of a target object from a specified camera viewpoint. The neural texture data is used to describe the differences between the small number of Gaussian spheres and other Gaussian spheres. The number of the small number of Gaussian spheres is less than a first threshold. A generation module is used to generate dense rendering data based on the sparse rendering data and the neural rendering model. The dense rendering data includes information on multiple Gaussian spheres, the number of which is greater than a second threshold, and the second threshold is not less than a first threshold. The second threshold is determined based on the number of Gaussian spheres required to render an image that meets the rendering quality requirements. The neural rendering model is used to generate a larger number of Gaussian spheres based on the neural texture data and the information on the small number of Gaussian spheres. The rendering module is used to render using the dense rendering data to generate a second rendered image of the target object from a specified camera viewpoint.

11. The apparatus as claimed in claim 10, characterized in that, The neural rendering model is used to filter Gaussian spheres based on the distance between the specified camera viewpoint and the Gaussian sphere; the distance between each Gaussian sphere generated by the neural rendering model and the specified camera viewpoint is no greater than a distance threshold.

12. The apparatus as claimed in claim 10, characterized in that, When the rendering module uses the plurality of Gaussian spheres to render and generate a second rendered image of the target object from a specified camera viewpoint, it is specifically used to: determine a plurality of target Gaussian spheres from the plurality of Gaussian spheres, wherein the plurality of target Gaussian spheres include Gaussian spheres whose distance from the target camera viewpoint is not greater than a distance threshold; and render based on the plurality of target Gaussian spheres to generate the second rendered image.

13. The apparatus according to any one of claims 10-12, characterized in that, The dense rendering data includes one or more layers of rendering data from the first output layer rendering data to the nth output layer rendering data generated by the neural rendering model based on the sparse rendering data. The first output layer rendering data is generated by the neural rendering model based on the sparse rendering data, the nth output layer rendering data is generated by the neural rendering model based on the (n-10)th output layer rendering data, the number of Gaussian spheres included in the i-th output layer rendering data is greater than the number of Gaussian spheres included in the (i-10)th output layer rendering data, and i takes any integer from 10 to n.

14. The apparatus according to any one of claims 10-13, characterized in that, The neural texture data includes each Gaussian sphere in the small number of Gaussian spheres, and the neural texture features represented by the corresponding raster in the trained neural texture space representation model; the neural texture space representation model includes at least one rasterized plane, each rasterized plane includes multiple rasters, and the multiple rasters are used to represent multiple neural texture features.

15. The apparatus according to any one of claims 10-14, characterized in that, The sparse rendering data includes a small number of Gaussian spheres generated based on multi-view images of the target object, which are obtained by acquiring or rendering multiple frames of the target object from multiple perspectives.

16. A rendering apparatus, characterized in that, The device includes: A generation module is used to generate sparse rendering data, which includes information about a small number of Gaussian spheres and neural texture data. The small number of Gaussian spheres is used to render a first rendered image of a target object from a specified camera viewpoint. The neural texture data is used to describe the differences between the small number of Gaussian spheres and multiple Gaussian spheres. The number of the small number of Gaussian spheres is less than a first threshold. The number of multiple Gaussian spheres is greater than a second threshold, and the second threshold is not less than the first threshold. The second threshold is determined based on the number of Gaussian spheres required to render an image that meets the rendering quality requirements. The sending module is used to send the sparse rendering data to the terminal device.

17. The apparatus as claimed in claim 16, characterized in that, The neural texture data includes the neural texture features represented by the corresponding grid in the trained neural texture space representation model for each Gaussian sphere in the small number of Gaussian spheres; the neural texture space representation model includes at least one rasterized plane, each rasterized plane includes multiple grids, and the multiple grids are used to represent multiple neural texture features.

18. The apparatus as claimed in claim 16 or 17, characterized in that, The sparse rendering data includes a small number of Gaussian spheres generated based on multi-view images of the target object, which are obtained by acquiring or rendering multiple frames of the target object from multiple perspectives.

19. A rendering system, characterized in that, Including servers and terminal devices; The terminal device is configured to perform the method as described in any one of claims 1 to 6; The server is configured to perform the method as described in any one of claims 7 to 9.

20. An electronic device, characterized in that, It includes at least one processor and at least one memory; wherein the one or more memories store one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the method as claimed in any one of claims 1 to 6, or to perform the method as claimed in any one of claims 7 to 9.

21. A chip, characterized in that, It includes a processor and a power supply circuit; the power supply circuit is used to supply power to the processor, and the processor is used to execute a computer program to implement the method as described in any one of claims 1 to 6, or to perform the method as described in any one of claims 7 to 9.

22. A cluster of devices, characterized in that, It includes at least one electronic device, each electronic device including a processor and memory; The processor of the at least one electronic device is configured to execute instructions stored in the memory of the at least one electronic device to cause the device cluster to perform the method as described in any one of claims 1 to 6, or to perform the method as described in any one of claims 7 to 9.

23. A computer-readable storage medium, characterized in that, The device stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 6, or the method as described in any one of claims 7 to 9.

24. A computer program product, characterized in that, It includes computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 6, or to perform the method as described in any one of claims 7 to 9.

Images