Rendering method and apparatus, device cluster, storage medium, and program product

Abstract

Disclosed are a rendering method and apparatus, a device cluster, a storage medium and a program product, relating to the field of image processing. The method comprises: acquiring a first image on the basis of a first SPP of an image to be rendered, rendering the image to be rendered again to obtain a second image when an image quality metric of the first image is lower than a preset threshold, and then fusing the first image and the second image to generate a third image. This allows users to set different preset thresholds according to different requirements for rendering effect and efficiency under different scenarios. Whenever an image quality metric of an image generated at a given level during hierarchical rendering is lower than the preset threshold, the rendered SPP is then adjusted to perform the next level of hierarchical rendering to generate a new image, and then generate an image with a higher SPP by means of image fusion, thereby satisfying different user requirements with respect to rendering effect and rendering efficiency, preventing wasted computing power resources and time resources due to the use of excessively high SPPs for rendering in scenarios with lower rendering effect requirements, and reducing resource waste.

Description

A rendering method, device, device cluster, storage medium and program product

[0001] The present application claims priority from the Chinese patent application No. 202411388102.5 filed on September 30, 2024, and entitled "A rendering method, device, device cluster, storage medium and program product", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of image processing, in particular to a rendering method, device, device cluster, storage medium and program product. BACKGROUND

[0003] Rendering refers to converting a model or a scene into an image. The rendering acceleration scheme based on software and hardware is to use an optimization algorithm or utilize the parallel processing capability of a graphics processing unit (GPU) to realize rendering acceleration in the process of generating an image by a renderer. The rendering acceleration scheme based on an artificial intelligence (AI) model is to use an AI model to perform denoising processing on an image after the image is generated by rendering, so as to reduce the calculation amount required for rendering while ensuring the image quality, thereby realizing rendering acceleration.

[0004] However, in the above rendering acceleration scheme, the user cannot flexibly adjust the rendering according to different requirements for the effect and efficiency of rendering for different scenes, resulting in waste of computing resources and time resources. SUMMARY

[0005] The present application provides a rendering method, device, device cluster, storage medium and program product, which can reduce resource waste during rendering acceleration.

[0006] In a first aspect, a rendering method is provided, which includes: obtaining a first image based on a to-be-rendered image, the sampling number per pixel (SPP) of a single pixel of the first image being a first SPP; if an image quality index of the first image is lower than a preset threshold, rendering the to-be-rendered image according to a second SPP to obtain a second image; and generating a third image by fusing the first image and the second image.

[0007] Based on the above rendering method, in a hierarchical rendering manner, when the image quality index of the first image based on the to-be-rendered image obtained is lower than a preset threshold, the to-be-rendered image is rendered again to obtain a second image, and a third image is generated by image fusion based on the first image and the second image. In this way, different preset thresholds can be set according to different requirements of users for rendering effects and efficiency in different scenes, and the to-be-rendered image is rendered hierarchically. When the image quality index of the image obtained by hierarchical rendering of the to-be-rendered image each time is lower than the preset threshold, the SPP of rendering is adjusted to generate a new image in the next hierarchical rendering, and an image with a higher SPP is generated through image fusion, so that the requirements of users for different rendering effects and rendering efficiency can be met, and the waste of algorithm resource and time resource caused by using a too high SPP for rendering in a scene with a relatively low requirement for rendering effect is avoided, and resource waste is reduced.

[0008] As a possible implementation manner, the second SPP can be a preset value, and the third SPP of the third image is the sum of the second SPP and the first SPP. In this way, the third SPP of the third image is higher than the first SPP of the first image, so that the quality of the image generated by rendering can be gradually improved, and resource waste caused by re-rendering when the image quality does not meet the requirement is avoided.

[0009] As a possible implementation manner, the second SPP can be the difference between the third SPP input by the user and the first SPP, and the third SPP is the SPP of the third image. In this way, the second SPP of the second image fused with the first image is determined according to the third SPP input by the user, and when the first image does not meet the requirement of the user, the image quality can be flexibly improved according to the requirement of the user, so that resource waste is reduced.

[0010] As a possible implementation manner, the first image and the second image are weighted and averaged according to a first proportion of the first SPP to the third SPP and a second proportion of the second SPP to the third SPP, to obtain the third image. In this way, image fusion is realized by weighted averaging of two images, so that the ability of continuous rendering based on the image generated by rendering is provided, and the image quality can be effectively improved.

[0011] As a possible implementation manner, the rendering of the first image and the second image is realized based on Monte Carlo sampling. In this way, through the superposition of Monte Carlo sampling, continuous rendering can be realized, and re-rendering caused by the image quality not meeting the requirement of the user is avoided, and time resource and algorithm resource are further saved.

[0012] As a possible implementation manner, the first image can be obtained by rendering the to-be-rendered image according to the preset first SPP, and can also be obtained by fusing the rendered image. In this way, the application does not limit the generation manner of the first image. The first image can be obtained by rendering according to the preset first SPP in the first rendering in the hierarchical rendering. The preset first SPP can be used to flexibly adjust the image quality of the image generated by the first rendering. When the user has a low requirement on the image quality, the first image can meet the user demand. The first image can be any image obtained by fusing after the first rendering in the hierarchical rendering. When the user has a high requirement on the image quality, image fusion can be performed on the basis of the first image to improve the image quality, thereby ensuring that the best performance and quality balance can be achieved in different rendering tasks.

[0013] As a possible implementation manner, the to-be-rendered image is rendered according to the preset first SPP to obtain a first rendered image; and the first rendered image is denoised to obtain the first image.

[0014] As a possible implementation manner, the second rendered image is fused according to the first image and the second image; and the second rendered image is denoised to obtain a third image.

[0015] In the above two possible implementation manners, the denoising process is integrated into the rendering process as a post-processing module after the rendering image is generated, and the flexibility and compatibility of the rendering process can be ensured.

[0016] As a possible implementation manner, the denoising strength of the denoising process is inversely proportional to the SPP at which the image is rendered. In this way, when the SPP is low, strong denoising processing is performed, and when the SPP is high, weak denoising processing is performed, thereby reducing resource waste while ensuring image quality.

[0017] Optionally, the denoising process is implemented based on an AI model, and the denoising strength of the denoising process is proportional to the parameter quantity of the AI model.

[0018] As a possible implementation manner, if the image quality index of the third image is greater than or equal to a preset threshold, the third image is output.

[0019] As a possible implementation manner, if the image quality index of the first image is greater than or equal to a preset threshold, the first image is output.

[0020] Based on the above two possible implementation manners, no matter how many times the image is rendered in the hierarchical rendering, the image is output when the image quality of the image meets the user demand, thereby ensuring the timeliness of the rendering result output.

[0021] In a second aspect, a rendering apparatus is provided, and the apparatus comprises various modules for performing the rendering method in the first aspect or any possible implementation manner of the first aspect.

[0022] The rendering apparatus in the second aspect can be a terminal device or a network device, or a chip (system), a network card or other components or assemblies arranged in the terminal device or the network device, or an apparatus comprising the terminal device or the network device, and the present application does not limit this.

[0023] In addition, the technical effects of the rendering apparatus in the second aspect can refer to the technical effects of the rendering method in the first aspect, which will not be repeated here.

[0024] In a third aspect, a computing device is provided, comprising a processor and a memory; the processor is coupled to the memory; the memory is configured to store computer instructions, and the computer instructions are executed by the processor to enable the computing device to implement the operation steps of the method in the first aspect or any possible implementation manner of the first aspect.

[0025] In a fourth aspect, a computing device cluster is provided, comprising at least one computing device, and each computing device comprises a processor and a memory.

[0026] The processor of the at least one computing device is configured to execute the instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the operation steps of the method in any possible implementation manner of the first aspect.

[0027] In a fifth aspect, a computer-readable storage medium is provided, comprising computer software instructions, when the computer software instructions are executed in a computer, the computer executes the operation steps of the method in the first aspect or any possible implementation manner of the first aspect.

[0028] In a sixth aspect, a chip system is provided. The chip system comprises a memory and at least one processor. The memory is configured to store a set of computer instructions, and when the processor executes the set of computer instructions, the operation steps of the method in any possible implementation manner of the first aspect are executed.

[0029] In a seventh aspect, a computer program product is provided, comprising computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the operation steps of the method in any possible implementation manner of the first aspect.

[0030] On the basis of the implementation manners of the above aspects provided by the present application, further combinations can be made to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0031] FIG. 1 is a scene diagram of a rendering process according to an embodiment of the present application;

[0032] FIG. 2 is a flow diagram of a rendering method according to an embodiment of the present application;

[0033] FIG. 3 is a flow diagram of another rendering method according to an embodiment of the present application;

[0034] FIG. 4 is a scene diagram of a rendering method according to an embodiment of the present application;

[0035] FIG. 5 is a diagram of a possible rendering device 500 according to an embodiment of the present application;

[0036] FIG. 6 is a diagram of a structure of a computing device according to an embodiment of the present application;

[0037] FIG. 7 is a diagram of a structure of a computing device cluster according to an embodiment of the present application;

[0038] FIG. 8 is a diagram of a structure of a connection between computing devices via a network according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] The present application can be applied not only to existing computing devices and rendering scenes having existing rendering functions, but also to future computing devices and rendering scenes having rendering functions. The terms used in the embodiments section of the present application are used only to explain the specific embodiments of the present application, and are not intended to limit the present application. For ease of understanding, the related terms involved in the embodiments of the present application will be introduced first.

[0040] (1) SPP

[0041] SPP refers to the number of sampling points allocated by a graphics card for each pixel. The higher the value of SPP, the less noise in the image generated by rendering, and the lower the value of SPP, the more noise in the image generated by rendering.

[0042] SPP can also be referred to as the number of light tracing samples. Light tracing can be referred to as ray tracing rendering, which refers to simulating the visual effect in a virtual three-dimensional scene that is almost consistent with the objective reality of the human visual world using computer graphics technology.

[0043] Light tracing sampling refers to projecting multiple light rays on a pixel point, and then averaging the color values of all the light rays after collision and coloring to obtain the final color value of the pixel point.

[0044] The number of light tracing samples refers to the number of emission points of the light rays sampled in a pixel.

[0045] (2) Monte Carlo sampling

[0046] Monte Carlo sampling is an approximate inference method that solves problems such as expectation, mean, area, integral, etc. by sampling a large number of particles.

[0047] The rendering in the embodiments of the application is to convert the solving problem of the rendering equation into a Monte Carlo numerical integration problem under the light tracing framework.

[0048] Monte Carlo integration is to sample the function value multiple times to obtain the mean value as the approximate value of the integral.

[0049] In order to facilitate understanding of the technical problems, the existing rendering method is described.

[0050] With the development of technology and the continuous improvement of aesthetic requirements, the images generated by rendering tend to be of higher quality, higher resolution and higher artistic effect, resulting in higher time resources and computing resources required for rendering.

[0051] In order to improve the speed and efficiency of rendering generated images and reduce the resources consumed by rendering, a rendering acceleration scheme is used to accelerate the rendering process. The rendering acceleration scheme includes a rendering acceleration scheme based on software and hardware and a rendering acceleration scheme based on an AI model.

[0052] In the rendering acceleration scheme based on software and hardware, the renderer performs denoising processing after generating a rendered image and obtains a final image. In the foregoing process, for example, the rendering acceleration scheme is to optimize the algorithm by improving the rasterization process and using a spatial partition data structure. For example, the rendering acceleration scheme is to use the parallel processing capability of GPU to realize rendering acceleration.

[0053] In the rendering acceleration scheme based on the AI model, the renderer generates a rendered image after rendering, and uses an AI model to fit the denoising process of the renderer. Through the AI model, the predicted and denoised image is generated, thereby reducing the calculation amount of rendering and realizing rendering acceleration.

[0054] For different scenes, users have different requirements for the effect and efficiency of rendering, but the rendering acceleration scheme based on the AI model uses a fixed rendering strategy for rendering, which cannot be flexibly adjusted according to the requirements of the user, thereby causing waste of computing resources and time resources.

[0055] To solve the above technical problems, the embodiment of the present application provides a rendering method, and in particular provides a hierarchical rendering method. A first image based on a first SPP of a to-be-rendered image is obtained, when an image quality index of the first image is lower than a preset threshold, the to-be-rendered image is rendered according to a second SPP to obtain a second image, and a third image is generated by image fusion according to the first image and the second image. In this way, different preset thresholds can be set according to different requirements of users for rendering effect and efficiency in different scenes, and the to-be-rendered image is rendered hierarchically. When the image quality index of the image obtained by hierarchical rendering of the to-be-rendered image each time is lower than the preset threshold, the SPP of the rendering is adjusted to generate a new image in the next hierarchical rendering, and an image with a higher SPP is generated through image fusion, so that the requirements of users for different rendering effects and rendering efficiencies can be met, and the waste of computing resources and time resources caused by using too high SPP for rendering in a scene with low rendering effect requirement is avoided, and resource waste is reduced.

[0056] The implementation manners of the embodiment of the present application are described in detail below with reference to the accompanying drawings.

[0057] FIG. 1 is a scene schematic diagram of a rendering process provided by the embodiment of the present application.

[0058] As shown in FIG. 1, the rendering device 100 includes a renderer 101, and the renderer 101 includes a rendering engine 1011. Optionally, the renderer 101 is computer software with rendering function. Optionally, the renderer 101 is hardware containing computer software with rendering function, for example, the hardware can be a processor or a controller, etc.

[0059] As a possible implementation manner, the rendering device 100 can further include an AI model 102 with de-noising processing capability, and the AI model 102 is used to perform de-noising processing on the image output by the renderer 101 in a sliding window manner.

[0060] The rendering device 100 can be a terminal, such as a mobile phone terminal, a tablet computer, a notebook computer, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, an extended reality (ER) device, etc., and can also be an edge device (for example, a box carrying a chip with processing capability), etc.

[0061] As a possible embodiment, the rendering device 100 can be a GPU, a central processing unit (CPU), other general-purpose processor, a digital signal processing (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The rendering device 100 can be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling program execution of the solutions of the present application.

[0062] The rendering device 100 obtains at least one frame of a to-be-rendered image of a to-be-rendered scene or a to-be-rendered model and a first image of the to-be-rendered image. The at least one frame of the to-be-rendered image can be pre-stored in the rendering device 100 or obtained from a user terminal or other storage device. The first image can be obtained by the renderer 101 in the rendering device 100 by rendering the to-be-rendered image according to a preset first SPP through the rendering engine 1011, or can be generated by image fusion of the rendering device 100.

[0063] If the image quality index of the first image is lower than a preset threshold, the renderer 101 in the rendering device 100 renders the to-be-rendered image according to a second SPP through the rendering engine 1011 to obtain a second image. The rendering device 100 fuses the first image and the second image to generate a third image.

[0064] The image quality index can be an image definition, color fidelity, signal-to-noise ratio, etc., or a numerical value obtained by an image quality evaluation algorithm. The preset threshold can be a threshold of the image quality index set by the user based on experience.

[0065] As a possible implementation manner, after the renderer 101 generates the second rendering image through the rendering engine 1011, the rendering device 100 performs denoising processing on the second rendering image through the AI model 102 to obtain the second image.

[0066] The following takes the rendering device 100 in FIG. 1 as an example for illustration.

[0067] FIG. 2 is a flowchart of a rendering method provided by an embodiment of the present application. As shown in FIG. 2, the rendering method can include steps 201 to 208.

[0068] In step 201, the rendering device 100 acquires a first image based on a to-be-rendered image.

[0069] As a possible implementation manner, the rendering device 100 acquires the first image by rendering the to-be-rendered image according to a preset first SPP.

[0070] Optionally, the renderer 101 in the rendering device 100 acquires the first image by rendering the to-be-rendered image according to a preset first SPP through the rendering engine 1011, and performing denoising processing on the first rendered image. The denoising intensity of the denoising processing is inversely proportional to the SPP at which the image is rendered.

[0071] As a first optional example, the renderer 101 performs denoising processing on the first rendered image to obtain the first image.

[0072] As a second optional example, the rendering device 100 performs denoising processing on the first rendered image according to a preset AI model to obtain the first image.

[0073] The AI model can be any model saved in the rendering device 100, and the AI model can also be determined according to the SPP at which the image is rendered. When the denoising processing is implemented based on the AI model, the denoising intensity of the denoising processing is proportional to the parameter quantity of the AI model, that is, the parameter quantity of the AI model is inversely proportional to the SPP at which the image is rendered.

[0074] As a possible implementation manner, the rendering device 100 acquires the first image fused and generated based on the to-be-rendered image last time.

[0075] In step 202, the rendering device 100 determines whether the image quality index of the first image is lower than a preset threshold. If not, step 203 is performed; if yes, step 204 is performed.

[0076] In step 203, the rendering device 100 outputs the first image when the image quality index of the first image is greater than or equal to the preset threshold.

[0077] The rendering device 100 determines the size relationship between the image quality index of the first image and the preset threshold. If it is determined that the image quality index of the first image is greater than or equal to the preset threshold, the rendering device 100 outputs the first image.

[0078] In a case that the first image is generated according to the first SPP, the rendering device 100 can directly obtain the first image meeting the user requirement after the first image is generated for the first time, thereby reducing the rendering time.

[0079] In a case that the first image is generated according to the first SPP, the rendering device 100 can directly obtain the first image meeting the user requirement after the first image is generated for the first time, thereby reducing the rendering time.

[0080] In a case that the first image is generated according to the first SPP, the rendering device 100 can directly obtain the first image meeting the user requirement after the first image is generated for the first time, thereby reducing the rendering time.

[0081] As a possible implementation manner, the rendering device 100 obtains the second image after the rendering of the to-be-rendered image and the denoising processing.

[0082] As a possible implementation manner, the second SPP is a preset value, for example, the second SPP is a fixed value preconfigured by the user, or the second SPP is a preset value input by the user.

[0083] As another possible implementation manner, the second SPP is a difference between the third SPP input by the user and the first SPP.

[0084] In a case that the first image is generated according to the first SPP, the rendering device 100 can directly obtain the first image meeting the user requirement after the first image is generated for the first time, thereby reducing the rendering time.

[0085] The rendering device 100 generates the third image by fusing the first image and the second image after the second image is generated. The SPP of the third image is the third SPP. The third SPP can be a value input by the user. When the second SPP is a preset value, the third SPP can also be a sum of the second SPP and the first SPP.

[0086] As a possible implementation manner, the rendering device 100 generates the second rendered image by fusing the first image and the second image. The rendering device 100 performs denoising processing on the second rendered image to obtain the second image. The denoising intensity of the denoising processing is inversely proportional to the SPP of the image, so the denoising intensity of the denoising processing on the second rendered image is greater than the denoising intensity of the denoising processing on the first rendered image.

[0087] Optionally, the renderer 101 in the rendering device 100 performs denoising processing on the second rendered image to obtain the second image.

[0088] Optionally, the rendering device 100 performs denoising processing on the second rendered image according to a preset AI model to obtain the second image.

[0089] The denoising strength of the denoising processing is proportional to the parameter quantity of the AI model.

[0090] Each preset AI model corresponds to an SPP of the rendered image, and the larger the SPP of the rendered image is, the smaller the parameter quantity of the corresponding AI model is. The rendering device 100 can determine the AI model for denoising processing according to the SPP of the rendered image.

[0091] As a possible implementation manner, the rendering device 100 performs weighted average on the first image and the second image according to a first proportion of the first SPP to the third SPP and a second proportion of the second SPP to the third SPP, to obtain the third image.

[0092] The rendering of the first image and the second image is based on Monte Carlo sampling. Due to the superposition of Monte Carlo sampling, the rendering device 100 can fuse the first image and the second image generated based on Monte Carlo sampling, to obtain the fused third image.

[0093] The first formula for the rendering device 100 to perform weighted average on the first image and the second image is as follows:

[0094] The first formula is as follows:

[0095] In the first formula, sample represents light tracing sampling rendering, image represents a to-be-rendered image, and x and a both represent SPP.

[0096] In the first formula, sample(image, x) represents an image obtained by rendering a to-be-rendered image with an SPP value of x, sample(image, a) represents an image obtained by rendering a to-be-rendered image with an SPP value of a, and sample(image, x-a) represents an image obtained by rendering a to-be-rendered image with an SPP value of (x-a).

[0097] When x is 512 and a is 256, the first formula represents that an image obtained by performing twice 256-SPP rendering and weighted average and an image obtained by performing single 512-SPP rendering can achieve the same effect.

[0098] Therefore, when the user is not satisfied with the effect of the image generated by rendering of 256SPP, the superimposability of Monte Carlo sampling can be used to re-render an image of 256SPP, and the two images of 256SPP are combined to obtain an image of 512SPP by using the first formula. Compared with the existing rendering method, when the user is not satisfied with the effect of the image generated by rendering, the scheme in the embodiment of the present application can effectively improve the image quality of the final image, and at the same time, since it is not necessary to render from the beginning, the computing resource and the calculation time are reduced, and the user experience is improved.

[0099] In step 206, the rendering device 100 determines whether the image quality index of the third image is lower than the preset threshold. If not, step 207 is performed, and if yes, step 208 is performed.

[0100] In step 207, the rendering device 100 outputs the third image when the image quality index of the third image is greater than or equal to the preset threshold.

[0101] When the image quality index of the third image is greater than or equal to the preset threshold, the third image generated by this hierarchical rendering meets the requirements of the user, and the rendering device 100 outputs the third image.

[0102] In step 208, the rendering device 100 determines that the first image is the third image when the image quality index of the third image is lower than the preset threshold, and re-executes step 201.

[0103] Based on the description of steps 201 to 208 in FIG. 2, the rendering device 100 obtains a first image based on a first SPP of a to-be-rendered image. When the image quality index of the first image is lower than a preset threshold, a second image is obtained by rendering the to-be-rendered image according to a second SPP, and a third image is generated by image fusion based on the first image and the second image. In this way, different preset thresholds can be set according to different requirements of the user for the effect and efficiency of rendering in different scenes, and the to-be-rendered image is rendered hierarchically. When the image quality index of the image obtained by hierarchical rendering of the to-be-rendered image each time is lower than the preset threshold, the SPP of the rendering is adjusted to generate a new image in the next hierarchical rendering, and a higher-SPP image is generated by image fusion, so that the requirements of the user for different rendering effects and rendering efficiencies can be met, and the waste of computing resources and time resources caused by using a too high SPP for rendering in a scene with a relatively low requirement for the rendering effect is avoided, and resource waste is reduced.

[0104] The FIG. 2 above is only one implementation of the rendering device 100 for rendering. Another possible implementation of the rendering method is described below, taking the rendering device 100 in FIG. 1 as an example for illustration. FIG. 3 is a flow diagram of another rendering method provided by an embodiment of the present application. For the specific implementation of the rendering device 100 in FIG. 3, refer to the related description of FIG. 1 above, which is not repeated here.

[0105] As shown in FIG. 3, the rendering method can include steps 301 to 309.

[0106] Step 301: The rendering device 100 initializes a scene to be rendered.

[0107] Step 302: The rendering device 100 configures rendering parameters and receives a user's selection instruction for an AI model.

[0108] The rendering parameters are the SPP of the generated image after rendering.

[0109] Step 303: The rendering device 100 renders at least one frame of image to be rendered of the scene to be rendered according to the rendering parameters in the light tracing rendering engine, to obtain a first rendered image.

[0110] The first rendered image can be equivalent to the first rendered image in step 201 of FIG. 2.

[0111] Step 304: The rendering device 100 performs denoising processing on the rendered image based on the AI model selected by the user, to obtain an image to be output.

[0112] The image to be output can be equivalent to the first image in step 201 and the third image in step 205 of FIG. 2.

[0113] Step 305: The rendering device 100 determines whether the image quality index of the image to be output is lower than a preset threshold. If not, step 306 is performed. If yes, step 307 is performed.

[0114] Step 305 is similar to steps 202 and 206 in FIG. 2, which is not repeated here.

[0115] Step 306: The rendering device 100 outputs the image to be output.

[0116] Step 305 is similar to steps 203 and 207 in FIG. 2, which is not repeated here. The image to be output can be equivalent to the first image in step 203 and the third image in step 207 of FIG. 2.

[0117] Step 307: The rendering device 100 takes the image to be output as a first image.

[0118] The step 307 is similar to the steps 201 and 208 in FIG. 2, and will not be described herein. The to-be-output image can be equivalent to the first image in the step 201 and the third image in the step 208 in FIG. 2.

[0119] In the step 308, the rendering device 100 receives the adjusted rendering parameter input by the user and the selection instruction of the AI model, and performs rendering and denoising processing on at least one frame of to-be-rendered image of the to-be-rendered scene according to the adjusted rendering parameter, to obtain a second image. The second image can be equivalent to the second image in the step 204 in FIG. 2.

[0120] In the step 309, the rendering device 100 fuses the first image and the second image to generate a second rendering image, and executes the step 304.

[0121] The second rendering image can be equivalent to the second rendering image described in the step 205 in FIG. 2.

[0122] Based on the steps 301 to 309 in FIG. 3, the rendering device 100 performs at least one rendering process in the hierarchical rendering. When the image quality index of the to-be-output image generated after the first rendering is greater than or equal to the preset threshold, the rendering device 100 outputs the to-be-output image, thereby significantly reducing the rendering time when the user has a low requirement on the rendering effect. When the image quality index of the to-be-output image generated after the first rendering is lower than the preset threshold, the rendering device 100 can continue to perform rendering by using the superposition of the Monte Carlo sampling, thereby effectively improving the sampling quality of the output image when the user has a high requirement on the rendering effect, while avoiding the problem of re-rendering and saving time and computing resources.

[0123] The overall process of the rendering method is described above, and another possible implementation manner of the rendering method will be described below through a specific embodiment. The rendering device 100 in FIG. 1 is taken as an example for description, and FIG. 4 is a scene schematic diagram of a rendering method provided in an embodiment of the present application. For the specific implementation manner of the rendering device 100 in FIG. 4, reference can be made to the related description of FIG. 1 above, and will not be described herein.

[0124] As shown in FIG. 4, the rendering method includes steps one to five.

[0125] In the step one, the rendering device 100 performs to-be-rendered scene initialization. The rendering device 100 initializes a to-be-rendered digital person broadcasting scene in a three-dimensional graphics software (blender) in the digital person broadcasting scene.

[0126] Step two, the rendering device 100 performs rendering under the requirement of the lowest quality and the highest efficiency. The rendering device 100 receives the SPP value 4 set by the user in the blender through the interactive editing mode, and performs rendering according to 4SPP in the light tracing rendering engine (cycles) to obtain a first rendering image. The rendering device 100 performs rendering on 1000 frames of the digital human broadcasting scene to be rendered, and the rendering time of each frame is about 2 seconds, and the total rendering time is expected to be 33 minutes.

[0127] The rendering device 100 performs denoising processing on the first rendering image according to the first AI model with the largest number of preset parameters to obtain a first to-be-output image of 4SPP. The rendering device 100 performs denoising in a sliding window mode of 30 frames, and the time consumption of denoising of each frame of image is 500 milliseconds, and the total time of denoising of the whole scene is about 9 minutes, so the total time required for the first hierarchical rendering is about 42 minutes. The first to-be-output image is equivalent to the first image in step 201 in FIG. 2.

[0128] The rendering device 100 receives the judgment result of the user that the first to-be-output image does not meet the requirement.

[0129] Step three, the rendering device 100 performs rendering under the requirement of lower quality and higher efficiency. The rendering device 100 receives the SPP value 64 set by the user in the blender through the user interface, and performs rendering according to 60SPP in the cycles to obtain a second rendering image of 60SPP. The rendering time of each frame is about 6 seconds, and the total rendering time is expected to be 100 minutes.

[0130] The rendering device 100 performs denoising processing on the second rendering image according to the second AI model with the second largest number of preset parameters, and fuses the first to-be-output image of 4SPP and the second rendering image of 60SPP after denoising processing to obtain a second to-be-output image of 64SPP. The second rendering image is equivalent to the second image in step 205 in FIG. 2, and the second to-be-output image is equivalent to the third image in step 205 in FIG. 2.

[0131] The number of parameters of the second AI model is less than that of the first AI model, and the denoising intensity of the second AI model in denoising processing is less than that of the first AI model in denoising processing.

[0132] The rendering device 100 performs denoising in a sliding window mode of 10 frames, and the time consumption of denoising of each frame of image is 300 milliseconds, and the total time of denoising of the whole scene is about 5 minutes, so the total time required for the second hierarchical rendering is about 105 minutes.

[0133] The rendering device 100 receives the judgment result of the user that the second to-be-output image does not meet the requirement.

[0134] Step four, the rendering device 100 performs rendering under higher quality and lower efficiency requirements. The rendering device 100 receives the SPP value set by the user in the blender as 512 in the user interface, and performs rendering according to 448 SPP in cycles to obtain a third rendering image of 448 SPP. The rendering time of each frame is about 30 seconds, and the total rendering time is expected to be 500 minutes.

[0135] The rendering device 100 performs denoising processing on the third rendering image according to the third AI model with the third largest number of parameters, and fuses the second to-be-output image of 64 SPP and the third rendering image of 448 SPP after denoising processing to obtain a third to-be-output image of 512 SPP. The rendering device 100 uses single-frame denoising technologies such as open image denoise (OIDN) to perform denoising, and the time consumption of denoising of each frame of image is 60 milliseconds, and the total time of denoising of the entire scene is about 1 minute, so the total time required for the third hierarchical rendering is about 501 minutes. Among them, the second to-be-output image is equivalent to the first image of step 205 in FIG. 2, the third rendering image is equivalent to the second image of step 205 in FIG. 2, and the third to-be-output image is equivalent to the third image of step 205 in FIG. 2.

[0136] Among them, the number of parameters of the third AI model is less than the number of parameters of the second AI model, and the denoising intensity of the denoising processing performed by the third AI model is less than the denoising intensity of the denoising processing performed by the second AI model.

[0137] The rendering device 100 receives the judgment result of the user that the third to-be-output image does not meet the requirements.

[0138] Step five, the rendering device 100 performs rendering under the highest quality requirement. The rendering device 100 receives the SPP value set by the user in the blender as 4096 in the user interface, and performs rendering according to 3584 SPP in cycles to obtain a fourth rendering image of 3584 SPP. The rendering time of each frame is about 50 seconds, and the total rendering time is expected to be 833 minutes.

[0139] The rendering device 100 fuses the third to-be-output image of 512 SPP and the fourth rendering image of 3584 SPP to obtain a fourth to-be-output image of 4096 SPP. Among them, the third to-be-output image is equivalent to the first image of step 205 in FIG. 2, the fourth rendering image is equivalent to the second image of step 205 in FIG. 2, and the fourth to-be-output image is equivalent to the third image of step 205 in FIG. 2.

[0140] In the above steps three to five, the smaller the value of SPP, the shorter the rendering waiting time, and the larger the value of SPP, the higher the quality of the image generated by rendering.

[0141] Based on the steps one to five in the above FIG. 4, the rendering device 100 realizes dynamic adjustment of the SPP according to specific quality and efficiency requirements through hierarchical rendering, so as to adapt to rendering tasks in different scenes and ensure that the best performance and quality balance can be achieved in rendering tasks in different scenes. In the scene of pursuing high efficiency, the rendering device 100 can significantly reduce the rendering time through a lower SPP value combined with the AI model denoising technology. In the embodiment of the present application, if the 64SPP image can meet the user's demand, the rendering device 100 only needs to spend 105 minutes at the shortest, compared with 833 minutes required for generating an image by using the highest image quality in the existing rendering method, which can save 733 minutes of rendering time.

[0142] In addition, the rendering device 100 can realize continuous rendering by using the superimposability of Monte Carlo sampling, avoiding re-rendering caused by the fact that the image quality does not meet the user's demand, and further saving time and resources. That is, in the embodiment of the present application, the image meeting the user's demand can be obtained in the shortest time of 42 minutes; if the image does not meet the user's demand, it can also be continuously rendered, assuming that the user's demand for the effect is met only after 4096SPP. The existing re-rendering scheme needs to waste (42+105+500) minutes of trial and error time, while in the embodiment of the present application, since the continuous rendering can be performed on the image rendered last time, trial and error is not needed, thereby reducing the waste of time and resources.

[0143] The rendering method provided by the embodiment is described in detail above, and the rendering apparatus provided by the embodiment will be described below in combination with FIG. 5.

[0144] FIG. 5 is a schematic diagram of a possible rendering apparatus 500 provided by the embodiment of the present application. The rendering apparatus 500 can be used to implement the rendering function of the rendering device 100 in the above method embodiment, and thus can also achieve the beneficial effects possessed by the above method embodiment. In the embodiment, the rendering apparatus 500 can be the rendering device 100 for implementing the rendering function in FIG. 1, and can also be a module (such as a chip) applied to the rendering device 100.

[0145] The rendering apparatus 500 includes an acquisition module 501, a rendering module 502, and a fusion module 503.

[0146] The acquisition module 501 is configured to acquire a first image based on a to-be-rendered image; and a sampling number SPP of a single pixel of the first image is a first SPP. For example, the acquisition module 501 is configured to perform step 201 in FIG. 2 or steps 303 and 307 in FIG. 3.

[0147] The rendering module 502 is configured to perform rendering on the image to be rendered according to a second SPP to obtain a second image, if the image quality indicator of the first image is lower than a preset threshold. For example, the rendering module 502 is configured to perform step 202 in FIG. 2 or step 308 in FIG. 3.

[0148] The fusion module 503 is configured to generate a third image by fusing the first image and the second image. For example, the fusion module 503 is configured to perform step 203 in FIG. 2 or step 309 in FIG. 3.

[0149] As a possible implementation, the second SPP is a preset value, and a third SPP of the third image is a sum of the second SPP and the first SPP.

[0150] As a possible implementation, the second SPP is a difference between a third SPP input by a user and the first SPP, and the third SPP is an SPP of the third image.

[0151] As a possible implementation, the fusion module 503 is specifically configured to perform weighted averaging on the first image and the second image according to a first proportion of the first SPP to the third SPP and a second proportion of the second SPP to the third SPP, to obtain the third image.

[0152] As a possible implementation, the acquisition module 501 is specifically configured to perform rendering on the image to be rendered according to a preset first SPP, to obtain the first image.

[0153] As a possible implementation, the acquisition module 501 is specifically configured to perform rendering on the image to be rendered according to a preset first SPP, to obtain a first rendering image; and perform denoising processing on the first rendering image, to obtain the first image.

[0154] As a possible implementation, the fusion module 503 is specifically configured to generate a second rendering image by fusing the first image and the second image; and perform denoising processing on the second rendering image, to obtain the third image.

[0155] As a possible implementation, a denoising intensity of the denoising processing is inversely proportional to an SPP of image rendering.

[0156] As a possible implementation, the denoising processing is implemented based on an artificial intelligence (AI) model, and a denoising intensity of the denoising processing is proportional to a parameter quantity of the AI model.

[0157] As a possible implementation manner, the apparatus further includes an output module configured to output the third image if the image quality indicator of the third image is greater than or equal to the preset threshold.

[0158] As a possible implementation manner, the apparatus further includes an output module configured to output the first image if the image quality indicator of the first image is greater than or equal to the preset threshold.

[0159] The obtaining module 501, the rendering module 502, and the fusion module 503 can be implemented by software or by hardware. For example, the implementation of the obtaining module 501 is described below. The implementation of the rendering module 502 and the fusion module 503 can be similar to the implementation of the obtaining module 501.

[0160] As an example of a software functional unit, the obtaining module 501 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the obtaining module 501 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers running the code can be distributed in the same availability zone (AZ), or in different AZs. Each AZ includes one data center or multiple data centers in a similar geographical location. Generally, one region includes multiple AZs.

[0161] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same virtual private cloud (VPC), or in multiple VPCs. Generally, one VPC is set in one region, and communication between two VPCs in the same region or between VPCs in different regions needs to be set through a communication gateway in each VPC to realize the interconnection between the VPCs.

[0162] As an example of a hardware functional unit, the obtaining module 501 can include at least one computing device, such as a server or the like. Alternatively, the obtaining module 501 can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the like. The PLD can be implemented by a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0163] The multiple computing devices included in the obtaining module 501 can be distributed in the same region or in different regions. The multiple computing devices included in the obtaining module 501 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the obtaining module 501 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0164] It should be noted that in other embodiments, any of the obtaining module 501, the rendering module 502, and the fusion module 503 can be used to perform any step of the rendering method. The steps implemented by the obtaining module 501, the rendering module 502, and the fusion module 503 can be specified as needed, and the overall function of the rendering device 500 can be achieved by implementing different steps of the rendering method by the obtaining module 501, the rendering module 502, and the fusion module 503 respectively.

[0165] The embodiments of the present application also provide a computing device, as shown in FIG. 6, the computing device 600 includes a bus 602, a processor 604, a memory 606, and a communication interface 608. The processor 604, the memory 606, and the communication interface 608 communicate through the bus 602. The computing device 600 can be a server or a terminal device. It should be understood that the number of processors and memories in the computing device 600 is not limited by the present application.

[0166] The bus 602 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is represented in FIG. 6, but it does not mean that there is only one bus or only one type of bus. The bus 604 can include a path for transmitting information between various components (e.g., the memory 606, the processor 604, the communication interface 608) of the computing device 600.

[0167] The processor 604 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0168] The memory 606 can include a volatile memory (e.g., a random access memory (RAM)), and the processor 604 can further include a non-volatile memory (e.g., a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD)).

[0169] The memory 606 stores executable program codes, and the processor 604 executes the executable program codes to respectively implement the functions of the modules included in the aforementioned rendering apparatus 500, thereby implementing the rendering method. That is, the memory 606 stores instructions for executing the rendering method.

[0170] Alternatively, the memory 606 stores executable program codes, and the processor 604 executes the executable program codes to respectively implement the functions of the modules included in the aforementioned rendering device 100, thereby implementing the rendering method. That is, the memory 606 stores instructions for executing the rendering method.

[0171] The communication interface 608 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to enable communication between the computing device 600 and other devices or communication networks.

[0172] The embodiments of the present application also provide a computing device cluster. The computing device cluster comprises at least one computing device. The computing device can be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0173] As shown in FIG. 7, the computing device cluster comprises at least one computing device 600. The same instructions for performing the rendering method can be stored in the memory 606 of one or more computing devices 600 in the computing device cluster.

[0174] In some possible implementation manners, partial instructions for performing the rendering method can also be respectively stored in the memory 606 of one or more computing devices 600 in the computing device cluster. In other words, the combination of one or more computing devices 600 can collectively execute the instructions for performing the rendering method.

[0175] It should be noted that the memory 606 in different computing devices 600 in the computing device cluster can store different instructions, respectively used for performing part of the functions of the rendering apparatus 500. That is, the instructions stored in the memory 606 in different computing devices 600 can implement the functions of one or more of the acquisition module 501, the rendering module 502, and the fusion module 503.

[0176] In some possible implementation manners, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. FIG. 8 shows a possible implementation manner. As shown in FIG. 8, two computing devices 600A and 600B are connected through a network. Specifically, the communication interface in each computing device is connected to the network. In this type of possible implementation manner, the memory 606 in the computing device 600A stores instructions for performing the functions of the acquisition module 501. Meanwhile, the memory 606 in the computing device 600B stores instructions for performing the functions of the rendering module 502 and the fusion module 503.

[0177] It should be understood that the functions of the computing device 600A shown in FIG. 8 can also be completed by multiple computing devices 600. Similarly, the functions of the computing device 600B can also be completed by multiple computing devices 600.

[0178] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be software or a program product containing instructions, capable of running on a computing device or stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device performs the rendering method shown in FIG. 2 or FIG. 3.

[0179] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium or data storage device that can be accessed by a computing device and can include one or more available media or data storage devices. The available medium can be a magnetic medium, (e.g., a floppy diskette, a hard disk drive, a magnetic tape), an optical medium, (e.g., a digital video disc (DVD)), a semiconductor medium, (e.g., a solid state drive (SSD)), or the like. The computer readable storage medium includes instructions that are indicative of the rendering method as shown in FIG. 2 or FIG. 3.

[0180] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are performed. The computer can be a general purpose computer, a special purpose computer, a computer network, a network device, a user equipment or other programmable apparatus. The computer programs or instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer programs or instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through a wired or wireless manner. The computer readable storage medium can be any available medium or data storage device that can be accessed by a computer or a server, data center or the like that integrates one or more available media. The available medium can be a magnetic medium, for example, a floppy disk, a hard disk, a magnetic tape; an optical medium, for example, a digital video disc (DVD); or a semiconductor medium, for example, a solid state drive (SSD). The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A rendering method, characterized by, The method comprises: acquiring a first image based on a to-be-rendered image; a sampling number SPP of a single pixel of the first image is a first SPP; if an image quality index of the first image is lower than a preset threshold, rendering the to-be-rendered image according to a second SPP to obtain a second image; generating a third image by fusing the first image and the second image.

2. The method of claim 1, wherein, The second SPP is a preset value, and a third SPP of the third image is a sum of the second SPP and the first SPP.

3. The method of claim 1, wherein, The second SPP is a difference between a third SPP input by a user and the first SPP, and the third SPP is an SPP of the third image.

4. The method according to claim 2 or 3, characterized in that, The generating a third image by fusing the first image and the second image comprises: performing weighted averaging on the first image and the second image according to a first proportion of the first SPP in the third SPP and a second proportion of the second SPP in the third SPP to obtain the third image.

5. The method according to any one of claims 1 to 4, characterized in that, The acquiring a first image comprises: rendering the to-be-rendered image according to the preset first SPP to obtain the first image.

6. The method of claim 5, wherein, The rendering the to-be-rendered image according to the preset first SPP to obtain the first image comprises: rendering the to-be-rendered image according to the preset first SPP to obtain a first rendered image; performing denoising processing on the first rendered image to obtain the first image.

7. The method according to any one of claims 1 to 6, characterized in that, The generating a third image by fusing the first image and the second image comprises: generating a second rendered image by fusing the first image and the second image; performing denoising processing on the second rendered image to obtain the third image.

8. The method according to claim 6 or 7, characterized in that, A denoising intensity of the denoising processing is inversely proportional to an SPP of image rendering.

9. The method according to any one of claims 6-8, characterized in that, The denoising processing is implemented based on an artificial intelligence AI model, and a denoising intensity of the denoising processing is proportional to a parameter quantity of the AI model.

10. The method of claim 1, wherein, The method further comprises: if an image quality index of the third image is greater than or equal to the preset threshold, outputting the third image.

11. The method of claim 1, wherein, The method further comprises: if an image quality index of the first image is greater than or equal to the preset threshold, outputting the first image.

12. A rendering device, characterized in that The apparatus comprises: an acquiring module configured to acquire a first image based on a to-be-rendered image; a sampling number SPP of a single pixel of the first image is a first SPP; a rendering module configured to, if an image quality index of the first image is lower than a preset threshold, render the to-be-rendered image according to a second SPP to obtain a second image; a fusing module configured to generate a third image by fusing the first image and the second image.

13. The apparatus of claim 12, wherein, The second SPP is a preset value, and a third SPP of the third image is a sum of the second SPP and the first SPP.

14. The apparatus of claim 12, wherein, The second SPP is a difference between a third SPP input by a user and the first SPP, and the third SPP is an SPP of the third image.

15. The apparatus of claim 13 or 14, wherein, The fusion module is specifically configured to perform weighted average on the first image and the second image according to a first proportion of the first SPP to the third SPP and a second proportion of the second SPP to the third SPP, to obtain the third image.

16. The apparatus of any one of claims 12-15, wherein, The acquisition module is specifically configured to render the to-be-rendered image according to the preset first SPP, to obtain the first image.

17. The apparatus of claim 16, wherein, The acquisition module is specifically configured to render the to-be-rendered image according to the preset first SPP, to obtain a first rendered image; and perform denoising processing on the first rendered image, to obtain the first image.

18. The apparatus of any of claims 12-17, wherein, The fusion module is specifically configured to generate a second rendered image by fusing the first image and the second image; and perform denoising processing on the second rendered image, to obtain the third image.

19. The apparatus of claim 17 or 18, wherein, The denoising strength of the denoising processing is inversely proportional to the SPP at which the image is rendered.

20. The apparatus of any of claims 17-19, wherein, The denoising processing is implemented based on an artificial intelligence (AI) model, and the denoising strength of the denoising processing is proportional to a parameter quantity of the AI model.

21. The apparatus of claim 12, wherein, The apparatus further includes: An output module configured to output the third image if the image quality indicator of the third image is greater than or equal to the preset threshold.

22. The apparatus of claim 12, wherein, The apparatus further includes: An output module configured to output the first image if the image quality indicator of the first image is greater than or equal to the preset threshold.

23. A cluster of computing devices, characterized in that, The at least one computing device includes a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the operation steps of the method of any one of claims 1-11.

24. A computer-readable storage medium, comprising: It includes: Computer software instructions; when the computer software instructions run in the controller, the controller performs the operation steps of the method of any one of claims 1-11.

25. A computer program product, characterised in that, It includes computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the operation steps of the method of any one of claims 1-11.

Images