Rendering method and device, equipment cluster, storage medium and program product
By employing hierarchical rendering and image fusion methods, combined with Monte Carlo sampling and AI model denoising, the problem of resource waste in existing rendering acceleration solutions is solved, enabling flexible adjustment of rendering effects and efficiency, and improving resource utilization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2026-03-31
AI Technical Summary
Existing rendering acceleration solutions cannot be flexibly adjusted according to users' different requirements for rendering effects and efficiency, resulting in a waste of computing and time resources.
A hierarchical rendering method is adopted. By obtaining the first image quality index of the image to be rendered, if it is lower than the preset threshold, the SPP is adjusted for re-rendering. A higher quality image is generated through image fusion. Combined with Monte Carlo sampling and AI model denoising processing, the rendering process can be flexibly adjusted.
It reduces resource waste, meets users' requirements for different rendering effects and efficiency, improves the flexibility and compatibility of the rendering process, and saves computing power and time resources.
Smart Images
Figure CN121767201A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing, and more particularly to a rendering method, apparatus, device cluster, storage medium, and program product. Background Technology
[0002] Rendering refers to converting a model or scene into an image. Hardware-based rendering acceleration solutions employ optimization algorithms or leverage the parallel processing capabilities of graphics processing units (GPUs) during the image generation process. Artificial intelligence (AI) model-based rendering acceleration solutions utilize AI models to denoise the image after it has been rendered, reducing the computational load while maintaining image quality, thus accelerating rendering.
[0003] However, in the above-mentioned rendering acceleration solutions, users cannot flexibly adjust the rendering according to different requirements for rendering effect and efficiency for different scenarios, resulting in a waste of computing power and time resources. Summary of the Invention
[0004] This application provides a rendering method, apparatus, device cluster, storage medium, and program product that can reduce resource waste during rendering acceleration.
[0005] Firstly, a rendering method is provided, comprising: obtaining a first image based on an image to be rendered, wherein the number of samples per pixel (SPP) of the first image is a first SPP; if the image quality index of the first image is lower than a preset threshold, rendering the image to be rendered according to a second SPP to obtain a second image; and fusing the first image and the second image to generate a third image.
[0006] Based on the above rendering method, a hierarchical rendering approach is adopted. When the image quality index of the first image based on the image to be rendered is lower than a preset threshold, the image to be rendered is rendered again to obtain a second image. The first and second images are then merged to generate a third image. In this way, users can set different preset thresholds for different rendering effects and efficiency requirements in different scenarios. The image to be rendered is rendered in stages. Each time the image quality index of the image obtained from the staged rendering is lower than the preset threshold, the rendering SPP is adjusted to perform the next staged rendering to generate a new image. The image with a higher SPP is generated through image fusion. This can meet the user's different rendering effect and rendering efficiency requirements, and avoid the waste of computing power and time resources caused by using excessively high SPP for rendering in scenarios with low rendering effect requirements, thus reducing resource waste.
[0007] As one possible implementation, the second SPP can be a preset value, and the third SPP of the third image is the sum of the second SPP and the first SPP. In this way, the third SPP of the third image is higher than the first SPP of the first image, thereby enabling a gradual improvement in the quality of the rendered image and avoiding the waste of resources caused by re-rendering when the image quality does not meet the requirements.
[0008] As one possible implementation, the second SPP can be the difference between the user-inputted third SPP and the first SPP, where the third SPP is the SPP of the third image. In this way, the second SPP of the second image fused with the first image is determined based on the user-inputted third SPP. When the first image does not meet the user's requirements, the image quality can be flexibly improved according to the user's needs, thereby reducing resource waste.
[0009] As one possible implementation, a weighted average is taken between the first image and the second image based on a first ratio of the first SPP to the third SPP and a second ratio of the second SPP to the third SPP to obtain the third image. Thus, image fusion is achieved by weighted averaging of the two images, providing the ability to continue rendering based on the generated image, effectively improving image quality.
[0010] As one possible implementation, the rendering of the first and second images is based on Monte Carlo sampling. Thus, by utilizing the superposition property of Monte Carlo sampling, sequential rendering can be achieved, avoiding re-rendering due to unsatisfactory image quality, further saving time and computing resources.
[0011] As one possible implementation, the first image can be obtained by rendering the image to be rendered according to a preset first SPP, or it can be generated by fusing rendered images. Thus, this application does not limit the generation method of the first image. The first image can be obtained by rendering according to a preset first SPP during the first rendering in hierarchical rendering. The preset first SPP allows for flexible adjustment of the image quality of the image generated in the first rendering. When the user's requirements for image quality are low, the first image can meet the user's needs. The first image can also be an image obtained by fusing in any subsequent rendering after the first rendering in hierarchical rendering. When the user's requirements for image quality are high, image fusing can be performed on the first image to improve image quality, thereby ensuring the best performance and quality balance in different rendering tasks.
[0012] As one possible implementation, the image to be rendered is rendered according to a preset first SPP to obtain a first rendered image; the first rendered image is then denoised to obtain a first image.
[0013] As one possible implementation, a second rendered image is generated by fusing the first and second images; the second rendered image is then denoised to obtain a third image.
[0014] In the two possible implementations mentioned above, denoising is integrated into the rendering process as a post-processing module after the rendered image is generated, which can ensure the flexibility and compatibility of the rendering process.
[0015] One possible approach is to inversely proportional to the SPP (Size, Power, Price) of the rendered image. This allows for stronger denoising at lower SPPs and weaker denoising at higher SPPs, thus reducing resource waste while maintaining image quality.
[0016] Optionally, the denoising process is implemented based on an AI model, and the denoising intensity is proportional to the number of parameters in the AI model.
[0017] As one possible implementation, if the image quality index of the third image is greater than or equal to a preset threshold, the third image is output.
[0018] As one possible implementation, if the image quality index of the first image is greater than or equal to a preset threshold, the first image is output.
[0019] Based on the two possible implementation methods mentioned above, regardless of how many times the image is generated in the hierarchical rendering process, the image is output when the image quality meets the user's requirements, thus ensuring the timeliness of the rendering result output.
[0020] In a second aspect, a rendering apparatus is provided, the apparatus comprising modules for performing the rendering method of the first aspect or any possible implementation thereof.
[0021] The rendering apparatus described in the second aspect may be a terminal device or a network device, or it may be a chip (system), network card or other component or assembly disposed in a terminal device or a network device, or it may be an apparatus containing a terminal device or a network device. This application does not limit it in this regard.
[0022] Furthermore, the technical effects of the rendering apparatus described in the second aspect can be referred to the technical effects of the rendering method described in the first aspect, and will not be repeated here.
[0023] Thirdly, a computing device is provided, the computing device including a processor and a memory; the processor is coupled to the memory; the memory is used to store computer instructions, which are recorded and executed by the processor to enable the computing device to perform the operation steps of the method described in the first aspect or any possible implementation of the first aspect.
[0024] Fourthly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and memory;
[0025] The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the operational steps of the method as described in any possible implementation of the first aspect.
[0026] Fifthly, a computer-readable storage medium is provided, comprising computer software instructions that, when executed in a computer, cause the computer to perform operational steps of the method as described in the first aspect or any possible implementation thereof.
[0027] In a sixth aspect, a chip system is provided. The chip system includes a memory and at least one processor. The memory stores a set of computer instructions, which, when executed by the processor, perform the operational steps of the method described in any possible implementation of the first aspect.
[0028] In a seventh aspect, a computer program product is provided, comprising computer program instructions that, when executed by a cluster of computing devices, perform operational steps of the method as described in any possible implementation of the first aspect.
[0029] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description
[0030] Figure 1 A scene diagram illustrating a rendering process provided in an embodiment of this application;
[0031] Figure 2 A flowchart illustrating a rendering method provided in an embodiment of this application;
[0032] Figure 3 A flowchart illustrating another rendering method provided in an embodiment of this application;
[0033] Figure 4 A scene diagram illustrating a rendering method provided in an embodiment of this application;
[0034] Figure 5 A schematic diagram of a possible rendering apparatus 500 provided for an embodiment of this application;
[0035] Figure 6 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;
[0036] Figure 7This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;
[0037] Figure 8 This is a schematic diagram of a network connection structure between computing devices provided in an embodiment of this application. Detailed Implementation
[0038] This application can be used not only with existing computing devices and rendering scenes with existing rendering capabilities, but also with future computing devices and rendering scenes with rendering capabilities. The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. For ease of understanding, the relevant terminology involved in the embodiments of this application will be introduced below.
[0039] (1)SPP
[0040] SPP refers to the number of sampling points allocated to each pixel by the graphics card. A higher SPP value results in a less noisy rendered image, while a lower SPP value results in a noisier rendered image.
[0041] SPP can also be called Ray Tracing Samples. Ray tracing can be called ray tracing rendering, which refers to using computer graphics technology to simulate visual effects in a virtual 3D scene that are almost identical to the objective real world as seen by humans.
[0042] Ray tracing sampling refers to projecting multiple rays onto a pixel, and then taking the average of the colors resulting from the collisions of all the rays at that pixel as the final color value of that pixel.
[0043] Ray tracing sampling count refers to the number of light emission points sampled within a single pixel.
[0044] (2) Monte Carlo sampling
[0045] Monte Carlo sampling is an approximate inference method that uses sampling of a large number of particles to solve problems such as expectation, mean, area, and integral.
[0046] In this embodiment of the application, the rendering problem is transformed into a Monte Carlo numerical integration problem within the framework of ray tracing by solving the rendering equation.
[0047] Monte Carlo integration is an approximation of the integral value by taking the average of multiple samples of the function value.
[0048] To make the technical issues easier to understand, the existing rendering methods will now be described.
[0049] With the development of technology and the continuous improvement of aesthetic requirements, the rendered images tend to have higher quality, higher resolution and more exquisite effects, which leads to higher time and computing resources required for rendering.
[0050] To improve the speed and efficiency of image rendering and reduce rendering resource consumption, rendering acceleration solutions are employed to speed up the rendering process. These solutions include hardware / software-based acceleration methods and AI model-based acceleration methods.
[0051] In hardware and software-based rendering acceleration solutions, after the renderer generates a rendered image, it performs noise reduction processing to obtain the final image. In the aforementioned process, for example, rendering acceleration solutions optimize algorithms by improving the rasterization process and using spatial partitioning data structures, or by utilizing the parallel processing capabilities of the GPU to achieve rendering acceleration.
[0052] In the AI model-based rendering acceleration solution, after the renderer generates the rendered image, an AI model is used to fit the renderer's denoising process. By using the AI model to predict and generate the denoised image, the computational load of rendering is reduced, thus accelerating the rendering process.
[0053] For different scenarios, users have different requirements for rendering effects and efficiency. However, AI model-based rendering acceleration solutions use fixed rendering strategies and cannot be flexibly adjusted according to user requirements, resulting in a waste of computing and time resources.
[0054] To address the aforementioned technical problems, embodiments of this application provide a rendering method, particularly a hierarchical rendering method. A first image is obtained based on a first SPP (Simplified Rendering Point) of the image to be rendered. When the image quality index of the first image is lower than a preset threshold, a second image is obtained by rendering the image to be rendered according to a second SPP. A third image is generated by fusing the first and second images. In this way, users can set different preset thresholds to meet different requirements for rendering effects and efficiency in different scenarios. Hierarchical rendering of the image to be rendered is performed, and each time the image quality index of the image obtained from hierarchical rendering is lower than the preset threshold, the rendering SPP is adjusted for the next hierarchical rendering to generate a new image. Image fusion is then used to generate an image with a higher SPP, thereby meeting users' different requirements for rendering effects and efficiency. This avoids wasting computing and time resources by using excessively high SPPs for rendering in scenarios with lower rendering effect requirements, thus reducing resource waste.
[0055] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0056] Figure 1This is a scene diagram illustrating a rendering process provided in an embodiment of this application.
[0057] like Figure 1 As shown, the rendering device 100 includes a renderer 101, and the renderer 101 includes a rendering engine 1011. Optionally, the renderer 101 is computer software with rendering capabilities. Alternatively, the renderer 101 is hardware containing computer software with rendering capabilities, such as a processor or controller.
[0058] As one possible implementation, the rendering device 100 may also include an AI model 102 with denoising capabilities, which is used to denoise the image output by the renderer 101 using a sliding window method.
[0059] The rendering device 100 can be a terminal, such as a mobile terminal, tablet computer, laptop computer, virtual reality (VR) device, augmented reality (AR) device, mixed reality (MR) device, extended reality (ER) device, etc., or it can be an edge device (e.g., a box carrying a chip with processing capabilities).
[0060] As one possible embodiment, the rendering device 100 may be a GPU, a central processing unit (CPU), other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The rendering device 100 may be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program according to the present application.
[0061] The rendering device 100 acquires at least one frame of the scene or model to be rendered, as well as a first image of the image to be rendered. The at least one frame of the image to be rendered may be pre-saved in the rendering device 100 or acquired from a user terminal or other storage device. The first image may be obtained by the renderer 101 rendering the image to be rendered using the rendering engine 1011 according to a preset first SPP, or optionally generated by the rendering device 100 through image fusion.
[0062] If the image quality index of the first image is lower than a preset threshold, the renderer 101 in the rendering device 100 renders the image to be rendered according to the second SPP through the rendering engine 1011 to obtain the second image. The rendering device 100 then merges the first image and the second image to generate a third image.
[0063] The image quality metrics can be indicators such as image sharpness, color fidelity, and signal-to-noise ratio, or values obtained through image quality assessment algorithms. The preset threshold can be a threshold set by the user based on experience for the image quality metrics.
[0064] As one possible implementation, after the renderer 101 generates the second rendered image through the rendering engine 1011, the rendering device 100 performs noise reduction processing on the second rendered image through the AI model 102 to obtain the second image.
[0065] The following is based on Figure 1 The following explanation uses rendering device 100 as an example.
[0066] Figure 2 This is a flowchart illustrating a rendering method provided in an embodiment of this application. Figure 2 As shown, the rendering method may include steps 201 to 208.
[0067] Step 201: The rendering device 100 acquires the first image based on the image to be rendered.
[0068] As one possible implementation, the rendering device 100 renders the image to be rendered according to a preset first SPP to obtain a first image.
[0069] Optionally, the renderer 101 in the rendering device 100 renders the image to be rendered according to a preset first SPP using the rendering engine 1011 to obtain a first rendered image. The rendering device 100 then performs denoising processing on the first rendered image to obtain the first image. The denoising intensity is inversely proportional to the SPP used for rendering the image.
[0070] As a first alternative example, renderer 101 performs noise reduction processing on the first rendered image to obtain the first image.
[0071] As a second alternative example, the rendering device 100 performs noise reduction processing on the first rendered image according to a preset AI model to obtain the first image.
[0072] The AI model can be any model stored in the rendering device 100, or it can be determined by the SPP used for rendering the image. When denoising is implemented based on the AI model, the denoising intensity is directly proportional to the number of parameters in the AI model, that is, the number of parameters in the AI model is inversely proportional to the SPP used for rendering the image.
[0073] As one possible implementation, the rendering device 100 acquires the first image generated by fusion after the previous hierarchical rendering based on the image to be rendered.
[0074] Step 202: The rendering device 100 determines whether the image quality index of the first image is lower than a preset threshold. If not, proceed to step 203; if yes, proceed to step 204.
[0075] Step 203: If the image quality index of the first image is greater than or equal to a preset threshold, the rendering device 100 outputs the first image.
[0076] The rendering device 100 determines the relationship between the image quality index of the first image and the preset threshold. If it is determined that the image quality index of the first image is greater than or equal to the preset threshold, the rendering device 100 outputs the first image.
[0077] When the first image is generated by rendering according to a preset first SPP, the rendering device 100 can directly obtain the first image that meets the user's requirements after the first rendering of the first image, thereby reducing the rendering time.
[0078] Step 204: If the image quality index of the first image is lower than the preset threshold, the rendering device 100 renders the image to be rendered according to the second SPP to obtain the second image.
[0079] The rendering device 100 determines the relationship between the image quality index of the first image and the preset threshold. If it is determined that the image quality index of the first image is lower than the preset threshold, the rendering device 100 renders the image to be rendered according to the second SPP to obtain the second image.
[0080] As one possible implementation, the rendering device 100 renders and denoises the image to be rendered to obtain a second image.
[0081] As one possible implementation, the second SPP is a preset value, such as a fixed value pre-configured by the user or a preset value input by the user.
[0082] As another possible implementation, the second SPP is the difference between the third SPP input by the user and the first SPP.
[0083] Step 205: The rendering device 100 merges the first image and the second image to generate a third image.
[0084] After generating the second image, the rendering device 100 merges the first and second images to generate a third image. The SPP of the third image is a third SPP. The third SPP can be a value input by the user. When the second SPP is a preset value, the third SPP can also be the sum of the second SPP and the first SPP.
[0085] As one possible implementation, the rendering device 100 fuses the first image and the second image to generate a second rendered image. The rendering device 100 then performs denoising processing on the second rendered image to obtain the second image. The denoising intensity is inversely proportional to the SPP (Single Pixel Scale) of the image rendering process; therefore, the denoising intensity of the second rendered image performed by the rendering device 100 is greater than the denoising intensity of the first rendered image.
[0086] Optionally, the renderer 101 in the rendering device 100 performs noise reduction processing on the second rendered image to obtain the second image.
[0087] Optionally, the rendering device 100 performs noise reduction processing on the second rendered image according to a preset AI model to obtain the second image.
[0088] The denoising intensity of the denoising process is directly proportional to the number of parameters in the AI model.
[0089] Each preset AI model corresponds to the SPP of the rendered image. The larger the SPP of the rendered image, the smaller the number of parameters of the corresponding AI model. The rendering device 100 can determine the AI model to be denoised based on the SPP of the rendered image.
[0090] As one possible implementation, the rendering device 100 performs a weighted average of the first image and the second image based on the first ratio of the first SPP to the third SPP and the second ratio of the second SPP to the third SPP to obtain the third image.
[0091] The rendering of the first and second images is based on Monte Carlo sampling. Due to the superposition of Monte Carlo sampling, the rendering device 100 can fuse the first and second images generated based on Monte Carlo sampling to obtain a fused third image.
[0092] The first formula for weighted averaging the first and second images by the rendering device 100 is shown below:
[0093] First formula:
[0094] In the first formula above, sample represents ray tracing sampling rendering, image represents the image to be rendered, and x and a both represent SPP.
[0095] In the first formula above, sample(image,x) represents the image obtained by rendering the image to be rendered with an SPP value of x, sample(image,a) represents the image obtained by rendering the image to be rendered with an SPP value of a, and sample(image,xa) represents the image obtained by rendering the image to be rendered with an SPP value of (xa).
[0096] When x is 512 and a is 256, the first formula above indicates that the image obtained by performing two 256SPP renderings and then weighting the average is the same as the image generated by performing a single 512SPP rendering.
[0097] Therefore, when a user is not satisfied with the effect of the image generated by rendering 256SPP, the superposition property of Monte Carlo sampling can be utilized to render another 256SPP image. The two 256SPP images are then combined using the first formula mentioned above to obtain a 512SPP image. Compared with existing rendering methods that require re-rendering when the user is not satisfied with the effect of the rendered image, the solution in this application embodiment can effectively improve the image quality of the final image. At the same time, since there is no need to render from scratch, it reduces computing resources and computation time, thereby improving the user experience.
[0098] Step 206: Rendering device 100 determines whether the image quality index of the third image is lower than the preset threshold. If not, proceed to step 207; if yes, proceed to step 208.
[0099] Step 207: If the image quality index of the third image is greater than or equal to a preset threshold, the rendering device 100 outputs the third image.
[0100] If the image quality index of the third image is greater than or equal to the preset threshold, the third image generated by this graded rendering meets the user's requirements, and the rendering device 100 outputs the third image.
[0101] Step 208: If the image quality index of the third image is lower than a preset threshold, the rendering device 100 determines the first image as the third image and re-executes step 201.
[0102] Based on the above Figure 2As described in steps 201 to 208, the rendering device 100 acquires a first image based on a first SPP of the image to be rendered. When the image quality index of the first image is lower than a preset threshold, the image to be rendered is rendered according to a second SPP to obtain a second image. The first image and the second image are then fused to generate a third image. In this way, users can set different preset thresholds for different rendering effects and efficiency requirements in different scenarios, performing graded rendering of the image to be rendered. Each time the image quality index of the image obtained from graded rendering is lower than the preset threshold, the rendering SPP is adjusted for the next graded rendering to generate a new image. Image fusion is then used to generate an image with a higher SPP, thereby meeting users' different rendering effect and efficiency requirements. This avoids wasting computing and time resources by using excessively high SPPs for rendering in scenarios with lower rendering effect requirements, thus reducing resource waste.
[0103] The above text Figure 2 This is one implementation method that renders only on rendering device 100. The following describes another possible implementation method for the rendering approach. Figure 1 Taking rendering device 100 as an example, this will be explained. Figure 3 This is a flowchart illustrating another rendering method provided in an embodiment of this application. (About...) Figure 3 The specific implementation of the rendering device 100 can be referred to the aforementioned Figure 1 The relevant descriptions will not be repeated here.
[0104] like Figure 3 As shown, the rendering method may include steps 301 to 309.
[0105] Step 301: Initialize the scene to be rendered using rendering device 100.
[0106] Step 302: Rendering device 100 configures rendering parameters and receives user selection instructions for AI models.
[0107] The rendering parameter is the SPP of the rendered image.
[0108] Step 303: In the ray tracing rendering engine, the rendering device 100 renders at least one frame of the scene to be rendered according to the rendering parameters to obtain the first rendered image.
[0109] Wherein, the first rendered image can be equivalent to Figure 2 The first rendered image in step 201.
[0110] Step 304: The rendering device 100 performs noise reduction processing on the rendered image based on the AI model selected by the user to obtain the image to be output.
[0111] The image to be output can be equivalent to Figure 2 The first image in step 201 and the third image in step 205.
[0112] Step 305: The rendering device 100 determines whether the image quality index of the image to be output is lower than a preset threshold. If not, proceed to step 306. If yes, proceed to step 307.
[0113] Among them, step 305 and Figure 2 Steps 202 and 206 are similar and will not be described in detail here.
[0114] Step 306: Rendering device 100 outputs the image to be output.
[0115] Among them, step 305 and Figure 2 Steps 203 and 207 are similar and will not be described in detail here. The output image can be equivalent to... Figure 2 The first image in step 203 and the third image in step 207.
[0116] Step 307: The rendering device 100 uses the image to be output as the first image.
[0117] Among them, step 307 and Figure 2 Steps 201 and 208 are similar and will not be described in detail here. The output image can be equivalent to... Figure 2 The first image in step 201 and the third image in step 208.
[0118] Step 308: The rendering device 100 receives the adjusted rendering parameters and the selection instruction for the AI model input by the user. Based on the adjusted rendering parameters, it renders and denoises at least one frame of the scene to be rendered, obtaining a second image. The second image can be equivalent to... Figure 2 The second image in step 204.
[0119] Step 309: The rendering device 100 merges the first image and the second image to generate a second rendered image, and then executes step 304.
[0120] The second rendered image can be equivalent to Figure 2 The second rendered image described in step 205.
[0121] Based on the above Figure 3In steps 301 to 309, the rendering device 100 performs at least one rendering process in the hierarchical rendering. When the image quality index of the output image generated after the first rendering is greater than or equal to a preset threshold, the rendering device 100 outputs the output image, thereby significantly reducing rendering time when the user's requirements for rendering effects are low. When the image quality index of the output image generated after the first rendering is lower than the preset threshold, the rendering device 100 can continue rendering by utilizing the superposition property of Monte Carlo sampling. This effectively improves the sampling quality of the output image when the user's requirements for rendering effects are high, while avoiding the problem of re-rendering and saving time and computing resources.
[0122] The above describes the overall process of the rendering method. The following section describes another possible implementation of the rendering method through a specific example. Figure 1 Taking rendering device 100 as an example, this will be explained. Figure 4 This is a scene illustration illustrating a rendering method provided in an embodiment of this application. About Figure 4 The specific implementation of the rendering device 100 can be referred to the aforementioned Figure 1 The relevant descriptions will not be repeated here.
[0123] like Figure 4 As shown, the rendering method includes steps one through five.
[0124] Step 1: Initialize the scene to be rendered using the rendering device 100. The rendering device 100 initializes a digital human broadcasting scene to be rendered in the 3D graphics software (Blender).
[0125] Step 2: Rendering device 100 performs rendering under the minimum quality and highest efficiency requirements. Rendering device 100 receives the SPP value of 4 set by the user in Blender using an interactive editing method, and renders the image in the ray tracing rendering engine (cycles) according to 4 SPP to obtain the first rendered image. Rendering device 100 renders 1000 frames of the digital human broadcasting scene to be rendered, with each frame taking approximately 2 seconds to render, and the total rendering time is estimated at 33 minutes.
[0126] Rendering device 100 performs denoising processing on the first rendered image based on the first AI model with the largest preset parameters, obtaining a 4SPP first output image. Rendering device 100 uses a 30-frame sliding window method for denoising, with each frame taking 500 milliseconds to denoise, resulting in a total denoising time of approximately 9 minutes for the entire scene. Therefore, the total time required for the first stage of rendering is approximately 42 minutes. The first output image is equivalent to... Figure 2 The first image in step 201.
[0127] The rendering device 100 receives the user's judgment that the first image to be output does not meet the requirements.
[0128] Step 3: Rendering device 100 performs rendering under conditions of lower quality and higher efficiency. Rendering device 100 receives the SPP value of 64 set by the user in the Blender interface, and renders according to 60 SPP in cycles, obtaining a second rendered image with 60 SPP. The rendering time for each frame is approximately 6 seconds, and the total rendering time is estimated to be 100 minutes.
[0129] Rendering device 100 performs denoising processing on the second rendered image based on a second AI model with the second largest preset parameter set, and then fuses the 4SPP first output image and the denoised 60SPP second rendered image to obtain a 64SPP second output image. The second rendered image is equivalent to... Figure 2 The second image in step 205, the second image to be output is equivalent to Figure 2 The third image in step 205.
[0130] The second AI model has fewer parameters than the first AI model, and the denoising intensity of the second AI model is less than that of the first AI model.
[0131] The rendering device 100 uses a 10-frame sliding window method for denoising. Each frame takes 300 milliseconds to denoise, and the total denoising time for the entire scene is about 5 minutes. Therefore, the total time required for the second graded rendering is about 105 minutes.
[0132] The rendering device 100 receives the user's judgment that the second image to be output does not meet the requirements.
[0133] Step 4: Rendering device 100 performs rendering with higher quality and lower efficiency requirements. Rendering device 100 receives the SPP value of 512 set by the user in the blender interface, and renders according to 448 SPP in cycles, obtaining a third rendered image of 448 SPP. The rendering time for each frame is approximately 30 seconds, and the total rendering time is estimated at 500 minutes.
[0134] Rendering device 100 denoises the third rendered image based on the third AI model with the third largest preset parameter set. It then fuses the 64-slice second output image with the denoised 448-slice third rendered image to obtain a 512-slice third output image. Rendering device 100 uses single-frame denoising techniques such as the open-source image denoising library (OIDN) for denoising. Each frame takes 60 milliseconds to denoise, and the total denoising time for the entire scene is approximately 1 minute. Therefore, the total time required for the third stage of rendering is approximately 501 minutes. The second output image is equivalent to... Figure 2 The first image and the third rendered image in step 205 are equivalent to Figure 2 The second image and the third image to be output in step 205 are equivalent to... Figure 2 The third image in step 205.
[0135] Among them, the number of parameters of the third AI model is less than that of the second AI model, and the denoising strength of the third AI model is less than that of the second AI model.
[0136] The rendering device 100 receives the user's judgment that the third image to be output does not meet the requirements.
[0137] Step 5: Rendering device 100 performs rendering at the highest quality requirements. The SPP value set by the user in the Blender interface is 4096, and the rendering device 100 renders at 3584 SPP in cycles, resulting in a fourth rendered image of 3584 SPP. Each frame takes approximately 50 seconds to render, and the total rendering time is estimated at 833 minutes.
[0138] Rendering device 100 fuses the third image to be output (512 SPP) and the fourth image (3584 SPP) to obtain a fourth image to be output (4096 SPP). The third image to be output is equivalent to... Figure 2 The first image and the fourth rendered image in step 205 are equivalent to Figure 2 The second image and the fourth image to be output in step 205 are equivalent to... Figure 2 The third image from step 205.
[0139] In steps three through five above, the smaller the value of SPP, the shorter the rendering wait time; the larger the value of SPP, the higher the quality of the rendered image.
[0140] Based on the above Figure 4In steps one through five, the rendering device 100 achieves dynamic adjustment of the SPP (Size, Length, and Proportion) based on specific quality and efficiency requirements through tiered rendering. This adapts to rendering tasks in different scenarios, ensuring optimal performance and quality balance across various rendering scenarios. In scenarios requiring high efficiency, the rendering device 100 significantly reduces rendering time by using a lower SPP value combined with AI model denoising technology. In this embodiment, if a 64-SPP image meets user needs, the rendering device 100 only needs to spend a minimum of 105 minutes, saving 733 minutes of rendering time compared to the 833 minutes required for rendering images at the highest image quality in existing rendering methods.
[0141] Furthermore, the rendering device 100 utilizes the superimposability of Monte Carlo sampling to achieve continuous rendering, avoiding re-rendering due to unsatisfactory image quality, thus further saving time and resources. Specifically, in this embodiment, an image meeting user requirements can be obtained in a minimum of 42 minutes. If the image does not meet user needs, rendering can continue. Assuming that the user's desired effect is met only after 4096 SPPs, existing re-rendering schemes would waste (42 + 105 + 500) minutes of trial and error time. However, in this embodiment, since rendering can continue on the previously rendered image, trial and error is unnecessary, reducing the waste of time resources.
[0142] The rendering method provided according to this embodiment has been described in detail above. The following will combine... Figure 5 This describes the rendering apparatus provided in this embodiment.
[0143] Figure 5 This is a schematic diagram of a possible rendering apparatus 500 provided in an embodiment of this application. The rendering apparatus 500 can be used to implement the rendering function of the rendering device 100 in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In this embodiment, the rendering apparatus 500 can be... Figure 1 The rendering device 100 used to implement the rendering function can also be a module (such as a chip) applied to the rendering device 100.
[0144] The rendering device 500 includes an acquisition module 501, a rendering module 502, and a fusion module 503.
[0145] The acquisition module 501 is used to acquire a first image based on the image to be rendered; the sampling number (SPP) of a single pixel in the first image is the first SPP. For example, the acquisition module 501 is used to perform the above... Figure 2 Step 201, or execution Figure 3 Steps 303 and 307 in the text.
[0146] The rendering module 502 is configured to render the image to be rendered according to the second SPP if the image quality index of the first image is lower than a preset threshold, thereby obtaining a second image. For example, the rendering module 502 is configured to perform the above... Figure 2 Step 202, or execution Figure 3 Step 308 in the process.
[0147] The fusion module 503 is used to fuse the first image and the second image to generate a third image. For example, the fusion module 503 is used to perform the above... Figure 2 Step 203, or execution Figure 3 Step 309 in the process.
[0148] As one possible implementation, the second SPP is a preset value, and the third SPP of the third image is the sum of the second SPP and the first SPP.
[0149] As one possible implementation, the second SPP is the difference between the third SPP input by the user and the first SPP, wherein the third SPP is the SPP of the third image.
[0150] As one possible implementation, the fusion module 503 is specifically used to perform a weighted average of the first image and the second image based on the first ratio of the first SPP to the third SPP and the second ratio of the second SPP to the third SPP to obtain the third image.
[0151] As one possible implementation, the acquisition module 501 is specifically used to render the image to be rendered according to the preset first SPP to obtain the first image.
[0152] As one possible implementation, the acquisition module 501 is specifically used to render the image to be rendered according to the preset first SPP to obtain a first rendered image; and to perform noise reduction processing on the first rendered image to obtain the first image.
[0153] As one possible implementation, the fusion module 503 is specifically used to fuse the first image and the second image to generate a second rendered image; and to perform noise reduction processing on the second rendered image to obtain the third image.
[0154] As one possible implementation, the denoising intensity of the denoising process is inversely proportional to the SPP of the image rendering.
[0155] As one possible implementation, the denoising process is based on an artificial intelligence (AI) model, and the denoising intensity is proportional to the number of parameters in the AI model.
[0156] As one possible implementation, the device further includes an output module, configured to output the third image if the image quality index of the third image is greater than or equal to the preset threshold.
[0157] As one possible implementation, the device further includes an output module, configured to output the first image if the image quality index of the first image is greater than or equal to the preset threshold.
[0158] The acquisition module 501, rendering module 502, and fusion module 503 can all be implemented in software or hardware. For example, the implementation of the acquisition module 501 will be described below. Similarly, the implementation of the rendering module 502 and fusion module 503 can refer to the implementation of the acquisition module 501.
[0159] As an example of a software functional unit, module 501 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, module 501 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.
[0160] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.
[0161] As an example of a hardware functional unit, the acquisition module 501 may include at least one computing device, such as a server. Alternatively, the acquisition module 501 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.
[0162] The multiple computing devices included in the acquisition module 501 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition module 501 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition module 501 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0163] It should be noted that, in other embodiments, any one of the acquisition module 501, rendering module 502, and fusion module 503 can be used to execute any step in the rendering method. The steps implemented by the acquisition module 501, rendering module 502, and fusion module 503 can be specified as needed. By implementing different steps in the rendering method through the acquisition module 501, rendering module 502, and fusion module 503, all functions of the rendering device 500 can be realized.
[0164] This application also provides a computing device, such as... Figure 6 As shown, the computing device 600 includes a bus 602, a processor 604, a memory 606, and a communication interface 608. The processor 604, the memory 606, and the communication interface 608 communicate with each other via the bus 602. The computing device 600 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 600.
[0165] Bus 602 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus 604 is represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 604 may include a path for transmitting information between various components of the computing device 600 (e.g., memory 606, processor 604, communication interface 608).
[0166] Processor 604 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0167] Memory 606 may include volatile memory, such as random access memory (RAM). Processor 604 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).
[0168] The memory 606 stores executable program code, which the processor 604 executes to implement the functions of each module included in the aforementioned rendering apparatus 500, thereby implementing the rendering method. In other words, the memory 606 stores instructions for executing the rendering method.
[0169] Alternatively, the memory 606 stores executable code, which the processor 604 executes to implement the functions of each module included in the aforementioned rendering device 100, thereby implementing the rendering method. That is, the memory 606 stores instructions for executing the rendering method.
[0170] The communication interface 608 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 600 and other devices or communication networks.
[0171] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.
[0172] like Figure 7 As shown, the computing device cluster includes at least one computing device 600. The memory 606 of one or more computing devices 600 in the computing device cluster may store the same instructions for executing rendering methods.
[0173] In some possible implementations, the memory 606 of one or more computing devices 600 in the computing device cluster may also store partial instructions for executing the rendering method. In other words, a combination of one or more computing devices 600 can jointly execute the instructions for executing the rendering method.
[0174] It should be noted that the memory 606 in different computing devices 600 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the rendering device 500. That is, the instructions stored in the memory 606 of different computing devices 600 can implement the functions of one or more modules among the acquisition module 501, rendering module 502, and fusion module 503.
[0175] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 8 One possible implementation is shown. For example... Figure 8 As shown, the two computing devices 600A and 600B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 606 in computing device 600A stores instructions for executing the functions of the acquisition module 501. Simultaneously, the memory 606 in computing device 600B stores instructions for executing the functions of the rendering module 502 and the fusion module 503.
[0176] It should be understood that Figure 8 The functions of computing device 600A shown can also be performed by multiple computing devices 600. Similarly, the functions of computing device 600B can also be performed by multiple computing devices 600.
[0177] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform actions such as... Figure 2 or Figure 3 The rendering method shown.
[0178] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to... Figure 2 or Figure 3 The rendering method shown.
[0179] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD). The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A rendering method, characterized by, The method comprises: acquiring a first image based on a to-be-rendered image; a sampling number SPP of a single pixel of the first image is a first SPP; if an image quality index of the first image is lower than a preset threshold, rendering the to-be-rendered image according to a second SPP to obtain a second image; generating a third image by fusing the first image and the second image.
2. The method of claim 1, wherein, The second SPP is a preset value, and a third SPP of the third image is a sum of the second SPP and the first SPP.
3. The method of claim 1, wherein, The second SPP is a difference between a third SPP input by a user and the first SPP, and the third SPP is an SPP of the third image.
4. The method according to claim 2 or 3, characterized in that, The generating a third image by fusing the first image and the second image comprises: performing weighted averaging on the first image and the second image according to a first proportion of the first SPP in the third SPP and a second proportion of the second SPP in the third SPP to obtain the third image.
5. The method according to any one of claims 1 to 4, characterized in that, The acquiring a first image comprises: rendering the to-be-rendered image according to the preset first SPP to obtain the first image.
6. The method of claim 5, wherein, The rendering the to-be-rendered image according to the preset first SPP to obtain the first image comprises: rendering the to-be-rendered image according to the preset first SPP to obtain a first rendered image; performing denoising processing on the first rendered image to obtain the first image.
7. The method according to any one of claims 1 to 6, characterized in that, The generating a third image by fusing the first image and the second image comprises: generating a second rendered image by fusing the first image and the second image; performing denoising processing on the second rendered image to obtain the third image.
8. The method according to claim 6 or 7, characterized in that, A denoising intensity of the denoising processing is inversely proportional to an SPP of image rendering.
9. The method according to any one of claims 6-8, characterized in that, The denoising processing is implemented based on an artificial intelligence AI model, and a denoising intensity of the denoising processing is proportional to a parameter quantity of the AI model.
10. The method of claim 1, wherein, The method further comprises: if an image quality index of the third image is greater than or equal to the preset threshold, outputting the third image.
11. The method of claim 1, wherein, The method further comprises: if an image quality index of the first image is greater than or equal to the preset threshold, outputting the first image.
12. A rendering device, characterized in that The apparatus comprises: an acquiring module configured to acquire a first image based on a to-be-rendered image; a sampling number SPP of a single pixel of the first image is a first SPP; a rendering module configured to, if an image quality index of the first image is lower than a preset threshold, render the to-be-rendered image according to a second SPP to obtain a second image; a fusing module configured to generate a third image by fusing the first image and the second image.
13. The apparatus of claim 12, wherein, The second SPP is a preset value, and a third SPP of the third image is a sum of the second SPP and the first SPP.
14. The apparatus of claim 12, wherein, The second SPP is a difference between a third SPP input by a user and the first SPP, and the third SPP is an SPP of the third image.
15. The apparatus of claim 13 or 14, wherein, The fusion module is specifically configured to perform weighted average on the first image and the second image according to a first proportion of the first SPP to the third SPP and a second proportion of the second SPP to the third SPP, to obtain the third image.
16. The apparatus of any one of claims 12-15, wherein, The acquisition module is specifically configured to render the to-be-rendered image according to the preset first SPP, to obtain the first image.
17. The apparatus of claim 16, wherein, The acquisition module is specifically configured to render the to-be-rendered image according to the preset first SPP, to obtain a first rendered image; and perform denoising processing on the first rendered image, to obtain the first image.
18. The apparatus of any of claims 12-17, wherein, The fusion module is specifically configured to generate a second rendered image by fusing the first image and the second image; and perform denoising processing on the second rendered image, to obtain the third image.
19. The apparatus of claim 17 or 18, wherein, The denoising strength of the denoising processing is inversely proportional to the SPP at which the image is rendered.
20. The apparatus of any of claims 17-19, wherein, The denoising processing is implemented based on an artificial intelligence (AI) model, and the denoising strength of the denoising processing is proportional to a parameter quantity of the AI model.
21. The apparatus of claim 12, wherein, The apparatus further includes: An output module configured to output the third image if the image quality indicator of the third image is greater than or equal to the preset threshold.
22. The apparatus of claim 12, wherein, The apparatus further includes: An output module configured to output the first image if the image quality indicator of the first image is greater than or equal to the preset threshold.
23. A cluster of computing devices, characterized in that, The at least one computing device includes a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the operation steps of the method of any one of claims 1-11.
24. A computer-readable storage medium, comprising: It includes: Computer software instructions; when the computer software instructions run in the controller, the controller performs the operation steps of the method of any one of claims 1-11.
25. A computer program product, characterised in that, It includes computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the operation steps of the method of any one of claims 1-11.