Image rendering method and related device thereof

Abstract

The present application discloses an image rendering method and a related device, which can make the rendering effect of a target object good enough no matter which one of diffuse reflection, specular reflection and highlight reflection is formed by light rays shooting on the target object. The method of the present application comprises: resampling the light ray path between the target object and a light source based on an original light ray path between the target object and the light source, to obtain a new light ray path, the original light ray path being obtained by ray tracing, the original light ray path having the target object as a starting point and the light source as a terminal point, the new light ray path having the light source as a starting point and the target object as a terminal point; obtaining a sample pool based on the new light ray path, the sample pool being used to indicate light rays shooting on the target object, the new light ray path being formed by the light rays; and rendering the target object based on the sample pool, to obtain an image of the target object.

Description

Technical Field

[0001] This application relates to the field of computer graphics technology, and in particular to an image rendering method and related equipment. Background Technology

[0002] With the rapid development of the computer industry, users' demands for image quality are increasing. Currently, electronic devices typically use ray tracing technology to render 3D scenes, thereby obtaining more realistic 3D scene images for users to view and use, thus improving the user experience.

[0003] Currently, a ray tracing algorithm based on path resampling has been proposed. In this algorithm, for an object to be rendered in a 3D scene, after determining the original ray path formed by the first ray intersecting the object, the ray path can be resampled based on this original ray path to acquire a new ray path formed by the new ray that first intersects the object. Next, a corresponding sample pool can be obtained based on this new ray path; this sample pool is used to indicate the new ray that formed the new ray path. Finally, the image of the object can be rendered based on this sample pool.

[0004] In this algorithm, due to the limitations of the object's material, if the light's effect on an object is diffuse reflection, the re-emitted light rays hitting that object often have a large reflection range. Even if the reflected light rays continue to hit other objects, there is a high probability that they will reach the light source. Therefore, it is easy to resample an effective new light path, resulting in a better rendering effect for that object. However, if the light's effect on the object is specular reflection or specular reflection, the re-emitted light rays hitting that object often have a small reflection range. Even if the reflected light rays continue to hit other objects, there is only a small probability that they will reach the light source. Therefore, it is difficult to resample an effective new light path, resulting in an insufficient high-quality sample pool and a generally poor rendering effect for that object. Summary of the Invention

[0005] This application provides an image rendering method and related equipment that can achieve sufficiently high-quality rendering effects for the target object regardless of whether the phenomenon produced by light rays hitting the target object is diffuse reflection, specular reflection, or specular reflection.

[0006] A first aspect of this application provides an image rendering method, the method comprising:

[0007] When rendering a target object in a 3D scene, the original ray path between the target object and the light source can be obtained first. Since the original ray path between the target object and the light source starts from the target object and ends at the light source, these two endpoints can be determined. The properties of the target object and the light source can then be swapped to reconstruct the ray path between them. This involves resampling the ray path between the two objects, resulting in a new ray path between the target object and the light source, starting from the light source and ending at the target object.

[0008] Obtaining the new ray path between the target object and the light source is equivalent to obtaining the rays that form this new path. Information related to these rays can then be acquired and used to construct a sample pool. For example, the intensity values ​​of these rays before they hit the target object can be obtained and stored in the sample pool. In this way, the sample pool can be used to indicate the rays that strike the target object.

[0009] Once the sample pool is obtained, it can be used to render the target object, thereby obtaining the image of the target object. This image of the target object is then displayed on the screen for user viewing and use.

[0010] As can be seen from the above method, after obtaining the original ray path between the target object and the light source, the ray path between the target object and the light source can be resampled based on the original ray path to obtain a new ray path. The original ray path is obtained through ray tracing, with the target object as the starting point and the light source as the ending point. The new ray path starts from the light source and ends at the target object. Then, a sample pool can be obtained based on the new ray path. The sample pool is used to indicate the rays that are directed towards the target object, and the new ray path is formed by these rays. Finally, the target object can be rendered based on the sample pool to obtain the image of the target object. Based on the foregoing process, this application provides a new ray path resampling method. This method reverses the direction of the effective ray path in ray tracing (starting from the light source and ending at the target object) and selectively resamples the ray path, which can improve the success rate of sampling effective new ray paths. That is, it can easily resample the effective new ray path between the target object and the light source. In this way, regardless of whether the phenomenon produced by the light that forms the new ray path hitting the target object is direct lighting, diffuse reflection, specular reflection, or specular reflection, a high-quality and sufficient sample pool can be successfully obtained, thereby making the rendering effect of the target object sufficiently high-quality.

[0011] In one possible implementation, the new ray path includes a first ray path and a second ray path. The first ray path starts from the light source, ends at the target object, and passes through other objects. The second ray path starts from the light source, ends at the target object, and does not pass through other objects. The sample pool includes a first sample pool and a second sample pool. The first sample pool is used to indicate the first ray directed toward the target object, and the first ray path is formed by the first ray. The second sample pool is used to indicate the second ray directed toward the target object, and the second ray path is formed by the second ray. In the aforementioned implementation, after obtaining the original ray path between the target object and the light source, the ray path between the target object and the light source can be reconstructed in a certain way based on the original ray path. That is, the ray path between the two is resampled for the first time to obtain the first ray path between the target object and the light source (also called the indirect ray path). The first ray path starts from the light source and ends at the target object, and passes through other objects in between (that is, the first ray path has the light source and the target object as the two endpoints, and passes through at least one intermediate node in between, which are the other objects in the space where the target object is located, excluding the target object). Alternatively, the ray path between the two can be reconstructed in another way based on the original ray path. That is, the ray path between the two is resampled for the first time to obtain the second ray path between the target object and the light source (also called the direct ray path). The second ray path starts from the light source and ends at the target object, and does not pass through other objects in between (that is, the second ray path has the light source and the target object as the two endpoints, and does not pass through any intermediate nodes in between, i.e., it does not pass through any other objects in the space where the target object is located, excluding the target object). Obtaining the first and second ray paths between the target object and the light source is equivalent to obtaining the first ray (also called indirect ray, which originates from the light source, hits other objects, and indirectly reaches the target object) and the second ray (also called direct ray, which originates from the light source and directly reaches the target object) between the target object and the light source. Information related to the first ray can then be acquired and used to construct a first sample pool. For example, the intensity value of the first ray after hitting other objects and before hitting the target object can be obtained and stored in the first sample pool. In this way, the first sample pool can be used to indicate the first ray striking the target object. Similarly, information related to the second ray can be acquired and used to construct a second sample pool. For example, the intensity value of the second ray before hitting the target object can be obtained and stored in the second sample pool. In this way, the second sample pool can be used to indicate the second ray striking the target object.Based on this, the target object can be rendered using both the first and second sample pools to obtain an image of the target object. It is evident that the aforementioned implementation provides a ray tracing technique based on multiple sample pools, where multiple sample pools refer to at least one first sample pool and at least one second sample pool. In a common configuration, two first sample pools and one second sample pool can be constructed. The rendering effects corresponding to the two first sample pools are specular reflection and specular reflection, respectively, while the rendering effect corresponding to the second sample pool is direct lighting. Therefore, even if the target object is an object with a special material, the image of the target object rendered based on multiple sample pools can present both direct lighting effects and indirect lighting effects such as specular reflection and specular reflection, resulting in a more realistic and exquisite image, which is beneficial to improving the user experience.

[0012] In one possible implementation, the phenomenon produced when the first ray hits the target object includes at least one of the following: diffuse reflection, specular reflection, or specular reflection. In the aforementioned implementation, there may be two situations: (1) If only a single round of first resampling is performed, a set of first ray paths between the target object and the light source can be obtained, which also results in a set of first rays forming the set of first ray paths. Therefore, the phenomenon produced when the set of first rays hits the target object can be controlled to be one of the following: diffuse reflection, specular reflection, or specular reflection. Correspondingly, the first sample pool used to indicate that the set of first rays corresponds to a certain rendering effect, which can be presented as one of the following: diffuse reflection, specular reflection, or specular reflection. For example, if the phenomenon produced when the set of first rays hits the target object is diffuse reflection, the rendering effect corresponding to the first sample pool is diffuse reflection, etc. (2) If only multiple rounds of first resampling are performed, multiple sets of first ray paths between the target object and the light source can be obtained, which also results in multiple sets of first rays forming these multiple sets of first ray paths. For any one of the multiple sets of first rays, the phenomenon produced when that set of first rays hits the target object can be controlled to be one of the following: diffuse reflection, specular reflection, or specular reflection. It should be noted that the phenomena produced by the multiple sets of first rays upon hitting the target object can be the same or different; this is not limited here. Correspondingly, for any one of the multiple first sample pools used to indicate these multiple sets of first rays, this first sample pool corresponds to a certain rendering effect, which can be one of the following: diffuse reflection, specular reflection, or specular reflection. It should be noted that the rendering effects corresponding to the multiple first sample pools can be the same or different. For example, a conventional sample pool configuration can be: constructing two first sample pools, where the first first sample pool corresponds to a specular reflection rendering effect, and the second first sample pool corresponds to a specular reflection rendering effect. It can be seen that the multi-sample pool technology provided by the aforementioned implementation method can make different first sample pools correspond to different rendering effects, or different first sample pools correspond to the same rendering effect, and can be selected according to the material of the target object, improving the flexibility and comprehensiveness of the solution.

[0013] In one possible implementation, the number of first rays indicated by the first sample pool is less than the number of pixels displayed on the screen, and the number of second rays indicated by the second sample pool is equal to the number of pixels displayed on the screen. The screen is used to display an image of the target object. In the aforementioned implementation, for any first sample pool, it may contain a set of light intensity values ​​of the first rays after hitting other objects and before hitting the target object. The size of the first sample pool is the number of these light intensity values, which is the number of the set of first rays (i.e., the size of the first sample pool is the number of first rays indicated by the first sample pool). It is worth noting that the size of the first sample pool is less than the screen resolution (i.e., the number of pixels displayed on the screen), and the screen is used to display the final image of the target object. For any second sample pool, it may contain a set of light intensity values ​​of the second rays before hitting the target object. The size of the second sample pool is the number of these light intensity values, which is the number of the set of second rays (i.e., the size of the second sample pool is the number of second rays indicated by the second sample pool). It is worth noting that the size of the second sample pool is equal to the screen resolution (i.e., the number of pixels displayed on the screen). Therefore, the aforementioned implementation method can make the size of the first sample pool smaller than the screen resolution, which can improve the efficiency of obtaining the first sample pool.

[0014] In one possible implementation, rendering the target object based on a first sample pool and a second sample pool to obtain an image of the target object includes: rendering the target object based on the first sample pool to obtain a first lighting image of the target object; rendering the target object based on the second sample pool to obtain a second lighting image of the target object; and fusing the first lighting image and the second lighting image to obtain an image of the target object. In the aforementioned implementation, since at least one first sample pool can be obtained, each of the at least one first sample pool can be shaded to obtain at least one indirect lighting image of the target object (i.e., the aforementioned first lighting image). Since at least one second sample pool can be obtained, each of the at least one second sample pool can be shaded to obtain at least one direct lighting image of the target object (i.e., the aforementioned second lighting image). After obtaining at least one indirect lighting image and at least one direct lighting image of the target object, these multiple images can be considered as multiple layers. Therefore, the at least one indirect lighting image and at least one direct lighting image of the target object can be superimposed to obtain the final image of the target object that can be displayed.

[0015] In one possible implementation, before fusing the first illumination image and the second illumination image to obtain an image of the target object, the method further includes: acquiring a noise-free component of the material information of the target object; performing a first processing on the first illumination image based on the noise-free component to obtain a processed first illumination image, the first processing including demodulation processing, noise reduction processing, and modulation processing; performing a second processing on the second illumination image based on the noise-free component to obtain a processed second illumination image, the second processing including demodulation processing, noise reduction processing, and modulation processing; fusing the first illumination image and the second illumination image to obtain an image of the target object includes: fusing the processed first illumination image and the processed second illumination image to obtain an image of the target object. In the aforementioned implementation, after acquiring the noise-free component of the material information of the target object, at least one indirect illumination image of the target object can be divided by the noise-free component to obtain at least one demodulated indirect illumination image of the target object. After obtaining at least one demodulated indirect lighting image of the target object, noise reduction can be performed on the demodulated indirect lighting image of the target object to obtain at least one denoised indirect lighting image of the target object. After obtaining at least one denoised indirect lighting image of the target object, the at least one denoised indirect lighting image of the target object can be multiplied by the noise-free component to obtain at least one modulated indirect lighting image of the target object, which can be used as at least one processed indirect lighting image of the target object (the processed indirect lighting image is the aforementioned processed first lighting image). At the same time, after obtaining the noise-free component of the material information of the target object, at least one direct lighting image of the target object can be divided by the noise-free component to obtain at least one demodulated direct lighting image of the target object. After obtaining at least one demodulated direct lighting image of the target object, noise reduction can be performed on the at least one demodulated direct lighting image of the target object to obtain at least one denoised direct lighting image of the target object. After obtaining at least one denoised direct illumination image of the target object, the noise-free component can be multiplied to obtain at least one modulated direct illumination image of the target object, i.e., at least one processed direct illumination image of the target object (the processed direct illumination image is the aforementioned processed second illumination image). After obtaining at least one processed indirect illumination image and at least one processed direct illumination image of the target object, the processed indirect illumination image and at least one processed direct illumination image of the target object can be superimposed to obtain the final image of the target object that can be displayed.As can be seen, the aforementioned implementation provides a new image denoising method that can demodulate, denoise, and modulate each layer based on noise-free classification of the target object's material information. This allows each layer to retain the details of the target object as much as possible, making the image of the target object obtained by superimposing each layer more realistic and clear, which is beneficial to improving the user's viewing experience and further enhancing the user experience.

[0016] In one possible implementation, the first processing further includes super-resolution processing, which is used to make the resolution of the processed first lighting image equal to the screen resolution. In the aforementioned implementation, since the size of at least one first sample pool is smaller than the screen resolution, the resolution of at least one modulated indirect lighting image of the target object is also smaller than the screen resolution. To achieve better fusion of the layers, after obtaining the modulated at least one indirect lighting image of the target object, super-resolution processing can be applied to the modulated at least one indirect lighting image of the target object, and the super-resolution at least one indirect lighting image of the target object can be used as the processed at least one indirect lighting image of the target object. At this time, the resolution of the processed at least one indirect lighting image of the target object is equal to the screen resolution. It can be seen that the aforementioned implementation, in the process of rendering the target object based on multi-sample pooling technology, can be combined with super-resolution processing to a certain extent to reduce the amount of computation required in this process, which is conducive to accelerating the implementation of ray tracing technology based on multi-sample pooling and further improving the user experience.

[0017] A second aspect of this application provides an image rendering apparatus, comprising: a resampling module, configured to resample the ray path between a target object and a light source based on the original ray path between the target object and the light source to obtain a new ray path, wherein the original ray path is obtained by ray tracing, the original ray path starts from the target object and ends at the light source, and the new ray path starts from the light source and ends at the target object; a first acquisition module, configured to acquire a sample pool based on the new ray path, the sample pool being used to indicate the rays directed toward the target object, and the new ray path being formed by the rays; and a rendering module, configured to render the target object based on the sample pool to obtain an image of the target object.

[0018] As can be seen from the above device, after obtaining the original ray path between the target object and the light source, the ray path between the target object and the light source can be resampled based on the original ray path to obtain a new ray path. The original ray path is obtained through ray tracing, with the target object as the starting point and the light source as the ending point. The new ray path starts from the light source and ends at the target object. Then, a sample pool can be obtained based on the new ray path. The sample pool is used to indicate the rays that are directed towards the target object, and the new ray path is formed by these rays. Finally, the target object can be rendered based on the sample pool to obtain an image of the target object. Based on the foregoing process, this application provides a new ray path resampling method. This method reverses the direction of the effective ray path in ray tracing (starting from the light source and ending at the target object) and selectively resamples the ray path, which can improve the success rate of sampling effective new ray paths. That is, it can easily resample the effective new ray path between the target object and the light source. In this way, regardless of whether the phenomenon produced by the light that forms the new ray path hitting the target object is direct lighting, diffuse reflection, specular reflection, or specular reflection, a high-quality and sufficient sample pool can be successfully obtained, thereby making the rendering effect of the target object sufficiently high-quality.

[0019] In one possible implementation, the new ray path includes a first ray path and a second ray path. The first ray path starts from the light source, ends at the target object, and passes through other objects. The second ray path starts from the light source, ends at the target object, and does not pass through other objects. The sample pool includes a first sample pool and a second sample pool. The first sample pool is used to indicate the first ray directed toward the target object, and the first ray path is formed by the first ray. The second sample pool is used to indicate the second ray directed toward the target object, and the second ray path is formed by the second ray.

[0020] In one possible implementation, the phenomenon produced when the first ray strikes the target object is specular reflection or specular reflection.

[0021] In one possible implementation, the number of first rays indicated by the first sample pool is less than the number of pixels displayed on the screen, and the number of second rays indicated by the second sample pool is equal to the number of pixels displayed on the screen, which is used to display an image of the target object.

[0022] In one possible implementation, the rendering module is configured to: render the target object based on a first sample pool to obtain a first lighting image of the target object; render the target object based on a second sample pool to obtain a second lighting image of the target object; and fuse the first lighting image and the second lighting image to obtain an image of the target object.

[0023] In one possible implementation, the rendering module is further configured to: acquire the noise-free component of the material information of the target object; perform a first processing on the first illumination image based on the noise-free component to obtain a processed first illumination image, the first processing including demodulation processing, noise reduction processing, and modulation processing; perform a second processing on the second illumination image based on the noise-free component to obtain a processed second illumination image, the second processing including demodulation processing, noise reduction processing, and modulation processing; and fuse the processed first illumination image and the processed second illumination image to obtain an image of the target object.

[0024] In one possible implementation, the first processing further includes super-resolution processing, which is used to make the resolution of the processed first illumination image equal to the resolution of the screen.

[0025] A third aspect of this application provides an electronic device including a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the electronic device performs the method described in the first aspect or any possible implementation thereof.

[0026] A fourth aspect of this application provides a circuit system including a processing circuit configured to perform the method as described in the first aspect and any possible implementation thereof.

[0027] A fifth aspect of this application provides a chip system including a processor for calling a computer program or computer instructions stored in a memory, such that the processor performs the method as described in the first aspect or any possible implementation thereof.

[0028] In one possible implementation, the processor is coupled to the memory via an interface.

[0029] In one possible implementation, the chip system also includes a memory that stores computer programs or computer instructions.

[0030] A sixth aspect of this application provides a computer storage medium storing one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method as described in the first aspect or any possible implementation thereof.

[0031] A seventh aspect of this application provides a computer program product storing instructions that, when executed by a computer, cause the computer to perform the method as described in the first aspect or any possible implementation thereof.

[0032] In this embodiment, after obtaining the original ray path between the target object and the light source, the ray path can be resampled based on the original ray path to obtain a new ray path. The original ray path is obtained through ray tracing, with the target object as the starting point and the light source as the ending point. The new ray path also starts from the light source and ends at the target object. Then, a sample pool can be obtained based on the new ray path. The sample pool is used to indicate the rays directed towards the target object, and the new ray path is formed by these rays. Finally, the target object can be rendered based on the sample pool to obtain an image of the target object. Based on the foregoing process, this application provides a new ray path resampling method. This method reverses the direction of the effective ray path in ray tracing (starting from the light source and ending at the target object) and selectively resamples the ray path, which can improve the success rate of sampling effective new ray paths. That is, it can easily resample the effective new ray path between the target object and the light source. In this way, regardless of whether the phenomenon produced by the light that forms the new ray path hitting the target object is direct lighting, diffuse reflection, specular reflection, or specular reflection, a high-quality and sufficient sample pool can be successfully obtained, thereby making the rendering effect of the target object sufficiently high-quality. Attached Figure Description

[0033] Figure 1 This is a schematic diagram illustrating the principle of ray tracing technology.

[0034] Figure 2 This is a schematic diagram illustrating the principle of rasterization technology.

[0035] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0036] Figure 4 A schematic flowchart of an image rendering method provided in an embodiment of this application;

[0037] Figure 5 A schematic diagram of a BVH tree provided in an embodiment of this application;

[0038] Figure 6 A schematic diagram of ray path resampling provided in an embodiment of this application;

[0039] Figure 7 This is a schematic diagram illustrating an application example of the image rendering method provided in the embodiments of this application;

[0040] Figure 8 A schematic diagram of the noise reduction process provided in the embodiments of this application;

[0041] Figure 9 A schematic diagram of the comparison results provided for an embodiment of this application;

[0042] Figure 10 Another schematic diagram illustrating the comparison results provided in the embodiments of this application;

[0043] Figure 11 Another schematic diagram illustrating the comparison results provided in the embodiments of this application;

[0044] Figure 12 Another schematic diagram illustrating the comparison results provided in the embodiments of this application;

[0045] Figure 13 Another schematic diagram illustrating the comparison results provided in the embodiments of this application;

[0046] Figure 14 Another schematic diagram illustrating the comparison results provided in the embodiments of this application;

[0047] Figure 15 This is a schematic diagram of the structure of an image rendering apparatus provided in an embodiment of this application. Detailed Implementation

[0048] This application provides an image rendering method and related equipment that can achieve sufficiently high-quality rendering effects for the target object regardless of whether the phenomenon produced by light rays hitting the target object is diffuse reflection, specular reflection, or specular reflection.

[0049] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0050] With the development of computer technology, more and more applications, such as games and video applications, require the display of high-quality images on electronic devices. These images are usually rendered by electronic devices based on models in a three-dimensional (3D) scene.

[0051] Traditional image processing methods typically employ rasterization to render 3D scenes, producing images capable of displaying them. However, images rendered using rasterization generally suffer from low quality and often fail to deliver realistic visuals. For instance, rendered images often struggle to accurately reproduce effects such as light reflection, refraction, and shadows within the scene. In response, a new rendering technique—ray tracing—emerged. Both ray tracing and rasterization are methods for image rendering, their primary purpose being to calculate and shade objects to project them into a two-dimensional screen space for display.

[0052] Figure 1 This is a schematic diagram illustrating the principle of ray tracing technology. Figure 1 As shown, the principle of ray tracing is as follows: From the camera's position, a ray of light is emitted through pixel positions on the image plane into the 3D scene. The nearest intersection point between the ray and the geometry is found, and then the shading of that intersection point is calculated. If the material of that intersection point is reflective, ray tracing can continue in the reflection direction from that intersection point, and the shading of the reflected intersection point can be calculated. In other words, the ray tracing method calculates projection and global illumination by tracking the propagation of light in a 3D scene, thereby rendering a 2D image.

[0053] Figure 2 This is a schematic diagram illustrating the principle of rasterization technology. (Example) Figure 2 As shown, the principle of rasterization is as follows: objects in a 3D scene are divided into triangles, the 3D coordinates of the triangle vertices are transformed into 2D coordinates on the image through coordinate transformation calculation, and finally, textures are filled into the triangles on the image to achieve image rendering.

[0054] Rasterization, which projects visible content directly onto the screen space to create an image, is relatively simple to process but produces less impressive lighting effects. Ray tracing, on the other hand, tracks every ray of light emitted from the camera to achieve realistic effects such as reflection, refraction, shadows, and ambient occlusion. Therefore, ray tracing provides truly lifelike lighting and shadow effects. Consequently, to render more realistic images, current electronic devices typically prioritize ray tracing for rendering 3D scenes, enhancing the user's viewing experience.

[0055] Currently, a ray tracing algorithm based on path resampling has been proposed. In this algorithm, for an object to be rendered in a 3D scene, after determining the original ray path formed by the first ray intersecting the object, the ray path can be resampled based on this original ray path to acquire a new ray path formed by the new ray that first intersects the object. Next, a corresponding sample pool can be obtained based on this new ray path; this sample pool indicates the new ray that formed the new ray path (i.e., the sample pool stores ray samples). Then, the image of the object can be rendered based on this sample pool.

[0056] In this algorithm, due to the limitations of the object's material, if the light's effect on an object is diffuse reflection, the re-emitted light rays hitting that object often have a large reflection range. Even if the reflected light rays continue to hit other objects, there is a high probability that they will reach the light source. Therefore, it is easy to resample an effective new light path, resulting in a better rendering effect for that object (i.e., the rendered image of that object is often of high quality). However, if the light's effect on the object is specular reflection or specular reflection, the re-emitted light rays hitting that object often have a smaller reflection range. The reflected light rays, after hitting other objects, have only a small probability of reaching the light source. Therefore, it is difficult to resample an effective new light path, resulting in an insufficient and high-quality sample pool. Consequently, the rendering effect for that object is often poor (i.e., the rendered image of that object is often of low quality).

[0057] Furthermore, this algorithm is not only inefficient in acquiring the sample pool, but the sample pool often contains a large amount of data, which makes the process of rendering objects based on the sample pool require a lot of computation, resulting in a high time cost for the entire rendering process and a poor user experience.

[0058] Furthermore, during the rendering process of objects, image denoising is often used to filter out noise in the image. However, the denoising process of these techniques often filters the entire image, resulting in a blurry final image of the object. Such an image leads to a poor viewing experience for the user and degrades the user experience.

[0059] To address the aforementioned problems, this application provides an image rendering method that can be executed by an electronic device. This electronic device includes a CPU and a GPU, capable of rendering images. Exemplarily, the electronic device may be a mobile phone, tablet computer, laptop computer, PC, mobile internet device (MID), wearable device, virtual reality (VR) device, augmented reality (AR) device, wireless electronic device in industrial control, wireless electronic device in self-driving, wireless electronic device in remote medical surgery, wireless electronic device in smart grid, wireless electronic device in transportation safety, wireless electronic device in smart city, wireless electronic device in smart home, etc. The electronic device may run Android, iOS, Windows, or other operating systems. Applications that require rendering 3D scenes to obtain 2D images, such as game applications, lock screen applications, map applications, or monitoring applications, can run on this electronic device.

[0060] To facilitate understanding, the following will be combined with... Figure 3 A detailed description of the specific structure of the electronic device is provided. Please refer to [reference needed]. Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 3 As shown, the electronic device 3000 may include: a central processing unit 3001, a graphics processor 3002, a display device 3003, and a memory 3004. Optionally, the electronic device 3000 may further include at least one communication bus. Figure 3 (Not shown in the image) is used to enable communication between various components.

[0061] It should be understood that the various components in the electronic device 3000 can also be coupled to each other via other connectors, which may include various interfaces, transmission lines, or buses. The various components in the electronic device 3000 can also be connected radially with the central processing unit 3001 at its center. In various embodiments of this application, coupling refers to mutual electrical connection or communication, including direct connection or indirect connection via other devices.

[0062] There are various ways to connect the central processing unit 3001 and the graphics processing unit 3002, and it is not limited to these methods. Figure 2 As shown in the diagram. The central processing unit 3001 and the graphics processing unit 3002 in the electronic device 3000 can be located on the same chip or they can be separate chips.

[0063] The functions of the central processing unit 3001, graphics processor 3002, display device 3003, and memory 3004 are briefly introduced below.

[0064] Central Processing Unit 3001: Used to run Operating System 3005 and Application Programming 3006. Application Programming 3006 can be a graphics application, such as a game, video player, etc. Operating System 3005 provides a system graphics library interface. Application Programming 3006 uses this system graphics library interface, as well as the drivers provided by Operating System 3005, such as user-mode and / or kernel-mode drivers for the graphics library, to generate an instruction stream for rendering graphics or image frames, and the necessary rendering data. The system graphics library includes, but is not limited to, OpenGL ES (Open Graphics Library for Embedded System), the Kronos Platform Graphics Interface, or Vulkan (a cross-platform graphics application programming interface). The instruction stream contains a series of instructions, which are typically calls to the system graphics library interface.

[0065] Optionally, the central processing unit 3001 may include at least one of the following types of processors: application processor, one or more microprocessors, digital signal processor (DSP), microcontroller unit (MCU), or artificial intelligence processor, etc.

[0066] The central processing unit 3001 may further include necessary hardware accelerators, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or integrated circuits for implementing logic operations. The processor 3001 may be coupled to one or more data buses for transferring data and instructions between the various components of the electronic device 3000.

[0067] Graphics processor 3002: Receives the graphics instruction stream sent by processor 3001, generates rendering targets through the rendering pipeline, and displays the rendering targets on display device 3003 through the operating system's layer compositing display module. The rendering pipeline, also known as the rendering pipeline, pixel pipeline, or pixel pipeline, is a parallel processing unit within the graphics processor 3002 used to process graphics signals. The graphics processor 3002 may include multiple rendering pipelines, which can process graphics signals independently and in parallel. For example, the rendering pipeline can perform a series of operations during the rendering of graphics or image frames; typical operations may include vertex processing, primitive processing, rasterization, fragment processing, etc.

[0068] Optionally, the graphics processor 3002 may include a general-purpose graphics processor that executes software, such as a GPU or other types of dedicated graphics processing units.

[0069] Display device 3003: Used to display various images generated by electronic device 3000, which may be the graphical user interface (GUI) of the operating system or image data (including still images and video data) processed by graphics processor 3002.

[0070] Optionally, the display device 3003 may include any suitable type of display screen, such as a liquid crystal display (LCD), a plasma display, or an organic light-emitting diode (OLED) display.

[0071] Memory 3004 is the transmission channel between the central processing unit 3001 and the graphics processor 3002, and can be double data rate synchronous dynamic random access memory (DDR SDRAM) or other types of cache.

[0072] The above describes the specific structure of the electronic device used in the image rendering method provided in the embodiments of this application. The following will describe in detail the process of the image rendering method provided in the embodiments of this application. Figure 4 This is a schematic flowchart of an image rendering method provided in an embodiment of this application, as shown below. Figure 4 As shown, the method includes:

[0073] 401. Obtain the original ray path between the target object and the light source. The original ray path is obtained through ray tracing. The original ray path starts from the target object and ends at the light source.

[0074] In this embodiment, after obtaining the model file of a certain 3D scene, the rendering information of each object in the 3D scene, the spatial information of the camera, and the spatial information of the light source, etc., can be parsed from it. The following is a brief introduction to these information:

[0075] (1) The rendering information of each object includes the spatial information of the object and the material information of the object. The spatial information of the object includes the vertex coordinates, vertex normals and triangle indices of the object, and the material information of the object includes the color, metallicity and roughness of the object.

[0076] It should be noted that a bounding volume hierarchy (BVH) tree can be constructed based on the spatial information of each object. The BVH tree can then be used to implement subsequent ray tracing operations. Specifically, the BVH tree can be constructed based on the vertex coordinates, vertex normals, and triangle indices of the objects. In essence, the BVH tree contains the spatial information of multiple bounding boxes. The spatial information of each bounding box includes the coordinates of its eight vertices and the vertical height of those eight vertices. Each bounding box is used to enclose at least one object. To further understand the aforementioned BVH tree, the following section will combine... Figure 5 To provide further details. Figure 5 A schematic diagram of a BVH tree provided in an embodiment of this application is shown below. Figure 5 As shown, suppose there are 6 objects in a 3D scene. Based on the spatial information of these 6 objects, the spatial information of bounding box A (enclosing these 6 objects) can be determined. Based on the spatial information of 4 of these objects, the spatial information of bounding box B (enclosing these 4 objects) can be determined. Based on the spatial information of the remaining 2 objects, the spatial information of bounding box C (enclosing these 2 objects) can be determined. The spatial information of bounding box A, bounding box B, and bounding box C are managed using a binary tree structure, resulting in a BVH tree. It can be seen that the BVH tree can centrally manage the spatial information of bounding boxes A, B, and C, which is equivalent to centrally managing bounding boxes A, B, and C. Bounding box A is the largest bounding box among all bounding boxes.

[0077] When implementing ray tracing based on a BVH tree, it's necessary to calculate whether the ray intersects with objects in the 3D scene (i.e., intersection calculation). Due to the existence of the BVH tree, a bounding box enclosing the objects can be determined based on the BVH tree. Then, it's determined whether the ray intersects with this bounding box. If the ray doesn't touch the bounding box, it means the ray will definitely not intersect with any object within that bounding box. If the ray does touch the bounding box, then it's calculated whether the ray intersects with any object within that bounding box. For example, if it's detected that the ray doesn't intersect with bounding box B in the binary tree, it means the ray will definitely not intersect with any of the four objects within bounding box B. Therefore, the step of detecting whether the ray intersects with the four objects in bounding box B can be omitted, and only the step of detecting whether the ray intersects with the two objects in bounding box C can be performed.

[0078] In addition, calculations can be performed based on the material information of each object to obtain the noise-free components of the material information of each object. The noise-free components of the material information of each object can then be processed into a queryable table to enable subsequent demodulation, noise reduction, and modulation processing of the object's image. This will not be elaborated on here.

[0079] (2) The camera's spatial information may include its vertical height and coordinates, etc., and the camera is used to capture images of a simulated 3D scene. It should be noted that when implementing ray tracing, the camera's spatial information can be used to determine its spatial position in the 3D scene. Therefore, the emission point of the light rays can be determined based on the camera's spatial position. That is, the camera emits light rays to various objects in the 3D scene to perform path calculations and intersection calculations, thereby determining the ray paths formed by these rays in the 3D scene, and thus enabling subsequent rendering operations for each object in the 3D scene.

[0080] (3) Spatial information of light sources is typically used to determine direct and indirect light sources in a 3D scene. Direct light sources are those that emit light themselves in a 3D scene, while indirect light sources are those that do not emit light themselves but rely on other light sources for illumination. For example, the aforementioned objects can all be considered indirect light sources. It should be noted that when implementing ray tracing, if a ray emitted by the camera intersects with at least one object in the 3D scene and is reflected on the object's surface, and is ultimately received by a direct light source, the ray path formed by such a ray is considered a valid ray path. Therefore, for some rays emitted by the camera, if these rays first intersect with an object in the 3D scene and are ultimately received by a direct light source (regardless of whether they intersect with other objects in between), then the valid ray path formed between this object and the direct light source can be used as the original ray path between this object and the direct light source (i.e., the ray path between this object and the direct light source has been initially sampled). A similar operation can be performed on the other objects in the 3D scene, thus obtaining the original ray paths between each object and the direct light source in the 3D scene. These original ray paths can be used to resample the ray paths between each object and the direct light source in the subsequent 3D scene, as well as to render the images of each object.

[0081] As can be seen, after obtaining the rendering information of each object in the 3D scene, the spatial information of the camera, and the spatial information of the light source, the original light path between each object and the direct light source (hereinafter referred to as the light source) can be obtained based on this information.

[0082] Since the operations performed on each object in this embodiment are similar, for ease of explanation, the following description will use one object as an illustration, and refer to this object as the target object. As described above, based on the rendering information of each object in the 3D scene, the spatial information of the camera, and the spatial information of the light source, the initial sampling of the light path between the target object and the direct light source can be completed, thereby obtaining the original light path between the target object and the light source. This original light path starts at the target object and ends at the light source, and may or may not pass through other objects in between.

[0083] It is worth noting that the original light path between the target object and the light source involved in this embodiment can also be understood as a set of original light paths between the target object and the light source. This is because there are usually multiple light rays emitted by the camera that first intersect with the target object, and these multiple light rays can be called multiple original light rays, that is, a set of original light rays. Therefore, the set of light paths formed by this set of original light rays between the target object and the light source is a set of original light paths between the target object and the light source.

[0084] 402. Based on the original light path between the target object and the light source, the light path between the target object and the light source is resampled to obtain a new light path. The new light path starts from the light source and ends at the target object.

[0085] After obtaining the original ray path between the target object and the light source, since the original ray path between the target object and the light source starts from the target object and ends at the light source, we can determine the two endpoints, the target object and the light source, and swap their properties to reconstruct the ray path between them in a certain way. That is, we perform a first resampling on the ray path between them to obtain the first ray path between the target object and the light source (also known as the indirect ray path). The first ray path starts from the light source, ends at the target object, and passes through other objects in between.

[0086] Specifically, since the first resampling can be performed in a single round or repeatedly in multiple rounds, the first ray path between the target object and the light source can be obtained in various ways:

[0087] (1) Based on a set of original ray paths between the target object and the light source, a first resampling is performed, which yields a set of first ray paths between the target object and the light source. Each of these original ray paths corresponds one-to-one with the multiple first ray paths contained within it. For any given original ray path, let it start at a surface point on the target object, pass through a surface point on another object, and end at a surface point on the light source. For ease of explanation, the surface point on the target object passed by the original ray path is referred to as the first surface point, the surface point on another object passed by the original ray path is referred to as the second surface point, and the surface point on the light source passed by the original ray path is referred to as the third surface point. Thus, the original ray path can be divided into two segments: the first segment is the ray path between the first and second surface points, and the second segment is the ray path between the second and third surface points. When performing the first resampling on the first path segment, a surface region can be drawn centered on the second surface point of the remaining objects. Within this surface region, a surface point is selected (e.g., randomly or according to a preset rule, without restriction), referred to as the fourth surface point (the fourth surface point can be the same as the second surface point or a surface point surrounding the second surface point, without restriction). The fourth surface point is then connected to the first surface point, forming the first new path segment pointing from the fourth surface point to the first surface point. Similarly, when performing the first resampling on the second path segment, a surface region can be drawn centered on the third surface point of the light source. Within this surface region, a surface point is selected, referred to as the fifth surface point (the fifth surface point can be the same as the third surface point or a surface point surrounding the third surface point, without restriction). The fifth surface point is then connected to the fourth surface point, forming the second new path segment pointing from the fifth surface point to the fourth surface point. In this way, the first new path and the second new path constitute the first ray path corresponding to the original ray path. This first ray path starts at the fifth surface point, passes through the fourth surface point, and ends at the first surface point. For the other original ray paths among the multiple original ray paths, the same operation as that performed on the original ray path can be performed on the other original ray paths. Therefore, multiple first ray paths corresponding one-to-one with the multiple original ray paths can be obtained, that is, a set of first ray paths between the target object and the light source.

[0088] It should be understood that the aforementioned example is merely illustrative, illustrating the scenario where the original ray path passes through one other object. It does not limit the number of other objects the original ray path can pass through. If the original ray path passes through multiple other objects, the first resampling operation on the original ray path is similar, and will not be elaborated upon here. Correspondingly, the first ray path obtained based on the original ray path that passes through multiple other objects can also pass through multiple other objects. To further understand the aforementioned first resampling process, the following section will combine... Figure 6 The example shown will be further explained. Figure 6 A schematic diagram of ray path resampling provided in an embodiment of this application is shown below. Figure 6 As shown, when performing ray tracing on a room, camera 100 emits a set of raw rays towards table 200. One of these raw rays is used for illustration. Let's assume that this raw ray, after originating from camera 100, first strikes surface point A on table 200, then reflects and strikes surface point B on wall 300, then reflects again and strikes surface point C on lampshade 400, and finally reflects again and strikes surface point D on light source 500. Therefore, the raw ray path between table 200 and light source 500 consists of three paths: the first path 601 points from surface point A to surface point B, the second path 602 points from surface point B to surface point C, and the third path 603 points from surface point C to surface point D. Then, we can take surface point E around surface point B on wall 300 and construct a new path 604 from surface point E to surface point A. Next, keeping surface point C on lampshade 400 unchanged, we construct a second new path 605 from surface point C to surface point E. Then, we take surface point F around surface point D on light source 500 and construct a third new path 606 from surface point F to surface point C. In this way, these three new paths constitute a first ray path. It can be understood that a similar operation can be performed on the original ray paths formed by the remaining original rays from camera 100 to table 200, thus obtaining multiple first ray paths between table 200 and light source 500, that is, a set of first ray paths between table 200 and light source 500.

[0089] (2) Repeat (1) multiple times to obtain multiple sets of first light paths between the target object and the light source. That is, after performing the first resampling based on a set of original light paths between the target object and the light source, and obtaining a set of first light paths between the target object and the light source, the first resampling can be performed again based on a set of original light paths between the target object and the light source to obtain another set of first light paths between the target object and the light source. This process is repeated. After performing the first resampling multiple times, multiple sets of first light paths between the target object and the light source can be obtained.

[0090] In addition, after obtaining the original light path between the target object and the light source, the light path between the two is reconstructed in another way based on the original light path between the two, that is, the light path between the two is resampled for the first time, so as to obtain the second light path (also known as the direct light path) between the target object and the light source. The second light path starts from the light source and ends at the target object, and does not pass through any other objects in between.

[0091] Specifically, since the second resampling can be performed in a single round or repeatedly in multiple rounds, the second ray path between the target object and the light source can be obtained in various ways:

[0092] (1) A second resampling is performed based on a set of original ray paths between the target object and the light source, resulting in a set of second ray paths between the target object and the light source. Each set of original ray paths contains multiple original ray paths, and each set of second ray paths contains multiple second ray paths. For any given original ray path, let it start at a certain surface point of the target object and end at a certain surface point of the light source. For ease of explanation, the surface point of the target object traversed by the original ray path is referred to as the first surface point, and the surface point of the light source traversed by the original ray path is referred to as the second surface point. Therefore, when performing a second resampling on the original ray path, a surface region can be delineated centered on the second surface point of the light source. Within this surface region, a surface point is selected (for example, randomly or according to a preset rule; no restriction is placed here), called the third surface point (the third surface point can be the same as the second surface point, or it can be one of the surface points surrounding the second surface point; no restriction is placed here). The third surface point is then connected to the first surface point, forming a second ray path pointing from the third surface point to the first surface point. This second ray path starts at the third surface point, does not pass through any other object surface points, and ends at the first surface point. For the other original ray paths besides this one, the same operation can be performed on them, thus obtaining multiple second ray paths corresponding one-to-one with the original ray paths—that is, a set of second ray paths between the target object and the light source.

[0093] (2) Repeat (1) multiple times to obtain multiple sets of second light paths between the target object and the light source. That is, after performing a second resampling based on a set of original light paths between the target object and the light source to obtain a set of second light paths between the target object and the light source, a second resampling can be performed again based on a set of original light paths between the target object and the light source to obtain another set of second light paths between the target object and the light source. This process is repeated. After performing multiple rounds of second resampling, multiple sets of second light paths between the target object and the light source can be obtained.

[0094] 403. Based on the new ray path between the target object and the light source, obtain a sample pool. The sample pool is used to indicate the rays that are directed toward the target object. The new ray path is formed by these rays.

[0095] Obtaining the first ray path between the target object and the light source is equivalent to obtaining the first ray (also known as an indirect ray) between the target object and the light source. The first ray is the ray that forms the first ray path. It can be regarded as the ray that is emitted from the light source, hits other objects in the middle, is reflected and hits the target object, and is then reflected again and received by the camera.

[0096] Information related to the first ray can then be acquired, and this information can be used to construct a first sample pool. For example, the intensity values ​​of the first ray after hitting other objects and before hitting the target object (e.g., the brightness and color of the first ray) can be acquired and stored in the first sample pool. In this way, the first sample pool can be used to indicate the first ray directed towards the target object.

[0097] Specifically, the first sample pool can be obtained in several ways:

[0098] (1) If only one set of first ray paths between the target object and the light source is obtained, since a first ray path is formed by a first ray, the set of first ray paths is formed by a set of first rays. Since the set of first rays contains multiple first rays, information related to these multiple first rays can be used to construct a first sample pool (for example, storing the light intensity values ​​of these multiple first rays after hitting other objects and before hitting the target object in a first sample pool). Therefore, the first sample pool can be used to indicate the set of first rays directed toward the target object. As in the example above, after obtaining a set of first ray paths between the target object and the light source, for the first ray paths formed by the first new path 604, the second new path 605, and the third new path 606, the light intensity values ​​of the first rays forming the first ray path after hitting the lampshade and the wall and before hitting the table can be obtained and stored in the first sample pool. Similarly, in the set of first ray paths between the table and the light source, in addition to the first ray path, the light intensity values ​​of the other first rays forming the other first ray paths after hitting other objects and before hitting the table can be obtained and stored in the first sample pool. In this way, a complete first sample pool can be obtained, which is used to indicate a set of first rays of light directed toward the table.

[0099] Furthermore, the phenomenon produced by the first ray after hitting the target object can be controlled to be one of the following: diffuse reflection, specular reflection, or specular reflection. Correspondingly, the first sample pool corresponds to a certain rendering effect, which can be presented as one of the following: diffuse reflection, specular reflection, or specular reflection. For example, if the phenomenon produced by the first ray after hitting the target object is diffuse reflection, the rendering effect corresponding to the first sample pool will be diffuse reflection, and so on.

[0100] (2) If multiple sets of first ray paths are obtained between the target object and the light source, these multiple sets of first ray paths are formed by multiple sets of first rays (the two correspond one-to-one). For any set of first rays, since the set of first rays contains multiple first rays, information related to these multiple first rays can be used to construct a first sample pool (for example, storing the light intensity values ​​of these multiple first rays after hitting other objects and before hitting the target object in a first sample pool). Similarly, for the other sets of first rays, the same operation as that performed on the set of first rays can be performed, so multiple first sample pools can be obtained in the end, and one first sample pool is used to indicate a set of first rays directed toward the target object.

[0101] Furthermore, for any one of the multiple sets of first rays, the phenomenon produced by that set of first rays after hitting the target object can be controlled to be one of the following: diffuse reflection, specular reflection, or specular reflection. It should be noted that the phenomena produced by multiple sets of first rays after hitting the target object can be the same or different; this is not limited here. Correspondingly, for any one of the multiple first sample pools, this first sample pool corresponds to a certain rendering effect, which can be presented as one of the following: diffuse reflection, specular reflection, or specular reflection. It should be noted that the rendering effects corresponding to multiple first sample pools can be the same or different; this is not limited here. For example, a typical sample pool configuration can be: constructing two first sample pools, where the rendering effect corresponding to the first first sample pool is specular reflection, and the rendering effect corresponding to the second first sample pool is specular reflection.

[0102] Furthermore, in both case (1) and case (2), for any first sample pool (which can also be understood as a first sample image), the first sample pool may contain a set of light intensity values ​​of the first rays after hitting the other objects and before hitting the target object. Then, the size of the first sample pool (the resolution of the first sample image) is the number of these light intensity values, which is the number of the first rays (that is, the size of the first sample pool (the resolution of the first sample image) is the number of first rays indicated by the first sample pool). It is worth noting that the size of the first sample pool is smaller than the resolution of the screen (i.e., the number of pixels displayed on the screen), which is used to display the final image of the target object.

[0103] Furthermore, obtaining the second ray path between the target object and the light source is equivalent to obtaining the second ray (also known as the direct ray) between the target object and the light source. The second ray is the ray that forms the second ray path. It can be regarded as the ray that is emitted from the light source, does not hit other objects in the middle, directly hits the target object, and is then reflected and received by the camera.

[0104] Information related to the second ray can then be acquired, and this information can be used to construct a second sample pool. For example, the intensity value of the second ray before it hits the target object (e.g., the brightness and color of the second ray) can be acquired and stored in the second sample pool. In this way, the second sample pool can be used to indicate the second ray directed towards the target object.

[0105] Specifically, a second sample pool can be obtained in several ways:

[0106] (1) If only one set of second ray paths between the target object and the light source is obtained, since one second ray path is formed by one second ray, the set of second ray paths is formed by a set of second rays. Since the set of second rays contains multiple second rays, information related to these multiple second rays can be used to construct a second sample pool (for example, storing the light intensity values ​​of these multiple second rays before hitting the target object in a second sample pool). Therefore, the second sample pool can be used to indicate the set of second rays directed toward the target object.

[0107] (2) If multiple sets of second ray paths are obtained between the target object and the light source, these multiple sets of second ray paths are formed by multiple sets of second rays (the two correspond one-to-one). For any set of second rays in the multiple sets of second rays, since the set of second rays contains multiple second rays, a second sample pool can be constructed using the information related to these multiple second rays (for example, storing the light intensity values ​​of these multiple second rays before hitting the target object in a second sample pool). Similarly, for the remaining sets of second rays, the same operation can be performed as for the set of second rays, so that multiple second sample pools can be obtained in the end, and one second sample pool is used to indicate a set of second rays that are directed toward the target object.

[0108] Furthermore, in both case (1) and case (2), for any second sample pool (which can also be understood as a second sample image), the second sample pool may contain a set of light intensity values ​​of the second rays before hitting the target object. Then, the size of the second sample pool (the resolution of the second sample image) is the number of these light intensity values, which is the number of the second rays in the set (that is, the size of the second sample pool (the resolution of the second sample image) is the number of second rays indicated by the second sample pool). It is worth noting that the size of the second sample pool is equal to the resolution of the screen (i.e., the number of pixels displayed on the screen), which is used to display the final image of the target object.

[0109] 404. Based on the sample pool, render the target object to obtain an image of the target object.

[0110] After obtaining the first sample pool, the target object can be rendered using the first sample pool to obtain the image of the target object. The image of the target object is then the image to be displayed on the screen for the user to view and use.

[0111] Specifically, images of the target object can be obtained in the following ways:

[0112] (1) Render the target object based on the first sample pool to obtain the indirect lighting image of the target object. It should be noted that since at least one first sample pool can be obtained (for example, only one first sample pool is obtained in the aforementioned case (1), or multiple first sample pools are obtained in the aforementioned case (2), at least one first sample pool can be shading (e.g., lighting operation, etc.) to obtain at least one indirect lighting image of the target object (for example, corresponding to the aforementioned case (1), one indirect lighting image of the target object can be obtained, or multiple indirect lighting images of the target object can be obtained corresponding to the aforementioned case (2).

[0113] (2) Render the target object based on the second sample pool to obtain the direct lighting image of the target object. It should be noted that since at least one second sample pool can be obtained (for example, only one second sample pool is obtained in the aforementioned case (1), or multiple second sample pools are obtained in the aforementioned case (2), at least one second sample pool can be shading (e.g., lighting operations, etc.) to obtain at least one direct lighting image of the target object (for example, one direct lighting image of the target object can be obtained corresponding to the aforementioned case (1), or multiple direct lighting images of the target object can be obtained corresponding to the aforementioned case (2).

[0114] (3) The indirect lighting image and the direct lighting image are fused to obtain the image of the target object. It should be noted that after obtaining at least one indirect lighting image and at least one direct lighting image of the target object, these multiple images can be regarded as multiple layers. Therefore, at least one indirect lighting image and at least one direct lighting image of the target object can be superimposed to obtain the final image of the target object that can be displayed.

[0115] Furthermore, in order to obtain higher quality images of the target object, the layers used to synthesize the target object image can be processed in advance:

[0116] (1) Obtain the noise-free component of the material information of the target object. It should be noted that before processing each layer, the noise-free component of the material information of the target object can be obtained from the aforementioned table.

[0117] (2) Based on the noise-free component of the target object's material information, a first processing step is performed on the indirect lighting image of the target object to obtain a processed indirect lighting image. The first processing step includes demodulation, noise reduction, and modulation. It should be noted that after obtaining the noise-free component of the target object's material information, at least one indirect lighting image of the target object can be divided by the noise-free component to obtain at least one demodulated indirect lighting image of the target object. After obtaining at least one demodulated indirect lighting image of the target object, noise reduction can be performed on the demodulated indirect lighting image of the target object to obtain at least one denoised indirect lighting image of the target object. After obtaining at least one denoised indirect lighting image of the target object, the at least one denoised indirect lighting image of the target object can be multiplied by the noise-free component to obtain at least one modulated indirect lighting image of the target object.

[0118] Since the size of at least one first sample pool is smaller than the screen resolution, the resolution of at least one modulated indirect lighting image of the target object is also smaller than the screen resolution. To achieve better fusion of the layers, after obtaining at least one modulated indirect lighting image of the target object, super-resolution processing can be applied to it to obtain at least one processed indirect lighting image of the target object. In this case, the resolution of the processed at least one indirect lighting image of the target object is equal to the screen resolution.

[0119] (3) A second processing step is performed on the direct illumination image of the target object based on the noise-free component of the target object's material information to obtain a processed direct illumination image. The second processing includes demodulation, noise reduction, and modulation. It should be noted that after obtaining the noise-free component of the target object's material information, at least one direct illumination image of the target object can be divided by the noise-free component to obtain at least one demodulated direct illumination image of the target object. After obtaining at least one demodulated direct illumination image of the target object, noise reduction can be performed on the demodulated direct illumination image of the target object to obtain at least one denoised direct illumination image of the target object. After obtaining at least one denoised direct illumination image of the target object, the at least one denoised direct illumination image of the target object can be multiplied by the noise-free component to obtain at least one modulated direct illumination image of the target object, i.e., at least one processed direct illumination image of the target object.

[0120] (4) The processed indirect lighting image and the processed direct lighting image are fused to obtain the image of the target object. It should be noted that after obtaining at least one processed indirect lighting image and at least one processed direct lighting image of the target object, the processed at least one indirect lighting image and at least one processed direct lighting image of the target object can be superimposed to obtain the final image of the target object that can be displayed.

[0121] It should be noted that in this embodiment, the indirect illumination image is the aforementioned first illumination image, the direct illumination image is the aforementioned second illumination image, the processed indirect illumination image is the aforementioned processed first illumination image, and the processed direct illumination image is the aforementioned processed second illumination image.

[0122] To further understand the image rendering method provided in the embodiments of this application, a specific application example is provided below to further introduce the method. Figure 7 This is a schematic diagram illustrating an application example of the image rendering method provided in the embodiments of this application, such as... Figure 7 As shown, this application example is mainly used to showcase a virtual auto show to users. The application example includes:

[0123] (1) Resample the light paths between objects (e.g., vehicles, floors, etc.) and light sources in the auto show to obtain indirect light paths and direct light paths.

[0124] (2) A first-class sample pool is constructed based on direct ray paths. The first-class sample pool contains one sample pool, and the rendering effect corresponding to this sample pool is direct lighting. The size of this sample pool is the same as the screen size (resolution). For example, if the screen resolution is 1920*1080, then the size of this sample pool is 1920*1080. This screen is used to display images of a virtual auto show. A second-class sample pool is constructed based on indirect ray paths. The second-class sample pool contains n sample pools. The rendering effect corresponding to the first sample pool is specular reflection, the rendering effect corresponding to the second sample pool is specular reflection, the rendering effect corresponding to the third sample pool is diffuse reflection, ..., and the rendering effect corresponding to the nth sample pool is specular reflection. The size of each of these n sample pools is 1 / 4 of the screen size, that is, the size of each sample pool is 480*270.

[0125] (3) Rendering is performed based on the first type of sample pool to obtain a direct lighting image, which is used to present objects that are directly visible to the camera.

[0126] (4) Rendering is performed based on the second type of sample to obtain an indirect lighting image, which is used to further present effects such as specular reflection and specular reflection on the basis of visible objects.

[0127] (5) The image obtained in step (3) is noisy and needs to be denoised. The denoising process mainly includes three steps, such as... Figure 8 As shown ( Figure 8 (A schematic diagram of the noise reduction process provided in the embodiments of this application): a. Calculate the noise-free component of the material information of the objects in the auto show; b. Divide the image obtained in step (3) by the noise-free component; c. Reduce the noise of the result after division, and then multiply the result after noise reduction by the noise-free component to obtain the direct illumination image after noise removal.

[0128] (6) The image obtained in step (4) is also noisy. It can be processed according to the noise reduction process in step (5) to obtain the indirect illumination image after noise removal.

[0129] (7) The resolution of the direct lighting image after noise removal obtained in step (5) is the same as that of the screen. The resolution of the indirect lighting image after noise removal obtained in step (6) is 1 / 4 of that of the screen. Therefore, the indirect lighting image after noise removal can be super-resolution processed. Finally, the super-resolution indirect lighting image and the direct lighting image after noise removal are superimposed to obtain the image of the virtual auto show and displayed on the screen.

[0130] Furthermore, the image rendering method provided in this application embodiment can be compared with the image rendering methods provided by related technologies, and the comparison results are as follows: Figures 9 to 14 As shown ( Figure 9 A schematic diagram illustrating the comparison results provided in an embodiment of this application. Figure 10 Another schematic diagram illustrating the comparison results provided in the embodiments of this application. Figure 11 Another schematic diagram illustrating the comparison results provided in the embodiments of this application. Figure 12 Another schematic diagram illustrating the comparison results provided in the embodiments of this application. Figure 13 Another schematic diagram illustrating the comparison results provided in the embodiments of this application. Figure 14 (Another schematic diagram of the comparison results provided in the embodiments of this application). The test scenario includes a vehicle, a ground, and light sources in the environment.

[0131] based on Figure 9 It can be seen that in the images rendered by the relevant technology, the vehicle body does not have a reflection of the ground, based on Figure 10 As can be seen, in the image rendered by the multi-pooling technique in this application embodiment, the vehicle body presents a clear reflection of the ground.

[0132] based on Figure 11 It is known that after the relevant rendering technology denoises the image, the details of the car seat are lost. Figure 12 As can be seen, after the image is denoised in this embodiment of the application, the details of the car seat can be well preserved.

[0133] based on Figure 13 It is known that after super-resolution processing of the image using related techniques, the resulting image exhibits significant aliasing. Figure 14 As can be seen, the embodiments of this application can significantly reduce distortion by differentiating the effects of different lighting (direct lighting and indirect lighting), and the resulting images are smooth and clear.

[0134] In this embodiment, after obtaining the original ray path between the target object and the light source, the ray path can be resampled based on the original ray path to obtain a new ray path. The original ray path is obtained through ray tracing, with the target object as the starting point and the light source as the ending point. The new ray path also starts from the light source and ends at the target object. Then, a sample pool can be obtained based on the new ray path. The sample pool is used to indicate the rays directed towards the target object, and the new ray path is formed by these rays. Finally, the target object can be rendered based on the sample pool to obtain an image of the target object. Based on the foregoing process, this application provides a new ray path resampling method. This method reverses the direction of the effective ray path in ray tracing (starting from the light source and ending at the target object) and selectively resamples the ray path, which can improve the success rate of sampling effective new ray paths. That is, it can easily resample the effective new ray path between the target object and the light source. In this way, regardless of whether the phenomenon produced by the light that forms the new ray path hitting the target object is direct lighting, diffuse reflection, specular reflection, or specular reflection, a high-quality and sufficient sample pool can be successfully obtained, thereby making the rendering effect of the target object sufficiently high-quality (i.e., the rendered image of the target object has sufficiently high quality).

[0135] Furthermore, this application provides a ray tracing technique based on multiple sample pools, where multiple sample pools refer to at least one first sample pool and at least one second sample pool. In a common configuration, two first sample pools and one second sample pool can be constructed. The rendering effects corresponding to the two first sample pools are specular reflection and specular reflection, respectively, while the rendering effect corresponding to the second sample pool is direct lighting. Therefore, even if the target object is an object with a special material, the image of the target object rendered based on multiple sample pools can present both direct lighting effects and indirect lighting effects such as specular reflection and specular reflection, resulting in a more realistic and exquisite image, which is beneficial to improving the user experience.

[0136] Furthermore, in this embodiment, the size of the first sample pool is smaller than the screen resolution, which can improve the efficiency of obtaining the first sample pool. In the process of rendering the target object based on the multi-sample pool technology, it can be combined with super-resolution processing to reduce the amount of computation required in the process to a certain extent, which is conducive to accelerating the implementation of ray tracing technology based on the multi-sample pool and further improving the user experience.

[0137] Furthermore, this application provides a novel image denoising method that can demodulate, denoise, and modulate each layer based on noise-free classification of the target object's material information. This allows each layer to retain the details of the target object as much as possible, making the image of the target object obtained by superimposing the layers more realistic and clear, which is beneficial to improving the user's viewing experience.

[0138] The above is a detailed description of the image rendering method provided in the embodiments of this application. The image rendering apparatus provided in the embodiments of this application will be described below. Figure 15 A schematic diagram of the structure of the image rendering apparatus provided in the embodiments of this application is shown below. Figure 15 As shown, the device includes:

[0139] The resampling module 1501 is used to resample the light path between the target object and the light source based on the original light path between the target object and the light source to obtain a new light path. The original light path is obtained by ray tracing. The original light path starts from the target object and ends at the light source. The new light path starts from the light source and ends at the target object.

[0140] The first acquisition module 1502 is used to acquire a sample pool based on the new ray path. The sample pool is used to indicate the rays that are directed toward the target object. The new ray path is formed by the rays.

[0141] Rendering module 1503 is used to render the target object based on the sample pool to obtain an image of the target object.

[0142] In this embodiment, after obtaining the original ray path between the target object and the light source, the ray path can be resampled based on the original ray path to obtain a new ray path. The original ray path is obtained through ray tracing, with the target object as the starting point and the light source as the ending point. The new ray path also starts from the light source and ends at the target object. Then, a sample pool can be obtained based on the new ray path. The sample pool is used to indicate the rays directed towards the target object, and the new ray path is formed by these rays. Finally, the target object can be rendered based on the sample pool to obtain an image of the target object. Based on the foregoing process, this application provides a new ray path resampling method. This method reverses the direction of the effective ray path in ray tracing (starting from the light source and ending at the target object) and selectively resamples the ray path, which can improve the success rate of sampling effective new ray paths. That is, it can easily resample the effective new ray path between the target object and the light source. In this way, regardless of whether the phenomenon produced by the light that forms the new ray path hitting the target object is direct lighting, diffuse reflection, specular reflection, or specular reflection, a high-quality and sufficient sample pool can be successfully obtained, thereby making the rendering effect of the target object sufficiently high-quality.

[0143] In one possible implementation, the new ray path includes a first ray path and a second ray path. The first ray path starts from the light source, ends at the target object, and passes through other objects. The second ray path starts from the light source, ends at the target object, and does not pass through other objects. The sample pool includes a first sample pool and a second sample pool. The first sample pool is used to indicate the first ray directed toward the target object, and the first ray path is formed by the first ray. The second sample pool is used to indicate the second ray directed toward the target object, and the second ray path is formed by the second ray.

[0144] In one possible implementation, the phenomenon produced when the first ray strikes the target object is specular reflection or specular reflection.

[0145] In one possible implementation, the number of first rays indicated by the first sample pool is less than the number of pixels displayed on the screen, and the number of second rays indicated by the second sample pool is equal to the number of pixels displayed on the screen, which is used to display an image of the target object.

[0146] In one possible implementation, the rendering module 1503 is configured to: render the target object based on a first sample pool to obtain a first lighting image of the target object; render the target object based on a second sample pool to obtain a second lighting image of the target object; and fuse the first lighting image and the second lighting image to obtain an image of the target object.

[0147] In one possible implementation, the rendering module 1503 is further configured to: acquire the noise-free component of the material information of the target object; perform a first processing on the first illumination image based on the noise-free component to obtain a processed first illumination image, the first processing including demodulation processing, noise reduction processing, and modulation processing; perform a second processing on the second illumination image based on the noise-free component to obtain a processed second illumination image, the second processing including demodulation processing, noise reduction processing, and modulation processing; and the rendering module 1503 is configured to fuse the processed first illumination image and the processed second illumination image to obtain an image of the target object.

[0148] In one possible implementation, the first processing further includes super-resolution processing, which is used to make the resolution of the processed first illumination image equal to the resolution of the screen.

[0149] It should be noted that the information interaction and execution process between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of this application, and the resulting technical effects are the same as those of the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in the embodiment of this application, and it will not be repeated here.

[0150] This application also relates to a circuit system including a processing circuit configured to perform the aforementioned actions. Figure 4 The steps in the illustrated embodiment.

[0151] This application also relates to a chip system, which includes a processor for calling computer programs or computer instructions stored in memory, so that the processor performs the aforementioned operations. Figure 4 The steps in the illustrated embodiment.

[0152] In one possible implementation, the processor is coupled to the memory via an interface.

[0153] In one possible implementation, the chip system also includes a memory that stores computer programs or computer instructions.

[0154] This application also relates to a computer storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform the aforementioned actions. Figure 4 The steps in the illustrated embodiment.

[0155] This application also relates to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform the aforementioned actions. Figure 4 The steps in the illustrated embodiment.

[0156] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0158] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0159] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0160] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An image rendering method, characterized in that, The method includes: Based on the original ray path between the target object and the light source, the ray path between the target object and the light source is resampled to obtain a new ray path. The original ray path is obtained through ray tracing. The original ray path starts from the target object and ends at the light source. The new ray path includes a first ray path and a second ray path. The first ray path starts from the light source and ends at the target object, and passes through other objects. The second ray path starts from the light source and ends at the target object, and does not pass through other objects. Based on the new ray path, a sample pool is obtained, which includes a first sample pool and a second sample pool. The first sample pool is used to indicate a first ray of light directed toward the target object, and the first ray path is formed by the first ray. The second sample pool is used to indicate a second ray of light directed toward the target object, and the second ray path is formed by the second ray. Based on the sample pool, the target object is rendered to obtain an image of the target object.

2. The method according to claim 1, characterized in that, The phenomenon produced when the first ray strikes the target object includes at least one of the following: specular reflection or specular reflection.

3. The method according to claim 1 or 2, characterized in that, The number of first rays indicated by the first sample pool is less than the number of pixels displayed on the screen, and the number of second rays indicated by the second sample pool is equal to the number of pixels displayed on the screen, which is used to display an image of the target object.

4. The method according to claim 1 or 2, characterized in that, The step of rendering the target object based on the sample pool to obtain an image of the target object includes: The target object is rendered based on the first sample pool to obtain the first lighting image of the target object; The target object is rendered based on the second sample pool to obtain a second lighting image of the target object; The first illumination image and the second illumination image are fused together to obtain an image of the target object.

5. The method according to claim 4, characterized in that, Before fusing the first illumination image and the second illumination image to obtain the image of the target object, the method further includes: Obtain the noise-free component of the material information of the target object; The first illumination image is processed based on the noise-free component to obtain the processed first illumination image. The first processing includes demodulation processing, noise reduction processing, and modulation processing. The second illumination image is processed based on the noise-free component to obtain the processed second illumination image. The second processing includes demodulation processing, noise reduction processing, and modulation processing. The process of fusing the first illumination image and the second illumination image to obtain an image of the target object includes: The processed first illumination image and the processed second illumination image are fused to obtain an image of the target object.

6. The method according to claim 5, characterized in that, The first process further includes super-resolution processing, which is used to make the resolution of the processed first illumination image equal to the resolution of the screen, the screen being used to display the image of the target object.

7. An image rendering apparatus, characterized in that, The device includes: The resampling module is used to resample the ray path between the target object and the light source based on the original ray path between the target object and the light source to obtain a new ray path. The original ray path is obtained by ray tracing, and the original ray path starts from the target object and ends at the light source. The new ray path includes a first ray path and a second ray path. The first ray path starts from the light source and ends at the target object, and passes through other objects. The second ray path starts from the light source and ends at the target object, and does not pass through other objects. The first acquisition module is used to acquire a sample pool based on the new ray path. The sample pool includes a first sample pool and a second sample pool. The first sample pool is used to indicate a first ray of light directed toward the target object. The first ray path is formed by the first ray. The second sample pool is used to indicate a second ray of light directed toward the target object. The second ray path is formed by the second ray. The rendering module is used to render the target object based on the sample pool to obtain an image of the target object.

8. The apparatus according to claim 7, characterized in that, The phenomenon produced when the first ray of light strikes the target object is specular reflection or specular reflection.

9. The apparatus according to claim 7 or 8, characterized in that, The number of first rays indicated by the first sample pool is less than the number of pixels displayed on the screen, and the number of second rays indicated by the second sample pool is equal to the number of pixels displayed on the screen, which is used to display an image of the target object.

10. The apparatus according to claim 7 or 8, characterized in that, The rendering module is used for: The target object is rendered based on the first sample pool to obtain the first lighting image of the target object; The target object is rendered based on the second sample pool to obtain a second lighting image of the target object; The first illumination image and the second illumination image are fused together to obtain an image of the target object.

11. The apparatus according to claim 10, characterized in that, The rendering module is also used for: Obtain the noise-free component of the material information of the target object; The first illumination image is processed based on the noise-free component to obtain the processed first illumination image. The first processing includes demodulation processing, noise reduction processing, and modulation processing. The second illumination image is processed based on the noise-free component to obtain the processed second illumination image. The second processing includes demodulation processing, noise reduction processing, and modulation processing. The rendering module is used to fuse the processed first lighting image and the processed second lighting image to obtain an image of the target object.

12. The apparatus according to claim 11, characterized in that, The first process further includes super-resolution processing, which is used to make the resolution of the processed first illumination image equal to the resolution of the screen, the screen being used to display the image of the target object.

13. An electronic device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the electronic device performs the method as described in any one of claims 1 to 6.

14. A circuit system, characterized in that, The circuit system includes processing circuitry configured to perform the method as described in any one of claims 1 to 6.

15. A chip system, characterized in that, The chip system includes a processor for calling a computer program or computer instructions stored in a memory to cause the processor to perform the method as described in any one of claims 1 to 6.

16. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 6.

17. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 6.

Images