Image rendering method and electronic equipment

Abstract

The invention provides an image rendering method and electronic equipment, relates to the technical field of image processing, and can solve the problems of large calculation amount, serious calculation power consumption and low rendering efficiency in a ray tracing process. The method comprises the following steps: for each pixel in a to-be-rendered image, calculating rendering information of the pixel, and determining light data of the pixel based on the rendering information of the pixel; different light data can be self-adapted by obtaining the rendering information required when the image quality effects of different pixels are obtained.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image rendering method and an electronic device. Background Art

[0002] In graphics rendering related applications, such as games, virtual simulation, augmented reality (AR) / virtual reality (VR), etc., improving the realism of rendered images has always been the main goal of the industry. Ray tracing technology has attracted much attention because it is based on real physical laws and can generate rendering results that are close to the real world.

[0003] When using ray tracing technology to render images, a large number of rays need to be constructed and emitted into the three-dimensional scene. In the process of reflection / refraction of the rays with virtual objects in the three-dimensional scene, a large number of new rays are generated, making the number of rays extremely large. In addition, using ray tracing technology to render images requires not only calculating the intersection points of all rays with virtual objects (i.e., ray intersection), but also coloring each pixel in the image, which consumes a lot of computing power. Therefore, how to reduce the amount of calculations in the ray tracing process, reduce computing power consumption, and improve rendering efficiency is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present application provide an image rendering method and an electronic device for solving the problems of large amount of calculation, severe consumption of computing power, and low rendering efficiency in a ray tracing process.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an image rendering method is provided, the method comprising: an electronic device obtains an image to be rendered, the image to be rendered includes pixels; for each pixel included in the image to be rendered, the electronic device calculates rendering information of the pixel in the image to be rendered, the rendering information of the pixel refers to the rendering information required to obtain the target image quality effect of the pixel. Then, the electronic device determines the light data of the pixel according to the rendering information of the pixel, wherein the light data corresponding to the rendering information of different pixels is different. Furthermore, the electronic device renders the pixels included in the image to be rendered according to the light data of the pixel.

[0007] Based on the first aspect, the solution of the present application is adopted to calculate the rendering information of the pixel for each pixel in the image to be rendered, and determine the light data of the pixel based on the rendering information of the pixel. Since different rendering information corresponds to different light data, the rendering information required for obtaining different pixel image quality effects is adaptively adapted to different light data to optimize the computing performance of ray tracing.

[0008] For example, for pixels that require higher rendering information to achieve the target image quality, more light data is used for ray tracing; and for pixels that require lower rendering information to achieve the target image quality, less light data is used for rendering. By reducing the light data, the problems of large amount of calculation, serious computing power consumption, and low rendering efficiency in the ray tracing process can be solved.

[0009] In a possible implementation of the first aspect, the rendering information of the pixel includes a first coefficient, which is used to indicate the light emission density required for the pixel to obtain a target image quality effect; wherein, the electronic device calculates the rendering information of the pixels included in the image to be rendered, which may include the following content.

[0010] The electronic device calculates the first normal coordinate and the first world coordinate of the pixel; the electronic device samples m first pixels in the first preset pixel area, and calculates the second normal coordinate and the second world coordinate of each of the m first pixels. For each first pixel, the pixel distance between the pixel and the first pixel is calculated according to the first world coordinate and the second world coordinate to obtain m first pixel distances, and the normal angle between the pixel and the first pixel is calculated according to the first normal coordinate and the second normal coordinate to obtain m first normal angles; m is a positive integer. Then, the electronic device calculates the first coefficient according to the m first pixel distances and the m first normal angles.

[0011] In this way, the pixel distance and the normal angle are calculated using the normal coordinates and the world coordinates, and the first coefficient is calculated using the pixel distance and the normal angle, so that the reliability of the calculated first coefficient can be ensured.

[0012] In a possible implementation manner of the first aspect, the electronic device may calculate the first coefficient according to the m first pixel distances and the m first normal angles, including the following content.

[0013] The electronic device calculates a first probability of light oscillation occurring in a pixel based on m first pixel distances and m first normal angles; the electronic device samples n second pixels within a second preset pixel area, and respectively calculates the probability of light oscillation occurring in each of the n second pixels to obtain n second probabilities; n is a positive integer; then, the electronic device smoothes the first probability and the n second probabilities to obtain a first coefficient.

[0014] In this implementation, the electronic device calculates the first probability of light oscillation occurring in a pixel and n second probabilities of light oscillation occurring in n second pixels within a second preset pixel area, and smoothes the first probability and the n second probabilities, that is, smoothes the pixel and the second pixels around the pixel to reduce noise, thereby making the calculated first coefficient more accurate.

[0015] In a possible implementation of the first aspect, the electronic device may calculate the first world coordinates of the pixel, including the following contents: the electronic device obtains a first depth value of the pixel and a first vertex coordinate of the pixel in a local coordinate system; the electronic device performs a matrix transformation on the first vertex coordinate using a space matrix and a projection matrix according to the first depth value to obtain the first world coordinates of the pixel.

[0016] In this way, the electronic device uses the space matrix and the projection matrix to perform matrix transformation on the first vertex coordinates according to the first depth value to obtain the first world coordinates of the pixel, which can reduce the load of the electronic device in calculating the first world coordinates.

[0017] In a possible implementation of the first aspect, the electronic device calculating the first normal coordinate of a pixel may include the following contents. The electronic device obtains multiple virtual objects in a three-dimensional scene corresponding to the image to be rendered; for each virtual object in the multiple virtual objects, the electronic device calculates the normal coordinate of each vertex in the virtual object in a vertex shader; the electronic device obtains the normal coordinate of each pixel corresponding to the virtual object by interpolating the normal coordinate of the vertex through a rasterization interpolation algorithm, and saves the normal coordinate of each pixel corresponding to the virtual object in a pixel shader; and then, the electronic device obtains the first normal coordinate of the pixel from the pixel shader.

[0018] In this way, the electronic device calculates the normal coordinates of each vertex in the virtual object in the vertex shader, and obtains the normal coordinates of each pixel corresponding to the virtual object by interpolating the normal coordinates of the vertex through a rasterization interpolation algorithm; compared with the related technology of transmitting all pixels corresponding to the virtual objects to the pixel shader, and the pixel shader calculates the normal coordinates of each pixel, the above-mentioned scheme of the present application can optimize the hardware memory access of the electronic device.

[0019] In a possible implementation of the first aspect, the light data of the pixel includes a first light quantity, where the first light quantity is the number of light rays passing through the pixel; and the electronic device determining the light data of the pixel according to the rendering information of the pixel may include the following content. The electronic device queries the first light quantity in a stored first light lookup table according to the first coefficient, and the first light lookup table records the correspondence between the first coefficient and the first light quantity.

[0020] Based on this implementation, the efficiency of the electronic device in determining the first light quantity based on the first coefficient can be improved.

[0021] In a possible implementation of the first aspect, the rendering information of the pixel also includes a first distance, and the first distance is used to indicate the distance between the pixel and the virtual camera; the electronic device calculating the rendering information of the pixel may include the following content. The electronic device obtains the first camera coordinates of the pixel included in the image to be rendered, that is, the electronic device obtains the coordinates of the pixel in the coordinate system established based on the virtual camera; then, the electronic device calculates the first distance according to the first camera coordinates.

[0022] In a possible implementation of the first aspect, the light data of the pixel includes a first iteration number, and the first iteration number is used to indicate the number of times each light ray passing through the pixel intersects with the virtual object in the three-dimensional scene; the electronic device determines the light data of the pixel based on the rendering information of the pixel, which may include the following content.

[0023] The electronic device searches for the first iteration number in a stored second light lookup table according to the first distance, and the second lookup table records the corresponding relationship between the first distance and the first iteration number.

[0024] In this way, the efficiency of the electronic device in determining the first iteration number can be improved.

[0025] In a possible implementation manner of the first aspect, the electronic device rendering pixels included in the to-be-rendered image according to light data of the pixels may include the following content.

[0026] The electronic device obtains acceleration structure and material information; the electronic device calculates multiple intersection coordinates after the pixel light intersects with the virtual object in the three-dimensional scene based on the acceleration structure and material information; the electronic device calculates the light energy at each of the multiple intersection coordinates based on the pixel light data; the electronic device takes the weighted sum of the light energy at all intersections as the color value of the pixel, and renders the pixel included in the image to be rendered based on the color value of the pixel.

[0027] The acceleration structure is used to accelerate the determination of multiple intersection coordinates, and the material information includes one or more of roughness, basic color, refractive index, and reflectivity.

[0028] It can be seen that the present application uses a ray tracing algorithm to render the pixels included in the image to be rendered, which can improve the image rendering effect while optimizing the computing performance of ray tracing.

[0029] In a second aspect, an electronic device is provided, which has the function of implementing any one of the methods in the first aspect, and the function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0030] According to a third aspect, an electronic device is provided, comprising: a memory and one or more processors, wherein the memory is used to store computer-executable instructions, and when the electronic device is running, the processor executes the computer-executable instructions stored in the memory so that the electronic device executes any one of the methods described in the first aspect.

[0031] In a fourth aspect, a chip system is provided, which is applied to an electronic device. The chip system includes a processor and an interface, wherein the interface is used to receive instructions and transmit them to at least one processor; at least one processor executes the instructions so that the electronic device executes any one of the methods described in the first aspect.

[0032] In a fifth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is executed on a computer, the computer can execute any of the methods described in the first aspect.

[0033] In a sixth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the methods described in the first aspect.

[0034] Among them, the technical effects brought about by any implementation method in the above-mentioned second to sixth aspects can refer to the technical effects brought about by different implementation methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0036] Figure 2 A schematic diagram of the principle of ray tracing provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of another principle of ray tracing provided in an embodiment of the present application;

[0038] Figure 4 A schematic diagram of the principle of rasterization processing provided in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of an image rendering process provided by an embodiment of the present application;

[0040] Figure 6 A schematic diagram of a BVH provided in an embodiment of the present application;

[0041] Figure 7 A schematic diagram of an image rendering provided in an embodiment of the present application;

[0042] Figure 8A schematic diagram of an image rendering process provided by an embodiment of the present application;

[0043] Fig. 9 A schematic diagram of a flow chart for calculating a first coefficient provided in an embodiment of the present application;

[0044] Fig.10 A diagram showing an example of the principle of frustum clipping provided in an embodiment of the present application;

[0045] Fig.11 A schematic diagram of a principle for calculating a first coefficient provided in an embodiment of the present application;

[0046] Fig.12 A schematic diagram of another principle for calculating the first coefficient provided in an embodiment of the present application;

[0047] Fig.13 A schematic diagram of a ray tracing process provided in an embodiment of the present application;

[0048] Fig.14 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or. For example, A / B can mean A or B. The “and / or” in this article is merely a way to describe the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0050] The following terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more.

[0051] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0052] In addition, in the embodiments of the present application, directional terms such as "up", "down", "left" and "right" may be defined including but not limited to the orientation relative to the schematic placement of the components in the drawings. It should be understood that these directional terms may be relative concepts, which are used for relative description and clarification, and may change accordingly according to changes in the orientation of the components in the drawings.

[0053] The image rendering method provided in the embodiment of the present application can be applied to electronic devices. For example, the electronic device can be a mobile phone, a tablet computer, a personal digital assistant (PDA), a personal computer (PC), an ultra-mobile personal computer (UMPC), a smart home device (e.g., a smart TV, a smart screen, a large screen, etc.), a wearable device (e.g., a smart watch, a smart bracelet, etc.), a vehicle-mounted device, an AR / VR device, and a console of an electronic device, etc. The embodiment of the present application does not impose any special restrictions on the specific form of the electronic device.

[0054] Please refer to Figure 1 , is a structural diagram of an electronic device 100 provided in an embodiment of the present application.

[0055] like Figure 1 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a positioning module 181, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0056] The processor 110 may include a central processing unit (CPU) and a graphics processor. The CPU is used to run the operating system and application programs of the electronic device. The application program may be a graphics application program, such as a game application, a video playback application program, etc. The operating system provides a system graphics library interface, and the application program generates an instruction stream for rendering graphics or image frames, and the required related rendering data through the system graphics library interface provided by the operating system and the driver provided by the operating system. The system graphics library interface includes, but is not limited to: an open graphics library for embedded system (OpenGL ES) interface, a Khronos platform graphics interface interface, or a graphics application program interface (Vulkan), etc. The instruction stream contains a series of instructions, which are usually call instructions for the system graphics library interface.

[0057] Exemplarily, the CPU may also include one or more of the following processing units: such as an application processor (AP), a modem processor, an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0058] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0059] The CPU may also include a hardware accelerator, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or an integrated circuit for implementing logic operations. The CPU may be coupled to one or more data buses for transmitting data and instructions between various components of the electronic device 100.

[0060] The graphics processor may include a general-purpose graphics processor that executes software, such as a graphics processing unit (GPU) or other types of dedicated graphics processing units. Figure 1 The processor 110 includes a CPU and a GPU as an example for illustration.

[0061] The GPU is used to receive the graphics instruction stream sent by the CPU, generate a rendering target through a rendering pipeline, and display the rendering target on the display screen 194 through the operating system of the electronic device 100. Among them, the rendering pipeline can also be called a rendering pipeline, a pixel pipeline or a pixel pipeline, which is a parallel processing unit inside the GPU for processing graphics signals. The GPU may include multiple rendering pipelines, and multiple rendering pipelines can process graphics signals in parallel independently of each other. For example, the rendering pipeline can perform a series of operations in the process of rendering graphics or image frames, and typical operations may include: vertex processing, primitive processing, rasterization, fragment processing, etc.

[0062] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0063] In some embodiments, the processor 110 may include one or more interfaces. It is understood that the interface connection relationship between the modules illustrated in this embodiment is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0064] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.

[0065] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the photosensitive element of camera 193 through the lens. The light signal is converted into an electrical signal, and the photosensitive element of camera 193 transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0066] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0067] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0068] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0069] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0070] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0071] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-LED, Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0072] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0073] The methods in the following embodiments can all be implemented in the electronic device 100 having the above hardware structure. To facilitate understanding, the technical terms involved in the embodiments of the present application are first explained.

[0074] 1. Ray tracing technology

[0075] Ray tracing, also known as ray tracing, light trail tracing or ray tracing, is a technology used to achieve image rendering. Its main purpose is to project virtual objects (also called objects, models, etc.) in three-dimensional space into two-dimensional space for display by calculating the color values ​​of each pixel on the screen image.

[0076] Specifically, ray tracing is to render an image by tracing the path of light in pixels in the image plane and simulating the effect of light meeting virtual objects. For example, from the position of the virtual camera, a beam of light is emitted into the three-dimensional scene through the pixels on the image plane, the intersection point of the first intersection of the light and the virtual object in the three-dimensional scene (hereinafter referred to as the main intersection point) is calculated, and then the light energy of the main intersection point (also referred to as light intensity, coloring, etc.) is calculated. If the material of the main intersection point is reflective (or refractive), it can continue to be traced in the reflection (or refraction) direction of the main intersection point, and continue to obtain the light energy of the intersection point after reflection (or refraction), and the weighted sum of the light energy of all intersection points is used as the color value of the pixel. In an embodiment of the present application, for each pixel on the image plane, the color value of each pixel can be calculated through the above process, so that a frame of image can be rendered. In other words, ray tracing is to calculate the projection and global illumination by tracing the propagation process of light in a three-dimensional scene, thereby rendering a two-dimensional image.

[0077] Please refer to Figure 2 , is a schematic diagram of the principle of ray tracing provided in the embodiment of the present application. Figure 2 As shown, the principle of ray tracing is: a beam of light is emitted from the position of the virtual camera along the pixel position on the image plane. After the light passes through the pixel on the image plane and intersects with the virtual object in the three-dimensional scene, reflection and refraction may occur. After reflection or refraction, a new light is formed, and the new light may also hit other virtual objects and be reflected or refracted. After multiple reflections or refractions, the light finally reaches the light source. In an embodiment of the present application, the path of the light emitted from the virtual camera that finally reaches the light source after multiple reflections or refractions can be called an effective light path.

[0078] For example, Figure 2 Four effective light paths are shown, and the initial light on each effective light path passes through a pixel on the image plane. The initial light can be understood as the light that starts from the virtual camera, passes through a pixel on the image plane, and intersects with the virtual object in the three-dimensional scene for the first time.

[0079] like Figure 2 As shown, the initial light a1 of the effective light path a intersects with the virtual object 1 and is reflected to form the light a2. The light a2 intersects with the virtual object 2 and is reflected to form the light a3. The light a3 intersects with the virtual object 3 and is reflected to form the light a4. The light a4 eventually reaches the light source.

[0080] After the initial light ray b1 of the effective light path b intersects with the virtual object 1, it is emitted to form the light ray b2. After the light ray b2 intersects with the virtual object 2, it is reflected to form the light ray b3. The light ray b3 finally reaches the light source.

[0081] After the initial light ray c1 of the effective light path c intersects with the virtual object 2, it is reflected to form the light ray c2. After the light ray c2 intersects with the virtual object 3, it is emitted to form the light ray c3. The light ray c3 finally reaches the light source.

[0082] The initial light d1 of the effective light path d intersects with the virtual object 3 and is reflected to form the light d2, which eventually reaches the light source.

[0083] It should be noted that not all light rays emitted from the position of the virtual camera will eventually reach the light source. Some light rays may be reflected or refracted by the virtual object and diffuse outward and away from the light source. In the embodiment of the present application, the path of light rays emitted from the virtual camera that does not reach the light source after multiple reflections or refractions can be called an invalid light path.

[0084] For ease of understanding, Figure 2 The number of intersections between the light and the virtual object on each valid light path shown is small. In actual implementation, the light emitted from the virtual camera usually undergoes multiple reflections or refractions before reaching the light source, and there are many intersections on each valid light path. For example, Figure 3 As shown in the figure, the initial light O1 emitted from the virtual camera intersects with the virtual object 1 at point P1, and forms light O2 after reflection at point P1; the light O2 intersects with the virtual object 2 at point P2, and forms light O3 after reflection at point P2; the light O3 intersects with the virtual object at point P3, and forms light O4 after reflection at point P3, and so on. Assuming that after k reflections, the light finally reaches the light source. Where k is an integer much greater than 1.

[0085] exist Figure 3 In the ray tracing process shown in FIG, the light energy at each intersection is calculated, and then the weighted sum of the light energy at all intersections is used as the color value of the corresponding pixel, and the corresponding pixel is rendered using the color value. Figure 3 The process shown calculates the color value of the pixel, thereby rendering each pixel and finally obtaining a rendered image.

[0086] For example, in Figure 3 In the effective light path shown, the color value of the pixel corresponding to the effective light path is the weighted sum of the light energy at P1, the light energy at P2, the light energy at P3, ... and the light energy at Pk.

[0087] (II) Rasterization

[0088] Rasterization is the process of converting virtual objects in a three-dimensional scene into a two-dimensional matrix composed of pixels in the image plane, then calculating the color value of each pixel and coloring each pixel according to the color value to generate the final rendered image.

[0089] Please refer to Figure 4 , is a schematic diagram of the principle of rasterization processing provided by the embodiment of the present application. Figure 4 As shown in the figure, the principle of rasterization is: divide the virtual objects in the 3D scene into triangles, convert the 3D coordinates of the vertices of the triangles in the 3D scene into 2D coordinates on the image plane through coordinate transformation, and interpolate inside the triangle to obtain the 2D coordinates of each pixel of the triangle on the image plane. Then calculate the color value according to the 2D coordinates of each pixel, and color each pixel according to the color value to generate the final rendered image.

[0090] For example, Figure 4 As shown, the three-dimensional coordinates of the vertices of the triangle in the three-dimensional scene include (a1, b1, c1), (a2, b2, c2), (a3, b3, c3). After mapping the three-dimensional coordinates of the vertices to the image plane, the two-dimensional coordinates of the vertices of the triangle on the image plane include (x1, y1), (x2, y2), (x3, y3).

[0091] It should be noted that the reason why virtual objects in a three-dimensional scene are divided into triangles is because triangles are the most basic polygons, and any other polygons can be split into triangles. The interior of a triangle must be planar, and the inside and outside of the triangle are clear. As long as the three vertices of the triangle are defined, the properties of any point inside the triangle can be obtained. For example, based on the two-dimensional coordinates of the three vertices of the triangle, the two-dimensional coordinates of other pixels inside the triangle can be obtained, that is, interpolation inside the triangle.

[0092] It can be seen that rasterization is to project virtual objects in a three-dimensional scene directly onto an image plane and color the pixels on the image plane. It is easy to process and provides poor lighting effects. Ray tracing is to achieve real effects such as reflection, refraction, shadow, ambient occlusion, material, and global illumination by tracing each ray emitted by a virtual camera. Therefore, ray tracing can provide more realistic lighting effects.

[0093] In the related art, the virtual camera emits the same number of rays to each pixel on the image plane, that is, the number of rays corresponding to each pixel on the image plane is uniform. In this way, when using the ray tracing method to render an image, the number of rays that need to be traced will be greater, resulting in a very large amount of calculation in the ray tracing process. In addition, ray tracing requires calculating the light energy at the intersection of the ray and the virtual object (i.e., finding the intersection), and the calculation of the light energy at each intersection is a deep iterative process, which further increases the amount of calculation for ray tracing.

[0094] Based on this, an embodiment of the present application provides an image rendering method, which can reduce the amount of calculation for ray tracing without affecting the rendering effect of the image quality, reduce the computing resources and computing time consumed by the electronic device, and improve the computing performance of ray tracing. Specifically, for each pixel in the image to be rendered, different light data are adaptively used. For example, for pixels that require higher rendering information to obtain the preset image quality effect, more light data is used for ray tracing; and for pixels that require lower rendering information to obtain the preset image quality effect, less light data is used for ray tracing, that is, different rendering information corresponds to different light data, thereby optimizing the computing performance of ray tracing.

[0095] It should be noted that the image to be rendered can be understood as the image plane described in the above embodiment. Figure 2 and Figure 4 It can be seen that the image plane is composed of multiple pixels, that is, the image to be rendered includes multiple pixels. Among them, a pixel refers to the smallest unit in an image represented by a digital sequence. In the embodiment of the present application, the color value of each pixel is calculated, and each pixel is colored according to the color value to achieve rendering of the image to be rendered, and the color value and position of the pixel can determine the final appearance of the image to be rendered.

[0096] In addition, the virtual objects described in the embodiments of the present application can be understood as static object elements in a three-dimensional scene, such as virtual grass, virtual buildings, etc.; or, the virtual objects can also be movable objects in a three-dimensional scene, such as virtual characters, virtual characters, virtual animals, etc. in game applications. For ease of understanding, the embodiments of the present application illustrate the virtual objects as geometric shapes such as cubes, cones, spheres, and cylinders.

[0097] The technical solution provided by the embodiments of the present application is described in detail below in conjunction with the accompanying drawings.

[0098] For example, an application (APP) may be installed in the electronic device. When the application needs to display an image through the electronic device, an instruction may be sent to the electronic device so that the electronic device renders the corresponding image according to the instruction, and then displays the rendered image through the display screen of the electronic device.

[0099] For example, combined with Figure 5 , which is a flowchart of an image rendering process. In this example, the electronic device can be provided with a CPU, a GPU, and a memory (such as Figure 1 The internal memory shown in ) etc. Among them, the CPU can be used for instruction processing and control. The GPU can render images under the control of the CPU. The memory can be used to provide storage functions, such as storing rendering data of GPU rendered images.

[0100] like Figure 5 As shown, the application can issue a rendering instruction to instruct the electronic device to render a frame of image (which can be understood as a frame of image to be rendered). The rendering instruction can correspond to a drawing command. The CPU can receive the rendering instruction and call the corresponding graphics drawing application programming interface (application programming interface, API) to instruct the GPU to perform the rendering operation corresponding to the rendering instruction. The GPU can then execute the rendering instruction and store the rendering data in the memory.

[0101] For example, Figure 5 As shown, in this example, the memory may include multiple pre-created frame buffers (FrameBuffer, FB), such as frame buffer 11, frame buffer 12, frame buffer 13, etc. Different frame buffers can be used to store different data in the image rendering process. For example, in this example, the GPU can store rendering data, such as acceleration structure, material information, light data, etc. in the frame buffer of the memory. Then, the GPU can read the rendering data from the frame buffer of the memory and use the ray tracing algorithm to render the image to be rendered.

[0102] Before introducing the method for image rendering using ray tracing provided in the present application, the rendering data required for the ray tracing involved in the present application is first introduced.

[0103] (1) Ray tracing acceleration structure

[0104] The ray tracing acceleration structure is obtained based on the three-dimensional scene in the above embodiment, and the ray tracing acceleration structure is used to quickly obtain the intersection of light rays, that is, by tracing rays through the ray tracing acceleration structure, the intersection of light rays and virtual objects in the three-dimensional scene can be quickly obtained. Exemplarily, the ray tracing acceleration structure may include a bounding volume hierarchy (BVH), a uniform grid (uniform grid), a k-dimensional tree (kd-tree), or an axis-aligned bounding box (AABB) and other structures. The embodiments of the present application do not specifically limit the ray tracing acceleration structure.

[0105] Among them, the ray tracing acceleration structure can quickly remove irrelevant objects (such as irrelevant virtual objects in the three-dimensional scene) by utilizing the space division structure, so that only a small part of the virtual objects can be traversed to find the nearest intersection. For example, taking the ray tracing acceleration structure as the above-mentioned BVH as an example, the virtual objects in the three-dimensional scene are surrounded by BVH, and before the light intersects with the virtual objects in the three-dimensional scene, it will first intersect with the BVH. If the light does not hit the BVH, it means that the light will not intersect with the virtual objects in the BVH; if the light hits the BVH, then it is calculated whether the light intersects with the virtual objects in the BVH.

[0106] Usually, a 3D scene includes multiple virtual objects. Using BVH actually uses a binary tree structure to manage virtual objects in the 3D scene. For example, please refer to Figure 6 , Figure 6 A schematic diagram of a BVH provided in an embodiment of the present application. Figure 6 As shown, different virtual objects are surrounded by bounding boxes of different sizes, and a corresponding binary tree structure is formed. Then, the binary tree structure is traversed in sequence to determine whether the light intersects with the virtual objects in the BVH. For example, when the light does not intersect with the bounding box B in the binary tree, it means that the light will not intersect with the four virtual objects in the bounding box B; when the light intersects with the bounding box C in the binary tree, it is further calculated whether the light intersects with the two virtual objects in the bounding box C. In this way, the process of detecting whether the light intersects with the four virtual objects in the bounding box B can be reduced, thereby quickly obtaining the intersection point of the light and the virtual objects in the three-dimensional scene.

[0107] In some embodiments, a ray tracing acceleration structure can be created by the following steps. For example, the steps for an electronic device to create a ray tracing acceleration structure include: Step ①: The electronic device divides the virtual object into triangles, then obtains the vertex data and index data of the triangles, and stores the vertex data of the triangles in a vertex buffer in the memory, and stores the index data of the triangles in an index buffer in the memory. Among them, the vertex buffer and the index buffer are a frame buffer created by the electronic device in the memory, and the vertex buffer and the index buffer are different buffers of the same frame buffer in the memory; or, the vertex buffer and the index buffer are different frame buffers in the memory.

[0108] Step ②: The electronic device creates an acceleration structure object by filling VkAccelerationStructureGeometryKHR and calling the BuildGeometryInfo function. Step ③: The electronic device calls the vkGetBufferDeviceAddressKHR function through the vulkan instruction to obtain the storage address of the vertex buffer and index buffer in the memory, and triggers the GPU to read the vertex data and index data of the triangle from the vertex buffer and index buffer based on the storage address of the vertex buffer and index buffer. Step ④: The electronic device calls the vkCmdBuildAccelerationStructuresKHR function through the vulkan instruction to fill the vertex data and index data of the triangle into the acceleration structure object to create a ray tracing acceleration structure.

[0109] (2) Material information

[0110] It is understandable that the three-dimensional scene includes multiple virtual objects, each of which corresponds to a material information, which is pre-configured by the electronic device. For example, the material information may include one or more of roughness, basic color, refractive index, and reflectivity.

[0111] (3) Light data

[0112] Ray data refers to the data required by an electronic device to execute a ray tracing algorithm. For example, ray data includes the number of rays and the number of ray iterations (referred to as iterations). The number of rays refers to the number of rays emitted by a virtual camera to a single pixel on the image plane, that is, the number of rays passing through a single pixel on the image plane. The iteration number is used to indicate the number of times each ray emitted by the virtual camera to a single pixel intersects with a virtual object in a three-dimensional scene. For example, the virtual camera emits three different rays to a single pixel, and each ray intersects with a virtual object three times in the three-dimensional scene.

[0113] For example, the electronic device can store the number of rays and the number of ray iterations in the ray data in a frame buffer of the memory, such as in different channels of a texture (or mask image) included in the frame buffer. In other words, the ray data can be saved in a texture, thereby achieving the effect of reducing the storage overhead of the ray data results.

[0114] For example, the electronic device may store the number of rays and the number of ray iterations in a preset format map on a frame buffer. The preset format map may include at least two channels. The two channels are used to store the number of rays and the number of ray iterations, respectively. For example, one channel may be used to store the number of rays, and the other channel may be used to store the number of ray iterations.

[0115] Taking the preset format of RG8 format as an example, illustratively, the electronic device can store the number of rays in the R channel of RG8 format on a frame buffer in the memory; the electronic device can store the number of ray iterations in the G channel of RG8 format on a frame buffer in the memory.

[0116] The following embodiments will describe the image rendering method provided by the embodiments of the present application in combination with the above examples. It should be understood that Figure 7 The description thereof is only an example of the solution provided in the embodiment of the present application. Figure 7 The composition of the image to be rendered shown does not constitute a limitation on the solution described in the embodiment of the present application. In other embodiments, the image to be rendered may also include Figure 7 More or fewer pixels as shown.

[0117] For example, Figure 7 The image to be rendered shown includes 25 pixels. For each pixel, the electronic device calculates the rendering information of the pixel included in the image to be rendered, and the rendering information of the pixel is the rendering information required to obtain the target image quality result of the pixel. Then, the electronic device determines the light data of the pixel according to the rendering information of the pixel (different rendering information corresponds to different light data), and renders the pixels included in the image to be rendered according to the light data of the pixel.

[0118] Still Figure 7 As shown below, Figure 7 Taking the pixels A and B included in the example, the specific implementation process of using different light data for ray tracing for different pixels and rendering based on the results of ray tracing is exemplified in the embodiment of the present application. Figure 7 In the image to be rendered shown, the electronic device uses the first light data to perform ray tracing, and renders pixel A based on the result of the ray tracing, and uses the second light data to perform ray tracing, and renders pixel B based on the result of the ray tracing.

[0119] As an example, Figure 8 As shown, the process of rendering pixel A and pixel B by the electronic device may include the following steps.

[0120] Step a: The electronic device calculates first rendering information of pixel A, where the first rendering information is rendering information required for pixel A to obtain a first image quality effect.

[0121] In some embodiments, the rendering information corresponding to pixel A includes a first coefficient, and the first coefficient is used to indicate the light emission density required for pixel A to obtain a first image quality effect. In other embodiments, the rendering information corresponding to pixel A includes a first distance, and the first distance is the distance between pixel A and the virtual camera in the camera coordinate system. In some other embodiments, the rendering information corresponding to pixel A includes the first coefficient and the first distance.

[0122] In an embodiment of the present application, when the rendering information corresponding to the pixel includes the first coefficient, the light data of the pixel includes the first light quantity. When the rendering information corresponding to the pixel includes the first distance, the light data of the pixel includes the first iteration number. When the rendering information corresponding to the pixel includes the first coefficient and the first distance, the light data of the pixel includes the first light quantity and the first iteration number.

[0123] Step b: the electronic device calculates second rendering information of pixel B, where the second rendering information is rendering information required for pixel B to obtain a second image quality effect.

[0124] It should be noted that, for the example description of the rendering information corresponding to pixel B, reference can be made to the example description of pixel A in step a above, which will not be repeated here. It should be understood that in the embodiment of the present application, the first rendering information is different from the second rendering information.

[0125] For example, if the first image quality effect is better than the second image quality effect, the first rendering information is higher than the second rendering information. Conversely, if the second image quality effect is better than the first image quality effect, the second rendering information is higher than the first rendering information.

[0126] Step c: The electronic device determines the first light data corresponding to pixel A according to the first rendering information.

[0127] Step d: The electronic device determines the second light data corresponding to the pixel B according to the second rendering information, and the first light data is different from the second light data.

[0128] Exemplarily, if the first rendering information is higher than the second rendering information, the first ray data is greater than the second ray data. Conversely, if the second rendering information is higher than the first rendering information, the second ray data is less than the first ray data.

[0129] Step e: The electronic device uses the first light data to render pixel A and uses the second light data to render pixel B, thereby rendering the image to be rendered.

[0130] Taking the first image quality effect as better than the second image quality effect as an example, the first rendering information is higher than the second rendering information. Exemplarily, for pixel A that requires higher first rendering information to obtain the first image quality effect, the electronic device uses more first light data for ray tracing, and renders pixel A based on the result of ray tracing; and for pixel B that requires lower second rendering information to obtain the second image quality effect, the electronic device uses less second light data for ray tracing, and renders pixel B based on the result of ray tracing.

[0131] In summary, using the solution of the embodiment of the present application, the electronic device calculates the first rendering information of pixel A, and determines the first light data corresponding to pixel A based on the first rendering information. Correspondingly, the electronic device calculates the second rendering information of pixel B, and determines the second light data corresponding to pixel B based on the second rendering information. Since the first rendering information is the rendering information required for pixel A to obtain the first image quality effect, and the second rendering information is the rendering information required for pixel B to obtain the second image quality effect, the first rendering information is different from the second rendering information, and therefore the first image quality effect is different from the second image quality effect, and thus the determined first light data is different from the second light data. Compared to the solution in the related art that uses the same light data for rendering each pixel in the rendered image, the present application uses different light data for rendering for different pixels, that is, the electronic device can adapt to the light data corresponding to different pixels to optimize the efficiency of ray tracing.

[0132] It should be noted that the rendering process of other pixels in the image to be rendered may refer to the rendering process of the above-mentioned pixel A and pixel B, which will not be described in detail here.

[0133] As an example, in an embodiment of the present application, the electronic device can render each pixel in the image to be rendered based on the ray tracing algorithm according to the ray tracing acceleration structure, material information of the virtual object, and light data, thereby obtaining a rendered image.

[0134] In some embodiments, the electronic device passes the ray tracing acceleration structure and the material information of the virtual object into the ray tracing rendering pipeline (RayTracingPipeline), and calls the VkDescriptorSets function to pass the map storing the light data into the ray tracing rendering pipeline. Then, the electronic device calls the execution instruction vkCmdTraceRay to start ray tracing for each pixel in the ray tracing rendering pipeline, and renders each pixel based on the result of ray tracing, thereby obtaining a rendered image.

[0135] Exemplarily, in the shader of the ray tracing rendering pipeline, for each pixel in the image to be rendered, the electronic device calculates the coordinates of the intersection after the light rays intersect with the virtual object based on the ray tracing acceleration structure and the material information of the virtual object. Then, based on the intersection coordinates of each intersection, the number of rays, and the number of iterations of each ray, the electronic device calculates the light energy at each intersection, takes the weighted sum of the light energy at all intersections as the color value of the corresponding pixel, and renders the corresponding pixel using the color value. At this point, the electronic device can output an image rendered based on ray tracing.

[0136] The following embodiments introduce a specific implementation method in which an electronic device determines light data of a pixel according to rendering information of the pixel.

[0137] Taking the case where the rendering information of a pixel includes a first coefficient and the light data of the pixel includes a first light quantity as an example, the electronic device can query the first light quantity in a stored first light lookup table according to the first coefficient. The first light lookup table records the correspondence between the first coefficient and the first light quantity.

[0138] For example, the electronic device stores a first light lookup table, that is, the electronic device stores a correspondence between the first coefficient and the first light quantity. The correspondence can be stored in the electronic device in the form of an array or a table. For example, the correspondence can be as shown in Table 1 below.

[0139] Table 1

[0140] First coefficient (e) First light number (lightNum) [0.0-0.1] 1 (0.1-0.3] 2 …… ……

[0141] It can be seen from Table 1 above that the larger the first coefficient is, the more the corresponding number of first rays is, that is, the larger the light emission density required by the pixel is, the more the corresponding number of first rays is. Correspondingly, the smaller the first coefficient is, the smaller the corresponding number of first rays is, that is, the smaller the light emission density required by the pixel is, the smaller the corresponding number of first rays is. Exemplarily, when the first coefficient is 0.1, the electronic device can query the number of first rays corresponding to the first coefficient based on Table 1 above, which is 1.

[0142] It should be noted that the above Table 1 is only an example of the present application and does not constitute a limitation of the present application. Of course, the above Table 1 may also include the corresponding relationship between the first coefficient of other values ​​and the number of first rays, which will not be described one by one here.

[0143] It can be seen that in the embodiment of the present application, for pixels that require a higher light emission density to obtain the expected image quality effect, a larger number of rays are used for ray tracing; and for pixels that require a lower light emission density to obtain the expected image quality effect, a smaller number of rays are used for ray tracing. By using this method of different pixels adapting to different numbers of rays, the number of rays is reduced while ensuring that the image quality effect is not affected, thereby achieving the purpose of reducing the computational amount of ray tracing and optimizing the computational performance of ray tracing.

[0144] Taking the case where the rendering information of a pixel includes a first distance and the light data of the pixel includes a first iteration number as an example, the electronic device can query the first iteration number in a stored second light lookup table according to the first distance. The second light lookup table records the correspondence between the first distance and the first iteration number.

[0145] For example, the electronic device stores a second light lookup table, that is, the electronic device stores a correspondence between the first distance and the first iteration number. The correspondence can be stored in the electronic device in the form of an array or a table. For example, the correspondence can be shown in Table 2 below.

[0146] Table 2

[0147] First distance First iteration number (depthMax) [0.0-100] 3 (100-5000] 2 …… ……

[0148] It can be seen from Table 2 above that the shorter the first distance, the more the corresponding first iteration number, that is, the shorter the distance between the pixel and the virtual camera, the more times the light intersects with the virtual object. Correspondingly, the longer the first distance, the fewer the corresponding first iteration number, that is, the longer the distance between the pixel and the virtual camera, the fewer times the light intersects with the virtual object. Exemplarily, when the first distance is 50, the electronic device can query the first iteration number corresponding to the first distance based on Table 2 above, which is 1.

[0149] It should be noted that the above Table 2 is only an example of the present application and does not constitute a limitation of the present application. Of course, the above Table 2 may also include other numerical correspondences between the first distance and the first iteration number, which will not be described one by one here.

[0150] It can be seen that in the embodiment of the present application, the electronic device calculates the distance between each pixel and the virtual camera. For pixels with a shorter distance, a larger number of iterations are used for ray tracing; and for pixels with a longer distance, a smaller number of iterations are used for ray tracing. By adaptively using different numbers of iterations for each ray corresponding to different pixels, the number of iterations corresponding to each ray is reduced while ensuring that the image quality effect is not affected, thereby achieving the purpose of reducing the amount of intersection calculations in the ray tracing process and optimizing the computational performance of ray tracing.

[0151] The following embodiments illustrate the specific implementation process of the electronic device calculating the first coefficient. Fig. 9 As shown, the electronic device may calculate the first coefficient by including the following steps, such as step I to step V.

[0152] Step I: The electronic device performs frustum clipping on the virtual object in the three-dimensional scene.

[0153] The visual cone clipping refers to determining whether the virtual object is located in the visual cone of the camera. The electronic device performing visual cone clipping on the virtual object in the three-dimensional scene can be understood as the electronic device determining whether the virtual object in the three-dimensional scene passes through the visual cone clipping.

[0154] In some embodiments, if any virtual object in the three-dimensional scene is located within the visual cone of the virtual camera, the electronic device determines that the virtual object passes through the visual cone clipping; if any virtual object in the three-dimensional scene is located outside the visual cone of the virtual camera, the electronic device determines that the virtual object does not pass through the visual cone clipping.

[0155] In other embodiments, the electronic device creates a ray tracing acceleration structure for the virtual object in the three-dimensional scene, and then performs frustum clipping based on the ray tracing acceleration structure, wherein the ray tracing acceleration structure may be a BVH, a uniform grid, a k-dimensional tree, or an AABB.

[0156] Taking the ray tracing acceleration structure as AABB as an example, illustratively, the electronic device determines whether the AABB of the virtual object intersects with the view frustum. If the AABB of the virtual object intersects with the view frustum or the AABB of the virtual object is inside the view frustum, the electronic device determines that the virtual object passes the view frustum clipping; if the AABB of the virtual object is outside the view frustum, the electronic device determines that the virtual object does not pass the view frustum clipping.

[0157] Please refer to Fig.10 , is a schematic diagram of the principle of frustum clipping provided by an embodiment of the present application. Fig.10 As shown in the figure, the view cone of the virtual camera is composed of a near clipping plane, a far clipping plane and four view cone surfaces. The near clipping plane is a surface set to prevent the virtual object from being too close to the virtual camera and not visible, the far clipping plane is a surface set to prevent the virtual object from being too far away from the virtual camera and not visible, and the four view cone surfaces are surfaces formed by four lines diverging from the virtual camera and tangent to the two clipping planes.

[0158] For example, Fig.10 As shown, virtual object 1 intersects with the view frustum, virtual objects 2, 3, and 4 are all located inside the view frustum, that is, virtual objects 1, 2, 3, and 4 are clipped by the view frustum. Virtual object 5 is located outside the view frustum, that is, virtual object 5 is not clipped by the view frustum.

[0159] As an example, the electronic device adds the virtual objects that pass the frustum clipping to the rendering list (mesh list), while the virtual objects that do not pass the frustum clipping do not need to be added to the rendering list. Subsequently, the electronic device can render the virtual objects in the rendering list, thereby reducing the load of the electronic device in the subsequent rendering process.

[0160] Step II: The electronic device performs rasterization processing on the virtual objects in the rendering list to generate geometric data (or G-Buffer data).

[0161] The geometric data may include depth information and normal information corresponding to each virtual object in the rendering list. The depth information is used to indicate the distance between the virtual object and the virtual camera, and the normal information is used to indicate the surface normal of the virtual object.

[0162] In some embodiments, the electronic device creates a depth map in preset format 1 and a normal map in preset format 2 as rendering targets for rasterization processing. The normal map is used as a normal rendering target (color attachment) and the depth map is used as a depth rendering target (depth attachment). Then, for each virtual object in the rendering list, the electronic device calls the vkCmdDrawIndex instruction for rasterization processing to generate geometric information.

[0163] The preset format 1 may be, for example, a depth24 format, and the preset format 2 may be, for example, an RGB16F format. Of course, the preset format 1 and the preset format 2 may also be other suitable formats, which are not listed here.

[0164] Exemplarily, each virtual object in the rendering list is divided into triangles, and the electronic device calculates the normal coordinates (wNormal) of each vertex of the triangle in the world space coordinate system in the vertex shader, and transmits the normal coordinates of each vertex to the pixel shader through the rasterization interpolation algorithm. The rasterization interpolation algorithm is used to interpolate the normal coordinates of each pixel in the triangle according to the normal coordinates of the triangle vertices.

[0165] After the electronic device transmits the normal coordinates of each vertex to the pixel shader through the rasterization interpolation algorithm, the normal coordinates of each pixel corresponding to the virtual object are stored in the pixel shader. Then, the pixel shader outputs the normal coordinates of each pixel into a normal map of the preset format 2.

[0166] Exemplarily, the electronic device performs a depth test on each virtual object in the rendering list to obtain the depth coordinates of each pixel in the virtual object in the world space coordinate system, and then the electronic device outputs the depth coordinates of each pixel to a depth map of preset format 1. At this point, the electronic device generates geometric data, including a normal map and a depth map. The normal map stores the normal coordinates of each pixel, and the depth map stores the depth coordinates of each pixel.

[0167] That is to say, the electronic device can store the normal coordinates and depth coordinates respectively on a map included in a frame buffer in the memory, that is, the normal coordinates can be saved on one map and the depth coordinates can be saved on another map, thereby achieving the effect of reducing the geometric data storage overhead.

[0168] It should be noted here that in the related art, the electronic device needs to transfer each pixel corresponding to each virtual object in the three-dimensional scene to the pixel shader of the GPU, and the pixel shader calculates the normal coordinates and depth coordinates of each pixel, which results in a large amount of data copying; and in the process of the pixel shader saving the normal coordinates and depth coordinates of each pixel to the memory, it will also cause the GPU to access a large amount of memory, that is, the amount of data accessed by the GPU to the memory is large.

[0169] In the embodiment of the present application, rasterization is used to generate geometric data, and only the vertices of the triangle need to be transmitted to the vertex shader, and the vertex shader calculates the normal coordinates of the vertices, thereby optimizing the amount of data copying. In addition, the electronic device uses a rasterization interpolation algorithm to transmit the normal coordinates of each pixel to the pixel shader, which can also optimize the amount of memory access of the GPU.

[0170] In addition, in an embodiment of the present application, the electronic device generates geometric data through rasterization processing, and renders each pixel in the image to be rendered through a ray tracing algorithm to obtain a rendered image. Compared with image rendering performed only through rasterization processing, or rendering performed only through a ray tracing algorithm, this optimizes the image rendering quality while also improving the rendering efficiency.

[0171] Step III: The electronic device calculates the first coefficient according to the geometric data.

[0172] Exemplarily, the electronic device creates a map of preset format 3 (which may be called shock ratio), in which the initial value of each pixel is 0. The map is used to store the probability of light shock occurring at each pixel. For example, the preset format 3 may be R32F format.

[0173] As an example, the electronic device uses the above map as a rendering target to construct a quadrilateral model (i.e., a Quad model) that covers the entire screen, and the quadrilateral model includes four vertices, two triangles, and an index. The electronic device can obtain the first coefficient corresponding to pixel A by rendering the quadrilateral model in the pixel shader.

[0174] For example, Fig.11 As shown, the size of the quadrilateral model constructed by the electronic device is a 5×5 pixel area, and the quadrilateral model includes pixel A. For pixel A, the electronic device obtains the depth value (D A ), as well as the vertex coordinates and normal coordinates (Normal(u,v).xyz) of pixel A in the local coordinate system. Then, the electronic device calculates the depth value (D A ), use the camera's space matrix and projection matrix to perform matrix transformation on the vertex coordinates of pixel A to obtain the world coordinates of pixel A (wPos(u,v).xyz).

[0175] For example, the world coordinates of pixel A satisfy the following expressions (1) and (2):

[0176] wPos(u,v).xyz=(tempData.x / tempData.w, tempData.y / tempData.w, tempData.z / tempData.w); expression (1)

[0177] tempData=[(u,v)*2.0-1.0,D A ×1.0]*Inverse(viewMatrix*projectionMatrix); expression (2)

[0178] In expression (1), wPos(u,v).xyz represents the world coordinates of pixel A, tempData.x represents the x-axis coordinate of pixel A in the four-dimensional vector, tempData.y represents the y-axis coordinate of pixel A in the four-dimensional vector, tempData.z represents the z-axis coordinate of pixel A in the four-dimensional vector, and tempData.w represents the normal coordinate of pixel A in the four-dimensional vector.

[0179] In expression (2), tempData represents the four components of the four-dimensional vector, (u, v) refers to the two-dimensional coordinates of pixel A, and its value range is [0.0, 1.0]. The purpose of (u, v)*2.0-1.0 is to transform the u and v coordinates in the range of [0.0, 1.0] to the range of [-1.0, 1.0]. D ARefers to the depth value of pixel A, Inverse means inverting the matrix, viewMatrix means the spatial matrix of the camera, and projectionMatrix means the projection matrix.

[0180] It should be noted that, in the embodiment of the present application, the depth value (D A ) can also be called the first depth value, and the vertex coordinates of pixel A in the local coordinate system can also be called the first vertex coordinates.

[0181] Accordingly, the electronic device samples m pixels i (or first pixels) in an N×N pixel area (or first preset pixel area), obtains the normal coordinates and depth value of each pixel i in the m pixels i, and calculates the world coordinates of each pixel i according to the depth value and the spatial matrix and projection matrix of the camera. Wherein, N and m are positive integers.

[0182] like Fig.11 As shown, the electronic device samples 3 (i.e., m=3) pixels i (e.g., pixel i1, pixel i2, and pixel i3) in a 3×3 (i.e., N=3) pixel area. For pixel i1, illustratively, the electronic device obtains a depth value (D i1 ), and according to the depth value (D i1 ) as well as the camera's space matrix and projection matrix, use the above expressions (1) and (2) to calculate the world coordinates of pixel i1 (wPos(u_i1,v_i1).xyz), and obtain the normal coordinates of pixel i1 (Normal(u_i1,v_i1).xyz).

[0183] On this basis, the distance L1 between pixel A and pixel i1 is calculated according to the world coordinates of pixel A (wPos(u,v).xyz) and the world coordinates of pixel i1 (wPos(u_i1,v_i1).xyz). Correspondingly, the normal angle θ1 between pixel A and pixel i1 is calculated according to the normal coordinates of pixel A (Normal(u,v).xyz) and the normal coordinates of pixel i1 (Normal(u_i1,v_i1).xyz). Then, the probability f1 is obtained according to the distance L1 between pixel A and pixel i1 and the normal angle θ1 between pixel A and pixel i1.

[0184] Still Fig.11 As shown, for pixel i2, the electronic device obtains the depth value (D i2 ), and according to the depth value (D i2) as well as the camera's space matrix and projection matrix, use the above expressions (1) and (2) to calculate the world coordinates of pixel i2 (wPos(u_i2,v_i2).xyz), and obtain the normal coordinates of pixel i2 (Normal(u_i2,v_i2).xyz).

[0185] On this basis, the distance L2 between pixel A and pixel i2 is calculated according to the world coordinates of pixel A (wPos(u,v).xyz) and the world coordinates of pixel i2 (wPos(u_i2,v_i2).xyz). Correspondingly, the normal angle θ2 between pixel A and pixel i2 is calculated according to the normal coordinates of pixel A (Normal(u,v).xyz) and the normal coordinates of pixel i2 (Normal(u_i2,v_i2).xyz). Then, the probability f2 is obtained according to the distance L2 between pixel A and pixel i2 and the normal angle θ2 between pixel A and pixel i2.

[0186] Still Fig.11 As shown, for pixel i3, the electronic device obtains the depth value (D i3 ), and according to the depth value (D i3 ) as well as the camera's space matrix and projection matrix, use the above expressions (1) and (2) to calculate the world coordinates of pixel i3 (wPos(u_i3,v_i3).xyz), and obtain the normal coordinates of pixel i3 (Normal(u_i3,v_i3).xyz).

[0187] On this basis, the distance L3 between pixel A and pixel i3 is calculated according to the world coordinates of pixel A (wPos(u,v).xyz) and the world coordinates of pixel i3 (wPos(u_i3,v_i3).xyz). Correspondingly, the normal angle θ3 between pixel A and pixel i3 is calculated according to the normal coordinates of pixel A (Normal(u,v).xyz) and the normal coordinates of pixel i3 (Normal(u_i3,v_i3).xyz). Then, the probability f3 is obtained according to the distance L3 between pixel A and pixel i3 and the normal angle θ3 between pixel A and pixel i3.

[0188] That is, for m sampling pixels (such as pixel i1, pixel i2 and pixel i3), the electronic device calculates the probability corresponding to each sampling pixel, and then uses the average value of the probabilities of the m sampling pixels as the first probability corresponding to pixel A.

[0189] It should be noted that, in the embodiment of the present application, the normal coordinates of the above-mentioned pixel A can be referred to as the first normal coordinates, and the world coordinates of the pixel A can be referred to as the first space coordinates. Correspondingly, the normal coordinates of the above-mentioned pixel i can be collectively referred to as the second normal coordinates, and the world coordinates of the pixel i can be collectively referred to as the second space coordinates. Further, the distance between the above-mentioned pixel A and the pixel i can be collectively referred to as the first pixel distance, and the normal angle between the above-mentioned pixel A and the pixel i can be collectively referred to as the first normal angle.

[0190] In the embodiment of the present application, the electronic device uses the average of the probability f1, the probability f2 and the probability f3 as the probability (also referred to as the first probability) that the light oscillation occurs in the pixel A. Fig.11 For each pixel in the quadrilateral model shown (i.e., a 5×5 pixel area), the electronic device samples m pixels i in the N×N pixel area, and calculates the probability of light oscillation occurring for each pixel (also referred to as the second probability) according to the above process, thereby obtaining the probability of light oscillation occurring for each pixel in the 5×5 pixel area.

[0191] As an example, the electronic device outputs the probability of light oscillation occurring at each pixel to a pre-created map of preset format 3. That is, the map of preset format 3 stores the probability of light oscillation occurring at each pixel.

[0192] In some embodiments of the present application, the electronic device may perform a smoothing convolution operation on the probability of light oscillation occurring in each pixel generated in the above embodiments to smooth the probability of light oscillation corresponding to each pixel, thereby obtaining a first coefficient corresponding to each pixel.

[0193] Taking the calculation of the first coefficient corresponding to pixel A by an electronic device as an example, illustratively, the electronic device samples n pixels z (or second pixels) in an M×M pixel area (or second preset pixel area) around pixel A, and obtains the probabilities corresponding to the n pixels z from a map of preset format 3, thereby obtaining n second probabilities. Correspondingly, the electronic device obtains the first probability corresponding to pixel A from the map of preset format 3. Then, the electronic device calculates the average of the first probability and the n second probabilities, thereby obtaining the first coefficient corresponding to pixel A. Wherein, M and n are positive integers.

[0194] For example, Fig.12As shown, the electronic device samples 8 (i.e., n=8) pixels z in a 3×3 (i.e., M=3) pixel area around pixel A, for example, pixel z1, pixel z2, pixel z3, pixel z4, pixel z5, pixel z6, pixel z7, and pixel z8. The electronic device obtains the probabilities corresponding to the above 8 pixels from the map of the preset format 3, and obtains the probability corresponding to pixel A. Then, the electronic device calculates the average value of the probabilities corresponding to the above 9 pixels to obtain the first coefficient corresponding to pixel A.

[0195] As an example, for Fig.11 For each pixel in the quadrilateral model (i.e., a 5×5 pixel area) shown in FIG. 1 , the electronic device samples n pixels in the M×M pixel area and performs the sampling according to Fig.12 The process shown calculates the first coefficient corresponding to each pixel. Then, the electronic device outputs the first coefficient corresponding to each pixel to a pre-created map of a preset format 4. The preset format 4 may be R32F, for example.

[0196] This application is combined here Figure 9-12 The specific implementation process of the electronic device calculating the first coefficient is introduced in detail, and the following embodiment introduces the specific implementation method of the electronic device calculating the first distance. It should be understood that the first distance is the distance between the pixel A and the virtual camera in the camera coordinate system.

[0197] Exemplarily, the electronic device obtains the world coordinates of pixel A (wPos(u,v).xyz) and uses the camera space matrix to perform a matrix transformation on the world coordinates of pixel A to obtain the camera coordinates of pixel A (vPos(u,v).xyz), which can also be referred to as the first camera coordinates.

[0198] For example, the camera coordinates of pixel A can satisfy the following expression:

[0199] vPos(u,v).xyz=wPos(u,v).xyz×viewMatrix; wherein vPos(u,v).xyz refers to the camera coordinates of pixel A, wPos(u,v).xyz refers to the world coordinates of pixel A, and viewMatrix refers to the camera space matrix.

[0200] Then, the electronic device calculates the first distance based on the camera coordinates of pixel A. For example, the first distance satisfies the following expression:

[0201] Among them, distance refers to the first distance, vPos(u,v).x refers to the x-axis coordinate of pixel A in the camera coordinate system, vPos(u,v).y refers to the y-axis coordinate of pixel A in the camera coordinate system, and vPos(u,v).z refers to the z-axis coordinate of pixel A in the camera coordinate system.

[0202] It should be noted that electronic devices can use the method provided in the above embodiments to render each pixel in the image to be rendered. The rendering process of each pixel is similar, and the rendering process of pixel A can be referred to, which will not be repeated here.

[0203] The above embodiments illustrate the specific implementation of the image rendering provided by the present application. Fig.13 The process shown is an exemplary description of the ray tracing process provided by the present application.

[0204] For example, Fig.13 As shown, the ray tracing process may include the following steps. Step i: The electronic device performs a view frustum clipping on the virtual objects in the three-dimensional scene. The electronic device may create a ray tracing acceleration structure to perform a view frustum clipping on the virtual objects in the three-dimensional scene. The electronic device adds the virtual objects clipped by the view frustum to a rendering list, that is, the rendering list includes a collection of virtual objects clipped by the view frustum.

[0205] Step ii: The electronic device performs rasterization processing on the virtual object clipped by the view cone to generate depth information and normal information. The depth information is stored in the depth map, and the normal information is stored in the normal map. Step iii: The electronic device calculates the probability of light oscillation occurring in the pixel based on the depth information and the normal information. The electronic device stores the probability of light oscillation occurring in each pixel in the light oscillation probability map (i.e., the map of the above-mentioned preset format 3). Step iv: The electronic device performs smooth convolution processing on the probability of light oscillation occurring in each pixel to determine the first coefficient corresponding to each pixel. The first coefficient corresponding to each pixel is stored in the coefficient map (i.e., the map of the above-mentioned preset format 4). Step v: The electronic device searches for the first number of rays corresponding to the first coefficient based on the first coefficient. Step vi: The electronic device performs ray tracing based on the first number of rays and the ray tracing acceleration structure.

[0206] It should be noted that Fig.13 The specific implementation methods of each step shown are all included in the description of the above embodiments. Please refer to the above embodiments and will not be described one by one here.

[0207] In addition, the contents recorded in each embodiment of the present application can explain the technical solutions in other embodiments of the embodiments of the present application. The technical features recorded in each embodiment can also be applied in other embodiments and combined with the technical features in other embodiments to form new solutions. The present application only exemplarily lists several embodiments for illustration, and does not mean that the present application is limited to this.

[0208] The present application provides an electronic device, which may include: a display screen, a memory, and one or more processors; the memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device performs various functions or steps performed in the above electronic device. The structure of the electronic device can refer to the above Figure 1 The structure of the electronic device 100 is shown.

[0209] The present application also provides a chip system for use in electronic devices. Fig.14 As shown, the chip system 1100 includes at least one processor 1101 and at least one interface circuit 1102. The processor 1101 may be Figure 1 The processor 110 is shown. The interface circuit 1102 may be, for example, an interface circuit between the processor 110 and an external memory; or an interface circuit between the processor and the internal memory 121 .

[0210] The processor 1101 and the interface circuit 1102 can be interconnected via lines. For example, the interface circuit 1102 can be used to receive signals from other devices (such as the memory of the electronic device 100). For another example, the interface circuit 1102 can be used to send signals to other devices (such as the processor 1101). Exemplarily, the interface circuit 1102 can read the instructions stored in the memory and send the instructions to the processor 1101. When the instructions are executed by the processor 1101, the electronic device can execute the various functions or steps executed by the mobile phone in the above embodiment. Of course, the chip system can also include other discrete devices, which is not specifically limited in the embodiments of the present application.

[0211] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes each function or step executed by the electronic device in the above method embodiment.

[0212] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step executed by the electronic device in the above method embodiment.

[0213] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules according to the system, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0214] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0215] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0216] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0217] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0218] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image rendering method, characterized in that: include: Acquire an image to be rendered, wherein the image to be rendered includes pixels; Calculating rendering information of pixels included in the image to be rendered; The rendering information of the pixel is the rendering information required to obtain the target image quality effect of the pixel; Determining light data of the pixel according to rendering information of the pixel; wherein different rendering information corresponds to different light data; Rendering pixels included in the to-be-rendered image according to the light data of the pixels.

2. The method according to claim 1, characterized in that The rendering information of the pixel includes a first coefficient, where the first coefficient is used to indicate the light emission density required for the pixel to obtain the target image quality effect; The calculating rendering information of pixels included in the image to be rendered includes: Calculating a first normal coordinate and a first world coordinate of the pixel; Sampling m first pixels in a first preset pixel area, and calculating a second normal coordinate and a second world coordinate of each of the m first pixels; For each of the first pixels, a pixel distance between the pixel and the first pixel is calculated according to the first world coordinate and the second world coordinate to obtain m first pixel distances, and a normal angle between the pixel and the first pixel is calculated according to the first normal coordinate and the second normal coordinate to obtain m first normal angles; m is a positive integer; The first coefficient is calculated according to the m first pixel distances and the m first normal angles.

3. The method according to claim 2, characterized in that The calculating the first coefficient according to the m first pixel distances and the m first normal angles includes: Calculating a first probability of light oscillation occurring at the pixel according to the m first pixel distances and the m first normal angles; Sampling n second pixels in the second preset pixel area, and respectively calculating the probability of light oscillation occurring in each of the n second pixels, to obtain n second probabilities, where n is a positive integer; Smoothing is performed on the first probability and the n second probabilities to obtain the first coefficient.

4. The method according to claim 2 or 3, characterized in that: The calculating the first world coordinate of the pixel comprises: Obtaining a first depth value of the pixel and a first vertex coordinate of the pixel in a local coordinate system; According to the first depth value, a space matrix and a projection matrix are used to perform a matrix transformation on the first vertex coordinates to obtain a first world coordinate of the pixel.

5. The method according to any one of claims 2 to 4, characterized in that: The calculating the first normal coordinate of the pixel comprises: Acquire multiple virtual objects in a three-dimensional scene corresponding to the image to be rendered; For each virtual object in the plurality of virtual objects, calculating the normal coordinates of each vertex in the virtual object in a vertex shader; The normal coordinates of each vertex are transmitted to the pixel shader by means of a rasterization interpolation algorithm; the rasterization interpolation algorithm is used to store the normal coordinates of each pixel corresponding to the virtual object obtained by interpolation in the pixel shader according to the normal coordinates of the vertex; The first normal coordinate of the pixel is obtained from the pixel shader.

6. The method according to any one of claims 2 to 5, characterized in that: The light data of the pixel includes a first light quantity, where the first light quantity is the quantity of light passing through the pixel; and determining the light data of the pixel according to the rendering information of the pixel includes: The first light quantity is queried in a stored first light lookup table according to the first coefficient, wherein the first light lookup table records a corresponding relationship between the first coefficient and the first light quantity.

7. The method according to any one of claims 1 to 6, characterized in that The rendering information of the pixel also includes a first distance, where the first distance is used to indicate the distance between the pixel and the virtual camera; and the calculating the rendering information of the pixel includes: Acquire first camera coordinates of pixels included in the image to be rendered; wherein the first camera coordinates are used to indicate coordinates of the pixels in a coordinate system established based on the virtual camera; The first distance is calculated according to the first camera coordinates.

8. The method according to claim 7, characterized in that The light data of the pixel includes a first iteration number, where the first iteration number is used to indicate the number of times each light ray passing through the pixel intersects with a virtual object in a three-dimensional scene; The determining the light data of the pixel according to the rendering information of the pixel includes: The first iteration number is queried in a stored second light lookup table according to the first distance, wherein the second light lookup table records a corresponding relationship between the first distance and the first iteration number.

9. The method according to any one of claims 1 to 8, characterized in that The step of rendering the pixels included in the image to be rendered according to the light data of the pixels comprises: Get acceleration structure and material information; Calculating, according to the acceleration structure and the material information, coordinates of a plurality of intersection points after the light passing through the pixel intersects with a virtual object in the three-dimensional scene; Calculating the light energy at each intersection coordinate of the plurality of intersection coordinates according to the light data of the pixel; The weighted sum of the light energies at all intersection points is taken as the color value of the pixel; Rendering pixels included in the image to be rendered based on the color values ​​of the pixels; wherein the acceleration structure is used to accelerate the determination of the multiple intersection coordinates, and the material information includes one or more of roughness, basic color, refractive index, and reflectivity.

10. An electronic device, characterized in that: The electronic device comprises: memory and one or more processors; The memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device executes the method as claimed in any one of claims 1 to 9.

11. A chip system, characterized in that: Applied to electronic equipment, the chip system comprises: at least one processor and an interface; The interface is used to receive instructions and transmit them to the at least one processor; the at least one processor executes the instructions so that the electronic device executes the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 9.

Images