Image processing method and related apparatus

Abstract

The application discloses an image processing method applied to an electronic device with limited computing power. The method comprises the following steps: obtaining to-be-rendered data; performing rasterization processing on the to-be-rendered data to obtain a first image; identifying a target object in the first image, and performing ray tracing processing on the target object to obtain a second image. Since the ray tracing processing is only performed on the local object in the image, the computing power requirement of image rendering is reduced, so that the electronic device with limited computing power can also realize image rendering by using the ray tracing method, and the effect of image rendering is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image processing method and related device. BACKGROUND

[0002] With the rapid development of the computer industry, people's requirements for images are increasing. The image obtained by using the traditional rendering method to render a three-dimensional scene is generally of poor quality and often cannot present a realistic picture. Therefore, ray tracing technology has emerged as the times require.

[0003] Ray tracing (RT) technology refers to tracking each light ray emitted from a camera to realize reflection, refraction, shadow or caustic effect, so as to simulate a real virtual scene and then render a realistic image. However, since each light ray in the scene needs to be tracked in the ray tracing process, the calculation overhead is very large.

[0004] In related technologies, ray tracing technology is often applied to devices with strong computing power (such as personal computers equipped with independent graphics cards), and devices with limited computing power (such as mobile devices) often cannot apply this ray tracing technology, thereby making it difficult to obtain better rendering effects on devices with limited computing power. SUMMARY

[0005] The present application provides an image processing method, which renders the to-be-rendered data once by using rasterization processing to obtain a first image, and then renders the object with an identifier in the first image by ray tracing processing to improve the rendering effect. Since only the local object in the image is subjected to ray tracing processing, the computing power requirement for image rendering is reduced, so that devices with limited computing power can also use ray tracing methods to realize image rendering, thereby improving the effect of image rendering.

[0006] The first aspect of the present application provides an image processing method, which can be applied to an electronic device with limited computing power, and the electronic device can perform an image rendering process. The method comprises: an electronic device acquires to-be-rendered data, which can include models in a 3D scene and attribute information of the models, such as models of the sky, houses, etc. and attribute information such as color and material of the models. The electronic device performs rasterization processing on the to-be-rendered data by using a forward rendering method or a deferred rendering method to obtain a first image. The electronic device performs ray tracing processing on a target object with an identifier in the first image to obtain a second image, and the identifier is used to mark an object to be subjected to ray tracing processing, i.e. the target object with the identifier is an object to be subjected to ray tracing processing. Exemplarily, the target object can be an object that can display obvious light and shadow effects, such as a floor, a mirror or a window, etc.

[0007] That is, the electronic device only performs ray tracing processing on the target object with the identifier in the first image, and does not perform ray tracing processing on the object without the identifier. The target object with the identifier can be realized in various ways. In one way, the target object has a corresponding specific field, and the target object can be considered to have an identifier. In another way, the target object has a corresponding specific field and the value of the specific field is a preset value, and the target object can be considered to have an identifier.

[0008] In this scheme, the first image is obtained by rendering the to-be-rendered data once by rasterization processing, and the object with the identifier in the first image is rendered again by ray tracing processing to improve the rendering effect. Since only the local object in the image is processed by ray tracing processing, the algorithm requirement of image rendering is reduced, so that the device with limited algorithm can also realize image rendering by ray tracing method, and the effect of image rendering is improved.

[0009] In one possible implementation, the identifier is also used to mark the ray tracing processing mode, which may include, for example, reflection, refraction, shadow, or focus scattering. In this way, during the ray tracing processing of the target object in the first image by the electronic device, the electronic device can determine the ray tracing processing mode required by the target object according to the identifier of the target object, and implement ray tracing processing based on the ray tracing processing mode. For example, when the target object in the first image is a floor and the value of the identifier of the floor is 0, the electronic device can perform ray tracing on the floor, and the ray tracing processing mode is reflection.

[0010] By marking the ray tracing processing mode with the identifier, the electronic device can avoid analyzing the material of the target object and then selecting a ray tracing processing method based on the material of the target object, thereby improving the efficiency of the electronic device in performing ray tracing processing.

[0011] In one possible implementation, the to-be-rendered data includes the target object and the material parameters of the target object; and the electronic device can determine the identifier of the target object according to the material parameters of the target object. For example, when the roughness in the material parameters of the floor is 0, the electronic device can determine and generate the identifier of the floor, and the value of the identifier is 0, that is, the ray tracing mode corresponding to the floor is reflection. By determining the identifier of the target object based on the material parameters of the target object by the electronic device, the process of manually adding the identifier to the target object can be saved, and manpower and resources are saved.

[0012] In a possible implementation, the electronic device performs ray tracing processing on the target object in the first image, including: the electronic device obtains the position of the target object in the three-dimensional scene in the first image, that is, the electronic device transforms the coordinate system of the target object in the first image into the coordinate system in the three-dimensional scene through coordinate system transformation, to obtain the position of the target object in the three-dimensional scene. According to the position of the target object in the three-dimensional scene, the electronic device performs ray tracing processing to obtain a ray tracing result. Finally, the electronic device updates the color of the target object in the first image according to the ray tracing result to obtain the second image.

[0013] In this solution, the electronic device performs ray tracing processing by obtaining the position of the target object in the three-dimensional scene in the image, thereby effectively improving the overall rendering effect of the image on the basis of the image obtained after rasterization processing, and the requirement for computing power is low.

[0014] In a possible implementation, the electronic device performs ray tracing processing on the target object in the first image to obtain a second image, including: the electronic device performs ray tracing on the target object in the first image according to the identifier of the target object to obtain a ray tracing result; and the electronic device updates the color of the target object in the first image according to the ray tracing result to obtain a second image.

[0015] In this solution, the electronic device updates the color of the target object in the first image based on the result of ray tracing to implement ray tracing processing, which can reduce the modification to the prior art as much as possible and improve the realizability of the solution.

[0016] In a possible implementation, the electronic device performs ray tracing on the target object in the first image according to the identifier of the target object to obtain a ray tracing result, which can include: the electronic device determines a target pixel in the first image, the target pixel has the identifier, and the target object includes one or more target pixels; the electronic device obtains a target position of the target pixel in the three-dimensional scene through coordinate transformation, and performs ray tracing according to the target position and the identifier to obtain the intersection of the ray and the three-dimensional scene. After determining the intersection, the electronic device can calculate the color of the intersection, and then fuse the color of the intersection with the original color of the target pixel based on the ray tracing manner, to obtain the new color of the target pixel. That is, in the actual process of ray tracing processing, the electronic device can perform ray tracing processing on each pixel in the first image in units of pixels, to implement ray tracing processing on the target object.

[0017] In this embodiment, the electronic device implements ray tracing processing by performing ray tracing processing on each pixel with an identifier, and updating the color of the pixel based on the intersection point obtained by ray tracing, thereby effectively improving the overall rendering effect of the image.

[0018] In a possible implementation, the electronic device updates the color of the target pixel according to the color of the intersection point, including: the electronic device calculates the projection of the intersection point on the image according to the position of the intersection point in the three-dimensional scene; if the intersection point has a corresponding projection pixel on the first image or the third image, the color of the target pixel is updated according to the color of the projection pixel; if the intersection point does not have a corresponding projection pixel on the first image or the third image, the color of the intersection point is calculated, and the color of the target pixel is updated according to the color of the intersection point; wherein the third image is a previous frame image of the second image.

[0019] In short, in the process of image rendering by the electronic device, the electronic device does not render all objects in the 3D scene in real time. The electronic device usually renders the objects to be currently displayed on the screen to obtain a rendered image and display it on the screen. If the intersection point has been rendered and displayed on the image in the process of rendering the previous frame image (i.e., the third image), or the intersection point has been rendered and displayed on the image in the process of rendering the current frame image (i.e., the first image), the color of the intersection point can be determined based on the color of the corresponding pixel point of the intersection point on the previous frame image or the current frame image. That is, the color of the intersection point is obtained by reusing the color of the pixel on the previous frame image or the current frame image, thereby avoiding the recalculation of the color of the intersection point and reducing the calculation amount.

[0020] In a possible implementation, the electronic device performs ray tracing according to the target position and the identifier to obtain an intersection point of a ray and the three-dimensional scene, including: the electronic device obtains an acceleration structure, which is obtained based on the three-dimensional scene, and the acceleration structure may include, but is not limited to, a Bounding Volume Hierarchy (BVH), a Uniform grid, or a k-dimensional tree (kd-tree), etc.; the electronic device performs ray tracing through the acceleration structure according to the target position and the identifier to obtain the intersection point of the ray and the three-dimensional scene. The acceleration structure is used to implement ray tracing processing, which accelerates the speed of the electronic device to find the intersection point and improves the efficiency of the electronic device to perform ray tracing.

[0021] In a possible implementation, the electronic device performs rasterization processing on the to-be-rendered data to obtain the first image, including: performing non-light rendering on the to-be-rendered data to obtain a fourth image; obtaining, according to attribute information of the to-be-rendered data, a geometry buffer corresponding to a pixel in the fourth image, the geometry buffer being used to store attribute parameters corresponding to the pixel; and performing light calculation on the pixel in the fourth image according to the geometry buffer to obtain the first image.

[0022] In a possible implementation, the electronic device obtains, according to attribute information of the to-be-rendered data, a geometry buffer corresponding to a pixel in the fourth image, including: if the to-be-rendered object in the fourth image is the target object, generating a first geometry buffer corresponding to the to-be-rendered object according to attribute information of the to-be-rendered object and a first resolution; if the to-be-rendered object in the fourth image is located in a peripheral area of the target object, generating a second geometry buffer corresponding to the to-be-rendered object according to attribute information of the to-be-rendered object and a second resolution; and if the to-be-rendered object in the fourth image is located in a background area, generating a third geometry buffer corresponding to the to-be-rendered object according to attribute information of the to-be-rendered object and a third resolution; wherein the to-be-rendered data includes the to-be-rendered object, the first resolution is greater than the second resolution, the second resolution is greater than the third resolution, and the first geometry buffer, the second geometry buffer and the third geometry buffer are used to store color attribute parameters.

[0023] In a possible implementation, the electronic device obtains, according to attribute information of the to-be-rendered data, a geometry buffer corresponding to a pixel in the fourth image, and further includes: generating, according to attribute information of the to-be-rendered object and a fourth resolution, a fourth geometry buffer corresponding to the to-be-rendered object, the fourth geometry buffer being used to store attribute parameters other than color attribute parameters; and the fourth resolution is less than the first resolution.

[0024] In a possible implementation, the electronic device obtains, according to attribute information of the to-be-rendered data, a geometry buffer corresponding to a pixel in the fourth image, and further includes: generating, according to attribute information of the to-be-rendered object and a fourth resolution, a fourth geometry buffer corresponding to the to-be-rendered object, the fourth geometry buffer being used to store attribute parameters other than color attribute parameters; and the fourth resolution is less than the first resolution.

[0024] In a possible implementation, the electronic device obtains, according to attribute information of the to-be-rendered data, a geometry buffer corresponding to a pixel in the fourth image, and further includes: generating, according to attribute information of the to-be-rendered object and a fourth resolution, a fourth geometry buffer corresponding to the to-be-rendered object, the fourth geometry buffer being used to store attribute parameters other than color attribute parameters; and the fourth resolution is less than the first resolution.

[0025] In a possible implementation, the electronic device acquires the data to be rendered, including: the electronic device acquires three-dimensional scene data and a fifth image sent by the server, the fifth image being a background image rendered by the server. In this way, the electronic device can only render part of the non-background region in the 3D scene, and fuse the rendered image with the background image sent by the server, to obtain a complete and rendered image. The background image sent by the server refers to an image including only a background region, that is, the background image includes only a background far away. For example, the server can render the sky, mountains, seas, or high-rise buildings far away, to obtain a corresponding background image.

[0026] Exemplarily, in a possible scenario, a game application can be running in the electronic device, and the server renders a background region in a 3D scene in real time, obtains a background image, and sends the background image to the electronic device. The electronic device renders a non-background region in the 3D scene during running of the game application, and combines the background image sent by the server to obtain a rendered image, to be displayed on the screen.

[0027] Optionally, when the game application running in the electronic device is a network game in which multiple users are online at the same time, the background image rendered by the server can also be sent to multiple different electronic devices respectively. Different electronic devices perform personalized rendering according to actual content to be displayed, to display different images on the screen.

[0028] By implementing rendering of the background region by the server, the rendering calculation amount of the electronic device can be reduced, and the requirement for the computing power of the electronic device is lowered.

[0029] The second aspect of the present application provides an electronic device, including an acquisition unit and a processing unit; the acquisition unit is configured to acquire data to be rendered; the processing unit is configured to perform rasterization processing on the data to be rendered to obtain a first image; the processing unit is further configured to perform ray tracing processing on a target object in the first image to obtain a second image; wherein the target object has an identifier, and the identifier is used to mark an object to be executed for ray tracing processing.

[0030] In a possible implementation, the identifier is also used to mark a ray tracing processing mode.

[0031] In a possible implementation, the ray tracing processing mode includes reflection, refraction, shadow, or caustics.

[0032] In a possible implementation, the acquisition unit is further configured to acquire a position of the target object in the three-dimensional scene in the first image; and the processing unit is further configured to perform ray tracing processing according to the position of the target object in the three-dimensional scene, to obtain a ray tracing result, and update the color of the target object in the first image according to the ray tracing result, to obtain the second image.

[0033] In a possible implementation, the processing unit is further configured to perform ray tracing on the target object in the first image according to the identification of the target object, to obtain a ray tracing result, and update the color of the target object in the first image according to the ray tracing result, to obtain the second image.

[0034] In a possible implementation, the processing unit is further configured to determine a target pixel in the first image, the target pixel having the identification, and the target object including one or more target pixels; the acquisition unit is further configured to acquire a target position of the target pixel in the three-dimensional scene; and the processing unit is further configured to perform ray tracing according to the target position and the identification, to obtain an intersection of a ray with the three-dimensional scene, and update the color of the target pixel according to the color of the intersection.

[0035] In a possible implementation, the processing unit is further configured to calculate a projection of the intersection on an image according to the position of the intersection in the three-dimensional scene, and if the intersection has a corresponding projection pixel on the first image or a third image, update the color of the target pixel according to the color of the projection pixel, or if the intersection does not have a corresponding projection pixel on the first image or the third image, calculate the color of the intersection and update the color of the target pixel according to the color of the intersection, where the third image is a previous frame image of the second image.

[0036] In a possible implementation, the acquisition unit is further configured to acquire an acceleration structure, the acceleration structure being obtained based on the three-dimensional scene; and the processing unit is further configured to perform ray tracing through the acceleration structure according to the target position and the identification, to obtain an intersection of a ray with the three-dimensional scene.

[0037] In a possible implementation, the processing unit is further configured to perform non-illumination rendering on the to-be-rendered data, to obtain a fourth image, obtain a geometry buffer corresponding to a pixel in the fourth image according to attribute information of the to-be-rendered data, the geometry buffer being configured to store attribute parameters corresponding to the pixel, and perform illumination calculation on the pixel in the fourth image according to the geometry buffer, to obtain the first image.

[0038] In a possible implementation, the processing unit is further configured to: if the object to be rendered in the fourth image is the target object, generate a first geometric buffer corresponding to the object to be rendered according to attribute information of the object to be rendered and the first resolution; if the object to be rendered in the fourth image is located in a peripheral region of the target object, generate a second geometric buffer corresponding to the object to be rendered according to attribute information of the object to be rendered and a second resolution; and if the object to be rendered in the fourth image is located in a background region, generate a third geometric buffer corresponding to the object to be rendered according to attribute information of the object to be rendered and a third resolution; wherein the data to be rendered includes the object to be rendered, the first resolution is greater than the second resolution, the second resolution is greater than the third resolution, and the first geometric buffer, the second geometric buffer, and the third geometric buffer are used to store color attribute parameters.

[0039] In a possible implementation, the processing unit is further configured to: generate a fourth geometric buffer corresponding to the object to be rendered according to attribute information of the object to be rendered and a fourth resolution, wherein attribute parameters stored in the fourth geometric buffer are not color attribute parameters, and the fourth resolution is less than the first resolution.

[0040] In a possible implementation, the acquisition unit is further configured to acquire three-dimensional scene data and a fifth image sent by a server, wherein the fifth image is a rendered background image.

[0041] In a possible implementation, the data to be rendered includes the target object and material parameters of the target object, and the processing unit is further configured to determine the identity of the target object according to the material parameters of the target object.

[0042] The third aspect of the present application provides an electronic device, which includes a processor, a non-volatile memory, and a volatile memory; wherein the non-volatile memory or the volatile memory stores computer readable instructions; the processor reads the computer readable instructions to enable the electronic device to implement the method of any one of the implementation manners of the first aspect.

[0043] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is run on a computer, the computer is enabled to execute the method of any one of the implementation manners of the first aspect.

[0044] The fifth aspect of the present application provides a computer program product, which, when run on a computer, enables the computer to execute the method of any one of the implementation manners of the first aspect.

[0045] The sixth aspect of the present application provides a chip comprising one or more processors. Part or all of the processors are used to read and execute a computer program stored in a memory to execute the method in any possible implementation manner of any aspect described above. Optionally, the chip comprises the memory, and the memory is connected to the processor by a circuit or a wire. Optionally, the chip further comprises a communication interface, and the processor is connected to the communication interface. The communication interface is used to receive data and / or information to be processed, and the processor obtains the data and / or information from the communication interface, processes the data and / or information, and outputs the processing result through the communication interface. The communication interface can be an input / output interface. The method provided by the present application can be implemented by one chip or by multiple chips cooperatively. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1a A schematic diagram of the principle of ray tracing;

[0047] Figure 1b A schematic diagram of the principle of rasterization processing;

[0048] Figure 2 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application;

[0049] Figure 3 A flowchart of an image processing method 300 provided by an embodiment of the present application;

[0050] Figure 4 A flowchart of ray tracing processing of an image provided by an embodiment of the present application;

[0051] Figure 5 A schematic diagram of a BVH provided by an embodiment of the present application;

[0052] Figure 6 A schematic diagram of a reflection scene provided by an embodiment of the present application;

[0053] Figure 7 A schematic diagram of determining an intersection color provided by an embodiment of the present application;

[0054] Figure 8 A flowchart of ray tracing processing provided by an embodiment of the present application;

[0055] Figure 9 A flowchart of rasterization processing of to-be-rendered data provided by an embodiment of the present application;

[0056] Figure 10 A flowchart of generating a G-buffer based on adaptive resolution provided by an embodiment of the present application;

[0057] Figure 11 A schematic diagram illustrating a process for rendering and distributing a background image on a server, provided as an embodiment of this application;

[0058] Figure 12 A schematic diagram of a pre-filtered ambient light map provided in an embodiment of this application;

[0059] Figure 13 A flowchart illustrating an edge-cloud combined image rendering method provided in an embodiment of this application;

[0060] Figure 14 This is a schematic diagram of a hybrid rendering pipeline provided in an embodiment of this application;

[0061] Figure 15(a) shows a first image after rasterization processing provided in an embodiment of this application;

[0062] Figure 15(b) is a second image after ray tracing processing provided in an embodiment of this application;

[0063] Figure 16 This is a schematic diagram of the structure of an electronic device 1600 provided in an embodiment of this application. Detailed Implementation

[0064] The embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. As those skilled in the art will recognize, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0065] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved.

[0066] With the development of computer technology, more and more applications, such as games and video applications, require the display of high-quality images on electronic devices. These images are usually rendered by electronic devices based on models in a three-dimensional (3D) scene.

[0067] Traditional image processing methods typically employ rasterization to render 3D scenes, producing images capable of displaying them. However, images rendered using rasterization generally suffer from low quality and often fail to deliver realistic visuals. For instance, rendered images often struggle to accurately reproduce effects such as light reflection, refraction, and shadows within the scene. In response, a new rendering technique—ray tracing—has emerged. Both ray tracing and rasterization are methods for image rendering, their primary purpose being to calculate and shade objects to project them into a two-dimensional screen space for display.

[0068] Please refer to Figure 1a , Figure 1a This is a schematic diagram illustrating the principle of ray tracing. Figure 1a As shown, the principle of ray tracing is as follows: From the camera's position, a ray of light is emitted through pixel positions on the image plane into the 3D scene. The nearest intersection point between the ray and the geometry is found, and then the shading of that intersection point is calculated. If the material at that intersection point is reflective, ray tracing can continue in the reflection direction from that intersection point, and the shading of the reflected intersection point can be calculated. In other words, the ray tracing method calculates projection and global illumination by tracking the propagation of light in a 3D scene, thereby rendering a 2D image.

[0069] Please refer to Figure 1b , Figure 1b This is a schematic diagram illustrating the principle of rasterization processing. (Example) Figure 1b As shown, the principle of rasterization is as follows: the model in the 3D scene is divided into triangles, the three-dimensional coordinates of the triangle vertices are transformed into two-dimensional coordinates on the image through coordinate transformation calculation, and finally the texture is filled in the triangle on the image to achieve image rendering.

[0070] Rasterization projects visible content directly onto the screen space to create an image; this process is relatively simple but produces less impressive lighting effects. Ray tracing, on the other hand, tracks every ray of light emitted from the camera to achieve realistic effects such as reflection, refraction, shadows, and ambient occlusion. Therefore, ray tracing provides lifelike lighting. However, because it requires tracking the trajectory of every ray, the computation is extremely complex, demanding high computing power from the devices used to perform ray tracing.

[0071] In related technologies, ray tracing technology is mainly used on devices with strong computing power, such as personal computers (PCs) equipped with dedicated graphics cards. However, devices with limited computing power, such as mobile devices like mobile phones and tablets, often cannot use ray tracing technology, which makes it difficult to obtain good rendering effects on devices with limited computing power.

[0072] In view of this, embodiments of this application provide an image processing method that uses rasterization to render the data to be rendered once to obtain a first image, and then uses ray tracing to render the marked objects in the first image a second time to improve the rendering effect. Since ray tracing is only performed on local objects in the image, the computational requirements for image rendering are reduced, enabling devices with limited computing power to use ray tracing to achieve image rendering, thus improving the image rendering effect.

[0073] The image processing method in this embodiment can be executed by an electronic device. This electronic device includes a CPU and a GPU, capable of rendering images. Exemplarily, this electronic device can be a mobile phone, tablet computer, laptop computer, PC, mobile internet device (MID), wearable device, virtual reality (VR) device, augmented reality (AR) device, wireless electronic device in industrial control, wireless electronic device in self-driving, wireless electronic device in remote medical surgery, wireless electronic device in smart grid, wireless electronic device in transportation safety, wireless electronic device in smart city, wireless electronic device in smart home, etc. This electronic device can run Android, iOS, Windows, and other operating systems. Applications that require rendering 3D scenes to obtain 2D images can run on this electronic device, such as game applications, lock screen applications, map applications, or monitoring applications.

[0074] To facilitate understanding, the following will be combined with... Figure 2 A detailed description of the specific structure of the electronic device is provided. Please refer to [reference needed]. Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0075] In one possible embodiment, such as Figure 2 As shown, the electronic device 2000 may include: a central processing unit 2001, a graphics processor 2002, a display device 2003, and a memory 2004. Optionally, the electronic device 2000 may further include at least one communication bus. Figure 2 (Not shown in the image) is used to enable communication between various components.

[0076] It should be understood that the various components in the electronic device 2000 can also be coupled to each other via other connectors, which may include various interfaces, transmission lines, or buses. The various components in the electronic device 2000 can also be connected radially with the central processing unit 2001 at its center. In various embodiments of this application, coupling refers to mutual electrical connection or communication, including direct connection or indirect connection via other devices.

[0077] There are various ways to connect the CPU 2001 and the GPU 2002, and it is not limited to these methods. Figure 2 As shown in the diagram. In electronic device 2000, the central processing unit 2001 and the graphics processor 2002 can be located on the same chip or they can be separate chips.

[0078] The following is a brief introduction to the functions of the central processing unit 2001, graphics processing unit 2002, display device 2003, and memory 2004.

[0079] Central Processing Unit 2001: Used to run Operating System 2005 and Applications 2006. Applications 2006 can be graphics-based applications, such as games, video players, etc. Operating System 2005 provides a system graphics library interface. Applications 2006 use this system graphics library interface, along with drivers provided by Operating System 2005, such as user-mode and / or kernel-mode graphics library drivers, to generate instruction streams for rendering graphics or image frames, as well as the necessary rendering data. The system graphics library includes, but is not limited to, OpenGL ES (Open Graphics Library for Embedded System), the Kronos Platform Graphics Interface, or Vulkan (a cross-platform graphics application programming interface). The instruction stream contains a series of instructions, which are typically calls to the system graphics library interface.

[0080] Optionally, the central processing unit 2001 may include at least one of the following types of processors: application processor, one or more microprocessors, digital signal processor (DSP), microcontroller unit (MCU), or artificial intelligence processor, etc.

[0081] The central processing unit 2001 may further include necessary hardware accelerators, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or integrated circuits for implementing logic operations. The processor 2001 may be coupled to one or more data buses for transferring data and instructions between the various components of the electronic device 2000.

[0082] The graphics processing unit (GPU) 2002 receives the graphics instruction stream sent by the processor 2001, generates rendering targets through the rendering pipeline, and displays the rendering targets on the display device 2003 through the operating system's layer compositing display module. The rendering pipeline, also known as the rendering pipeline, pixel pipeline, or pixel pipeline, is a parallel processing unit within the GPU 2002 used to process graphics signals. The GPU 2002 may include multiple rendering pipelines, which can process graphics signals independently and in parallel. For example, the rendering pipeline can perform a series of operations during the rendering of graphics or image frames; typical operations may include vertex processing, primitive processing, rasterization, fragment processing, etc.

[0083] Optionally, the graphics processor 2002 may include a general-purpose graphics processor that executes software, such as a GPU or other types of dedicated graphics processing units.

[0084] Display device 2003: Used to display various images generated by electronic device 2000, which may be the graphical user interface (GUI) of the operating system or image data (including still images and video data) processed by graphics processor 2002.

[0085] Optionally, the display device 2003 may include any suitable type of display screen, such as a liquid crystal display (LCD), a plasma display, or an organic light-emitting diode (OLED) display.

[0086] Memory 2004 is the transmission channel between the central processing unit 2001 and the graphics processor 2002, and can be double data rate synchronous dynamic random access memory (DDR SDRAM) or other types of cache.

[0087] The above describes the specific structure of the electronic device used in the image processing method provided in the embodiments of this application. The following will describe in detail the flow of the image processing method provided in the embodiments of this application.

[0088] Please refer to Figure 3 , Figure 3 This is a schematic flowchart of an image processing method 300 provided in an embodiment of this application. Figure 3 As shown, the image processing method 300 includes the following steps.

[0089] Step 301: Obtain the data to be rendered.

[0090] In this embodiment, the data to be rendered may include models in a 3D scene and the model's attribute information. For example, the models in the 3D scene may include models of the sky, houses, bridges, people, boxes, or trees. The model's attribute information may include attributes such as the model's color and material.

[0091] It should be understood that during the download and installation of applications, electronic devices download and save the relevant data to be rendered within the application. While running the application, the electronic device can obtain this data by loading the application's related data. Alternatively, the electronic device can also obtain the data to be rendered by receiving data sent from a server while browsing the web, enabling the device to render images based on this data.

[0092] In other words, electronic devices can obtain the data to be rendered by reading local data or by receiving data sent in real time from other devices. This embodiment does not specifically limit the way electronic devices obtain the data to be rendered.

[0093] Step 302: Rasterize the data to be rendered to obtain the first image.

[0094] In this embodiment, there are multiple ways to rasterize the data to be rendered.

[0095] Method 1: Rasterize the data to be rendered using forward rendering.

[0096] Forward rendering refers to the process of projecting and breaking down the geometry in the data to be rendered into vertices, then transforming and breaking down the vertices into fragments or pixels, and finally rendering these fragments or pixels before they are passed to the screen. A key characteristic of forward rendering is that the entire process of processing a geometry from the beginning to the final display of the image on the screen is uninterrupted; that is, forward rendering is a linear process.

[0097] In simple terms, in a 3D scene, the GPU in an electronic device performs lighting calculations for an object based on all light sources to render that object before rendering the next object, and so on. However, for each object that needs to be rendered, the CPU needs to iteratively render each fragment of that object in the fragment shader to obtain the shading result for each fragment. Since the shading results of most fragments are overwritten by the shading results of subsequent fragments, forward rendering can often waste a lot of time rendering useless fragments.

[0098] Method 2: Rasterize the data to be rendered using deferred rendering.

[0099] Deferred rendering refers to projecting and breaking down the geometry in the data to be rendered into vertices, transforming and further breaking down the vertices into fragments or pixels, acquiring various geometric information of these fragments or pixels, and storing this geometric information in a geometry buffer (G-buffer). This geometric information may include position vectors, color vectors, and / or normal vectors. Finally, based on this geometric information stored in the G-buffer, lighting calculations are performed on the fragments or pixels to obtain the final rendering result. In the lighting calculation stage, according to the fragments or pixels that need to be displayed in screen space, the geometric information in the G-buffer is used to perform scene lighting calculations on the corresponding fragments or pixels to output the image to be displayed in screen space. Compared to forward rendering, deferred rendering does not need to repeatedly perform lighting calculations on a large number of fragments or pixels, but only on the fragments or pixels that need to be displayed in screen space, thus saving a lot of useless lighting calculation steps.

[0100] Step 303: Perform ray tracing on the target object in the first image to obtain the second image.

[0101] In this embodiment, the target object in the first image is marked with an identifier. This identifier is used to mark the object to be subjected to ray tracing processing; that is, the target object with this identifier is the object to be subjected to ray tracing processing. For example, the identifier can be a field, where a value of 0 or 1 indicates that the target object needs to undergo ray tracing processing. Alternatively, the identifier can be a specific value within a field, where a value of 0 indicates that the target object corresponding to the identifier needs to undergo ray tracing processing, and a value of 1 indicates that the object does not need to undergo ray tracing processing. In short, the identification of a target object can be implemented in several ways. In one way, if the target object has a corresponding specific field, it can be considered to have an identifier; in another way, if the target object has a corresponding specific field and the value of that specific field is a preset value, it can be considered to have an identifier. This embodiment does not limit the implementation method of identifying the target object.

[0102] Optionally, the identifier can also be used to mark ray tracing processing methods, such as reflection, refraction, shadowing, or caustics. Reflection refers to the phenomenon where light changes its direction of propagation at the interface between different materials and returns to the original material. For example, if the floor identifier in the first image is used to mark reflection, and the electronic device tracks a ray of light emanating from the floor and intersects with the car chassis (i.e., the intersection point is the car chassis), then the floor can be considered to reflect the color of the car chassis. Reflection can include diffuse reflection or specular reflection. Diffuse reflection refers to the reflection of light in an indeterminate direction on or within a diffuse reflective object. Specular reflection refers to the reflection of a parallel beam of light onto a relatively smooth reflective surface, where the light beam still travels parallel in one direction. Refraction refers to the phenomenon where the direction of light propagation changes when light travels obliquely from one transparent medium into another. A shadow is a dark area formed when light travels straight through an opaque object. Caustics refers to the phenomenon where light rays, when passing through a transparent object, do not refract parallelly due to the unevenness of the object's surface, resulting in diffuse refraction and photon dispersion on the projected surface.

[0103] For example, the identifier can be a field where a value of 0 indicates reflection, a value of 1 indicates refraction, a value of 2 indicates shadow, and a value of 3 indicates caustics. In short, the electronic device can determine whether an object in the first image is a target object by judging whether it has an identifier; and the electronic device can determine the ray tracing processing method to be performed on the target object by determining the value of its identifier.

[0104] Thus, during the ray tracing process of the target object in the first image by the electronic device, the electronic device can determine the ray tracing method to be performed on the target object based on the object's identifier, and implement the ray tracing process based on that method. For example, if the target object in the first image is a floor, and the identifier value of the floor is 0, the electronic device can perform ray tracing on the floor, and the ray tracing method is reflection.

[0105] Optionally, if the identifier is only used to mark the object to be ray-traced, and not to mark the ray-tracing method, the electronic device can perform ray-tracing on the target object based on the material parameters of the target object included in the rendering data during the ray-tracing process. Generally, the rendering data includes the model's attribute information, which includes the model's material parameters. Since models are usually classified by object type, the material parameters of the target object can be obtained through the attribute information of the model corresponding to the target object.

[0106] In one possible example, the model's attribute information in the rendering data acquired by the electronic device may include an identifier corresponding to the model, which is used to mark the ray tracing process to be performed. During the rasterization process of the rendering data, the electronic device can generate an identifier for the target object in the first image based on the identifier corresponding to the model in the rendering data. That is, when the electronic device projects the model onto the screen space to form the target object in the first image, it records the identifier of the target object based on the model's identifier. For example, the model's identifier can be added during the model's design process. For instance, during the model's design process, the floor model is assigned an identifier with a value of 0, the diamond model is assigned an identifier with a value of 1, and the wall model is not assigned an identifier. By adding corresponding identifiers to specific models according to the desired lighting and shadow effects during the design process, such as only adding identifiers to models that can effectively improve the lighting and shadow effects, it is possible to selectively determine which models need to be ray-traced, avoiding ray tracing processing for all models and reducing the computing power requirements of the rendering device.

[0107] In another possible example, the rendering data acquired by the electronic device includes a model of the target object, and the model's attribute information includes the object's material parameters. The electronic device can determine and generate the target object's identifier based on these material parameters. Thus, after performing rasterization and obtaining the first image, the electronic device can obtain the identifier of the target object in the first image.

[0108] For example, when the roughness parameter in the material parameters of the floor is 0, the electronic device can determine and generate a mark for the floor with a value of 0, meaning the ray tracing method corresponding to the floor is reflection. Similarly, when the metallicity parameter in the material parameters of silver tableware is 1, the electronic device can determine and generate a mark for the silver tableware with a value of 0, meaning the ray tracing method corresponding to the silver tableware is reflection.

[0109] It should be understood that when an electronic device uses forward rendering to implement rasterization, the device can allocate specific storage space to store the identifiers of target objects in the first image. Generally, a target object can consist of one or more pixels, and for each pixel of the target object, its corresponding identifier needs to be stored.

[0110] When an electronic device uses deferred rendering to perform rasterization, it can generate a G-buffer to store the identifier of the target object model during the G-buffer generation process. That is, for each pixel in the first image, in addition to having a G-buffer to store the pixel's position vector, color vector, and / or normal vector, it also has a G-buffer to store the identifier corresponding to that pixel.

[0111] In this embodiment, the data to be rendered is rendered once using rasterization to obtain a first image. Then, the marked objects in the first image are rendered a second time using ray tracing to improve the rendering effect. Since ray tracing is only performed on local objects in the image, the computational requirements for image rendering are reduced, enabling devices with limited computing power to use ray tracing to achieve image rendering and improving the image rendering effect.

[0112] The above describes the process of electronic devices performing hybrid rendering of images based on rasterization and ray tracing. To facilitate understanding, the following will describe in detail the specific process of electronic devices performing ray tracing on images.

[0113] Please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating a process for performing ray tracing processing on an image, as provided in an embodiment of this application. Figure 4As shown, in one possible example, step 303 above may also include the following steps.

[0114] Step 3031: Determine the target pixel in the first image.

[0115] In this embodiment, the target object may include one or more target pixels, and the target pixels have the identifier. For the same target object, the one or more target pixels included in the target object have the same identifier. In the actual process of ray tracing processing, ray tracing processing can be performed on each pixel in the first image, thereby realizing ray tracing processing of the target object.

[0116] It should be understood that for a model in the data to be rendered, the model may include multiple parts. Different parts may have different identifiers, or some parts may have identifiers while others do not. In this case, the target object can be understood as a part of the model. Simply put, the material parameters of any part on the same target object are the same, and one or more pixels corresponding to the target object have the same identifier. For example, for a car model, which includes parts such as windows, body shell, and tires, the windows may have a corresponding identifier with a value of 0, while the body shell and tires may not have corresponding identifiers.

[0117] Step 3032: Obtain the target position of the target pixel in the three-dimensional scene.

[0118] Since the target pixel is actually a pixel in a two-dimensional image (i.e., the first image), and this target pixel is obtained by projection transformation of a certain part in the 3D scene, coordinate transformation can be performed on the target pixel to obtain its target position in the 3D scene. For example, an electronic device can transform the two-dimensional coordinates of the target pixel in the first image to obtain its world coordinates in the 3D scene, thereby determining its target position in the 3D scene.

[0119] Step 3033: Perform ray tracing based on the target location and the marker to obtain the intersection point of the ray with the three-dimensional scene.

[0120] After determining the target pixel's location in the 3D scene, the corresponding ray can be identified based on the pixel's identifier. For example, if the pixel's identifier is 0, the ray to be tracked is determined to be a reflected ray; if the pixel's identifier is 1, the ray to be tracked is determined to be a refracted ray. By tracing the path of the ray corresponding to the target pixel in the 3D scene, the intersection points of the ray with other objects in the 3D scene after it leaves the target location can be obtained. For example, if tracing reveals that a reflected ray from the floor intersects with the chassis of a car in the 3D scene, the intersection point can be determined to be the car chassis.

[0121] It should be understood that during ray tracing, in order to detect the intersection points of rays with objects, each ray needs to be intersected with all objects in the scene, such as complex objects like spheres and triangles. In related technologies, this is done by traversing every object in the scene and finding the object closest to the ray's origin as the intersection point. In complex 3D scenes with a large number of objects, ray tracing can be very time-consuming. In reality, most objects are very far from the rays, and only a small fraction of them are likely to intersect with them, so it's unnecessary to traverse all objects in the scene.

[0122] Based on this, in one possible example, the electronic device can acquire an acceleration structure based on the three-dimensional scene, which is used to quickly find the intersection of light rays; then, the electronic device performs ray tracing through the acceleration structure according to the target position and the identifier to obtain the intersection of the light rays with the three-dimensional scene.

[0123] For example, the acceleration structure may include structures such as a bounding volume hierarchy (BVH), a uniform grid, or a k-dimensional tree (kd-tree). This embodiment does not specifically limit the acceleration structure. By utilizing a spatial partitioning structure, this acceleration structure can quickly remove irrelevant objects, thus finding the nearest intersection point by traversing only a very small subset.

[0124] For example, the BVH mentioned above can be simply described as enclosing an object in a bounding box. Before a ray intersects with an object in the scene, it first intersects with this bounding box. If the ray does not touch the bounding box, it means that the ray will definitely not intersect with an object in the bounding box; if the ray touches the bounding box, then it calculates whether the ray intersects with an object in the bounding box.

[0125] Generally, 3D scenes contain many objects. Using BVH essentially means using a binary tree structure to manage these objects. (See also...) Figure 5 , Figure 5 This is a schematic diagram of a BVH provided in an embodiment of this application. Figure 5 As shown, different objects are bounded by bounding boxes of varying sizes, forming corresponding binary tree structures. When detecting whether each ray intersects with an object in the scene, the binary tree is traversed sequentially. For example, if a ray is detected not to intersect with bounding box B in the binary tree, it means that the ray will definitely not intersect with the four objects in bounding box B. Therefore, the step of detecting whether the ray intersects with the four objects in bounding box B can be omitted, and only the step of detecting whether the ray intersects with the two objects in bounding box C can be performed.

[0126] Step 3034: Update the color of the target pixel based on the color of the intersection point.

[0127] After tracing the intersection of the light rays with the 3D scene, the electronic device can calculate the color of the intersection point, and then, based on the ray tracing method, merge the color of the intersection point with the original color of the target pixel to update the new color of the target pixel.

[0128] For example, see Figure 6 , Figure 6 This is a schematic diagram of a reflection scene provided in an embodiment of this application. When the target object is the floor, the ray tracing method is reflection, and the ray intersection point is the car body shell; then, if the color of the car body shell is red and the color of the target pixel is light yellow, the color of the target pixel can be updated based on the red color of the car body shell and the original light yellow color of the target pixel.

[0129] In real-time ray tracing, due to computational limitations, the number of rays sampled per pixel is typically limited. However, this usually introduces noise and affects rendering quality. The main solution is to cache historical image frames and project and accumulate them with the current image frame to increase the number of sampling points. In this embodiment, the amount of shading computation at ray intersections can be reduced by utilizing cached historical image frames and the current image frame.

[0130] In one possible embodiment, the electronic device can calculate the projection of the intersection point onto the image based on its position in the 3D scene, i.e., by coordinate transformation, calculate the corresponding pixel on the 2D image of the intersection point in the 3D scene. If the intersection point has a corresponding projected pixel on the first or third image, the color of the target pixel is updated according to the color of the projected pixel; if the intersection point does not have a corresponding projected pixel on the first or third image, the color of the intersection point is calculated, and the color of the target pixel is updated according to the color of the intersection point; wherein, the third image is the frame preceding the second image. It should be understood that the electronic device can obtain continuous video frames by displaying the rendered images frame by frame on the screen, with the second image being the image currently to be displayed on the screen, and the third image being the frame preceding the second image.

[0131] In simple terms, during image rendering on an electronic device, the device does not render all objects in the 3D scene in real time. The device typically renders only the object to be displayed on the screen at any given time, resulting in a rendered image. If an intersection point has already been rendered and displayed in the previous frame (i.e., the third image), or the current frame (i.e., the first image), its color can be determined based on the color of the corresponding pixel in either the previous or current frame. That is, by reusing the colors of pixels from the previous or current frame, the color of the intersection point is obtained, avoiding recalculation and reducing computational complexity.

[0132] For example, see Figure 7 , Figure 7 This is a schematic diagram illustrating how to determine the color of an intersection point, as provided in an embodiment of this application. Figure 7 As shown, the image on the left is the previous frame, and the image on the right is the current frame. In both the previous and current frames, the electronic device rendered the car's body, showing the side profile of the car.

[0133] For the floor area marked by the rectangle in the current frame image on the right, during ray tracing processing of this area, the intersection of the reflected rays from the floor and the scene represents the side of the car body. Therefore, the electronic device needs to determine the color of the car body's side. At this point, the electronic device can calculate the projected pixels of the car body's side on the image. Since the corresponding projected pixels of the car body's side can be found in both the previous and current frames, the electronic device can reuse the colors of these projected pixels to determine the color of the intersection of the reflected rays and the scene, ultimately achieving the shading and rendering of the floor.

[0134] As shown in the blue ellipse in the current frame image on the right, the floor area is where, during ray tracing, the intersection of reflected rays from the floor and the scene represents the car's chassis. Therefore, the electronic device needs to determine the color of the car's chassis. However, since there are no projected pixels corresponding to the car chassis in either the previous or current frame, the electronic device needs to recalculate the color of the ray tracing intersection (i.e., the car chassis).

[0135] To facilitate understanding, the following will describe in detail the process of ray tracing in electronic devices, using a processing flowchart.

[0136] Please refer to Figure 8 , Figure 8 This is a schematic diagram illustrating a ray tracing process provided in an embodiment of this application. Figure 8 As shown, the ray tracing process includes the following steps.

[0137] Step 801: Calculate the reflected / refracted rays.

[0138] After the electronic device identifies the target pixel, it can acquire the corresponding G-buffer to obtain the position vector, normal vector, and identifier stored in the G-buffer. The position vector determines the target pixel's location, the normal vector determines the direction of the reflected / refracted light ray, and the identifier determines whether the ray is reflected or refracted. Based on the information stored in the G-buffer, the electronic device can calculate the reflected / refracted light ray originating from the target pixel's location.

[0139] Step 802: Calculate the intersection of the light rays and the scene based on the acceleration structure.

[0140] The acceleration structure can include, but is not limited to, the aforementioned BVH, Uniform grid, or kd-tree structures.

[0141] Step 803: Determine whether there is an intersection between the light and the scene.

[0142] Step 804: If the light does not intersect with the scene, then display the color of the current pixel.

[0143] If the calculation shows that the light ray does not intersect with the scene, it means that the target pixel's location does not produce reflection or refraction. Therefore, no further rendering processing is needed for the target pixel. The color of the target pixel can be rendered based on the color vector stored in the G-buffer of the current image frame.

[0144] Step 805: If the light rays intersect with the scene, then perform projection calculations on the intersection points.

[0145] Specifically, the electronic device can calculate the projection of the intersection point onto the image based on the position of the intersection point in the three-dimensional scene, that is, calculate the corresponding pixel point of the intersection point in the 3D scene on the two-dimensional image through coordinate transformation.

[0146] Step 806: Determine whether the projected pixel is located in the current image frame.

[0147] The electronic device determines whether there is a projected pixel corresponding to the intersection point in the current image frame.

[0148] Step 807: If the projected pixel is located in the current image frame, then the color of the projected pixel is used as the color of reflection / refraction.

[0149] If it is determined that a projection pixel exists corresponding to an intersection point in the current image frame, the color of that projection pixel is used as the reflected / refracted color. This means the color of the projection pixel is fused with the color of the target pixel to update the target pixel's color. The color of the projection pixel can be obtained based on the G-buffer in the current image frame.

[0150] Step 808: If the projected pixel is not located in the current image frame, then determine whether the projected pixel is located in the previous image frame.

[0151] The electronic device determines whether there is a projected pixel corresponding to the intersection point in the previous image frame.

[0152] Step 809: If the projected pixel is located in the previous image frame, then the color of the projected pixel is used as the color of reflection / refraction.

[0153] If it is determined that a corresponding projected pixel exists at an intersection point in the previous image frame, the color of that projected pixel is used as the reflected / refracted color. This means the color of the projected pixel is fused with the color of the target pixel to update the target pixel's color. The color of the projected pixel can be obtained based on the G-buffer from the previous image frame.

[0154] Step 810: If the projected pixel is not located in the previous image frame, recalculate the color of the intersection as the color of reflection / refraction.

[0155] If it is determined that there is no corresponding projected pixel in the previous image frame, the color of the intersection point is recalculated to serve as the color of reflection / refraction. This recalculated color of the intersection point is then fused with the color of the target pixel to update the target pixel's color.

[0156] The above describes the process of ray tracing processing in electronic devices. To further reduce the computational load in the rendering process, the following describes the process of rasterization processing of the data to be rendered by an electronic device provided in this embodiment.

[0157] In one possible example, see Figure 9 , Figure 9 This is a schematic diagram illustrating a rasterization process for data to be rendered, provided as an embodiment of this application. Figure 9 As shown, step 302 may specifically include:

[0158] 3021, Perform no-light rendering on the data to be rendered to obtain the fourth image.

[0159] In this embodiment, the electronic device uses deferred rendering to perform rasterization processing. After obtaining the data to be rendered, the electronic device can perform preliminary rendering of the data, i.e., rendering without lighting calculations, to obtain the fourth image. Specific steps can be found in the description of step 302 above, and will not be repeated here.

[0160] 3022. Based on the attribute information of the data to be rendered, a geometric buffer corresponding to the pixel in the fourth image is obtained, and the geometric buffer is used to store the attribute parameters corresponding to the pixel.

[0161] In the no-light rendering stage, the electronic device can generate a G-buffer corresponding to each pixel in the fourth image based on the attribute information of the data to be rendered. The G-buffer can store attribute parameters such as the position vector, normal vector, and color vector corresponding to each pixel.

[0162] In one possible embodiment, the electronic device can generate the G-buffer in a variety of ways.

[0163] First, before generating the G-buffer corresponding to the pixels in the fourth image, the electronic device can determine the corresponding object to be rendered, that is, the object to be displayed in the fourth image. Then, based on the specific information of the object to be rendered, it determines the resolution for generating the G-buffer. Finally, based on this resolution, it generates the G-buffer corresponding to the object to be rendered, thereby obtaining the G-buffer corresponding to the pixels in the fourth image.

[0164] In case one, if the object to be rendered in the fourth image is the target object, then a first G-buffer corresponding to the object to be rendered is generated based on the attribute information of the object to be rendered and the first resolution. The first G-buffer is used to store color attribute parameters.

[0165] If the object to be rendered is a target object, then the pixels rendered based on the target object are located in the region of interest (ROI) of the fourth image. Therefore, for the object to be rendered, the electronic device can generate a G-buffer corresponding to the object based on a first resolution, where the first resolution can be the same as the resolution of the fourth image itself; that is, the electronic device generates the G-buffer corresponding to the object at full resolution. In this way, a corresponding G-buffer is generated for each pixel of the object to be rendered in the fourth image. For example, if the resolution of the fourth image is 1000*1000, the electronic device generates the first G-buffer corresponding to the object to be rendered based on the 1000*1000 resolution.

[0166] In the second scenario, if the object to be rendered in the fourth image is located in the surrounding area of ​​the target object, then a second G-buffer corresponding to the object to be rendered is generated based on the attribute information of the object to be rendered and the second resolution. The second G-buffer is used to store color attribute parameters.

[0167] If the object to be rendered is located in the surrounding area of ​​the target object, then the pixels rendered based on the object can be considered to be located in the surrounding area of ​​the region of interest. Therefore, for the object to be rendered, the electronic device can generate a second G-buffer corresponding to the object based on a second resolution, where the second resolution is smaller than the first resolution. For example, the second resolution can be 3 / 4 of the first resolution; that is, if the first resolution is 1000*1000, the second resolution can be 750*750. Furthermore, the second resolution can also be any other specific value smaller than the first resolution; no specific limitation is made on the second resolution here.

[0168] Specifically, determining whether an object to be rendered is located in the surrounding area of ​​a target object can be done by determining the distance between the pixels corresponding to the object to be rendered and the pixels corresponding to the target object. For example, if the distance between the pixels corresponding to the object to be rendered and the pixels corresponding to the target object is less than a first preset threshold, then it can be determined that the object to be rendered is located in the surrounding area of ​​the target object. This first preset threshold can be, for example, 10 pixels; that is, when the distance between the pixels corresponding to the object to be rendered and the pixels corresponding to the target object is less than 10 pixels, it can be determined that the object to be rendered is located in the surrounding area of ​​the target object.

[0169] Scenario 3: If the object to be rendered in the fourth image is located in the background area, then a third G-buffer corresponding to the object to be rendered is generated based on the attribute information of the object to be rendered and the third resolution. The third G-buffer is used to store color attribute parameters.

[0170] If the object to be rendered is located in the background area of ​​the target object, then the pixels rendered based on the object can be considered to be located in the background area. Therefore, for the object to be rendered, the electronic device can generate a third G-buffer corresponding to the object based on a third resolution, where the third resolution is smaller than the second resolution. For example, the third resolution can be half of the first resolution; that is, if the first resolution is 1000*1000, the second resolution can be 500*500. Furthermore, the third resolution can also be any other specific value smaller than the first resolution; no specific limitations are imposed on the third resolution here.

[0171] It should be understood that the distance between the pixels corresponding to the object to be rendered and the pixels corresponding to the target object is used to determine whether the object to be rendered is located in the background region. For example, if the distance between the pixels corresponding to the object to be rendered and the pixels corresponding to the target object is less than a second preset threshold, it can be determined that the object to be rendered is located in the surrounding area of ​​the target object. This second preset threshold can be, for example, 50 pixels; that is, if the distance between the pixels corresponding to the object to be rendered and the pixels corresponding to the target object is greater than 50 pixels, it can be determined that the object to be rendered is located in the background region.

[0172] The data to be rendered includes the object to be rendered. The first resolution is greater than the second resolution, the second resolution is greater than the third resolution, and the first, second, and third G-buffers are all used to store color attribute parameters, i.e., color vectors. In other words, the electronic device only selects the appropriate resolution from the three resolutions mentioned above based on the specific information of the object to be rendered when generating the G-buffer corresponding to the object for storing color attribute parameters.

[0173] The above three scenarios describe how electronic devices generate G-buffers for storing color attribute parameters. The following will describe how electronic devices generate G-buffers for storing other attribute parameters.

[0174] Case 4: Generate a fourth G-buffer corresponding to the object to be rendered based on the attribute information of the object to be rendered and the fourth resolution. The attribute parameters stored in the fourth G-buffer are not color attribute parameters, and the fourth resolution is smaller than the first resolution.

[0175] In simple terms, when the electronic device generates a G-buffer that is not used to store color attribute parameters, it uses a fourth resolution to generate the G-buffer. For example, when the electronic device generates a G-buffer used to store position attribute parameters or normal vector attribute parameters, it uses a fourth resolution to generate the G-buffer corresponding to the object to be rendered, regardless of whether the object to be rendered is the target object or located in the surrounding area of ​​the target object.

[0176] It should be understood that the higher the resolution used to generate the G-buffer, the more G-buffers are generated within the same image area, resulting in higher rendering accuracy. Conversely, the lower the resolution used to generate the G-buffer, the fewer G-buffers are generated within the same image area, resulting in lower rendering accuracy. For example, for an image composed of 1000*1000 pixels, generating the G-buffer at a resolution of 1000*1000 will yield 1000*1000 G-buffers; generating it at a resolution of 500*500 will yield 500*500 G-buffers. In other words, for the target object in the image, a higher resolution can be used to generate the G-buffer to ensure the rendering accuracy of the target object; for the surrounding and background areas of the target object, a lower resolution can be used to generate the G-buffer, reducing the computational load on the electronic device, saving storage space, and lowering the input / output (I / O) bandwidth requirements of the electronic device.

[0177] 3023, Perform illumination calculations on the pixels in the fourth image based on the G-buffer to obtain the first image.

[0178] After obtaining the G-buffer corresponding to each pixel in the fourth image, the electronic device can perform lighting calculations on the fourth image based on the attribute parameters such as position vector, normal vector, and color vector stored in the G-buffer to obtain the rendered first image.

[0179] It is understandable that when an electronic device generates a G-buffer at a lower resolution, some pixels in the fourth image may not have a corresponding G-buffer, meaning the electronic device does not store the attribute parameters corresponding to these pixels. In this case, during the illumination calculation of the fourth image, the electronic device can obtain the attribute parameters corresponding to the pixels that do not have a G-buffer through interpolation, thereby realizing the illumination calculation for these pixels.

[0180] Please refer to Figure 10 , Figure 10 This is a schematic diagram illustrating a process for generating a G-buffer based on adaptive resolution, provided as an embodiment of this application. Figure 10 As shown, the process of generating a G-buffer based on adaptive resolution includes the following steps.

[0181] Step 1001: Determine whether the G-buffer to be generated is used to store color attribute parameters.

[0182] Since electronic devices need to generate G-buffers for storing different attribute parameters, during the process of generating the G-buffer corresponding to the object to be rendered, the electronic device can first determine whether the G-buffer to be generated is used to store color attribute parameters.

[0183] Step 1002: If the G-buffer to be generated is not used to store color attribute parameters, then generate the G-buffer at 1 / 2 resolution.

[0184] When the G-buffer to be generated is not used to store color attribute parameters, but rather to store position attribute parameters or normal vector attribute parameters, the electronic device can generate the G-buffer at a lower resolution. For example, the electronic device can generate the G-buffer at half the original resolution of the image to be generated (i.e., the half resolution mentioned above). For instance, if the original resolution of the image to be generated is 1000*1000, half resolution is 500*500 resolution.

[0185] Step 1003: If the G-buffer to be generated is used to store color attribute parameters, then determine whether the object to be rendered is the target object.

[0186] One way electronic devices determine whether an object to be rendered is a target object is by judging whether the object has a corresponding identifier. If the object has a corresponding identifier, it can be determined that the object is a target object; if the object does not have a corresponding identifier, it can be determined that the object is not a target object.

[0187] Step 1004: If the object to be rendered is the target object, then generate a G-buffer at full resolution.

[0188] If the object to be rendered is a target object, the pixels rendered based on the target object are located in the region of interest (ROI) of the image. Therefore, for the object to be rendered, there is no need to reduce the rendering precision. Here, full resolution refers to the original resolution of the image to be generated; that is, when the electronic device generates the G-buffer for the object to be rendered, it generates the G-buffer at the normal resolution to ensure the rendering precision of the target object.

[0189] Step 1005: If the object to be rendered is not the target object, then determine whether the object to be rendered is located in the surrounding area of ​​the target object.

[0190] Step 1006: If the object to be rendered is located in the surrounding area of ​​the target object, then generate a G-buffer at 3 / 4 resolution.

[0191] If the object to be rendered is located in the surrounding area of ​​the target object, then the pixels rendered based on the object can be considered to be located in the surrounding area of ​​the region of interest. Therefore, for the object to be rendered, the electronic device can slightly reduce the rendering precision to reduce the computational load on the electronic device. For example, the electronic device can generate the G-buffer at 3 / 4 of the original resolution of the image to be generated (i.e., the 3 / 4 resolution mentioned above). For example, if the original resolution of the image to be generated is 1000*1000, then 1 / 2 resolution is 750*750 resolution.

[0192] Step 1007: If the object to be rendered is not located in the surrounding area of ​​the target object, then determine whether the object to be rendered is located in the background area.

[0193] Step 1008: If the object to be rendered is located in the background area, then generate the G-buffer at 1 / 2 resolution.

[0194] If the object to be rendered is located in the background region, then the pixels rendered based on the object can be considered to be located in the background region with low attention. Therefore, for the object to be rendered, the electronic device can further reduce the rendering precision to reduce the computational load on the electronic device. For example, the electronic device can generate the G-buffer at half the original resolution of the image to be generated to further reduce the computational load on the electronic device.

[0195] The above describes the process of rasterization of the data to be rendered by an electronic device. In order to further reduce the amount of computation in the rendering process, the following will describe the process of an electronic device acquiring the data to be rendered provided in this embodiment.

[0196] In one possible embodiment, the electronic device acquiring the data to be rendered may include: the electronic device acquiring 3D scene data and a fifth image sent by the server, wherein the fifth image is a rendered background image. That is, the server can render the background area in the 3D scene and send the rendered background image to the electronic device via the network. In this way, the electronic device can render only the non-background areas of the 3D scene and merge the rendered image with the background image sent by the server to obtain a complete and rendered image.

[0197] The background image sent by the server refers to an image that only includes the background area, meaning it only includes distant backgrounds. For example, the server can render backgrounds such as the sky, mountains, the sea, or distant skyscrapers to obtain the corresponding background image.

[0198] For example, in one possible scenario, an electronic device may be running a game application. The server can render the background area in the 3D scene in real time, obtain a background image, and send the background image to the electronic device. While the electronic device is running the game application, it renders the non-background areas in the 3D scene and combines them with the background image sent by the server to obtain a rendered image, which is then displayed on the screen.

[0199] Optionally, when the game application running on the electronic device is a multi-user online game, the background image rendered by the server can be distributed to multiple different electronic devices. Each electronic device performs personalized rendering based on the content it needs to display, resulting in different images on the screen.

[0200] For details, please refer to [link / reference]. Figure 11 , Figure 11 This is a schematic diagram illustrating a process for rendering and distributing a background image on a server, as provided in an embodiment of this application. Figure 11 As shown, the process of rendering and distributing a background image on the server includes the following steps.

[0201] Step 1101: The server renders the background area in the 3D scene to obtain the background image.

[0202] In one possible example, the server renders a background area in a 3D scene, generating six textures. These six textures can be used to compose a cube map as the background image. Simply put, a cube map is a texture containing six 2D textures, each forming a face of the cube, thus creating a textured cube. This cube map can encompass all background areas in the 3D scene; that is, objects in the 3D scene can be considered to be wrapped within this cube map. For example, see [link to relevant documentation]. Figure 11 The background image generated by the server consists of 6 textures, and each texture can form a face of a cube, thus forming a cube texture. This cube texture includes background areas such as distant skyscrapers, lights, and the night sky. Objects in the 3D scene that are not part of the background are all enclosed in this cube texture.

[0203] Optionally, when the light source of the 3D scene changes or the background area of ​​the 3D scene is updated, the server can re-render the changed 3D scene to obtain a new background image, so as to update the background image in real time.

[0204] Step 1102: The server calculates the diffuse irradiance corresponding to the background image.

[0205] During the rendering process, electronic devices may need to perform ray tracing to determine if there are intersections between light rays and objects in the background area. Therefore, in addition to obtaining the background image corresponding to the background area, the electronic device may also need to obtain the diffuse irradiance corresponding to the background objects in the background image in order to enable the electronic device to perform ray tracing processing.

[0206] Specifically, the irradiance corresponding to the background image can be calculated based on the reflection equation, since diffuse reflection k d and mirror k s They are independent of each other, therefore the reflection equation can be integrated into two parts as shown in Equation 1:

[0207]

[0208] Where Lo represents the irradiance reflected from point p when viewed from the direction ωo. Li() represents the radiance at a point through an infinitesimally small solid angle ωi, which can be considered as the incident direction vector ωi. Li(p,wi) represents the incident light intensity at point p. wi represents the incident ray vector of the solid angle. N represents the normal. (wi*n) represents the incident light attenuation caused by the incident angle, and the multiplication symbol here represents the dot product. ∫Ω....dwi represents the hemispherical integral over the incident ray vector of the incident direction hemisphere. k s k represents the mirror scaling factor. d Represents the diffuse scaling factor, and k s +k d Less than or equal to 1.

[0209] Step 1103: The server calculates the pre-filtered ambient light map.

[0210] In this embodiment, since the electronic device performs ray tracing on the target object in a specific way, it mainly focuses on the specular reflection part of the reflection equation, that is, the second half of the right side of Equation 1. By transforming the second half of the right side of Equation 1, we can obtain Equation 2:

[0211]

[0212] Here, fr is called the reflectance equation, and the bidirectional reflectance distribution function (BRDF) is generally used. Since the integral term depends not only on the input ray w... i It also depends on the output light w oTherefore, it is impossible to sample the cube map texture using two direction vectors. In this implementation, a partitioned summation approximation method is used to divide the pre-calculation into two separate parts for solution, and then the two parts are combined to obtain the pre-calculation result. Specifically, the partitioned summation approximation method divides the pre-calculation into two separate parts for solution as shown in Equation 3:

[0213]

[0214] The pre-calculation results can be as follows: Figure 12 As shown, Figure 12 This is a schematic diagram of a pre-filtered ambient light map provided in an embodiment of this application. The first part of the convolution calculation is called the pre-filtered ambient light map, which can be pre-calculated. In this embodiment, the electronic device can obtain different levels of filtered maps through the roughness value.

[0215] Step 1104: The server calculates the lookup table (LUT) corresponding to the integral map of the BRDF.

[0216] The latter part of Formula 3, which is the BRDF component of the specular reflection integral, can also be pre-calculated to obtain the pre-calculated result, i.e., the LUT corresponding to the BRDF integral map. This allows electronic devices to operate under given surface roughness and the angle nw between the incident ray and the normal. i Find the corresponding BRDF integral map in the LUT.

[0217] Step 1105: The server sends a background image and related data to the electronic device.

[0218] After the server renders the background image and calculates the corresponding pre-calculated results, it can send the background image and the corresponding pre-calculated results to the electronic device.

[0219] Step 1106: The electronic device performs image rendering based on the background image and related data to obtain the image to be displayed.

[0220] Finally, the electronic device can perform preliminary rendering based on the background image to obtain the first image mentioned above, and perform ray tracing processing based on relevant data and the first image to obtain the second image for display.

[0221] For ease of understanding, the edge-cloud combined image rendering method provided in the embodiments of this application will be described below with reference to the accompanying drawings. See also... Figure 13 , Figure 13 This is a flowchart illustrating an edge-cloud combined image rendering method provided in an embodiment of this application. Figure 13 As shown, the image rendering method combining edge and cloud computing includes the following steps.

[0222] Step 1301: The cloud server renders the background area in the 3D scene to obtain the rendered ambient light map.

[0223] In this embodiment, the cloud server can determine the background area in the 3D scene, and then render the background area based on light sources such as static and dynamic light sources to obtain an ambient light map. This ambient light map can be the cube map described above. Optionally, when the light source in the 3D scene changes or the background area of ​​the 3D scene is updated, the server can re-render the changed 3D scene to obtain a new ambient light map, thereby updating the ambient light map in real time.

[0224] Step 1302: The electronic device performs rasterization processing on the rendering data of the 3D scene and the ambient light map sent by the server to obtain a first image and a G-buffer that stores the attribute parameters of the pixels in the first image.

[0225] After obtaining the ambient light map from the server, the electronic device can perform rasterization processing based on the local 3D scene rendering data and the received ambient light map to obtain a first image and a G-buffer. The G-buffer stores the attribute parameters of the pixels in the first image, such as position vector attribute parameters, color vector attribute parameters, normal vector attribute parameters, and other attribute parameters.

[0226] Step 1303: The electronic device constructs an acceleration structure.

[0227] To accelerate the ray tracing process performed by electronic devices, the devices can construct acceleration structures based on 3D scenes. These acceleration structures can include, for example, BVH, Uniform grid, or kd-tree structures.

[0228] Step 1303 can be executed before or after step 1302. This embodiment does not impose any specific restrictions on the execution order of step 1303.

[0229] Step 1304: The electronic device performs ray tracing processing based on the first image and the ambient light map to obtain the second image.

[0230] After the accelerated structure is constructed, the electronic device can perform ray tracing based on the first image and the ambient light map, and find the intersection points of the rays with the 3D scene. Then, the color of the corresponding pixel in the first image is updated based on the color of the intersection point to obtain the second image.

[0231] Step 1305: The electronic device denoises the second image to obtain the image to be displayed.

[0232] During ray tracing, electronic devices typically limit the number of sampled rays per pixel, introducing noise into the second image. Therefore, electronic devices can use denoising algorithms to denoise the second image, obtaining a denoised image (i.e., the image to be displayed), and then display this denoised image on the screen. Denoising algorithms can include, for example, temporal denoising algorithms.

[0233] Please refer to Figure 14 , Figure 14 This is a schematic diagram of a hybrid rendering pipeline provided in an embodiment of this application.

[0234] In module 1401, the vertex shader and fragment shader perform the first rendering process, namely rasterization. The vertex shader and fragment shader perform preliminary rendering of the data to be rendered to obtain the first image. That is, the vertex shader and fragment shader in module 1401 perform the above-mentioned step 1302 to obtain the first image and a G-buffer that stores the attribute parameters of the pixels in the first image. Refer to Figure 15(a), which is a first image after rasterization provided in an embodiment of this application. As shown in Figure 15(a), in the first image obtained after rasterization, there is no reflection of the vehicle on the floor, and the reflection effect of the floor is not rendered in the first image.

[0235] In module 1402, the electronic device generates and stores a G-buffer corresponding to the first image. This G-buffer can store information such as the identifier, world coordinates, normal vector, and color of the pixels in the first image. The world coordinates of the pixels in the first image can be saved as a screen-space texture, or they can be obtained by using the depth mapping built into the rendering pipeline combined with the inverse of the view projection matrix.

[0236] In module 1403, the CPU or GPU of the electronic device builds an acceleration structure and acquires global vertex information for shading in ray tracing processing.

[0237] In module 1404, a second rendering process, namely ray tracing, is executed by a compute shader or fragment shader. Ray tracing effects such as reflection, refraction, or shadows of local objects are achieved through the compute shader or fragment shader. Referring to Figure 15(b), which shows a second image after ray tracing processing according to an embodiment of this application, as shown in Figure 15(a), the second image obtained after ray tracing processing contains the reflection of a vehicle on the floor, and the reflection effect of the floor is rendered on the second image.

[0238] Optionally, in the case where the electronic device implements ray tracing processing based on computational shaders, in module 1405, the rendered image is further processed by a full-screen vertex shader and a full-screen fragment shader to obtain an image for display on the screen.

[0239] Based on the embodiments corresponding to Figures 1 to 15(b), in order to better implement the above-described solutions of this application, related equipment for implementing the above-described solutions is also provided below. For details, please refer to... Figure 16 , Figure 16 This is a schematic diagram of the structure of an electronic device 1600 provided in an embodiment of this application. The electronic device 1600 includes an acquisition unit 1601 and a processing unit 1602. The acquisition unit 1601 is used to acquire data to be rendered; the processing unit 1602 is used to perform rasterization processing on the data to be rendered to obtain a first image; the processing unit 1602 is also used to perform ray tracing processing on a target object in the first image to obtain a second image; wherein the target object has an identifier, and the identifier is used to mark the object to be subjected to ray tracing processing.

[0240] In one possible implementation, the identifier is also used to mark the ray tracing processing method.

[0241] In one possible implementation, the ray tracing processing method includes reflection, refraction, shadowing, or caustics.

[0242] In one possible implementation, the processing unit 1602 is further configured to: perform ray tracing on the target object in the first image according to the identifier of the target object to obtain a ray tracing result; and update the color of the target object in the first image according to the ray tracing result to obtain a second image.

[0243] In one possible implementation, the processing unit 1602 is further configured to determine a target pixel in the first image, the target pixel having the identifier, and the target object including one or more target pixels; the acquisition unit 1601 is further configured to acquire the target position of the target pixel in a three-dimensional scene; the processing unit 1602 is further configured to perform ray tracing based on the target position and the identifier to obtain the intersection point of the ray and the three-dimensional scene; the processing unit 1602 is further configured to update the color of the target pixel based on the color of the intersection point.

[0244] In one possible implementation, the processing unit 1602 is further configured to: calculate the projection of the intersection point onto an image based on the position of the intersection point in the three-dimensional scene; if the intersection point has a corresponding projected pixel on the first image or the third image, update the color of the target pixel according to the color of the projected pixel; if the intersection point does not have a corresponding projected pixel on the first image or the third image, calculate the color of the intersection point and update the color of the target pixel according to the color of the intersection point; wherein the third image is the previous frame of the second image.

[0245] In one possible implementation, the acquisition unit 1601 is further configured to acquire an acceleration structure, which is obtained based on the three-dimensional scene; the processing unit 1602 is further configured to perform ray tracing through the acceleration structure according to the target position and the identifier, to obtain the intersection point of the ray with the three-dimensional scene.

[0246] In one possible implementation, the processing unit 1602 is further configured to: perform unlit rendering on the data to be rendered to obtain a fourth image; obtain a geometric buffer corresponding to the pixels in the fourth image based on the attribute information of the data to be rendered, the geometric buffer being used to store the attribute parameters corresponding to the pixels; and perform lighting calculations on the pixels in the fourth image based on the geometric buffer to obtain the first image.

[0247] In one possible implementation, the processing unit 1602 is further configured to: if the object to be rendered in the fourth image is the target object, generate a first geometric buffer corresponding to the object to be rendered based on the attribute information of the object to be rendered and a first resolution; if the object to be rendered in the fourth image is located in the surrounding area of ​​the target object, generate a second geometric buffer corresponding to the object to be rendered based on the attribute information of the object to be rendered and a second resolution; if the object to be rendered in the fourth image is located in the background area, generate a third geometric buffer corresponding to the object to be rendered based on the attribute information of the object to be rendered and a third resolution; wherein the data to be rendered includes the object to be rendered, the first resolution is greater than the second resolution, the second resolution is greater than the third resolution, and the first geometric buffer, the second geometric buffer, and the third geometric buffer are used to store color attribute parameters.

[0248] In one possible implementation, the processing unit 1602 is further configured to: generate a fourth geometric buffer corresponding to the object to be rendered based on the attribute information of the object to be rendered and the fourth resolution, wherein the attribute parameters stored in the fourth geometric buffer are not color attribute parameters; and the fourth resolution is less than the first resolution.

[0249] In one possible implementation, the acquisition unit 1601 is further configured to acquire three-dimensional scene data and a fifth image sent by the server, wherein the fifth image is a rendered background image.

[0250] In one possible implementation, the data to be rendered includes the target object and the material parameters of the target object; the processing unit 1602 is further configured to determine the identifier of the target object based on the material parameters of the target object.

[0251] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0252] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0253] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0254] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0255] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

Claims

1. An image processing method, characterized by, The method comprises the following steps: acquiring to-be-rendered data, the to-be-rendered data comprising three-dimensional scene data and a rendered background image sent by a server, the background image comprising a background region in the three-dimensional scene; performing rasterization processing on the to-be-rendered data to obtain a first image; performing ray tracing processing on a target object in the first image to obtain a second image; the ray tracing processing on the target object in the first image to obtain the second image comprises: determining a target pixel in the first image, the target pixel having an identifier, the target object comprising one or more target pixels; acquiring a target position of the target pixel in the three-dimensional scene; performing ray tracing according to the target position and the identifier to obtain a ray intersection point of the three-dimensional scene; calculating a projection of the intersection point on the image according to the position of the intersection point in the three-dimensional scene; if the intersection point has a corresponding projection pixel on the first image or a third image, updating the color of the target pixel according to the color of the projection pixel; if the intersection point does not have a corresponding projection pixel on the first image or the third image, calculating the color of the intersection point and updating the color of the target pixel according to the color of the intersection point to obtain the second image; wherein the identifier is used to mark an object to be executed for ray tracing processing, and the third image is a previous frame image of the second image.

2. The image processing method of claim 1, wherein, The identifier is also used to mark a ray tracing processing mode.

3. The image processing method of claim 2, wherein, The ray tracing processing mode comprises reflection, refraction, shadow or caustics.

4. The image processing method according to any one of claims 1 to 3, characterized in that, The ray tracing processing on the target object in the first image comprises: acquiring a position of the target object in the three-dimensional scene; performing ray tracing processing according to the position of the target object in the three-dimensional scene to obtain a ray tracing result; updating the color of the target object in the first image according to the ray tracing result to obtain the second image.

5. The image processing method of claim 1, wherein, The ray tracing according to the target position and the identifier to obtain the ray intersection point of the three-dimensional scene comprises: acquiring an acceleration structure, the acceleration structure being obtained based on the three-dimensional scene; performing ray tracing through the acceleration structure according to the target position and the identifier to obtain the ray intersection point of the three-dimensional scene.

6. The image processing method according to any one of claims 1 to 3, characterized by, The rasterization processing on the to-be-rendered data to obtain the first image comprises: performing non-lighting rendering on the to-be-rendered data to obtain a fourth image; obtaining a geometric buffer corresponding to a pixel in the fourth image according to attribute information of the to-be-rendered data, the geometric buffer being used to store attribute parameters corresponding to the pixel; performing lighting calculation on the pixel in the fourth image according to the geometric buffer to obtain the first image.

7. The image processing method of claim 6, wherein, The obtaining of the geometric buffer corresponding to the pixel in the fourth image according to the attribute information of the to-be-rendered data comprises: if a to-be-rendered object in the fourth image is the target object, generating a first geometric buffer corresponding to the to-be-rendered object according to attribute information of the to-be-rendered object and a first resolution; if the object to be rendered in the fourth image is located in the peripheral region of the target object, a second geometric buffer corresponding to the object to be rendered is generated according to attribute information of the object to be rendered and the second resolution if the object to be rendered in the fourth image is located in the background region, a third geometric buffer corresponding to the object to be rendered is generated according to attribute information of the object to be rendered and the third resolution. The rendering data includes the object to be rendered, the first resolution is greater than the second resolution, the second resolution is greater than the third resolution, and the first geometric buffer, the second geometric buffer and the third geometric buffer are used to store color attribute parameters.

8. The image processing method of claim 7, wherein, The attribute information of the rendering data includes attribute information of the target object and attribute information of the material parameter of the target object. The method further includes: determining the identity of the target object according to the material parameter of the target object.

9. The image processing method according to any one of claims 1 to 3, characterized by, The electronic device includes a memory and a processor, the memory stores code, and the processor is configured to execute the code, when the code is executed, the electronic device executes the method of any one of claims 1 to 9. The computer readable instructions, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer readable instructions, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 9.

10. An electronic device, comprising: ​ 11. A computer readable storage medium characterized by, ​ 12. A computer program product, characterised in that, ​

Images