Image processing method and electronic device

Abstract

An image processing method and an electronic device, relating to the technical field of image rendering, and capable of reducing the amount of computation of a ray tracing algorithm and reducing the power consumption of an electronic device when the electronic device renders an image by means of a ray tracing algorithm. In a process of an electronic device running a first application, the method comprises: in response to a rendering event for a first image frame, using a first number of rays for rendering to obtain the first image frame; receiving a first event, wherein the first event is used for triggering the electronic device to adjust the view of the image frame, and the movement amplitude between the image frames before and after adjustment exceeds a first amplitude; and in response to a rendering event for a second image frame, using a second number of rays for rendering to obtain a third image frame, and processing the third image frame to obtain the second image frame, wherein the second number is less than the first number, andthe first image frame and the second image frame are two consecutive image frames of the first application.

Description

Image processing method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on August 31, 2023, with application number 202311138355.2 and invention name “A Method and Electronic Device for Image Processing”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of image rendering technology, and in particular to an image processing method and electronic device. Background Art

[0003] Improving the realism of rendered images has long been a major industry priority in gaming, virtual reality (VR), and augmented reality (AR). Consequently, ray tracing, a technology that can generate rendering results close to the real world based on real-world physics, has also garnered significant attention.

[0004] However, ray tracing is a heavy-duty technology. Implementing it in electronic devices like mobile phones and tablets results in high power consumption, making it difficult to sustainably use ray tracing for image rendering on these devices.

[0005] Summary of the Invention

[0006] The present application provides an image processing method and an electronic device, which can reduce the computational complexity of a ray tracing algorithm and reduce the power consumption of an electronic device when using the ray tracing algorithm to render an image.

[0007] In the first aspect, the present application provides an image processing method, which is applied to an electronic device, wherein the electronic device includes a first application (such as a game application, a VR application, etc.), and the electronic device uses a ray tracing algorithm to render the image frame of the first application. In the process of the electronic device running the first application, the method includes: responding to a rendering event of the first image frame, using a first number of rays to render to obtain the first image frame. Receiving a first event (such as an operation event of accelerating forward in a game application), the first event is used to trigger the electronic device to adjust the viewing angle of the image frame, and the amplitude of movement between the image frames before and after the adjustment exceeds the first amplitude. In response to the rendering event of the second image frame, using a second number of rays to render to obtain a third image frame, processing the third image frame to obtain a second image frame, and the second number is less than the first number. Wherein, the first image frame and the second image frame are two consecutive image frames of the first application.

[0008] To sum up, using the present application, before receiving a preset event, the electronic device can use a larger number of rays to render the image frame of the first application; after receiving the preset event, the electronic device can use a lower number of rays to render the image, and then further improve the image quality of the rendered image (such as improving resolution and reducing noise) to obtain the image frame of the first application.

[0009] On the one hand, electronic devices use fewer rays to render images, which can reduce the amount of calculation in the rendering process, reduce rendering power consumption, and improve rendering efficiency.

[0010] On the other hand, when the amplitude of motion between two frames of images is large, a proper reduction in image quality has little impact on the user's perception. At the same time, when an electronic device uses a ray tracing algorithm to render an image, the fewer the number of rays, the more difficult it is to guarantee the quality of the rendered image, and even if the image quality is improved later, it is difficult to achieve the effect of rendering with a large number of rays. Therefore, after detecting the first event, the electronic device uses a smaller number of rays to render the image and obtains the image frame of the first application after processing, which can reduce the amount of computation in the rendering process while reducing the impact on the user's perception.

[0011] In a possible design of the first aspect, the first image frame and the second image frame both have a first resolution, and the third image frame has a second resolution, which is lower than the first resolution. The first number is related to the number of pixels included in the first image frame at the first resolution, and the second number is related to the number of pixels included in the third image frame at the second resolution.

[0012] That is, electronic devices can reduce the number of rays required to render a frame of image by reducing the image resolution.

[0013] In a possible design manner of the first aspect, processing the third image frame to obtain the second image frame includes: performing upsampling processing on the third image frame to obtain the second image frame.

[0014] It is understandable that resolution is a significant factor affecting image clarity. Reducing image quality to reduce rendering power consumption is clearly unacceptable. Therefore, after rendering the third image, the electronic device may also perform upsampling on the third image to restore the image resolution (e.g., to the first resolution).

[0015] Furthermore, the electronic device may also perform sharpening processing after the upsampling processing to enhance the boundaries of the image.

[0016] In a possible design of the first aspect, different first events correspond to different second resolutions. After receiving the first event, the second resolution corresponding to the received first event is determined.

[0017] Different first events can cause different motion amplitudes. Accordingly, the electronic device can configure different resolution parameters for different first events, that is, different first events correspond to different resolutions. In this way, the electronic device can reduce the amount of light to different degrees according to different motion amplitudes.

[0018] In a possible design of the first aspect, before receiving the first event, the number of light rays passing through each pixel during rendering of a frame of image is a third number, and after receiving the first event, the number of light rays passing through each pixel during rendering of the frame of image is a fourth number, where the fourth number is less than the third number. The first number is related to the third number, and the second number is related to the fourth number.

[0019] That is, electronic devices can reduce the number of rays required to render a frame of image by reducing the number of rays passing through each pixel.

[0020] In a possible design of the first aspect, processing the third image frame to obtain the second image frame includes: performing noise reduction processing on the third image frame, and processing the noise-reduced third image frame to obtain the second image frame.

[0021] In particular, when reducing the number of rays passing through each pixel, the fewer rays passing through each pixel, the higher the probability of pixel color calculation failure. In other words, noise is more likely to appear. Therefore, after rendering the third image frame, the electronic device further performs noise reduction on the third image frame, significantly reducing noise in the image.

[0022] It should be noted that when the implementation method of reducing the image resolution and the implementation method of reducing the number of light rays passing through each pixel are used in combination, the electronic device can further perform upsampling processing on the third image to improve the image resolution after performing noise reduction processing on the third image to obtain the noise reduced third image.

[0023] In one possible design of the first aspect, denoising the third image frame includes: using at least one of an albedo map, a normal map, and a mask map, and the third image frame as input, and running a preset denoising model to obtain a denoised third image frame. The preset denoising model is capable of removing noise from the noisy image frame based on the at least one of the albedo map, the normal map, and the mask map, and the noisy image frame.

[0024] That is to say, the electronic device can further combine at least one of the albedo map, the normal map and the mask map on the basis of the third image to complete the noise reduction processing to improve the effect of the noise reduction processing.

[0025] In a possible design of the first aspect, the RGB color information of any pixel in the albedo map is the RGB color information of the first intersection of the first ray passing through the pixel and the three-dimensional scene. The RGB color information of any pixel in the normal map corresponds to the normal coordinates of the first intersection of the first ray passing through the pixel and the three-dimensional scene. The RGB color information of any pixel in the mask map corresponds to the number of effective light paths in the light paths corresponding to the fourth number of light rays passing through the pixel, where the effective light paths refer to the light paths that intersect with the light sources in the three-dimensional scene. The three-dimensional scene is a scene used to render the image frames of the first application.

[0026] In a possible design of the first aspect, different first events correspond to different fourth quantities.After receiving the first event, the fourth quantity corresponding to the received first event is determined.

[0027] That is to say, the electronic device can reduce the amount of light to different degrees according to different first events.

[0028] In one possible design of the first aspect, receiving the first event includes receiving an operation event to move a character in the image frame, particularly an operation event to accelerate the movement of the character (e.g., speed up forward), which can cause a relatively large movement. Alternatively, receiving the first event includes receiving an operation event to change the viewing angle of the image frame.

[0029] In a possible design of the first aspect, the electronic device includes a central processing unit (CPU) and a graphics processing unit (GPU). Receiving a first event includes: a first application receives the first event. After the first application receives the first event, the first application sends a first instruction to the CPU, and the first instruction indicates that the first event has been received. The CPU determines a resolution parameter corresponding to the first event, and the resolution parameter indicates that the image frame resolution is adjusted to a second resolution. The CPU sends a resolution parameter and a light parameter to the GPU, and the light parameter indicates that the number of rays passing through each pixel is a fourth number. Accordingly, in response to the rendering event of the second image frame, a third image frame is rendered using the second number of rays, including: in response to the rendering event of the second image frame, the GPU uses the second number of rays to render to obtain the third image frame, and the second number is the product of the fourth number and the number of pixels included in the third image frame of the second resolution.

[0030] That is to say, the CPU can determine the resolution parameters that match the first event and pass the resolution parameters and light parameters to the GPU. Subsequently, the GPU can reduce the number of rays for rendering a frame of image based on the resolution parameters and light parameters, reduce the amount of rendering calculations, and reduce rendering power consumption.

[0031] In a possible design method of the first aspect, the electronic device also includes a neural network processor NPU. In the process of the GPU using the second number of rays to render and obtain the third image frame, the GPU renders and obtains at least one of the following maps: an albedo map, a normal map, and a mask map. The CPU obtains and sends at least one of the albedo map, the normal map, and the mask map, and the third image frame to the NPU. Accordingly, the third image frame is processed to obtain the second image frame, including: the NPU uses at least one of the albedo map, the normal map, and the mask map, and the third image frame to perform noise reduction processing on the third image frame to obtain a noise-reduced image frame. The CPU obtains and sends the noise-reduced image frame to the GPU. The GPU performs upsampling processing and sharpening processing on the noise-reduced image frame to obtain a second image frame.

[0032] That is to say, in the process of rendering the image, the CPU will also render at least one of the albedo map, normal map and mask map for the NPU to complete the noise reduction processing later.

[0033] In one possible design of the first aspect, the noise reduction process is performed asynchronously with the upsampling process and the sharpening process. Due to the asynchronous execution, after the GPU sends the rendered third image frame to the NPU, it can start rendering the next image frame without waiting for the NPU to receive the noise-reduced third image frame.

[0034] In a second aspect, the present application further provides an electronic device comprising a display screen, a memory, and one or more processors. The display screen, the memory, and the processor are coupled. The memory is configured to store computer program code, which comprises computer instructions. When the computer instructions are executed by the processor, the electronic device performs the method of the first aspect and any possible design thereof.

[0035] In a third aspect, the present application provides a chip system, which is applied to an electronic device including a display screen and a memory; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected by lines; the interface circuit is used to receive signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the method as in the first aspect and any possible design method thereof.

[0036] In a fourth aspect, the present application provides a computer storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method of the first aspect and any possible design thereof.

[0037] In a fifth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method according to the first aspect and any possible design thereof.

[0038] It can be understood that the beneficial effects that can be achieved by the electronic device of the second aspect, the chip system of the third aspect, the computer storage medium of the fourth aspect, and the computer program product of the fifth aspect provided above can be referred to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] FIG1 is a schematic diagram of a ray tracing algorithm provided in an embodiment of the present application;

[0040] FIG2 is a hardware structure diagram of a mobile phone provided in an embodiment of the present application;

[0041] FIG3 is a diagram of the software and hardware architecture of a mobile phone provided in an embodiment of the present application;

[0042] FIG4 is a simplified diagram of an image rendering process provided by an embodiment of the present application;

[0043] FIG5 is a schematic diagram of a preset event provided in an embodiment of the present application;

[0044] FIG6 is a flowchart of an image processing method provided in an embodiment of the present application;

[0045] FIG7 is a schematic diagram of an upsampling process and a sharpening process provided by an embodiment of the present application;

[0046] FIG8 is a flowchart of another image processing method provided by an embodiment of the present application;

[0047] FIG9 is a schematic diagram of a method for obtaining an albedo map and a normal map according to an embodiment of the present application;

[0048] FIG10 is a schematic diagram of a method for obtaining a mask map according to an embodiment of the present application;

[0049] FIG11 is a process diagram of a noise reduction process provided by an embodiment of the present application;

[0050] FIG12 is a flowchart of another image processing method provided in an embodiment of the present application;

[0051] FIG13 is a timing interaction diagram of the image processing method provided in an embodiment of the present application;

[0052] FIG14 is a simplified flowchart of the image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0054] References to "one embodiment" or "some embodiments" etc. described in this specification mean that the specific features, structures or characteristics described in conjunction with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in another way. The term "connected" includes direct and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0055] In the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0056] In certain scenarios, electronic devices need to perform a large amount of image rendering. Typically, when running gaming applications, AR applications, VR applications, or modeling applications on electronic devices, these applications often require the electronic devices to perform a large amount of image rendering. The electronic devices then display the rendered images.

[0057] For example, the electronic device may be installed with the aforementioned various applications (Application, APP). When the application needs to display an image through the electronic device, it may send instructions to the electronic device so that the electronic device completes the image rendering according to the instructions and finally displays the rendered image on the display screen.

[0058] In the following examples, the present application scheme is described using a game application as an example. It is understood that during operation, the game application can display game images to the user via an electronic device. The game images can be video images, which can be composed of multiple continuously played frame images.

[0059] The frame image is the result of the electronic device rendering the three-dimensional scene of the game. The three-dimensional scene includes models of buildings, grass, trees, light sources (such as the sun, sky, desk lamp, etc.). It is understandable that the three-dimensional scene is usually represented by a triangle mesh.

[0060] The above image rendering can be achieved based on a ray tracing algorithm.

[0061] Referring to Figure 1, the three-dimensional scene includes a sphere, a cube, and a light source, and the image to be rendered is Image 1. Using a ray tracing algorithm, an electronic device can split the rendering task of a pixel point in a frame of an image (such as pixel point 1 in Figure 1) into the impact of a number of light rays (light rays ①, ②...x in total x rays in Figure 1) emitted from a camera and passing through the pixel point on the three-dimensional scene. It can be understood that the camera refers to a virtual camera, and the camera posture is determined by the electronic device according to the user's operation in the game. Among them, for the user's operations of moving forward, backward, switching perspectives, etc. in the game, the electronic device can convert them into camera rotation or translation accordingly, so that the corresponding camera posture can be determined.

[0062] Specifically, the electronic device can determine the color information of the pixel point based on the optical paths of several light rays in the three-dimensional scene, such as the electronic device determines the color information of pixel point 1 based on the optical paths of light rays ①, ②, ..., x in the three-dimensional scene in Figure 1. The following takes light ① as an example to illustrate the tracing process of one of the light rays by the electronic device in the ray tracing algorithm: the electronic device can intersect light ① with a large number of triangular meshes corresponding to the sphere, cube, and light source in the three-dimensional scene, and obtain the nearest intersection point that intersects with light ① as intersection point A. The electronic device uses light ① as the incident light and intersection point A as the incident point, and obtains a reflected light ray and a refracted light ray through integral operation, such as obtaining a reflected light ray as light (1). The electronic device continues to intersect light ray (1) with a large number of triangular meshes corresponding to the sphere, cube, and light source in the three-dimensional scene, and obtains the nearest intersection point that intersects with light ray (1) as intersection point B. The electronic device continues to use light (1) as the incident light and intersection point B as the incident point, and obtains the secondary reflected light and the secondary refracted light through integration operation. If the secondary reflected light is obtained as light (2), etc., the tracking process for light 1 ends until it intersects with the light source or the above iterative process reaches a preset number of times. According to a similar process, the electronic device completes the tracking of multiple light rays.

[0063] Finally, the electronic device can obtain the color information of the pixel based on the color information at the intersection points (such as intersection A and intersection B) of the 3D scene during the tracing of multiple rays. For example, the color information of pixel 1 in Figure 1 can be obtained based on the color information at the intersection points of rays ①, ②, ..., and x in Figure 1.

[0064] In other words, by using ray tracing technology, electronic devices can simulate the path of light in the real world and generate rendering results that are close to the real world.

[0065] At the same time, ray tracing algorithms require numerous intersection and integration calculations, which in turn require significant computing resources. Therefore, ray tracing is a highly resource-intensive technology. Implementing ray tracing algorithms in electronic devices results in high power consumption.

[0066] In order to reduce the power consumption of electronic devices when using ray tracing algorithms, the industry has subsequently proposed a hardware acceleration strategy. That is, electronic devices use hardware to complete some calculations. For example, since the logic of intersecting light with triangular meshes is simple, and mainly because there are a large number of triangular networks, the intersection process requires a lot of calculations. This task with simple logic but high computational load is just suitable for hardware to do. Therefore, a typical hardware acceleration strategy is: electronic devices use hardware to complete the intersection calculation in the process of ray tracing algorithms, thereby reducing the power consumption of electronic devices using ray tracing algorithms.

[0067] The hardware-accelerated ray tracing algorithm can be applied in electronic devices such as personal computers (PCs), consoles, mobile phones, and tablets.

[0068] However, to improve the user experience, higher rendering efficiency requirements are being placed on the market. For example, in high-frame-rate gaming applications, more images need to be rendered per unit time, and the amount of computation required by electronic devices per unit time is also greater, which still leads to high power consumption of electronic devices. Ray tracing algorithms remain a high-load technology, especially for mobile devices. This makes it difficult to consistently use ray tracing algorithms for image rendering on mobile devices such as phones and tablets.

[0069] Based on the above problems, an embodiment of the present application provides an image processing method that can accelerate the ray tracing algorithm from the software level, further reducing the amount of computation required to render each frame of an image using the ray tracing algorithm, thereby improving rendering efficiency.

[0070] In the process of rendering an image using a ray tracing algorithm, the main computational resources consumed are the intersection operations of light rays and triangular meshes and the integral operations for determining reflected light rays, refracted light rays, etc. In addition, in the process of rendering a frame of an image, the more light rays emitted by the camera, the more light rays the electronic device needs to perform intersection operations with the triangular meshes, and the amount of intersection operations will increase accordingly; and the more light rays the electronic device needs to perform integral operations, the more integral operations will increase accordingly. Therefore, in an embodiment of the present application, in the process of rendering a frame of an image using a ray tracing algorithm, the electronic device can reduce the amount of operations in the rendering process by reducing the number of light rays emitted by the camera, thereby reducing rendering power consumption and improving rendering efficiency. It can be understood that the light rays emitted by the camera refer to the light rays emitted from the camera as the starting point and passing through each pixel point.

[0071] For example, the electronic device may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), artificial intelligence (AI) device, wearable device, vehicle-mounted device, smart home device and / or smart city device, etc. The embodiments of the present application do not impose any particular restrictions on the specific form of the electronic device.

[0072] FIG2 is a hardware structure diagram of an electronic device. As shown in FIG2 , taking a mobile phone as an example, the electronic device may include a processor 210, an external memory interface 220, an internal memory (hereinafter referred to as memory) 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone jack 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display 294, and a subscriber identification module (SIM) card interface 295.

[0073] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the mobile phone. In other embodiments, the mobile phone may include more or fewer components than shown, or some components may be combined or separated, or arranged differently. The components shown in the illustrations may be implemented in hardware, software, or a combination of software and hardware.

[0074] The processor 210 may include one or more processing units, for example, an application processor (AP), a modem processor, a central processing unit (CPU), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0075] The charging management module 240 is configured to receive charging input from a charger. The power management module 241 is configured to connect the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 to power the processor 210, the internal memory 221, the display 294, the camera 293, and the wireless communication module 260. The power management module 241 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance), etc.

[0076] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modulation and demodulation processor and baseband processor. The mobile communication module 250 can provide solutions for wireless communication including 2G / 3G / 4G / 5G applied in electronic devices. The wireless communication module 260 can provide solutions for wireless communication including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), Bluetooth low energy (BLE), ultra wide band (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR) and other wireless communication solutions applied in electronic devices.

[0077] The electronic device can implement display functions through a GPU, display screen 294, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or modify display information.

[0078] The electronic device can realize the camera function through the camera 293, ISP, video codec, GPU, display screen 294, application processor AP, neural network processor NPU, etc.

[0079] The electronic device can implement audio functions such as music playback and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone jack 270D, and the application processor.

[0080] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. TM ) system as an example to illustrate the software structure of an electronic device. The layered architecture divides the software system of an electronic device into several layers. Each layer has a clear role and division of labor, and the layers communicate with each other through software interfaces.

[0081] 3 , taking a mobile phone as an example, the software and hardware architecture of an electronic device may include an application (APP) layer, a framework layer, a system library, and a hardware layer.

[0082] The application layer can also be called the application program layer. The application program layer can include a series of application packages. The application package can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc. In some embodiments, the application program layer can also include applications that need to display images or videos to users after image rendering (also called first applications). For example, the application program layer can also include game applications, such as Etc. As another example, the application layer may also include AR applications, VR applications, modeling applications, etc.

[0083] The framework layer is also called the application framework layer. It provides an application programming interface (API) and programming framework for applications in the application layer. The framework layer includes predefined functions. For example, the framework layer may include a window manager, content provider, view system, resource manager, notification manager, activity manager, and so on.

[0084] In the embodiment of the present application, one or more functional modules may be provided in the framework layer for implementing the image processing method provided in the embodiment of the present application. It is understood that these functional modules are controlled by the CPU to perform corresponding functions.

[0085] The system library can include multiple functional modules, such as the surface manager, media framework, standard C library (libc), SQLite, graphics library, etc.

[0086] The graphics library may include at least one of the following: Open Graphics Library (OpenGL), OpenGL for Embedded Systems (OpenGL ES), Vulkan, etc. The graphics library may provide drawing and manipulation of 2D and 3D graphics in the application. In some embodiments, the CPU or GPU may call the graphics library to implement image rendering.

[0087] The hardware layer may include a processor, such as a CPU, a GPU, an NPU, etc. Furthermore, the hardware layer may also include a memory. In an embodiment of the present application, a frame buffer may be created in the memory to store rendering results and noise reduction processing results.

[0088] Referring to Figure 4, during the image rendering process, the application, CPU, GPU, memory, and display screen in the aforementioned hardware and software structure of the electronic device operate as follows: the application can issue a rendering instruction to instruct the rendering of an image. After receiving the rendering instruction, the CPU can call the corresponding graphics drawing application programming interface (API) in the image library to instruct the GPU to perform the rendering operation corresponding to the rendering instruction. The GPU executes the rendering instruction and stores the rendered image in the memory. Finally, the rendered image in the memory can be sent to the display screen for display.

[0089] In some embodiments, after receiving the rendering instruction issued by the application, the CPU may also determine parameters for reducing the number of rays emitted by the camera (such as the resolution parameter and / or ray parameter described below). After receiving the parameters for reducing the number of rays emitted by the camera, the GPU may perform a process for reducing the rays.

[0090] For example, the GPU may reduce the resolution of the image to be rendered according to the resolution parameter, thereby reducing the number of rays emitted by the camera during the process of rendering the entire image.

[0091] As another example, the GPU may reduce the light passing through each pixel according to the light parameters, thereby reducing the number of rays emitted by the camera during the rendering of the entire image.

[0092] When reducing light by lowering the image resolution, the NPU in the electronic device can perform noise reduction on the GPU-rendered image and store the resulting noise-reduced image in memory. Finally, the noise-reduced image in memory can be sent to the display for display.

[0093] By reducing the amount of light emitted by each pixel, the GPU in the electronic device can also upsample the rendered image to increase the image resolution and store the upsampled high-resolution image in memory. Finally, the high-resolution image in memory can be sent to the display for display.

[0094] The image processing method provided in the embodiment of the present application can be implemented in an electronic device having the above-mentioned software and hardware structure.

[0095] In an embodiment of the present application, the electronic device reduces the amount of computation in the rendering process by reducing the number of rays emitted by the camera. In practice, the electronic device may reduce the amount of ray emitted by the camera when rendering each frame of an image. This reduces the amount of computation in the rendering process of each frame of an image. Alternatively, after detecting a first event, the electronic device may reduce the amount of ray emitted by the camera when rendering each frame of an image (also referred to as a second image frame), and use a lower number of rays (also referred to as a second number) to render the image. This reduces the amount of computation in the rendering process of each frame of an image after the first event. The first event may include: an event that triggers the electronic device to adjust the viewing angle of the displayed image, and the amplitude of motion between the image frames before and after the adjustment exceeds the first amplitude.

[0096] It should be noted that when the amplitude of motion between two frames of images is large, a proper reduction in image quality has little impact on the user's perception. At the same time, when an electronic device uses a ray tracing algorithm to render an image, the fewer the number of rays, the more difficult it is to guarantee the quality of the rendered image. Even if subsequent processing to improve the image quality is performed (such as noise reduction, upsampling, sharpening, etc. described below), it is difficult to achieve the effect of rendering with a large number of rays. Therefore, after the electronic device detects the first event, when rendering each frame of the image, the electronic device reduces the number of rays emitted by the camera, which can reduce the amount of computation in the rendering process while reducing the impact on the user's perception.

[0097] Taking a gaming application as an example, the first event could be an accelerated forward event. When a character in a game video frame moves rapidly, not only does the perspective change, but due to the high speed of movement, the displayed image between the previous and next frames also changes significantly. For example, a long press of the forward button triggers accelerated forward. After receiving a long press of forward button 5011 in the game interface 501 shown in FIG5 , the electronic device can reduce the amount of light emitted by the camera when rendering each subsequent frame. This not only reduces the amount of computation required during the rendering process, but also does not significantly degrade the user's viewing experience.

[0098] Continuing with the example of a gaming application, users can directly change the viewing angle of the screen by sliding, and sliding operations typically result in significant changes in the viewing angle. Therefore, the first event can also be a sliding operation to change the viewing angle. For example, after receiving a user sliding operation on the game interface 502 shown in Figure 5, the electronic device can reduce the number of rays emitted by the camera when rendering each subsequent frame. This not only reduces the amount of computation required during the rendering process, but also does not significantly degrade the user's viewing experience.

[0099] It is understood that before the electronic device detects the first event, the camera may emit a default amount (which may be referred to as the first amount) of light to render a frame of image (which may be referred to as the first image frame). In other words, before and after the electronic device detects the preset event, the amount of light emitted by the camera is different during the process of rendering two consecutive frames of image.

[0100] And, after the first event ends, the electronic device can return to using the default number of rays to render the image. For example, after the user ends the long press operation on the forward button 5011 or after the user ends the sliding operation in the game screen 502, the electronic device can return to using the default number of rays to render the image.

[0101] Among them, the electronic device can reduce the number of rays emitted by different frames of images to the same number. For example, the electronic device can reduce the number of rays emitted by the camera during the rendering process of each frame of image from a larger value to a smaller value. Alternatively, the electronic device can also reduce the number of rays emitted by the camera during the rendering process of different frames of image to different numbers. Exemplarily, the electronic device can reduce the number of rays emitted by the camera during the rendering process of a frame of image to a number that matches the first event. For example, if the first event is event 1, then the number of rays emitted by the camera during the rendering process of a frame of image is number 1; if the first event is event 2, then the number of rays emitted by the camera during the rendering process of a frame of image is number 2. Another example is that the electronic device can dynamically adjust the number of rays emitted by the camera based on power consumption, such as the higher the power consumption, the less light the camera emits, and the lower the power consumption, the more light the camera emits.

[0102] Specifically, the electronic device can reduce the amount of light emitted by the camera by reducing the image resolution; or the electronic device can reduce the amount of light emitted by the camera by reducing the amount of light passing through each pixel; or the electronic device can reduce the amount of light emitted by the camera by combining reducing the image resolution and reducing the amount of light passing through each pixel. The image processing method provided in the embodiments of the present application will be described below with respect to each of these methods.

[0103] Method 1 is to reduce the amount of light emitted by the camera by reducing the resolution of the image.

[0104] It's understandable that as image resolution decreases, the number of image pixels decreases. When using ray ray rendering technology to render an image, the camera needs to emit light rays that pass through each pixel. Therefore, when the number of pixels is reduced, the electronic device only needs to emit light rays that pass through fewer pixels, thereby reducing the number of rays emitted by the camera. In other words, image resolution or the number of image pixels is positively correlated with the number of rays emitted by the camera.

[0105] Referring to S1 in FIG6 , before reducing the resolution, the image resolution is M*N (also referred to as the first resolution), which includes M*N pixels. The number of rays emitted by the camera passing through each pixel is x. Therefore, during the rendering process of one frame of image, the camera needs to emit a total of M*N*x rays.

[0106] After reducing the resolution, for example, by reducing it to d times the original resolution (also called the second resolution), the width and height of the image will be reduced to d times the original. Correspondingly, the number of pixels is also reduced from M*N to (d*M)*(d*N). The number of rays emitted by the camera passing through each pixel is still x. Therefore, during the rendering process of one frame of image, the camera needs to emit a total of (d*M)*(d*N)*x rays, and the number of rays is reduced.

[0107] For example, if d=0.5, then after reducing the resolution, the number of rays emitted by the camera is only (0.5M*0.5N*x) / (M*N*x)=0.25 times the number of rays emitted by the camera before reducing the resolution.

[0108] That is, before reducing the resolution, the number of rays the camera needs to shoot to render a frame is related to the number of pixels in the unreduced image. After reducing the resolution, the number of rays the camera needs to shoot to render a frame is related to the number of pixels in the reduced-resolution image. Generally speaking, the more pixels there are, the more rays the camera needs to shoot to render a frame.

[0109] Continuing to refer to S2 in FIG. 6 , after reducing the resolution, the electronic device may use a ray tracing algorithm to render the low-resolution image to obtain a rendered image (also referred to as a third image frame), such as an ellipse 601 included in the rendered image.

[0110] It is understandable that resolution is an important factor affecting image clarity. It is obviously unacceptable to reduce image quality in order to reduce rendering power consumption. Therefore, referring to S3 in Figure 6, after obtaining the rendered image, the electronic device can also upsample the rendered image to restore the resolution of the image (such as restoring it to the first resolution) and obtain a high-resolution image (also referred to as a fourth image frame). In the embodiment of the present application, the specific implementation of upsampling is not limited.

[0111] In a specific implementation, the electronic device may implement image upsampling in a manner similar to that shown in FIG7 :

[0112] Taking the example that the reduced resolution is 0.5 of the unreduced resolution, the electronic device can use the color information uv of the pixel in the uth row and vth column in the rendered image as the color information of the pixel in the 2u-1th row and 2v-1th column in the high-resolution image.

[0113] 7 , the electronic device may use the color information 11 of the pixel in the 1st row and 1st column of the rendered image as the color information of the pixel in the 1st row and 1st column of the high-resolution image, use the color information 12 of the pixel in the 1st row and 2nd column of the rendered image as the color information of the pixel in the 1st row and 3rd column of the high-resolution image… use the color information 21 of the pixel in the 2nd row and 1st column of the rendered image as the color information of the pixel in the 3rd row and 1st column of the high-resolution image, use the color information 22 of the pixel in the 2nd row and 2nd column of the rendered image as the color information of the pixel in the 3rd row and 3rd column of the high-resolution image… until the color information of all pixels in the rendered image is mapped to the corresponding pixels in the high-resolution image.

[0114] For pixels with unknown color information in a high-resolution image, the electronic device can calculate the color information of the pixel based on the color information of its neighboring pixels. Two pixels that share a border are called neighboring pixels.

[0115] Continuing to refer to Figure 7, the electronic device can average the color information 11 of the pixel in the 1st row and 1st column and the color information 12 of the pixel in the 1st row and 3rd column in the high-resolution image to obtain color information ① of the pixel in the 1st row and 2nd column; average the color information 11 of the pixel in the 1st row and 1st column and the color information 21 of the pixel in the 3rd row and 1st column in the high-resolution image to obtain color information ② of the pixel in the 2nd row and 1st column; average the color information 12 of the pixel in the 1st row and 3rd column and the color information 22 of the pixel in the 3rd row and 3rd column in the high-resolution image to obtain color information ③ of the pixel in the 2nd row and 3rd column; average the color information 21 of the pixel in the 3rd row and 1st column and the color information 22 of the pixel in the 3rd row and 3rd column in the high-resolution image to obtain color information ④ of the pixel in the 3rd row and 2nd column; average color information ①, color information ②, color information ③ and color information ④ to obtain color information ⑤ of the 2nd row and 2nd column... until the color information of all pixels in the high-resolution image is obtained.

[0116] In some embodiments, after upsampling to obtain a high-resolution image, the electronic device may further perform sharpening processing on the high-resolution image to further improve the clarity of the image. In the embodiments of the present application, the specific implementation of the sharpening processing is not limited.

[0117] In a specific implementation, the electronic device may calculate the color information of the pixel to be sharpened after sharpening based on the maximum and minimum values ​​of the color of the g channel in the color information of the pixel to be sharpened and multiple adjacent pixels in the high-resolution image, as well as the color information of the pixel to be sharpened and the multiple pixels. It is understood that the color information of each pixel includes the colors of three channels, namely the R channel, the G channel, and the B channel.

[0118] The pixel to be sharpened may be any pixel in the high-resolution image, and the adjacent pixels include pixels that are adjacent to the edge of the pixel to be sharpened.

[0119] Taking the pixel to be sharpened as the pixel P in the bold dotted box of the high-resolution image shown in Figure 7 as an example, the multiple adjacent pixel points may include 8 pixel points in the dotted box except the pixel P, such as pixel a to pixel h.

[0120] Compared to using only the four or even fewer pixels above, below, left, and right of the pixel to be sharpened, this implementation uses the eight pixels immediately adjacent to the edge of the pixel to be sharpened, resulting in better sharpening results without significantly increasing the amount of computation and access consumption for the sharpening process. This improves the sharpening effect while maintaining essentially the same computing power consumption.

[0121] For example, the electronic device may calculate the maximum and minimum values ​​of the color of the G channel in the color information of pixel a, pixel h, and pixel P:

[0122] For example, maximum value max_g=max(Pg, ag, bg, cg, dg, eg, fg, gg, hg);

[0123] For example, the minimum value min_g=min(Pg, ag, bg, cg, dg, eg, fg, gg, hg).

[0124] Among them, ××.g is the color of the G channel in the color information of pixel point ××.

[0125] Furthermore, the electronic device may calculate a sharpening coefficient based on the maximum value and the minimum value, and then use the sharpening coefficient, the pixel to be sharpened, and the color information of multiple adjacent pixels to calculate the sharpened color information of the pixel to be sharpened.

[0126] For example, the calculation formula of the sharpen coefficient is as follows:

[0127] That is to say, if min_g is less than 1-max_g, the sharpening coefficient is positively correlated with min_g / max_g; if mun_g is greater than 1-max_g, the sharpening coefficient is positively correlated with (1-max_g) / max_g.

[0128] And, the calculation formula of the color information of the pixel to be sharpened after sharpening is as follows:

[0129] Wherein, ××.rgb refers to the color information of pixel x. w is the correction constant of the sharpening coefficient, 0<w<1.

[0130] The second method is to reduce the amount of light emitted by the camera by reducing the amount of light passing through each pixel.

[0131] When rendering an image using a ray tracing algorithm, the camera emits rays of light that pass through each pixel. Therefore, by reducing the number of rays passing through each pixel, the number of rays emitted by the camera during the rendering of the entire image will also decrease. In other words, the number of rays passing through each pixel is positively correlated with the number of rays emitted by the camera to render the entire image.

[0132] Referring to S1 in FIG8 , before reducing the light passing through each pixel, x light rays (also referred to as the third number) are emitted through each pixel, and a frame of image includes M*N pixels. Therefore, during the rendering process of a frame of image, the camera needs to emit a total of M*N*x light rays.

[0133] After reducing the light passing through each pixel, y light rays (also called the fourth number) are emitted through each pixel, where y is less than x. A frame of image also includes M*N pixels. Therefore, during the rendering process of a frame of image, the camera needs to emit a total of M*N*y light rays.

[0134] Obviously, the number of rays emitted by the camera after reducing the rays passing through each pixel is only (M*N*y) / (M*N*x)=y / x, which is the amount of rays emitted by the camera before reducing the rays passing through each pixel.

[0135] That is, if the number of rays passing through each pixel is y, the number of rays the camera needs to emit to render a frame is related to y. If the number of rays passing through each pixel is x, the number of rays the camera needs to emit to render a frame is related to x. Generally speaking, the larger y or x is, the more rays the camera needs to emit to render a frame.

[0136] Continuing to refer to S2 in FIG8 , a ray tracing algorithm is used, and the number of rays emitted through each pixel is y (eg, 4, 5, etc.), and a rendered image (also referred to as a third image) can be obtained.

[0137] It is understandable that the less light passes through each pixel, the higher the probability that the pixel's color calculation will fail. In other words, noise is more likely to appear. For example, the rendered image in Figure 8 includes multiple noise points (shown in black).

[0138] Therefore, referring to S3 in FIG8 , after obtaining the rendered image, the electronic device may further perform noise reduction processing on the rendered image to eliminate noise in the rendered image and obtain a noise-reduced image (also referred to as a third image after noise reduction).

[0139] In the embodiments of the present application, the specific implementation of noise reduction is not limited. For example, the electronic device can use an artificial intelligence (AI) noise reduction model, such as the industry-recognized OptiX 5 with AI-accelerated Denoising, to perform noise reduction on the rendered image.

[0140] In some embodiments, the electronic device may also use a self-developed preset AI denoising model to perform denoising on the rendered image. The input of the preset AI denoising model includes at least the rendered image. Furthermore, the output of the preset AI denoising model may also include at least one of the following: an albedo map, a mask map, and a normal map.

[0141] The Albedo map records the base color of each model in the 3D scene when there is no lighting. The Normal map records the normal information at the intersection of light and the 3D scene. Both the Albedo map and the Normal map are RGB images.

[0142] In a specific implementation, the color information of each pixel in the above-mentioned Albedo map is the color information of the first intersection point of the first ray passing through the pixel in the image to be rendered and the three-dimensional scene. Also, the color information of each pixel in the above-mentioned Normal map corresponds to the normal coordinates of the first intersection point, and different normal coordinates correspond to different color information. Among them, the color information of each pixel in the Normal map includes the color values ​​of three color channels, namely the R channel, the G channel and the B channel, and the normal coordinates include the coordinate values ​​of three axes, namely the x-axis, the y-axis and the z-axis. Therefore, the color values ​​of the three color channels can correspond to the coordinate values ​​of the three axes respectively. For example, the color value of the R channel corresponds to the coordinate value of the x-axis, the color value of the G channel corresponds to the coordinate value of the y-axis, and the color value of the B channel corresponds to the coordinate value of the z-axis.

[0143] Taking the image to be rendered as image 1 shown in Figure 9, the first light ray passing through pixel 1 in image 1 is light ray ①, and the first intersection of light ray ① and the three-dimensional scene is intersection A as an example, the color information of pixel 1 in the Albedo map is the color information of intersection A, and the color information of pixel 1 in the Normal map corresponds to the normal coordinates of intersection A.

[0144] For example, the color information and normal coordinates of the first intersection point can be obtained as follows:

[0145] Still taking the example of the first intersection being intersection A shown in FIG9 , intersection A is located in the triangle index shown in the shaded area. The electronic device can obtain the material map (such as a color map) of the three vertices in the triangle index and the normal coordinates of the three vertices. It can be understood that the normal coordinates of each vertex here refer to the normal coordinates in the local coordinate system of the triangle index. The electronic device can obtain the color information of intersection A based on the color map of the three vertices, and can obtain the normal coordinates of intersection A based on the normal coordinates of the three vertices. For example, the electronic device can obtain the color information and normal coordinates of intersection A by using barycentric interpolation.

[0146] It should be noted that the normal coordinates of the three vertices refer to the coordinates in the local coordinate system of the triangle index. Therefore, the normal coordinates of the intersection point A obtained based on the normal coordinates of the three vertices are also the coordinates in the local coordinate system. Therefore, the electronic device can further convert the normal coordinates of the intersection point A to the camera coordinate system so that the normal coordinates of the intersection point A are the coordinates in the camera coordinate system. For example, the electronic device can use the following formula to convert the normal coordinates of the intersection point A to the camera coordinate system: cameraNormalInter = localNormalInter * viewMatrix,

[0147] Among them, cameraNormalInter is the normal coordinate of the intersection point A in the camera coordinate system, localNormalInter is the normal coordinate of the intersection point A in the local coordinate system, and viewMatrix is ​​the spatial position matrix of the camera coordinate system.

[0148] The mask map records: information related to the number or ratio of valid light paths for each pixel. The number of valid light paths for each pixel refers to the number or ratio of valid light paths among the multiple light paths corresponding to the multiple light rays passing through each pixel. Alternatively, the ratio of valid light paths for each pixel refers to the ratio of the number of valid light paths to the multiple light paths corresponding to the multiple light rays passing through each pixel. In addition, the mask map is a grayscale image. In the following, the ratio of valid light paths is mainly used as an example. A valid light path refers to a light path that can reach the light source, and an invalid light path refers to a light path that cannot reach the light source.

[0149] For example, consider the image to be rendered, Image 1 (Figure 10). For example, there are four light rays passing through pixel 1: ray ①, ray ②, ray ③, and ray ④. Light path 1 corresponding to ray ①, light path 2 corresponding to ray ②, and light path 4 corresponding to ray ④ all reach the light source. Therefore, light paths 1, 2, and 4 are valid light paths. Light path 3 corresponding to ray ③ cannot reach the light source, so it is an invalid light path. The electronic device can calculate that the proportion of valid light paths for pixel 1 is 3 / 4.

[0150] In a specific implementation, the color information of each pixel in the above-mentioned Mask map corresponds to the ratio of the effective light path of the pixel in the image to be rendered, and different ratios of the effective light path correspond to different color information.

[0151] Take the corresponding relationship between the ratio of effective light paths and color information as shown in Table 1 below as an example:

[0152] Table 1

[0153] Continuing with the example of FIG10 , the ratio of the effective light path of pixel 1 is 3 / 4. Based on Table 1 above, it can be determined that the color information of pixel 1 in the Mask map is 200.

[0154] That is, in the process of rendering to obtain a rendered image, the electronic device can obtain the above-mentioned Albedo map, Mask map and Normal map.

[0155] Subsequently, the electronic device uses the Albedo map, Mask map, and Normal map, as well as the rendered image, as inputs and runs a preset AI denoising model to output a denoised image. Referring to Figure 11 , the electronic device can use the rendered image, Albedo map, and Mask map as inputs to the preset AI denoising model to predict a denoised image.

[0156] The present application embodiment will exemplarily illustrate the process of training and obtaining a preset AI noise reduction model:

[0157] Multiple noise image samples and the Albedo map samples, Mask map samples, and Normal map samples corresponding to each noise image sample can be collected to obtain multiple groups of input samples. Each group of input samples includes a noise image sample and the corresponding Albedo map sample, Mask map sample, and Normal map sample. In addition, the denoised image samples corresponding to each noise image can be collected to obtain multiple output samples. The preset network model is trained using the multiple groups of input samples and the multiple output samples, and the model parameters of the network model are adjusted according to the error between the actual training output and the output sample. When the error between the actual training output and the output sample is less than the preset error, the training is terminated. The network model obtained at the end of the training is the preset AU denoising model.

[0158] The previous section describes the specific implementations of Methods 1 and 2, respectively. In actual implementation, Methods 1 and 2 can be combined. In other words, the resolution can be reduced, and the amount of light passing through each pixel can be reduced to further reduce the amount of light emitted by the camera. Accordingly, after obtaining the rendered image, the electronic device needs to perform noise reduction on the rendered image and increase its resolution.

[0159] For example, referring to S1 in FIG12 , the electronic device may reduce the image resolution from M*N to 0.5M*0.5N; and, referring to S2 in FIG12 , the electronic device may reduce the number of rays passing through each pixel from x to 4, where x is much larger than 4. Continuing to refer to S3 in FIG12 , using a ray tracing algorithm, the camera emits four rays passing through each pixel in an image with a resolution of 0.5M*0.5N, which can render a rendered image with a resolution of 0.5M*0.5N, but with noise at multiple pixels (shown as black areas in the figure). Continuing to refer to S4 in FIG12 , the electronic device may perform noise reduction processing on the rendered image to obtain a noise-reduced image with a resolution of 0.5M*0.5N and reduced noise (indicated by the disappearance of the black areas in the figure). And, continuing to refer to S5 in FIG12 , the electronic device may perform upsampling processing on the noise-reduced image to obtain a high-resolution image with a resolution increased to M*N.

[0160] The following mainly uses the combination of method 1 and method 2, and the use of rendered images, Albedo maps, and Normal maps for noise reduction as an example, and combines the software and hardware structure of electronic devices to illustrate the present application solution. Specifically, referring to Figure 13, the image processing method provided in the embodiment of the present application includes:

[0161] S1300: The game application receives a first event.

[0162] Regarding the first incident, please refer to the previous description and I will not repeat it here.

[0163] S1301 : The game application sends instruction 1 to the CPU, where instruction 1 indicates receipt of a first event.

[0164] Instruction 1 may also be referred to as the first instruction.

[0165] S1302: In response to instruction 1, the CPU determines a resolution parameter that matches the first event, where the resolution parameter is used to adjust image resolution.

[0166] The resolution parameter can be the ratio of the adjusted image resolution to the default resolution of the game application, such as 0.5, 0.75, etc. For example, a resolution parameter of 0.5 indicates that the adjusted image resolution is 0.5 of the initial resolution. For example, if the initial resolution is M*N, the adjusted image resolution should be 0.5M*0.5N.

[0167] Different first events can cause different motion amplitudes. Accordingly, the electronic device can configure different resolution parameters for different first events, that is, different first events correspond to different resolutions. In this way, the electronic device can reduce the amount of light to different degrees according to different motion amplitudes.

[0168] Taking the resolution parameter as an example, if the first event is an operation event of accelerating forward, the resolution parameter may be 0.75; if the first event is an operation event of sliding to change the viewing angle, the resolution parameter may be 0.5.

[0169] Furthermore, the same first event with different event parameters can also result in different motion amplitudes. Accordingly, different resolution parameters can be configured for the same first event with different event parameters. This allows the amount of light emitted by the camera to be reduced to varying degrees for different motion amplitudes. Accordingly, the event parameters can be included in instruction 1 for the CPU to determine the resolution parameters.

[0170] The event parameters include parameters that can affect the amplitude of the movement. For example, the event parameters include but are not limited to at least one of the following: operation type (such as single click, double click, long press, slide, etc.), control information, and slide distance, speed, etc.

[0171] Furthermore, the greater the motion caused by the event parameters, the smaller the resolution parameter is configured. The greater the motion, the greater the degradation in image quality, while having minimal impact on the user's perception. This allows the amount of light emitted by the camera to be reduced to a greater extent while minimizing the impact on the user's perception.

[0172] Taking the sliding operation to change the viewing angle as an example, a large sliding distance can trigger a large-angle view switching, resulting in a larger motion amplitude. The resolution parameter can be multiplied by d1. A small sliding distance can trigger a small-angle view switching, resulting in a smaller motion amplitude. The resolution parameter can be multiplied by d2. Both d1 and d2 are natural numbers greater than 0 and less than 1, with d1 being less than d2. For example, d1 = 0.5 and d2 = 0.75. This not only reduces the amount of light emitted by the camera to varying degrees, but also does not significantly affect the user's viewing experience.

[0173] S1303: The CPU determines light parameters, where the light parameters are used to indicate the amount of emitted light.

[0174] The light parameter can be the number of rays passing through each pixel. For example, a light parameter of 4 means that the number of rays passing through each pixel is 4. For another example, a light parameter of 5 means that the number of rays passing through each pixel is 5.

[0175] Typically, the light parameter is a fixed value. For example, the light parameter corresponding to different first events is 4. Of course, the light parameter can also be based on the first event (i.e., different first events correspond to different light parameters), or further based on the dynamic change of event parameters. This embodiment of the present application does not specifically limit this.

[0176] It should be noted that if the light parameters are fixed values, the above S1303 can be omitted.

[0177] The CPU can send the resolution parameter and the light parameter to the GPU, so that the GPU can render the image using a smaller number of rays according to the resolution parameter and the light parameter. For details, please refer to the description of S1304-S1317 below.

[0178] S1304: The CPU sends resolution parameters to the GPU.

[0179] S1305 , the GPU creates render targets (Render Target) of the rendered image, Albedo map, Normal map, and denoised image according to the resolution parameter and the default resolution.

[0180] It is understood that the render target of the rendered image, Albedo map, or Normal map is used to store the corresponding image rendered by the GPU, and the render target of the denoised image is used to store the denoised image obtained after denoising.

[0181] For example, if the resolution parameter is the ratio of the adjusted image resolution to the game application's default resolution, the resolution of the render target created based on the resolution parameter and the default resolution is resolution*scale, where resolution is the default resolution and scale is the ratio of the adjusted image resolution to the game application's default resolution.

[0182] For example, the GPU may call the glTexImage2D interface to configure the resolution size (including width and height) for each rendering target.

[0183] At this point, it should be noted that while the number of rays can be reduced by lowering the resolution during rendering, upsampling will be used later to restore the image resolution. Therefore, in S1305, there is no need to create a render target for storing the final processing result based on the resolution parameter. Instead, the render target before the resolution reduction can be used, such as the render target color_final with a resolution of resolution.

[0184] S1306. The CPU sends the resolution parameter to the NPU.

[0185] S1307. The NPU creates storage space for the rendered image, the Albedo map, the Normal map, and the denoised image according to the resolution parameter and the default resolution.

[0186] It can be understood that the storage space of the rendered image, Albedo map or Normal map is used to store the corresponding input data of the denoising process, and the storage space of the denoised image is used to store the output data of the denoising process.

[0187] Similarly, if the resolution parameter is the ratio of the adjusted image resolution to the default resolution of the game application, the size of the storage space created based on the resolution parameter and the default resolution is resolution*scale*r bytes, where r is the number of color channels of the corresponding image.

[0188] Taking resolution = 1280 * 720 and scale = 0.5 as an example, and the rendered image, Albedo map, Normal map and denoised image are all three-channel images, that is, r = 3, then the storage space for creating the rendered image, Albedo map, Normal map and denoised image respectively is 1280 * 0.5 * 720 * 0.5 * 3 bytes.

[0189] S1308: The game application sends a rendering instruction to the CPU.

[0190] When the game application needs to render a new frame of image, a rendering instruction may be sent to the CPU to instruct the start of rendering of the new frame of image.

[0191] S1309. The CPU creates a hardware acceleration structure for ray tracing.

[0192] Among them, the hardware acceleration structure is mainly used for the intersection of light and triangular meshes.

[0193] It is understandable that the process of creating a hardware acceleration structure is essentially the process of calling various interfaces provided by the GPU, and the embodiments of this application will not explain this in detail.

[0194] Taking the Vulkan graphics library as an example, the steps to create a hardware acceleration structure are as follows:

[0195] Step 1: The CPU calls the vkGetBufferDeviceAddressKHR interface provided by the Vulkan instruction to obtain the GPU addresses of the vertex buffers and index buffers of all models in the 3D scene.

[0196] Step 2: The CPU fills in the VkAccelerationStructureGeometryKHR interface and BuildGeometryInfo interface data structures provided by the Vulkan instructions;

[0197] Step 3: The CPU calls the vkCmdBuildAccelerationStructuresKHR interface provided by the Vulkan instruction to build the ray tracing acceleration structure.

[0198] It should be noted that the CPU can create a hardware acceleration structure each time it receives a rendering instruction to be used for intersection calculations between rays and triangle meshes during the rendering of the current frame. Alternatively, after receiving a rendering instruction, the CPU can detect whether the degree of change in the three-dimensional scene exceeds a preset degree and, if so, create a hardware acceleration structure. This embodiment of the present application is not specifically limited to this.

[0199] In addition, it should be noted that in S1309, the CPU is the main body of execution for creating the hardware acceleration structure. Under normal circumstances, the hardware acceleration structure is essentially created by the GPU driver. However, since the GPU driver is executed by the CPU, it is considered that the CPU is creating the hardware acceleration structure. Of course, in other embodiments, the GPU can also create the hardware acceleration structure. This embodiment of the present application does not specifically limit this.

[0200] S1310 . The CPU sends a binding instruction of the rendering pipeline to the GPU. The binding instruction instructs binding of a target shader and loading rendering targets of a rendered image, an Albedo map, and a Normal map into the target shader.

[0201] It is understood that after binding the target shader, it is equivalent to determining how to draw the graphics. In addition, loading the render target into the target shader is equivalent to determining the writing area of ​​the rendering result.

[0202] In addition, in the bind instruction, you can specify that the render target be loaded into the target shader as a readable and writable memory type (Storage Umage). This allows you to write data to the render target, such as writing rendering results to the render target, and read data from the render target, such as reading rendering results from the render target and using them for denoising.

[0203] S1311. The GPU performs a binding operation.

[0204] S1312. The CPU sends a rendering instruction 1 to the GPU. The rendering instruction 1 is used to instruct the GPU to perform a rendering operation of the first ray.

[0205] For example, the CPU may send rendering instruction 1 to the GPU by calling the vkCmdTraceRayKHR interface.

[0206] S1313. For each pixel point, the GPU uses a target shader to emit a first ray from the pixel point into the three-dimensional scene, and uses a hardware acceleration structure to obtain a first intersection point between the first ray and the three-dimensional scene.

[0207] It is understandable that after reducing the resolution, the number of pixels that the GPU needs to process will also be reduced accordingly. Therefore, the GPU can emit light and perform subsequent calculations on a smaller number of pixels, thereby reducing the amount of calculation.

[0208] The GPU emitting light can be understood as follows: the GPU determines a ray starting from the camera and extending along the line passing through the pixel points.

[0209] S1314. The GPU writes the color information of the first intersection point corresponding to each pixel point into the rendering target of the Albedo map, and writes data into the rendering target of the Normal map according to the normal coordinates of the first intersection point corresponding to each pixel point.

[0210] After the above S1313 and S1314, the Albedo map and the Normal map can be obtained. For the specific implementation of S13131 and S1314, please refer to Figure 9 and its related descriptions, which will not be repeated here.

[0211] S1315 : The GPU writes the color information of the first intersection point corresponding to each pixel point into the rendering target of the rendered image.

[0212] For example, after obtaining the color information of the first intersection corresponding to the pixel in the u-th row and the v-th column, the GPU may use the color information as the color information of the pixel in the u-th row and the v-th column in the rendering target of the rendered image.

[0213] S1316. The CPU sends a rendering instruction 2 to the GPU. The rendering instruction 2 is used to instruct the GPU to perform a rendering operation on the remaining light rays. The rendering instruction 2 includes light parameters.

[0214] Similarly, the CPU can send rendering instructions 2 to the GPU by calling the vkCmdTraceRayKHR interface.

[0215] S1317. For each pixel point, the GPU generates the remaining light emitted by the pixel point into the three-dimensional scene in the target shader according to the light parameters, and uses the hardware acceleration structure to obtain the intersection points of all light rays (including the first light ray and the remaining light rays) with the three-dimensional scene to obtain the color information of the pixel point in the rendered image.

[0216] Among them, for any light ray, if the light ray is within a preset number of iterations (such as 7 times, 8 times, etc.), it can intersect with the light source in the 3D scene, and the GPU can accumulate the color information of each intersection point of the light path of the light ray with the 3D scene into the rendering target of the rendered image. It should be noted that the color information of the first intersection point of the first light ray with the 3D scene has been written into the rendering target of the rendered image in S1315. Therefore, if it is determined in S1317 that the first light ray is within the preset number of iterations, it can intersect with the light source in the 3D scene, and there is no need to repeatedly accumulate the color information of the first intersection point of the first light ray with the 3D scene in S1317.

[0217] Alternatively, for any ray, if the ray still does not intersect with a light source in the 3D scene after a preset number of iterations, the GPU may not accumulate the color information of each intersection point of the ray's light path with the 3D scene in the render target of the rendered image. Similarly, the color information of the first intersection point of the first ray with the 3D scene has already been written to the render target of the rendered image in S1315. Therefore, if it is determined in S1317 that the first ray still does not intersect with a light source in the 3D scene after a preset number of iterations, the GPU needs to clear the data written to the corresponding pixel in the render target of the rendered image in S1315.

[0218] The GPU can average the accumulated values ​​of the color information at each pixel in the rendering target of the rendered image to obtain the average color information of the pixel, and write the average color information to the corresponding pixel in the rendering target of the rendered image to obtain the final rendered image.

[0219] Regarding the ray tracing process in S1317, please refer to Figure 1 and its related descriptions above, which will not be repeated here. Regarding the determination of the color information of the intersection point, please refer to the description of the determination process of the color information of the first intersection point (such as intersection point A) above, which will not be repeated here.

[0220] S1318. The CPU copies the rendered image, Albedo map, and Normal map obtained by GPU rendering in sequence to the storage space for the rendered image, Albedo map, and Normal map created by the NPU.

[0221] S1319. The CPU sends a noise reduction instruction to the NPU.

[0222] Exemplarily, the CPU may send a noise reduction instruction to the NPU by calling the snpe.execute interface of the NPU.

[0223] S1320: The NPU performs noise reduction processing according to the rendered image, the Albedo map, and the Normal map, and stores the obtained noise reduction image in a noise reduction image storage space.

[0224] For example, the NPU can use the data in the storage space of the rendered image, Albedo map, and Normal map as the input of the preset AI denoising model, run the preset AI denoising model, and then output a denoised image. It is understandable that the preset AI denoising model can be pre-loaded into the NPU. After the CPU sends a denoising instruction to the NPU, the NPU can directly use the pre-loaded preset AI denoising model for denoising.

[0225] For the specific implementation of the noise reduction process, please refer to FIG8 and its related descriptions above, which will not be repeated here.

[0226] S1321. The CPU copies the denoised image in the NPU to the rendering target of the denoised image in the GPU.

[0227] S1322. The GPU performs upsampling processing on the denoised image to obtain a high-resolution image.

[0228] Specifically, upsampling can be performed using GPU rasterization. Compared to CPU upsampling, using the GPU for upsampling allows the ray tracing algorithm for rendering, denoising, upsampling, and sharpening to be performed entirely on the GPU and NPU, without having to jump to the CPU. This avoids scheduling between additional modules.

[0229] In some embodiments, the NPU performs the noise reduction process asynchronously with the GPU's upsampling and sharpening processes. This allows the GPU to start processing the next frame after receiving the rendered image and syncing it to the NPU without waiting for the NPU's noise reduction process to complete.

[0230] S1323. The display screen acquires and displays a high-resolution image.

[0231] For example, the high-resolution image may be stored in the rendering target color_final, and the display screen may obtain the high-resolution image from the rendering target color_final and display it.

[0232] Furthermore, the GPU can also perform sharpening processing on high-resolution images to obtain sharpened images, and accordingly, the display screen displays the sharpened images.

[0233] For the specific implementation of upsampling and sharpening, please refer to Figures 6 and 7 above and their related descriptions, which will not be repeated here.

[0234] In summary, the embodiment shown in FIG13 can combine the aforementioned methods 1 and 2 to reduce the number of emitted rays, further reducing the amount of computation and improving efficiency. Furthermore, upsampling can compensate for the insufficient image resolution caused by method 1, while noise reduction can compensate for the high image noise caused by method 2. This reduces the amount of computation while ensuring image quality. In this way, even in high-frame-rate scenarios, high-quality images can be quickly generated and displayed.

[0235] It should be noted that the embodiment of FIG. 13 only illustrates the processing of one frame of image. In actual implementation, if multiple frames of image are processed continuously, each frame of image can be processed according to the process shown in FIG. 13 .

[0236] Of course, if the game application does not receive the first event, the existing processing flow can still be used to perform image processing. For example, in the existing processing flow, the number of emitted light rays will not be reduced.

[0237] The following briefly describes the work of the CPU, GPU, and NPU in the process shown in Figure 13, with reference to the simplified process diagram of Figure 14:

[0238] S1401: NPU loads the preset AI noise reduction model.

[0239] That is, the preset AI model is loaded into the NPU. For details, please refer to the description of S1320.

[0240] S1402: After receiving the first event, the GPU and NPU update rendering resources. For details, please refer to the description of S1301-S1305.

[0241] Among them, updating rendering resources includes GPU creating rendering targets and NPU creating storage space.

[0242] S1403: After receiving the rendering instruction, the CPU creates a hardware acceleration structure for ray tracing. For details, please refer to the description of S1308-S1309.

[0243] It is understandable that the CPU needs to create a hardware acceleration structure based on the three-dimensional scene.

[0244] S1404: The GPU generates auxiliary data for noise reduction. For details, please refer to the description of S1312-S1314.

[0245] The auxiliary data includes at least one of an Albedo map, a Normal map, and a Mask map.

[0246] S1405: The GPU performs ray tracing to obtain a rendered image. For details, see S1315-S1317.

[0247] In the process of ray tracing, the GPU can determine the number of rays emitted through each pixel based on the light parameters.

[0248] S1406: The NPU performs noise reduction processing to obtain a noise-reduced image. For details, see the description of S1318-S1320.

[0249] S1407: The GPU performs upsampling and sharpening to obtain the final color image. For details, please refer to the description of S1321-S1322.

[0250] S1408: The display screen acquires and displays the final color image. For details, please refer to the description of S1323.

[0251] An embodiment of the present application further provides an electronic device, which may include: a memory and one or more processors (such as a CPU, GPU, NPU, etc.). The memory and the processor are coupled. The memory is used to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device may perform the various functions or steps performed by the device in the above method embodiment.

[0252] An embodiment of the present application further provides a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected via lines. For example, the interface circuit can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit can be used to send signals to other devices (such as a processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which is not specifically limited in the embodiment of the present application.

[0253] This embodiment further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the image processing method in the above-mentioned embodiment.

[0254] This embodiment further provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

[0255] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the image processing method in the above-mentioned method embodiments.

[0256] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0257] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0258] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0259] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0260] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0261] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0262] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the electronic device includes a first application, the electronic device uses a ray tracing algorithm to render an image frame of the first application, and during the process of the electronic device running the first application, the method includes: In response to a rendering event of a first image frame, rendering using a first number of rays to obtain the first image frame; receiving a first event, wherein the first event is used to trigger the electronic device to adjust the viewing angle of the image frame, and the movement amplitude between the image frames before and after the adjustment exceeds a first amplitude; In response to a rendering event of the second image frame, rendering using a second number of rays to obtain a third image frame, and processing the third image frame to obtain the second image frame, wherein the second number is less than the first number; The first image frame and the second image frame are two consecutive image frames of the first application.

2. The method according to claim 1, characterized in that The resolutions of the first image frame and the second image frame are both a first resolution, the resolution of the third image frame is a second resolution, and the second resolution is lower than the first resolution; The first number is related to the number of pixels included in the first image frame of the first resolution, and the second number is related to the number of pixels included in the third image frame of the second resolution.

3. The method according to claim 2, characterized in that The step of processing the third image frame to obtain the second image frame includes: After upsampling the third image frame, the second image frame is obtained.

4. The method according to claim 2 or 3, characterized in that: Different first events correspond to different second resolutions; After receiving the first event, the method further includes: The second resolution corresponding to the received first event is determined.

5. The method according to any one of claims 1 to 4, characterized in that Before receiving the first event, the number of light rays passing through each pixel in the process of rendering one frame of image is a third number, and after receiving the first event, the number of light rays passing through each pixel in the process of rendering one frame of image is a fourth number, and the fourth number is less than the third number; The first quantity is related to the third quantity, and the second quantity is related to the fourth quantity.

6. The method according to any one of claims 1 to 5, characterized in that The step of processing the third image frame to obtain the second image frame includes: Performing noise reduction processing on the third image frame; The third image frame after the noise reduction process is processed to obtain the second image frame.

7. The method according to claim 6, characterized in that The performing noise reduction on the third image frame comprises: Taking at least one of an albedo map, a normal map, and a mask map, and the third image frame as input, running a preset denoising model, and obtaining the third image frame after denoising; The preset denoising model has the ability to remove noise from the noisy image frame based on at least one of an albedo map, a normal map and a mask map, and the noisy image frame.

8. The method according to claim 7, characterized in that The RGB color information of any pixel in the albedo map is the RGB color information of the first intersection of the first light ray passing through the pixel and the three-dimensional scene; The RGB color information of any pixel in the normal map corresponds to the normal coordinates of the first intersection point of the first light ray passing through the pixel and the three-dimensional scene; The RGB color information of any pixel in the mask map corresponds to the number of effective light paths in the light paths corresponding to the fourth number of light rays passing through the pixel, where the effective light path refers to a light path intersecting with a light source in the three-dimensional scene; The three-dimensional scene is a scene used to render the image frame of the first application.

9. The method according to claim 5, characterized in that Different first events correspond to different fourth quantities; After receiving the first event, the method further includes: The fourth quantity corresponding to the received first event is determined.

10. The method according to any one of claims 1 to 9, characterized in that The receiving of the first event comprises: An operation event of moving a character in an image frame is received, or an operation event of changing a viewing angle of an image frame is received.

11. The method according to any one of claims 1 to 10, characterized in that The electronic device includes a central processing unit (CPU) and a graphics processing unit (GPU); The receiving of the first event comprises: The first application receives the first event; After the first application receives the first event, the method further includes: The first application sends a first instruction to the CPU, where the first instruction indicates that the first event is received; The CPU determines a resolution parameter corresponding to the first event, wherein the resolution parameter indicates adjusting the image frame resolution to a second resolution; The CPU sends the resolution parameter and the light parameter to the GPU, wherein the light parameter indicates that the number of light rays passing through each pixel is a fourth number; The step of rendering a third image frame using a second number of rays in response to a rendering event of the second image frame comprises: In response to the rendering event of the second image frame, the GPU uses the second number of rays to render the third image frame, where the second number is the product of the fourth number and the number of pixels included in the third image frame of the second resolution.

12. The method according to claim 11, characterized in that The electronic device also includes a neural network processor NPU; In the process of the GPU using the second number of rays to render and obtain the third image frame, the method further includes: The GPU rendering obtains at least one of the following maps: an albedo map, a normal map, and a mask map; The CPU acquires and sends at least one of the albedo map, the normal map, and the mask map, and the third image frame to the NPU; The step of processing the third image frame to obtain the second image frame includes: The NPU uses at least one of the albedo map, the normal map, and the mask map, and the third image frame to perform noise reduction processing on the third image frame to obtain a noise-reduced image frame; The CPU acquires and sends the denoised image frame to the GPU; The GPU performs upsampling and sharpening processing on the denoised image frame to obtain the second image frame.

13. The method according to claim 12, characterized in that The noise reduction process is performed asynchronously with the upsampling process and the sharpening process.

14. An electronic device, characterized in that: The electronic device includes a display screen, a memory and a processor, and the display screen, the memory and the processor are coupled; wherein the memory stores computer program code, and the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1-13.

15. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 13.

Images