Image processing method and electronic equipment

Abstract

The invention discloses an image processing method and electronic equipment, relates to the technical field of image rendering, and can reduce the operand of a ray tracing algorithm and reduce the power consumption when the electronic equipment uses the ray tracing algorithm to render an image. In the process of running a first application by the electronic equipment, the method comprises the following steps: responding to a rendering event for a first image frame, and rendering by using a first number of light rays to obtain the first image frame. A first event is received, the first event is used for triggering the electronic equipment to adjust the view angle of the image frame, and the motion amplitude between the image frames before and after adjustment exceeds the first amplitude; and in response to a rendering event of a second image frame, rendering by using a second number of light rays to obtain a third image frame, and processing the third image frame to obtain a second image frame, the second number being less than the first number. Wherein the first image frame and the second image frame are two continuous image frames of the first application.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image rendering technology, and in particular to an image processing method and electronic device. Background Art

[0002] In scenarios such as gaming applications, virtual reality (VR), and augmented reality (AR), improving the realism of rendered images has always been a major focus of the industry. Therefore, ray tracing algorithms, as a technology that can generate rendering results close to the real world based on real physical laws, have also attracted much attention.

[0003] However, the ray tracing algorithm is a heavy-load technology. Implementing the ray tracing algorithm in electronic devices such as mobile phones and tablets will result in high power consumption of the electronic devices. As a result, it is difficult to continue using the ray tracing algorithm for image rendering in electronic devices such as mobile phones and tablets. Summary of the invention

[0004] The present application provides an image processing method and an electronic device, which can reduce the amount of calculation of a ray tracing algorithm and reduce the power consumption of an electronic device when using a ray tracing algorithm to render an image.

[0005] In a first aspect, the present application provides an image processing method, which is applied to an electronic device, wherein the electronic device includes a first application (such as a game application, a VR application, etc.), and the electronic device uses a ray tracing algorithm to render the image frame of the first application. In the process of the electronic device running the first application, the method includes: responding to a rendering event of the first image frame, using a first number of rays to render the first image frame. Receive a first event (such as an operation event of accelerating forward in a game application), the first event is used to trigger the electronic device to adjust the viewing angle of the image frame, and the movement amplitude between the image frames before and after the adjustment exceeds the first amplitude. In response to the rendering event of the second image frame, a second number of rays is used to render a third image frame, and the third image frame is processed to obtain a second image frame, and the second number is less than the first number. Among them, the first image frame and the second image frame are two consecutive image frames of the first application.

[0006] To summarize, by using the present application, before receiving a preset event, the electronic device can use a larger number of rays to render an image frame of a first application; after receiving the preset event, the electronic device can use a lower number of rays to render the image, and then further improve the image quality of the rendered image (such as improving resolution and reducing noise) to obtain an image frame of the first application.

[0007] On the one hand, electronic devices use fewer rays of light to render images, which can reduce the amount of calculations in the rendering process, reduce rendering power consumption, and improve rendering efficiency.

[0008] On the other hand, when the amplitude of motion between two frames of images is large, a proper reduction in image quality has little impact on the user's perception. At the same time, when an electronic device uses a ray tracing algorithm to render an image, the fewer the number of rays, the more difficult it is to guarantee the quality of the rendered image, and even if the image quality is improved later, it is difficult to achieve the effect of rendering with a large number of rays. Therefore, after detecting the first event, the electronic device uses a smaller number of rays to render the image and obtains the image frame of the first application after processing, which can reduce the amount of calculations in the rendering process while reducing the impact on the user's perception.

[0009] In a possible design manner of the first aspect, the resolutions of the first image frame and the second image frame are both the first resolution, and the resolution of the third image frame is the second resolution, which is lower than the first resolution. The first number is related to the number of pixels included in the first image frame of the first resolution, and the second number is related to the number of pixels included in the third image frame of the second resolution.

[0010] That is, electronic devices can reduce the amount of light required to render a frame of an image by reducing the image resolution.

[0011] In a possible design manner of the first aspect, processing the third image frame to obtain the second image frame includes: upsampling the third image frame to obtain the second image frame.

[0012] It is understandable that resolution is an important factor affecting image clarity. It is obviously unacceptable to reduce image quality in order to reduce rendering power consumption. Therefore, after rendering to obtain the third image, the electronic device may also perform upsampling processing on the third image to restore the resolution of the image (such as restoring to the first resolution).

[0013] Furthermore, the electronic device may also perform sharpening processing after the upsampling processing to enhance the boundaries of the image.

[0014] In a possible design of the first aspect, different first events correspond to different second resolutions. After receiving the first event, the second resolution corresponding to the received first event is determined.

[0015] Different first events may cause different motion amplitudes, and accordingly, the electronic device may configure different resolution parameters for different first events, that is, different first events correspond to different resolutions. In this way, the electronic device may reduce the amount of light to different degrees for different motion amplitudes.

[0016] In a possible design of the first aspect, before receiving the first event, the number of light rays passing through each pixel in the process of rendering a frame of image is a third number, and after receiving the first event, the number of light rays passing through each pixel in the process of rendering a frame of image is a fourth number, and the fourth number is less than the third number. The first number is related to the third number, and the second number is related to the fourth number.

[0017] That is, electronic devices can reduce the amount of light required to render a frame of an image by reducing the amount of light passing through each pixel.

[0018] In a possible design of the first aspect, processing the third image frame to obtain the second image frame includes: performing noise reduction processing on the third image frame and processing the third image frame after the noise reduction processing to obtain the second image frame.

[0019] In particular, in the implementation method of reducing the number of light rays passing through each pixel, the fewer the number of light rays passing through each pixel, the higher the possibility of color calculation failure of the pixel. That is, the more likely it is that noise will appear. Therefore, after rendering the third image frame, the electronic device further performs noise reduction processing on the third image frame, which can significantly reduce the noise in the image.

[0020] It should be noted that when the implementation method of reducing the image resolution and the implementation method of reducing the number of light rays passing through each pixel are used in combination, the electronic device can further perform upsampling processing on the third image to improve the resolution of the image after performing noise reduction processing on the third image to obtain the noise reduced third image.

[0021] In a possible design of the first aspect, denoising the third image frame includes: taking at least one of an albedo map, a normal map, and a mask map, and the third image frame as input, running a preset denoising model, and obtaining a denoised third image frame. The preset denoising model has the ability to remove noise from the noisy image frame based on at least one of the albedo map, the normal map, and the mask map, and the noisy image frame.

[0022] That is to say, the electronic device can further combine at least one of the albedo map, the normal map and the mask map on the basis of the third image to complete the noise reduction processing to improve the effect of the noise reduction processing.

[0023] In a possible design method of the first aspect, the RGB color information of any pixel in the albedo map is the RGB color information of the first intersection of the first light passing through the pixel and the three-dimensional scene. The RGB color information of any pixel in the normal map corresponds to the normal coordinates of the first intersection of the first light passing through the pixel and the three-dimensional scene. The RGB color information of any pixel in the mask map corresponds to the number of effective light paths in the light paths corresponding to the fourth number of light rays passing through the pixel, and the effective light path refers to the light path that intersects with the light source in the three-dimensional scene. Among them, the three-dimensional scene is a scene used to render the image frame of the first application.

[0024] In a possible design of the first aspect, different first events correspond to different fourth quantities. After receiving the first event, the fourth quantity corresponding to the received first event is determined.

[0025] That is to say, the electronic device can reduce the amount of light to different degrees for different first events.

[0026] In a possible design of the first aspect, receiving the first event includes: receiving an operation event of moving a character in an image frame, especially an operation event of accelerating the moving character (such as accelerating forward), which can cause a relatively large movement. Alternatively, receiving the first event includes: receiving an operation event of changing the viewing angle of the image frame.

[0027] In a possible design of the first aspect, the electronic device includes a central processing unit (CPU) and a graphics processing unit (GPU). Receiving a first event includes: a first application receives the first event. After the first application receives the first event, the first application sends a first instruction to the CPU, and the first instruction indicates that the first event is received. The CPU determines a resolution parameter corresponding to the first event, and the resolution parameter indicates that the image frame resolution is adjusted to a second resolution. The CPU sends a resolution parameter and a light parameter to the GPU, and the light parameter indicates that the number of rays passing through each pixel is a fourth number. Accordingly, in response to a rendering event of the second image frame, a third image frame is rendered using a second number of rays, including: in response to a rendering event of the second image frame, the GPU uses a second number of rays to render the third image frame, and the second number is the product of the fourth number and the number of pixels included in the third image frame of the second resolution.

[0028] That is to say, the CPU can determine the resolution parameters that match the first event, and pass the resolution parameters and light parameters to the GPU. Subsequently, the GPU can reduce the number of rays for rendering a frame of image based on the resolution parameters and light parameters, reduce the amount of rendering calculations, and reduce rendering power consumption.

[0029] In a possible design of the first aspect, the electronic device also includes a neural network processor NPU. In the process of the GPU using the second number of rays to render the third image frame, the GPU renders and obtains at least one of the following maps: an albedo map, a normal map, and a mask map. The CPU obtains and sends at least one of the albedo map, the normal map, and the mask map, and the third image frame to the NPU. Accordingly, the third image frame is processed to obtain the second image frame, including: the NPU uses at least one of the albedo map, the normal map, and the mask map, and the third image frame to perform noise reduction processing on the third image frame to obtain a noise-reduced image frame. The CPU obtains and sends the noise-reduced image frame to the GPU. The GPU performs upsampling and sharpening processing on the noise-reduced image frame to obtain a second image frame.

[0030] That is to say, in the process of rendering the image, the CPU will also render at least one of the albedo map, normal map and mask map for the NPU to complete the noise reduction processing later.

[0031] In a possible design of the first aspect, the noise reduction process is performed asynchronously with the upsampling process and the sharpening process. Through the asynchronous execution, after the GPU sends the rendered third image frame to the NPU, it does not need to wait for the NPU to obtain the third image frame after the noise reduction process before starting the rendering process of the next frame of the image.

[0032] In a second aspect, the present application further provides an electronic device, the electronic device comprising a display screen, a memory and one or more processors. The display screen, the memory and the processor are coupled. The memory is used to store computer program code, the computer program code comprising computer instructions, when the computer instructions are executed by the processor, the electronic device executes the method in the first aspect and any possible design thereof.

[0033] In a third aspect, the present application provides a chip system, which is applied to an electronic device including a display screen and a memory; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the method as in the first aspect and any possible design method thereof.

[0034] In a fourth aspect, the present application provides a computer storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method of the first aspect and any possible design thereof.

[0035] In a fifth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method of the first aspect and any possible design thereof.

[0036] It can be understood that the beneficial effects that can be achieved by the electronic device of the second aspect, the chip system of the third aspect, the computer storage medium of the fourth aspect, and the computer program product of the fifth aspect provided above can be referred to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of a ray tracing algorithm provided in an embodiment of the present application;

[0038] Figure 2 A hardware structure diagram of a mobile phone provided in an embodiment of the present application;

[0039] Figure 3 A diagram of the software and hardware architecture of a mobile phone provided in an embodiment of the present application;

[0040] Figure 4 A simplified diagram of an image rendering process provided by an embodiment of the present application;

[0041] Figure 5 A schematic diagram of a preset event provided in an embodiment of the present application;

[0042] Figure 6 A flowchart of an image processing method provided in an embodiment of the present application;

[0043] Figure 7 A schematic diagram of upsampling and sharpening processing provided in an embodiment of the present application;

[0044] Figure 8 A flowchart of another image processing method provided in an embodiment of the present application;

[0045] Fig. 9 A schematic diagram of obtaining an albedo map and a normal map provided in an embodiment of the present application;

[0046] Fig.10 A schematic diagram of a method for obtaining a mask map provided in an embodiment of the present application;

[0047] Fig.11 A process diagram of noise reduction processing provided in an embodiment of the present application;

[0048] Fig.12 A flowchart of another image processing method provided in an embodiment of the present application;

[0049] Fig.13A timing interaction diagram of the image processing method provided in an embodiment of the present application;

[0050] Fig.14 A simplified flowchart of the image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear indication to the contrary in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in a "or" relationship.

[0052] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connection" includes direct connection and indirect connection, unless otherwise specified. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0053] In the embodiments of the present application, the words "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0054] In some specific scenarios, electronic devices need to complete a large amount of image rendering. Typically, in scenarios where game applications, AR applications, VR applications, and modeling applications are run in electronic devices, electronic devices are often required to complete a large amount of image rendering. Then, the electronic device displays the rendered image.

[0055] Exemplarily, the electronic device may be installed with the above-mentioned various applications (Application, APP). When the application needs to display an image through the electronic device, it can send instructions to the electronic device so that the electronic device completes the rendering of the image according to the instructions, and finally displays the rendered image through the display screen.

[0056] In the following examples, the application is taken as an example to illustrate the present application scheme. It is understandable that the game application can display the game screen to the user through the electronic device during operation. The game screen can be a video screen, and the video screen can be composed of multiple frame images played continuously.

[0057] The frame image is the rendering result of the electronic device rendering the three-dimensional scene of the game. The three-dimensional scene includes models of buildings, grass, trees, light sources (such as the sun, sky, desk lamps, etc.). It can be understood that the three-dimensional scene is usually represented by a triangular mesh.

[0058] The above image rendering can be achieved based on a ray tracing algorithm.

[0059] See also Figure 1 , the three-dimensional scene includes a sphere, a cube and a light source, and the image to be rendered is image 1. Using the ray tracing algorithm, the electronic device can transform a pixel point (such as Figure 1 The rendering task of pixel 1) in is split into several rays (such as Figure 1 The influence of light rays ①, ②, ..., x (x light rays in total) on the three-dimensional scene. It can be understood that the camera refers to a virtual camera, and the camera posture is determined by the electronic device according to the user's operation in the game. Among them, the electronic device can convert the user's operation of moving forward, backward, switching perspectives, etc. in the game into rotation or translation of the camera accordingly, so as to determine the corresponding camera posture.

[0060] Specifically, the electronic device may determine the color information of the pixel point according to the light paths of the plurality of light rays in the three-dimensional scene. Figure 1The light path of light rays ①, ②, ..., x in the three-dimensional scene is used to determine the color information of pixel 1. The following takes light ① as an example to illustrate the tracing process of one of the light rays by the electronic device in the ray tracing algorithm: the electronic device can intersect light ① with a large number of triangular meshes corresponding to the sphere, cube, and light source in the three-dimensional scene, and obtain the nearest intersection point that intersects with light ① as intersection point A. The electronic device uses light ① as the incident light and intersection point A as the incident point, and obtains the first reflected light and the first refracted light through integral calculation, such as the first reflected light being light (1). The electronic device continues to intersect light (1) with a large number of triangular meshes corresponding to the sphere, cube, and light source in the three-dimensional scene, and obtains the nearest intersection point that intersects with light (1) as intersection point B. The electronic device continues to use light (1) as the incident light and intersection point B as the incident point, and obtains the second reflected light and the second refracted light through integral calculation, such as the second reflected light being light (2)... and so on, until it intersects with the light source, or the above-mentioned iterative process reaches a preset number of times, then the tracing process for light ① ends. Following a similar process, the electronic device completes the tracing of several light rays.

[0061] Finally, the electronic device can obtain the color information of the pixel point based on the color information at the intersection points (such as intersection point A and intersection point B) of the three-dimensional scene during the tracing process of the plurality of rays. Figure 1 The color information of the intersection of light rays ①, ②, ..., x with the three-dimensional scene during the tracing process is obtained. Figure 1 The color information of pixel 1 in the image.

[0062] In other words, by using ray tracing technology, electronic devices can simulate the path of light in the real world and generate rendering results that are close to the real world.

[0063] At the same time, the ray tracing algorithm requires a large number of intersection calculations and integral operations, which in turn requires a large amount of computing resources. Therefore, the ray tracing algorithm is also a heavy-load technology. Implementing the ray tracing algorithm in electronic devices will result in high power consumption of the electronic devices.

[0064] In order to reduce the power consumption of electronic devices when using ray tracing algorithms, the industry has proposed a hardware acceleration strategy. That is, electronic devices use hardware to complete some operations. For example, since the logic of the intersection of light and triangular meshes is simple, and mainly because of the large number of triangular networks, the intersection process requires a lot of calculations. This task with simple logic but high computational volume is just suitable for hardware. Therefore, a typical hardware acceleration strategy is: electronic devices use hardware to complete the intersection operation in the process of ray tracing algorithms, thereby reducing the power consumption of electronic devices using ray tracing algorithms.

[0065] The hardware-accelerated ray tracing algorithm can be applied in electronic devices such as personal computers (PCs), consoles, mobile phones, and tablets.

[0066] However, in order to improve the user experience, higher requirements are placed on rendering efficiency. For example, in high-frame-rate game applications, more images need to be rendered per unit time, and the amount of computation required by electronic devices per unit time is also greater, which still leads to high power consumption of electronic devices. Especially for mobile terminals, ray tracing algorithms are still a high-load technology. This makes it difficult to continue using ray tracing algorithms for image rendering in mobile terminals such as mobile phones and tablets.

[0067] Based on the above problems, an embodiment of the present application provides an image processing method, which can accelerate the ray tracing algorithm from the software level, further reduce the amount of computation required to render each frame of an image using the ray tracing algorithm, and thus improve rendering efficiency.

[0068] In the process of rendering an image using a ray tracing algorithm, the main computational resources consumed are the intersection operations of light rays and triangular meshes and the integral operations for determining reflected light rays, refracted light rays, etc. In addition, in the process of rendering a frame of an image, the more light rays emitted by the camera, the more light rays and triangular meshes the electronic device needs to perform intersection operations on, and the amount of intersection operations will increase accordingly; and the more light rays the electronic device needs to perform integral operations on, the more integral operations will increase accordingly. Therefore, in an embodiment of the present application, in the process of rendering a frame of an image using a ray tracing algorithm, the electronic device can reduce the amount of operations in the rendering process by reducing the number of light rays emitted by the camera, thereby reducing rendering power consumption and improving rendering efficiency. It can be understood that the light emitted by the camera refers to the light rays emitted from the camera as the starting point and passing through each pixel.

[0069] Exemplarily, the electronic device may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, and / or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic device.

[0070] See also Figure 2 , is a hardware structure diagram of an electronic device. Figure 2As shown, taking the electronic device as a mobile phone as an example, the electronic device may include a processor 210, an external memory interface 220, an internal memory (referred to as memory) 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc.

[0071] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the mobile phone. In other embodiments, the mobile phone may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0072] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a central processing unit (CPU), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0073] The charging management module 240 is used to receive the charging input of the charger. The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives the input of the battery 242 and / or the charging management module 240, and supplies power to the processor 210, the internal memory 221, the display screen 294, the camera 293, and the wireless communication module 260. The power management module 241 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc.

[0074] The wireless communication function of the electronic device can be realized by antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modulation and demodulation processor and baseband processor. Mobile communication module 250 can provide solutions for wireless communication including 2G / 3G / 4G / 5G applied in electronic devices. Wireless communication module 260 can provide solutions for wireless communication including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), Bluetooth low energy (BLE), ultra wide band (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. applied in electronic devices.

[0075] The electronic device can realize the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0076] The electronic device can realize the camera function through the camera 293, ISP, video codec, GPU, display screen 294, application processor AP, neural network processor NPU, etc.

[0077] The electronic device can implement audio functions such as music playing and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone jack 270D, and the application processor.

[0078] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. TM ) system as an example to illustrate the software structure of electronic devices. The layered architecture divides the software system of electronic devices into several layers, each layer has a clear role and division of labor, and the layers communicate with each other through software interfaces.

[0079] See also Figure 3Taking mobile phones as an example, the software and hardware architecture of electronic devices can include application (APP) layer, framework layer, system library and hardware layer.

[0080] The application layer may also be referred to as the application program layer. The application program layer may include a series of application packages. The application packages may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc. In some embodiments, the application program layer may also include applications (also referred to as first applications) that need to display images or videos to users after image rendering. For example, the application program layer may also include game applications, such as Etc. As another example, the application layer may also include AR applications, VR applications, modeling applications, etc.

[0081] The framework layer can also be called the application framework layer. The framework layer can provide an application programming interface (API) and a programming framework for the application of the application layer. The framework layer includes some predefined functions. Exemplarily, the framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, etc.

[0082] In the embodiment of the present application, one or more functional modules may be provided in the framework layer to implement the image processing method provided in the embodiment of the present application. It is understandable that these functional modules are controlled by the CPU to perform corresponding functions.

[0083] The system library can include multiple functional modules, such as surface manager, media framework, standard C library (libc), SQLite, graphics library, etc.

[0084] The graphics library may include at least one of the following: Open Graphics Library (OpenGL), OpenGL for Embedded Systems (OpenGL ES), Vulkan, etc. The graphics library may provide drawing and operation of 2D graphics and 3D graphics in the application. In some embodiments, the CPU or GPU may call the graphics library to implement image rendering.

[0085] The hardware layer may include a processor, such as a CPU, a GPU, an NPU, etc. Also, the hardware layer may include a memory. In an embodiment of the present application, a frame buffer may be created in the memory for storing rendering results and noise reduction processing results.

[0086] See also Figure 4 In the process of image rendering, the application, CPU, GPU, memory and display screen in the above-mentioned hardware and software structure of the electronic device work as follows: the application can issue a rendering instruction to instruct the rendering of an image. After receiving the rendering instruction, the CPU can call the corresponding graphics drawing application programming interface (Application Programming Interface, API) in the image library to instruct the GPU to perform the rendering operation corresponding to the rendering instruction. The GPU executes the rendering instruction and stores the rendered image in the memory. Finally, the rendered image in the memory can be sent to the display screen for display.

[0087] In some embodiments, after receiving the rendering instruction sent by the application, the CPU may also determine parameters for reducing the number of rays emitted by the camera (such as the resolution parameter and / or ray parameter described below). After receiving the parameters for reducing the number of rays emitted by the camera, the GPU may perform a process for reducing the rays.

[0088] Exemplarily, the GPU may reduce the resolution of the image to be rendered according to the resolution parameter, thereby reducing the number of rays emitted by the camera during the process of rendering the entire image.

[0089] As another example, the GPU may reduce the light passing through each pixel according to the light parameters, thereby reducing the number of light emitted by the camera during the process of rendering the entire image.

[0090] When reducing light by reducing the image resolution, the NPU in the electronic device can perform noise reduction on the GPU rendered image and store the noise-reduced image in the memory. Finally, the noise-reduced image in the memory can be sent to the display for display.

[0091] By reducing the light emitted by each pixel, the GPU in the electronic device can also upsample the rendered image to increase the image resolution and store the upsampled high-resolution image in the memory. Finally, the high-resolution image in the memory can be sent to the display for display.

[0092] The image processing method provided in the embodiment of the present application can be implemented in an electronic device having the above-mentioned software and hardware structure.

[0093] In an embodiment of the present application, the electronic device reduces the amount of computation in the rendering process by reducing the number of rays emitted by the camera. In practice, the electronic device may reduce the number of rays emitted by the camera when rendering each frame of an image. Thereby reducing the amount of computation in the rendering process of each frame of an image. Alternatively, after detecting a first event, the electronic device may reduce the number of rays emitted by the camera when rendering each frame of an image (also referred to as a second image frame), and use a lower number of rays (also referred to as a second number) to render the image. Thereby reducing the amount of computation in the rendering process of each frame of an image after the first event. Among them, the first event may include: an event that triggers the electronic device to adjust the viewing angle of the displayed image, and the amplitude of motion between the image frames before and after the adjustment exceeds the first amplitude.

[0094] It should be noted that when the amplitude of motion between two frames of images is large, a proper reduction in image quality will have little impact on the user's perception. At the same time, when an electronic device uses a ray tracing algorithm to render an image, the fewer the number of rays, the more difficult it is to guarantee the quality of the rendered image, and even if subsequent processing to improve the image quality is performed (such as noise reduction, upsampling, sharpening, etc. described below), it is difficult to achieve the effect of rendering with a large number of rays. Therefore, after the electronic device detects the first event, when rendering each frame of the image, the electronic device reduces the number of rays emitted by the camera, which can reduce the amount of calculations in the rendering process while reducing the impact on the user's perception.

[0095] Taking a game application as an example, the first event may be an accelerated forward event. When a character in a game video frame moves quickly, not only the perspective will change, but also, due to the fast moving speed, a large change will occur between the two frames of displayed images. For example, long pressing the forward button triggers accelerated forward. When the electronic device receives the user's Figure 5 After the forward button 5011 in the game interface 501 is long pressed, the electronic device can reduce the amount of light emitted by the camera when rendering each subsequent frame of the image. In this way, not only can the amount of calculation in the rendering process be reduced, but also the user's viewing experience will not be significantly reduced.

[0096] Still taking the game application as an example, the user can directly change the viewing angle of the screen by sliding, and the sliding operation usually causes a significant change in the viewing angle. The first event can also be the operation of sliding to change the viewing angle. Figure 5 After the sliding operation in the game interface 502, the number of rays emitted by the camera can be reduced when rendering each subsequent frame of the image, which not only reduces the amount of calculation in the rendering process, but also does not significantly reduce the user's viewing experience.

[0097] It is understandable that before the electronic device detects the first event, the camera may emit a default amount (which may be referred to as the first amount) of light to render a frame of image (which may be referred to as the first image frame). That is, before and after the electronic device detects the preset event, the amount of light emitted by the camera in the process of continuously rendering two frames of image is different.

[0098] And, after the first event ends, the electronic device can restore to using the default number of rays to render the image. For example, after the user ends the long press operation on the forward button 5011 or after the user ends the sliding operation in the game screen 502, the electronic device can restore to using the default number of rays to render the image.

[0099] Among them, the electronic device can reduce the number of light emitted by different frames of images to the same number. For example, the electronic device can reduce the number of light emitted by the camera during the rendering process of each frame of image from a larger value to a smaller value. Alternatively, the electronic device can also reduce the number of light emitted by the camera during the rendering process of different frames of image to different numbers. Exemplarily, the electronic device can reduce the number of light emitted by the camera during the rendering process of a frame of image to a number that matches the first event. For example, if the first event is event 1, the number of light emitted by the camera during the rendering process of a frame of image is number 1; if the first event is event 2, the number of light emitted by the camera during the rendering process of a frame of image is number 2. Another example is that the electronic device can dynamically adjust the number of light emitted by the camera based on power consumption, such as the higher the power consumption, the less light the camera emits, and the lower the power consumption, the more light the camera emits.

[0100] Specifically, the electronic device can reduce the amount of light emitted by the camera by reducing the resolution of the image; or the electronic device can reduce the amount of light emitted by the camera by reducing the amount of light passing through each pixel; or the electronic device can reduce the amount of light emitted by the camera by combining reducing the resolution of the image and reducing the amount of light passing through each pixel. The image processing method provided by the embodiments of the present application will be described below with respect to these methods respectively.

[0101] Method one is to reduce the amount of light emitted by the camera by lowering the resolution of the image.

[0102] It is understandable that when the image resolution is reduced, the number of image pixels is reduced. In the process of rendering an image using ray rendering technology, the camera needs to emit light passing through each pixel. Then, after the number of pixels is reduced, the electronic device only needs to emit light passing through a smaller number of pixels, thereby reducing the number of light emitted by the camera. That is, the image resolution or the number of image pixels is positively correlated with the number of light emitted by the camera.

[0103] See also Figure 6 In S1, before reducing the resolution, the image resolution is M*N (also called the first resolution), which includes M*N pixels. The number of rays emitted by the camera passing through each pixel is x. In the rendering process of one frame of image, the camera needs to emit a total of M*N*x rays.

[0104] After reducing the resolution, for example, reducing the resolution to d times of the original (also called the second resolution), the width and height of the image will be reduced to d times of the original. Correspondingly, the number of pixels is also reduced from M*N to (d*M)*(d*N). The number of rays emitted by the camera passing through each pixel is still x. In the rendering process of one frame of image, the camera needs to emit a total of (d*M)*(d*N)*x rays, and the number of rays is reduced.

[0105] For example, if d=0.5, then after reducing the resolution, the number of rays emitted by the camera is only (0.5M*0.5N*x) / (M*N*x)=0.25 times the number of rays emitted by the camera before reducing the resolution.

[0106] That is to say, before reducing the resolution, the number of rays that the camera needs to emit during the rendering of a frame of image is related to the number of pixels in the image without reducing the resolution. After reducing the resolution, the number of rays that the camera needs to emit during the rendering of a frame of image is related to the number of pixels in the image after reducing the resolution. Generally speaking, the more pixels there are, the more rays the camera needs to emit during the rendering of a frame of image.

[0107] Continue to see Figure 6 In S2, after reducing the resolution, the electronic device may use a ray tracing algorithm to render the low-resolution image to obtain a rendered image (also referred to as a third image frame), such as an ellipse 601 included in the rendered image.

[0108] It is understandable that resolution is an important factor affecting image clarity. It is obviously unacceptable to reduce image quality in order to reduce rendering power consumption. Therefore, you can continue to refer to Figure 6 In S3, after obtaining the rendered image, the electronic device may also perform upsampling processing on the rendered image, thereby restoring the resolution of the image (such as restoring it to the first resolution) and obtaining a high-resolution image (also referred to as a fourth image frame). In the embodiments of the present application, the specific implementation of upsampling is not limited.

[0109] In a specific implementation, the electronic device may use a similar Figure 7 The way shown is to achieve image upsampling:

[0110] Taking the example that the resolution after reduction is 0.5 of the resolution before reduction, the electronic device can use the color information uv of the pixel in the uth row and vth column in the rendered image as the color information of the pixel in the 2u-1th row and 2v-1th column in the high-resolution image.

[0111] See also Figure 7 The electronic device can use the color information 11 of the pixel in the 1st row and 1st column of the rendered image as the color information of the pixel in the 1st row and 1st column of the high-resolution image, use the color information 12 of the pixel in the 1st row and 2nd column of the rendered image as the color information of the pixel in the 1st row and 3rd column of the high-resolution image… use the color information 21 of the pixel in the 2nd row and 1st column of the rendered image as the color information of the pixel in the 3rd row and 1st column of the high-resolution image, use the color information 22 of the pixel in the 2nd row and 2nd column of the rendered image as the color information of the pixel in the 3rd row and 3rd column of the high-resolution image… until the color information of all pixels in the rendered image is mapped to the corresponding pixels in the high-resolution image.

[0112] For a pixel whose color information is unknown in a high-resolution image, the electronic device can calculate the color information of the pixel based on the color information of the pixel adjacent to the pixel. Among them, two pixels that are adjacent to each other by a side are called adjacent pixels.

[0113] Continue to see Figure 7 , the electronic device can average the color information 11 of the pixel in the 1st row and the 1st column and the color information 12 of the pixel in the 1st row and the 3rd column in the high-resolution image to obtain the color information ① of the pixel in the 1st row and the 2nd column; average the color information 11 of the pixel in the 1st row and the 1st column and the color information 21 of the pixel in the 3rd row and the 1st column in the high-resolution image to obtain the color information ② of the pixel in the 2nd row and the 1st column; average the color information 12 of the pixel in the 1st row and the 3rd column and the color information 22 of the pixel in the 3rd row and the 3rd column in the high-resolution image to obtain the color information ③ of the pixel in the 2nd row and the 3rd column; average the color information 21 of the pixel in the 3rd row and the 1st column and the color information 22 of the pixel in the 3rd row and the 3rd column in the high-resolution image to obtain the color information ④ of the pixel in the 3rd row and the 2nd column; average the color information ①, the color information ②, the color information ③ and the color information ④ to obtain the color information ⑤ of the 2nd row and the 2nd column ... until the color information of all the pixels in the high-resolution image is obtained.

[0114] In some embodiments, after upsampling to obtain a high-resolution image, the electronic device may further perform sharpening processing on the high-resolution image to further improve the clarity of the image. In the embodiments of the present application, the specific implementation of the sharpening processing is not limited.

[0115] In a specific implementation, the electronic device can calculate the color information of the pixel to be sharpened after sharpening based on the maximum and minimum values ​​of the color of the g channel in the color information of the pixel to be sharpened and multiple adjacent pixels in the high-resolution image, and the color information of the pixel to be sharpened and the multiple pixels. It can be understood that the color information of each pixel includes the colors of three channels, namely the R channel, the G channel and the B channel.

[0116] The pixel to be sharpened may be any pixel in the high-resolution image, and the adjacent pixels include pixels that are adjacent to the edge of the pixel to be sharpened.

[0117] The pixel to be sharpened is Figure 7 Taking the pixel point P in the bold dashed frame of the high-resolution image as an example, the adjacent multiple pixel points may include 8 pixel points in the dashed frame except the pixel point P, such as pixel point a to pixel point h.

[0118] Compared with only using the pixels in the four directions of the top, bottom, left and right of the pixel to be sharpened or even fewer directions for sharpening, this implementation uses the eight pixels adjacent to the edge of the pixel to be sharpened for sharpening, which can obtain better sharpening results and will not significantly increase the amount of calculation and access consumption of the sharpening process. Therefore, the sharpening effect can be improved while the computing power consumption remains basically unchanged.

[0119] Exemplarily, the electronic device may calculate the maximum and minimum values ​​of the colors of the G channel in the color information of the pixel points a to h and the pixel point P:

[0120] For example, the maximum value max_g = max(Pg, ag, bg, cg, dg, eg, fg, gg, hg);

[0121] For example, the minimum value min_g=min(Pg, ag, bg, cg, dg, eg, fg, gg, hg).

[0122] Among them, ××.g is the color of the G channel in the color information of pixel point ××.

[0123] Furthermore, the electronic device may calculate a sharpening coefficient based on the maximum value and the minimum value, and then use the sharpening coefficient, the pixel to be sharpened and the color information of multiple adjacent pixels to calculate the sharpened color information of the pixel to be sharpened.

[0124] Exemplarily, the calculation formula of the sharpening coefficient sharpen is as follows:

[0125]

[0126] That is to say, if min_g is less than 1-max_g, the sharpening coefficient is positively correlated with min_g / max_g; if mun_g is greater than 1-max_g, the sharpening coefficient is positively correlated with (1-max_g) / max_g.

[0127] And, the calculation formula of the color information color of the pixel to be sharpened after sharpening is as follows:

[0128]

[0129] Among them, ××.rgb refers to the color information of pixel x. w is the correction constant of the sharpening coefficient, 0<w<1.

[0130] Method 2: Reduce the amount of light emitted by the camera by reducing the amount of light passing through each pixel.

[0131] When rendering an image using a ray tracing algorithm, the camera will emit light that passes through each pixel. Therefore, after reducing the light that passes through each pixel, the number of rays emitted by the camera during the rendering of the entire image will also decrease. In other words, the number of rays that pass through each pixel is positively correlated with the number of rays emitted by the camera to render the entire image.

[0132] See also Figure 8 In S1, before reducing the light passing through each pixel, x light rays (also referred to as the third number) are emitted through each pixel, and a frame of image includes M*N pixels. In the rendering process of a frame of image, the camera needs to emit a total of M*N*x light rays.

[0133] After reducing the light passing through each pixel, y light rays (also referred to as the fourth number) are emitted through each pixel, where y is less than x. A frame of image also includes M*N pixels. Therefore, during the rendering process of a frame of image, the camera needs to emit a total of M*N*y light rays.

[0134] Obviously, reducing the number of rays emitted by the camera after passing through each pixel is only equivalent to reducing the number of rays emitted by the camera before passing through each pixel by (M*N*y) / (M*N*x)=y / x, and the number of rays is reduced.

[0135] That is to say, when the number of rays passing through each pixel is y, the number of rays that the camera needs to emit during the rendering of a frame of image is related to y. When the number of rays passing through each pixel becomes x, the number of rays that the camera needs to emit during the rendering of a frame of image is related to x. Generally, the larger y or x is, the more rays the camera needs to emit during the rendering of a frame of image.

[0136] Continue to see Figure 8 In S2, a ray tracing algorithm is used, and the number of light rays emitted through each pixel is y (such as 4, 5, etc.), and a rendered image (also referred to as a third image) can be obtained.

[0137] It is understandable that the less light passes through each pixel, the higher the possibility of color calculation failure of the pixel. That is, the more likely it is that noise will appear. For example, Figure 8 The rendered image obtained by rendering includes many noise points (shown in black in the figure).

[0138] Therefore, continue to see Figure 8 In S3, after obtaining the rendered image, the electronic device may further perform noise reduction processing on the rendered image to eliminate noise in the rendered image and obtain a noise-reduced image (also referred to as a third image after noise reduction).

[0139] In the embodiments of the present application, the specific implementation of noise reduction is not limited. For example, the electronic device can use an artificial intelligence (AI) noise reduction model, such as the industry-recognized OptiX 5 with AI-acceleratedDenoising, to perform noise reduction on the rendered image.

[0140] In some embodiments, the electronic device may also use a self-developed preset AI denoising model to perform denoising on the rendered image. The input of the preset AI denoising model includes at least the rendered image, and further, the output of the preset AI denoising model may also include at least one of the following: an albedo map, a mask map, and a normal map.

[0141] The Albedo map records the base color of each model in the 3D scene without lighting. The Normal map is used to record the normal information of the intersection of the light and the 3D scene. Both the Albedo map and the Normal map are RGB images.

[0142] In a specific implementation, the color information of each pixel in the above-mentioned Albedo map is the color information of the first intersection point of the first light ray passing through the pixel in the image to be rendered and the three-dimensional scene. Also, the color information of each pixel in the above-mentioned Normal map corresponds to the normal coordinates of the first intersection point, and different normal coordinates correspond to different color information. Among them, the color information of each pixel in the Normal map includes the color values ​​of three color channels, namely the R channel, the G channel and the B channel, and the normal coordinates include the coordinate values ​​of three axes, namely the x-axis, the y-axis and the z-axis. Therefore, the color values ​​of the three color channels can correspond to the coordinate values ​​of the three axes respectively. For example, the color value of the R channel corresponds to the coordinate value of the x-axis, the color value of the G channel corresponds to the coordinate value of the y-axis, and the color value of the B channel corresponds to the coordinate value of the z-axis.

[0143] The image to be rendered is Fig. 9 Taking the image 1 shown, the first light ray passing through the pixel 1 in the image 1 is the light ray ①, and the first intersection point of the light ray ① with the three-dimensional scene is the intersection point A. Then, the color information of the pixel 1 in the Albedo map is the color information of the intersection point A, and the color information of the pixel 1 in the Normal map corresponds to the normal coordinate of the intersection point A.

[0144] Exemplarily, the color information and normal coordinates of the first intersection point can be obtained as follows:

[0145] The first intersection point is Fig. 9 Taking the intersection A shown as an example, intersection A is located in the triangle index shown in the shaded area. The electronic device can obtain the material map (such as a color map) of the three vertices in the triangle index and the normal coordinates of the three vertices. It can be understood that the normal coordinates of each vertex here refer to the normal coordinates in the local coordinate system of the triangle index. The electronic device can obtain the color information of intersection A based on the color map of the three vertices, and can obtain the normal coordinates of intersection A based on the normal coordinates of the three vertices. For example, the electronic device can obtain the color information and normal coordinates of intersection A by using barycentric interpolation.

[0146] It should be noted that the normal coordinates of the three vertices refer to the coordinates in the local coordinate system of the triangle index, so the normal coordinates of the intersection point A obtained according to the normal coordinates of the three vertices are also the coordinates in the local coordinate system. Therefore, the electronic device can further convert the normal coordinates of the intersection point A to the camera coordinate system, so that the normal coordinates of the intersection point A are the coordinates in the camera coordinate system. For example, the electronic device can use the following formula to convert the normal coordinates of the intersection point A to the camera coordinate system:

[0147] cameraNormalInter=localNormalInter*viewMatrix,

[0148] Among them, cameraNormalInter is the normal coordinate of the intersection point A in the camera coordinate system, localNormalInter is the normal coordinate of the intersection point A in the local coordinate system, and viewMatrix is ​​the spatial position matrix of the camera coordinate system.

[0149] The Mask map records: information about the number or proportion of effective light paths for each pixel. The number of effective light paths for each pixel refers to the number or proportion of effective light paths among the multiple light paths corresponding to the multiple light rays passing through each pixel. Alternatively, the proportion of effective light paths for each pixel refers to the proportion of the number of effective light paths among the multiple light paths corresponding to the multiple light rays passing through each pixel. In addition, the Mask map is a grayscale image. In the following, the proportion of effective light paths is mainly used as an example. An effective light path refers to a light path that can reach the light source, and an invalid light path refers to a light path that cannot reach the light source.

[0150] The image to be rendered is Fig.10 For example, in the image 1 shown, there are 4 rays passing through pixel 1, namely, ray ①, ray ②, ray ③ and ray ④. Light path 1 corresponding to ray ①, light path 2 corresponding to ray ② and light path 4 corresponding to ray ④ can all reach the light source. Therefore, light paths 1, 2 and 4 are valid light paths. Light path 3 corresponding to ray ③ cannot reach the light source. Therefore, light path 3 is an invalid light path. The electronic device can calculate that the ratio of valid light paths of pixel 1 is 3 / 4.

[0151] In a specific implementation, the color information of each pixel in the above-mentioned Mask map corresponds to the ratio of the effective light path of the pixel in the image to be rendered, and different ratios of effective light paths correspond to different color information.

[0152] Take the corresponding relationship between the ratio of effective light path and color information as shown in Table 1 below as an example:

[0153] Table 1

[0154] Ratio of effective optical path 0 1 / 4 2 / 4 3 / 4 1 Color Information 0 65 130 200 255

[0155] Continue to use Fig.10 In the example, the ratio of the effective light path of pixel 1 is 3 / 4, then based on Table 1 above, it can be determined that the color information of pixel 1 in the Mask map is 200.

[0156] That is, in the process of rendering to obtain a rendered image, the electronic device can obtain the above-mentioned Albedo map, Mask map and Normal map.

[0157] Subsequently, the electronic device uses at least one of the Albedo map, the Mask map, and the Normal map, as well as the rendered image as input, runs a preset AI denoising model, and can output a denoised image. Fig.11 , the electronic device can use the rendered image, Albedo map and Mask map as input of the preset AI denoising model to predict the denoised image.

[0158] The present application embodiment will exemplarily illustrate the process of training to obtain a preset AI noise reduction model:

[0159] Multiple noise image samples and Albedo map samples, Mask map samples and Normal map samples corresponding to each noise image sample can be collected to obtain multiple groups of input samples. Each group of input samples includes a noise image sample and corresponding Albedo map samples, Mask map samples and Normal map samples. In addition, the denoised image samples corresponding to each noise image can be collected to obtain multiple output samples. The preset network model is trained using multiple groups of input samples and multiple output samples, and the model parameters of the network model are adjusted according to the error between the actual training output and the output sample until the error between the actual training output and the output sample is less than the preset error, and the training is terminated. The network model obtained at the end of the training is the preset AU denoising model.

[0160] The above article describes the specific implementation of method 1 and method 2 respectively. In actual implementation, method 1 and method 2 can also be combined. In other words, the resolution can be reduced, and the light passing through each pixel can be reduced to further reduce the amount of light emitted by the camera. Accordingly, after obtaining the rendered image, the electronic device needs to perform noise reduction on the rendered image on the one hand, and increase the resolution of the rendered image on the other hand.

[0161] For example, see Fig.12 In S1, the electronic device can reduce the image resolution from M*N to 0.5M*0.5N; and, see Fig.12 In S2, the electronic device can reduce the light passing through each pixel from x to 4, where x is much larger than 4. Fig.12 In S3, the ray tracing algorithm is used. The camera emits four rays passing through each pixel in the image with a resolution of 0.5M*0.5N. A rendered image with a resolution of 0.5M*0.5N can be obtained, but multiple pixels (shown as black areas in the figure) have noise. Fig.12 In S4, the electronic device can perform noise reduction processing on the rendered image to obtain a noise-reduced image with a resolution of 0.5M*0.5N and reduced noise (indicated by the disappearance of the black area in the figure). And, continue to refer to Fig.12In S5, the electronic device can upsample the noise-reduced image to obtain a high-resolution image with a resolution increased to M*N.

[0162] In the following, the present application scheme is mainly described by combining the first method with the second method, and using the rendered image, Albedo map and Normal map for noise reduction processing as an example, and combining the hardware and software structure of the electronic device. Fig.13 , the image processing method provided in the embodiment of the present application includes:

[0163] S1300: The game application receives a first event.

[0164] Regarding the first incident, please refer to the previous description and I will not repeat it here.

[0165] S1301. The game application sends instruction 1 to the CPU, where instruction 1 indicates that a first event has been received.

[0166] Instruction 1 may also be referred to as the first instruction.

[0167] S1302: In response to instruction 1, the CPU determines a resolution parameter that matches the first event, where the resolution parameter is used to adjust image resolution.

[0168] The resolution parameter may be a ratio of the adjusted image resolution to the default resolution of the game application, such as 0.5, 0.75, etc. For example, if the resolution parameter is 0.5, it indicates that the adjusted image resolution is 0.5 of the initial resolution. For example, if the initial resolution is M*N, the adjusted image resolution should be 0.5M*0.5N.

[0169] Different first events may cause different motion amplitudes, and accordingly, the electronic device may configure different resolution parameters for different first events, that is, different first events correspond to different resolutions. In this way, the electronic device may reduce the amount of light to different degrees for different motion amplitudes.

[0170] Taking the resolution parameter as an example, if the first event is an operation event of accelerating forward, the resolution parameter may be 0.75; if the first event is an operation event of sliding to change the viewing angle, the resolution parameter may be 0.5.

[0171] Furthermore, the same first event but different event parameters may also lead to different motion amplitudes. Accordingly, different resolution parameters may be configured for the same first event with different event parameters. In this way, the amount of light emitted by the camera may be reduced to different degrees for different motion amplitudes. Accordingly, the event parameters may be carried in instruction 1 for the CPU to determine the resolution parameters.

[0172] The event parameters include parameters that can affect the range of motion. Exemplarily, the event parameters include but are not limited to at least one of the following: operation type (such as single click, double click, long press, slide, etc.), control information, and slide distance, speed, etc.

[0173] Furthermore, the larger the motion amplitude caused by the event parameter, the smaller the resolution parameter is configured. The larger the motion amplitude, the greater the reduction in image quality and the smaller the impact on the user's perception. In this way, the amount of light emitted by the camera can be reduced to a greater extent while ensuring a smaller impact on the user's perception.

[0174] Take the operation event of sliding to change the viewing angle as an example. If the sliding distance is large, a large-angle viewing angle switch can be triggered, resulting in a large movement amplitude. The resolution parameter can be d1 times. If the sliding distance is small, a small-angle viewing angle switch can be triggered, resulting in a small movement amplitude. The resolution parameter can be d2 times. Both d1 and d2 are natural numbers greater than 0 and less than 1, and d1 is less than d2. For example, d1 = 0.5, d2 = 0.75. In this way, not only can the amount of light emitted by the camera be reduced to varying degrees, but it will not have a significant impact on the user's perception.

[0175] S1303: The CPU determines light parameters, where the light parameters are used to indicate the amount of emitted light.

[0176] The light parameter may be the number of light rays passing through each pixel. For example, if the light parameter is 4, it means that the number of light rays passing through each pixel is 4. For another example, if the light parameter is 5, it means that the number of light rays passing through each pixel is 5.

[0177] Typically, the light parameter is a fixed value. For example, the light parameters corresponding to different first events are all 4. Of course, the light parameter can also be based on the first event (that is, different first events correspond to different light parameters), or further based on the dynamic change of event parameters. This embodiment of the application does not specifically limit this.

[0178] It should be noted that if the light parameter is a fixed value, the above S1303 can be omitted.

[0179] The CPU may send the resolution parameter and the light parameter to the GPU, so that the GPU can render the image using a smaller number of light rays according to the resolution parameter and the light parameter. For details, see the description of S1304-S1317 below.

[0180] S1304. The CPU sends resolution parameters to the GPU.

[0181] S1305. The GPU creates rendering targets (Render Target) of the rendered image, Albedo map, Normal map, and denoised image respectively according to the resolution parameter and the default resolution.

[0182] It is understandable that the rendering target of the rendered image, Albedo map or Normal map is used to store the corresponding image rendered by the GPU, and the rendering target of the denoised image is used to store the denoised image obtained after the denoising process.

[0183] For example, if the resolution parameter is the ratio of the adjusted image resolution to the default resolution of the game application, the resolution of the rendering target created according to the resolution parameter and the default resolution is resolution*scale, where resolution is the default resolution and scale is the ratio of the adjusted image resolution to the default resolution of the game application.

[0184] Exemplarily, the GPU may call the glTexImage2D interface to configure the resolution size (including width and height) for each rendering target.

[0185] At this point, it should be noted that: although the number of rays can be reduced by reducing the resolution during rendering, upsampling will be used to restore the image resolution later. Therefore, in S1305, there is no need to create a rendering target for storing the final processing result according to the resolution parameter, but the rendering target before reducing the resolution can continue to be used, such as the rendering target color_final with a resolution of resolution.

[0186] S1306. The CPU sends a resolution parameter to the NPU.

[0187] S1307. The NPU creates storage spaces for the rendered image, the Albedo map, the Normal map, and the denoised image according to the resolution parameter and the default resolution.

[0188] It can be understood that the storage space of the rendered image, Albedo map or Normal map is used to store the corresponding input data of the denoising process, and the storage space of the denoised image is used to store the output data of the denoising process.

[0189] Similarly, taking the resolution parameter as the ratio of the adjusted image resolution to the default resolution of the game application as an example, the size of the storage space created according to the resolution parameter and the default resolution is resolution*scale*r bytes, where r is the number of color channels of the corresponding image.

[0190] Taking resolution = 1280*720 and scale = 0.5 as an example, and the rendered image, Albedo map, Normal map and denoised image are all three-channel images, that is, r = 3, the storage space for the rendered image, Albedo map, Normal map and denoised image can be created as 1280*0.5*720*0.5*3 bytes respectively.

[0191] S1308: The game application sends a rendering instruction to the CPU.

[0192] When the game application needs to render a new frame of image, a rendering instruction may be sent to the CPU to instruct the start of rendering of the new frame of image.

[0193] S1309. The CPU creates a hardware acceleration structure for ray tracing.

[0194] Among them, the hardware acceleration structure is mainly used for the intersection of light and triangular meshes.

[0195] It is understandable that the process of creating a hardware acceleration structure is essentially the process of calling various interfaces provided by the GPU, and the embodiments of the present application will not explain this in detail.

[0196] Taking the Vulkan graphics library as an example, the steps to create a hardware acceleration structure are as follows:

[0197] Step 1: The CPU calls the vkGetBufferDeviceAddressKHR interface provided by the vulkan instruction to obtain the GPU addresses of the vertex buffer and index buffer of all models in the 3D scene;

[0198] Step 2: The CPU fills in the data structures of the VkAccelerationStructureGeometryKHR interface and the BuildGeometryInfo interface provided by the vulkan instruction.

[0199] Step 3: The CPU calls the vkCmdBuildAccelerationStructuresKHR interface provided by the vulkan instruction to build a ray tracing acceleration structure.

[0200] It should be noted that the CPU can create a hardware acceleration structure each time it receives a rendering instruction, for use in the intersection of light and triangular meshes in the current frame image rendering process. Alternatively, after receiving a rendering instruction, the CPU can detect whether the degree of change of the three-dimensional scene exceeds a preset degree, and create a hardware acceleration structure if it exceeds the preset degree. The embodiments of the present application do not specifically limit this.

[0201] In addition, it should be noted that in S1309, the execution subject of creating the hardware acceleration structure is the CPU, and usually, the hardware acceleration structure is essentially created by the GPU driver, but the GPU driver is executed by the CPU, so it is considered that the CPU is creating the hardware acceleration structure. Of course, in other embodiments, the hardware acceleration structure can also be created by the GPU. The embodiments of the present application do not specifically limit this.

[0202] S1310. The CPU sends a binding instruction of the rendering pipeline to the GPU, where the binding instruction instructs binding of a target shader and instructs loading rendering targets of a rendered image, an Albedo map, and a Normal map into the target shader.

[0203] It is understandable that after binding the target shader, it is equivalent to determining how to draw the graphics. And, loading the rendering target into the target shader is equivalent to determining the writing area of ​​the rendering result.

[0204] In addition, in the binding instruction, you can specify that the render target is loaded into the target shader as a readable and writable memory type (StorageUmage). In this way, you can write data to the render target, such as writing the rendering result to the render target, and read data from the render target, such as reading the rendering result from the render target and using it for denoising.

[0205] S1311. GPU performs a binding operation.

[0206] S1312. The CPU sends a rendering instruction 1 to the GPU, where the rendering instruction 1 is used to instruct the GPU to perform a rendering operation on the first ray.

[0207] Exemplarily, the CPU may send rendering instruction 1 to the GPU by calling the vkCmdTraceRayKHR interface.

[0208] S1313. For each pixel point, the GPU uses a target shader to emit a first ray from the pixel point to the three-dimensional scene, and uses a hardware acceleration structure to obtain a first intersection point between the first ray and the three-dimensional scene.

[0209] It is understandable that after reducing the resolution, the number of pixels that the GPU needs to process will also be reduced accordingly, so the GPU can emit light and perform subsequent calculations on a smaller number of pixels, thereby reducing the amount of calculations.

[0210] The GPU emitting light can be understood as: the GPU determines a ray starting from the camera and extending along the line passing through the pixel points.

[0211] S1314. The GPU writes the color information of the first intersection point corresponding to each pixel point into the rendering target of the Albedo map, and writes data into the rendering target of the Normal map according to the normal coordinates of the first intersection point corresponding to each pixel point.

[0212] After the above S1313 and S1314, the Albedo map and the Normal map can be obtained. For the specific implementation of S13131 and S1314, please refer to the previous article. Fig. 9 The related instructions will not be repeated here.

[0213] S1315. The GPU writes the color information of the first intersection point corresponding to each pixel point into the rendering target of the rendered image.

[0214] Exemplarily, after obtaining the color information of the first intersection corresponding to the pixel in the uth row and the vth column, the GPU may use the color information as the color information of the pixel in the uth row and the vth column in the rendering target of the rendered image.

[0215] S1316. The CPU sends a rendering instruction 2 to the GPU. The rendering instruction 2 is used to instruct the GPU to perform a rendering operation on the remaining light rays. The rendering instruction 2 includes light parameters.

[0216] Similarly, the CPU can send rendering instructions 2 to the GPU by calling the vkCmdTraceRayKHR interface.

[0217] S1317. For each pixel point, the GPU generates the remaining light rays emitted by the pixel point into the three-dimensional scene in the target shader according to the light parameters, and uses the hardware acceleration structure to obtain the intersection points of all light rays (including the first light ray and the remaining light rays) with the three-dimensional scene to obtain the color information of the pixel point in the rendered image.

[0218] Among them, for any light ray, if the light ray is within the preset number of iterations (such as 7 times, 8 times, etc.), it can intersect with the light source in the three-dimensional scene, and the GPU can accumulate the color information of each intersection point of the light path of the light ray and the three-dimensional scene into the rendering target of the rendered image. It should be noted that the color information of the first intersection point of the first light ray with the three-dimensional scene has been written into the rendering target of the rendered image in S1315. Therefore, if it is determined in S1317 that the first light ray is within the preset number of iterations, it can intersect with the light source in the three-dimensional scene, and there is no need to repeatedly accumulate the color information of the first intersection point of the first light ray with the three-dimensional scene in S1317.

[0219] Alternatively, for any ray, if the ray still does not intersect with the light source in the three-dimensional scene after iterating a preset number of times, the GPU may not accumulate the color information of each intersection point of the light path of the ray and the three-dimensional scene in the rendering target of the rendered image. Similarly, the color information of the first intersection point of the first ray and the three-dimensional scene has been written into the rendering target of the rendered image in S1315. Therefore, if it is determined in S1317 that the first ray still does not intersect with the light source in the three-dimensional scene after iterating a preset number of times, the GPU needs to clear the data written to the corresponding pixel point in the rendering target of the rendered image in S1315.

[0220] The GPU can average the accumulated values ​​of the color information at each pixel in the rendering target of the rendered image to obtain the average color information of the pixel, and write the average color information to the corresponding pixel in the rendering target of the rendered image to obtain the final rendered image.

[0221] For the ray tracing process in S1317, please refer to the previous article Figure 1 Regarding the determination of the color information of the intersection point, reference may be made to the description of the determination process of the color information of the first intersection point (such as intersection point A) in the previous text, which will not be repeated here.

[0222] S1318. The CPU copies the rendered image, Albedo map, and Normal map obtained by GPU rendering in sequence to the storage space of the rendered image, Albedo map, and Normal map created by the NPU.

[0223] S1319. The CPU sends a noise reduction instruction to the NPU.

[0224] Exemplarily, the CPU may send a noise reduction instruction to the NPU by calling the snpe.execute interface of the NPU.

[0225] S1320: The NPU performs noise reduction processing according to the rendered image, the Albedo map, and the Normal map, and stores the obtained noise reduction image in a storage space for the noise reduction image.

[0226] Exemplarily, the NPU can use the data in the storage space of the rendered image, Albedo map, and Normal map as the input of the preset AI denoising model, and run the preset AI denoising model to output a denoised image. It is understandable that the preset AI denoising model can be pre-loaded into the NPU, and after the CPU sends a denoising instruction to the NPU, the NPU can directly use the pre-loaded preset AI denoising model for denoising.

[0227] For the specific implementation of noise reduction processing, please refer to the previous article Figure 8The related instructions will not be repeated here.

[0228] S1321. The CPU copies the denoised image in the NPU to the rendering target of the denoised image in the GPU.

[0229] S1322, GPU performs upsampling processing on the denoised image to obtain a high-resolution image.

[0230] Specifically, the GPU rasterization can be used for upsampling. Compared with the CPU upsampling, the GPU upsampling allows the ray tracing algorithm to be used for rendering, noise reduction, upsampling, and sharpening to be completed in the GPU and NPU without jumping out to the CPU. In this way, scheduling between more modules can be avoided.

[0231] In some embodiments, the NPU performs the noise reduction process asynchronously with the GPU performing the upsampling process and the sharpening process. In this way, after the GPU obtains the rendered image and synchronizes the rendered image to the NPU, it can start processing the next frame of the image without waiting for the NPU to complete the noise reduction process.

[0232] S1323. The display screen acquires and displays a high-resolution image.

[0233] Exemplarily, the high-resolution image may be stored in the rendering target color_final, and the display screen may obtain the high-resolution image from the rendering target color_final and display it.

[0234] Furthermore, the GPU can also perform sharpening processing on the high-resolution image to obtain a sharpened image, and correspondingly, the display screen displays the sharpened image.

[0235] For the specific implementation of upsampling and sharpening, please refer to the previous article. Figure 6 , Figure 7 The related instructions will not be repeated here.

[0236] In summary, Fig.13 In the embodiment shown, the number of emitted light rays can be reduced by combining the above-mentioned method 1 and method 2, thereby reducing the amount of computation and improving efficiency. In addition, the defect of insufficient image resolution caused by method 1 can be compensated by upsampling, and the defect of high image noise caused by method 2 can be compensated by noise reduction. Thus, the image quality is guaranteed while reducing the amount of computation. In this way, even in a high frame rate scene, a high-quality image can be quickly generated and used for display.

[0237] It should be noted that the above Fig.13In the embodiment of FIG. 1 , only one frame of image processing is described. In actual implementation, if multiple frames of image are processed continuously, each frame of image may be processed according to Fig.13 Process as shown.

[0238] Of course, if the game application does not receive the first event, the existing processing flow can still be used to perform image processing. For example, in the existing processing flow, the number of emitted light rays will not be reduced.

[0239] Combine the following Fig.14 The process diagram of the above Fig.13 The work of the CPU, GPU, and NPU in the process shown is briefly described:

[0240] S1401. NPU loads the preset AI noise reduction model.

[0241] That is, the preset AI model is loaded into the NPU. For details, please refer to the description of S1320.

[0242] S1402: After receiving the first event, the GPU and the NPU update rendering resources. For details, please refer to the description of S1301-S1305.

[0243] Among them, updating the rendering resources includes the GPU creating a rendering target and the NPU creating a storage space.

[0244] S1403: After receiving the rendering instruction, the CPU creates a hardware acceleration structure for ray tracing. For details, please refer to the description of S1308-S1309.

[0245] It is understandable that the CPU needs to create a hardware acceleration structure based on the three-dimensional scene.

[0246] S1404: GPU generates auxiliary data for noise reduction. For details, please refer to the description of S1312-S1314.

[0247] The auxiliary data includes at least one of an Albedo map, a Normal map, and a Mask map.

[0248] S1405: GPU performs ray tracing to obtain a rendered image. For details, see S1315-S1317.

[0249] In the process of ray tracing, the GPU can determine the number of rays emitted through each pixel based on the ray parameters.

[0250] S1406: NPU performs noise reduction processing to obtain a noise-reduced image. For details, please refer to the description of S1318-S1320.

[0251] S1407: GPU performs upsampling and sharpening to obtain a final color image. For details, please refer to the description of S1321-S1322.

[0252] S1408: The display screen acquires and displays the final color image. For details, please refer to the description of S1323.

[0253] The present application also provides an electronic device, which may include: a memory and one or more processors (such as a CPU, a GPU, an NPU, etc.). The memory is coupled to the processor. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device may perform the various functions or steps performed by the device in the above method embodiment.

[0254] The embodiment of the present application also provides a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected by lines. For example, the interface circuit can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit can be used to send signals to other devices (such as a processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiment of the present application.

[0255] This embodiment further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the image processing method in the above-mentioned embodiment.

[0256] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

[0257] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the image processing method in the above-mentioned method embodiments.

[0258] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.

[0259] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0260] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0261] The unit described as a separate component may or may not be physically separated, and the component shown as a unit may be one physical unit or multiple physical units, that is, it may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

[0262] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0263] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0264] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the electronic device includes a first application, the electronic device uses a ray tracing algorithm to render an image frame of the first application, and during the process of the electronic device running the first application, the method includes: In response to a rendering event of a first image frame, rendering using a first number of rays to obtain the first image frame; receiving a first event, wherein the first event is used to trigger the electronic device to adjust the viewing angle of the image frame, and the movement amplitude between the image frames before and after the adjustment exceeds a first amplitude; In response to a rendering event of the second image frame, rendering using a second number of rays to obtain a third image frame, and processing the third image frame to obtain the second image frame, wherein the second number is less than the first number; The first image frame and the second image frame are two consecutive image frames of the first application.

2. The method according to claim 1, characterized in that The resolutions of the first image frame and the second image frame are both a first resolution, the resolution of the third image frame is a second resolution, and the second resolution is lower than the first resolution; The first number is related to the number of pixels included in the first image frame of the first resolution, and the second number is related to the number of pixels included in the third image frame of the second resolution.

3. The method according to claim 2, characterized in that The step of processing the third image frame to obtain the second image frame includes: After upsampling the third image frame, the second image frame is obtained.

4. The method according to claim 2 or 3, characterized in that: Different first events correspond to different second resolutions; After receiving the first event, the method further includes: The second resolution corresponding to the received first event is determined.

5. The method according to any one of claims 1 to 4, characterized in that Before receiving the first event, the number of light rays passing through each pixel in the process of rendering one frame of image is a third number, and after receiving the first event, the number of light rays passing through each pixel in the process of rendering one frame of image is a fourth number, and the fourth number is less than the third number; The first quantity is related to the third quantity, and the second quantity is related to the fourth quantity.

6. The method according to any one of claims 1 to 5, characterized in that The step of processing the third image frame to obtain the second image frame includes: Performing noise reduction processing on the third image frame; The third image frame after the noise reduction process is processed to obtain the second image frame.

7. The method according to claim 6, characterized in that The performing noise reduction on the third image frame comprises: Taking at least one of an albedo map, a normal map, and a mask map, and the third image frame as input, running a preset denoising model, and obtaining the third image frame after denoising; The preset denoising model has the ability to remove noise from the noisy image frame based on at least one of an albedo map, a normal map and a mask map, and the noisy image frame.

8. The method according to claim 7, characterized in that The RGB color information of any pixel in the albedo map is the RGB color information of the first intersection of the first light ray passing through the pixel and the three-dimensional scene; The RGB color information of any pixel in the normal map corresponds to the normal coordinates of the first intersection point of the first light ray passing through the pixel and the three-dimensional scene; The RGB color information of any pixel in the mask map corresponds to the number of effective light paths in the light paths corresponding to the fourth number of light rays passing through the pixel, where the effective light path refers to a light path intersecting with a light source in the three-dimensional scene; The three-dimensional scene is a scene used to render the image frame of the first application.

9. The method according to claim 5, characterized in that Different first events correspond to different fourth quantities; After receiving the first event, the method further includes: The fourth quantity corresponding to the received first event is determined.

10. The method according to any one of claims 1 to 9, characterized in that The receiving of the first event comprises: An operation event of moving a character in an image frame is received, or an operation event of changing a viewing angle of an image frame is received.

11. The method according to any one of claims 1 to 10, characterized in that The electronic device includes a central processing unit (CPU) and a graphics processing unit (GPU); The receiving of the first event comprises: The first application receives the first event; After the first application receives the first event, the method further includes: The first application sends a first instruction to the CPU, where the first instruction indicates that the first event is received; The CPU determines a resolution parameter corresponding to the first event, wherein the resolution parameter indicates adjusting the image frame resolution to a second resolution; The CPU sends the resolution parameter and the light parameter to the GPU, wherein the light parameter indicates that the number of light rays passing through each pixel is a fourth number; The step of rendering a third image frame using a second number of rays in response to a rendering event of the second image frame comprises: In response to the rendering event of the second image frame, the GPU uses the second number of rays to render the third image frame, where the second number is the product of the fourth number and the number of pixels included in the third image frame of the second resolution.

12. The method according to claim 11, characterized in that The electronic device also includes a neural network processor NPU; In the process of the GPU using the second number of rays to render and obtain the third image frame, the method further includes: The GPU rendering obtains at least one of the following maps: an albedo map, a normal map, and a mask map; The CPU acquires and sends at least one of the albedo map, the normal map, and the mask map, and the third image frame to the NPU; The step of processing the third image frame to obtain the second image frame includes: The NPU uses at least one of the albedo map, the normal map, and the mask map, and the third image frame to perform noise reduction processing on the third image frame to obtain a noise-reduced image frame; The CPU acquires and sends the denoised image frame to the GPU; The GPU performs upsampling and sharpening processing on the denoised image frame to obtain the second image frame.

13. The method according to claim 12, characterized in that The noise reduction process is performed asynchronously with the upsampling process and the sharpening process.

14. An electronic device, characterized in that: The electronic device includes a display screen, a memory and a processor, and the display screen, the memory and the processor are coupled; wherein the memory stores computer program code, and the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1-13.

15. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 13.

Images