Rendering method and apparatus

Abstract

The application provides a rendering method and device, wherein a first processor receives a rendering command issued by an application, the rendering command being used to instruct a second processor to render a first image based on a first resolution; the first processor sends a rendering instruction to the second processor, the rendering instruction being used to instruct the second processor to render the first image; the second processor generates image data of the first image at a second resolution based on the rendering instruction, the second resolution being not greater than the first resolution; the second processor writes the image data of the first image at the second resolution into a first memory; the second processor reads image data of the first image at a third resolution from the first memory, the third resolution being greater than the second resolution; and the second processor generates the first image based on the image data of the first image at the third resolution.

Description

[0001] This application claims priority to Chinese Patent Application No. 202111364418.7, filed on November 17, 2021, entitled "Rendering Method and Apparatus", the entire contents of which are incorporated herein by reference.

[0002] This application is a divisional application of Chinese Patent Application No. 202111554611.7, filed on December 17, 2021, entitled "Rendering Method and Apparatus". Technical Field

[0003] This application relates to the field of image processing technology, and in particular to a rendering method and apparatus. Background Technology

[0004] With the development of display technology, image resolution has evolved towards higher resolutions, such as from 720P to 1080P, and then from 1080P to 2K. Here, P represents the total number of rows of pixels (e.g., 720P has 720 rows), and k represents the total number of columns of pixels (e.g., 2K has 2000 columns). When rendering high-resolution or ultra-high-resolution images, electronic devices can consume excessive computing power, leading to increased power consumption, overheating, and even performance lag. Summary of the Invention

[0005] The rendering method and apparatus provided in this application solve the problems of increased power consumption, severe heat generation, and lag when rendering images on electronic devices.

[0006] To achieve the above objectives, this application adopts the following technical solution: Firstly, this application provides a rendering method applied in an electronic device. The electronic device runs an application program and includes a first processor and a second processor. The method includes: the first processor receiving a rendering command from the application program, the rendering command instructing the second processor to render a first image based on a first resolution; the first processor sending a rendering instruction to the second processor, the rendering instruction instructing the second processor to render the first image; the second processor generating image data of the first image at a second resolution, where the second resolution is no greater than the first resolution, based on the rendering instruction; the second processor writing the image data of the first image at the second resolution into a first memory; the second processor reading image data of the first image at a third resolution, where the third resolution is greater than the second resolution, from the first memory; and the second processor generating the first image based on the image data of the first image at the third resolution. After the second processor writes the generated image data of the first image at the second resolution into the first memory, the second processor can read the image data of the first image at the third resolution from the first memory, eliminating the need for the second processor to generate the image data of the first image at the third resolution. This saves the computing power of the second processor, shortens the rendering time, thereby reducing power consumption and improving rendering smoothness, and solving the problems of severe device overheating and lag. Furthermore, the third resolution is greater than the second resolution, indicating that the second processor can obtain image data with a relatively high resolution. The second processor can then draw the first image based on the image data with the relatively high resolution, thereby improving the image quality of the first image.

[0007] Optionally, if the application uses forward rendering, the rendering instructions correspond to the first framebuffer, and the number of drawing instructions executed in the first framebuffer exceeds a preset threshold; if the application uses deferred rendering, the rendering instructions correspond to all framebuffers except the last one issued by the application. When the application uses different rendering methods, the rendering instructions sent by the first processor to the second processor target different framebuffers, indicating that the timing of the first processor sending rendering instructions differs depending on the rendering method used by the application, thus achieving framebuffer-based control of rendering instruction sending under different rendering methods.

[0008] Optionally, the first framebuffer is the framebuffer that executes the most drawing instructions among all framebuffers.

[0009] Optionally, before the first processor sends rendering instructions to the second processor, the method further includes: the first processor obtaining the rendering method of the application from the application's configuration file.

[0010] Optionally, the rendering instruction is used to instruct the second processor to render the first image based on a second resolution, which is smaller than the first resolution. The rendering instruction sent by the first processor to the second processor carries the second resolution of the first image, specifying the resolution of the image data generated by the second processor. This prevents the second processor from generating image data that does not match the resolution required by the application, thus improving the accuracy of the first image. Furthermore, since the second resolution is smaller than the first resolution, the smaller the resolution, the smaller the data volume. Therefore, by specifying a second resolution smaller than the first resolution, the amount of data processed by the second processor is reduced, lowering the power consumption of the electronic device and solving the problem of severe overheating.

[0011] Optionally, the second resolution is lower than the first resolution; the third resolution is the same as the first resolution, or the third resolution is higher than the first resolution. When the second resolution is lower than the first resolution and the third resolution is the same as the first resolution, the amount of data processed by the second processor is reduced when generating image data for the first image at the second resolution, because the second resolution is lower than the first resolution. However, when the third resolution is the same as the first resolution, it means that the second processor can read the image data corresponding to the first resolution and draw the first image with the first resolution, ensuring that the image quality of the drawn first image meets the image quality requirements of the application. If the third resolution is higher than the first resolution, the second processor can read image data with a resolution higher than the first resolution, and the image quality of the drawn first image is better than the image quality required by the application, thus improving the image quality of the first image.

[0012] Optionally, the second resolution is equal to the first resolution. If the second resolution is equal to the first resolution, but the third resolution is greater than the second resolution, then the third resolution is also greater than the first resolution. When the application requires rendering the first image based on the first resolution, the second processor can render the first image based on the third resolution. The image quality of the rendered first image is better than the image quality required by the application, thus improving the image quality of the first image.

[0013] Optionally, the electronic device also includes a third processor, which generates the image data of the first image at a third resolution. After the second processor generates the image data of the first image at the second resolution, the third processor generates the image data of the first image at the third resolution. This allows for active utilization of the computing power of the third processor, meeting the computing power requirements for high-resolution and / or ultra-high-resolution images, reducing the power consumption of the electronic device, and minimizing the occurrence of severe overheating. Furthermore, the third processor can share the computational load with the second processor, shortening the rendering process time, thereby improving rendering smoothness and resolving the issue of performance lag.

[0014] Optionally, after the second processor writes the image data of the first image at the second resolution to the first memory, the method further includes: a third processor reading the image data of the first image at the second resolution from the first memory; the third processor generating image data of the first image at the third resolution based on the image data of the first image at the second resolution; and the third processor writing the image data of the first image at the third resolution to the first memory.

[0015] Optionally, before the second processor writes the image data of the first image at the second resolution to the first memory, the method further includes: the second processor writing the image data of the first image at the second resolution to the second memory, the second processor having access to both the second and first memory, and a third processor having access to the first memory; the second processor sending a first notification to the first processor, the first notification indicating that the image data of the first image at the second resolution has been successfully written to the second memory; in response to receiving the first notification, the first processor sending a second notification to the second processor, the second notification indicating that the second processor writes the image data of the first image at the second resolution to the first memory, the second notification carrying an address pointer to the first memory; in response to receiving the second notification, the second processor reading the image data of the first image at the second resolution from the second memory. The first processor can monitor the image data read / write operations of the second processor to promptly trigger the second processor to write the image data of the first image at the second resolution to the first memory after the second processor writes the image data of the first image at the second resolution to the second memory, improving efficiency. Furthermore, the second notification sent by the first processor carries an address pointer to the first memory, allowing the second processor to write the image data of the first image at the second resolution to the first memory based on the address pointer, improving the accuracy of the write operation.

[0016] Optionally, after the second processor writes the image data of the first image at the second resolution to the first memory, and before the third processor reads the image data of the first image at the second resolution from the first memory, the method further includes: the first processor sending a third notification to the third processor, the third notification instructing the third processor to read the image data of the first image at the second resolution from the first memory. After the second processor writes the image data of the first image at the second resolution to the first memory, the first processor can promptly notify the third processor to read, triggering the third processor to generate the image data of the first image at the third resolution, thereby improving efficiency.

[0017] Optionally, after the third processor writes the image data of the first image at the third resolution to the first memory, and before the second processor reads the image data of the first image at the third resolution from the first memory, the method further includes: the first processor sending a fourth notification to the second processor. The fourth notification instructs the second processor to read the image data of the first image at the third resolution from the first memory, and the fourth notification carries an address pointer to the first memory. The first processor can monitor the image data read / write operations of the third processor, and promptly trigger the second processor to read the image data of the first image at the third resolution after the third processor writes the image data of the first image at the third resolution to the first memory, improving efficiency. Furthermore, since the fourth notification sent by the first processor carries an address pointer to the first memory, the second processor can read the image data of the first image at the third resolution from the correct address based on the address pointer, improving the accuracy of the read image data.

[0018] Optionally, after the second processor reads the image data of the first image at the third resolution from the first memory, the method further includes: the second processor writing the image data of the first image at the third resolution into the second memory, the second processor having permission to access the second memory and the first memory, and the third processor having permission to access the first memory.

[0019] Optionally, the electronic device further includes a third processor, wherein image data of the first image at a third resolution is generated by the third processor; before the first processor sends rendering instructions to the second processor, the method further includes: the first processor allocating first memory from third memory, the first processor having permission to access the third memory; the first processor sending a pointer address of the first memory to the third processor, the third processor writing image data of the first image at a third resolution into the first memory based on the pointer address, and reading image data of the first image at a second resolution from the first memory based on the pointer address.

[0020] Optionally, the electronic device further includes a third processor, wherein image data of the first image at a third resolution is generated by the third processor; before the first processor sends rendering instructions to the second processor, the method further includes: the first processor allocating first memory from a hardware buffer; the first processor sending a pointer address of the first memory to the third and second processors, wherein the first, second, and third processors have access to the first memory, and the third and second processors perform read and write operations on the first memory based on the pointer address. In this embodiment, the first processor can allocate first memory from the hardware buffer, and the second processor does not need to write the image data of the first image at the second resolution to other memory before writing the image data of the first image at the second resolution to the first memory. The second and third processors can share data based on the first memory, achieving zero-copy data sharing between the second and third processors and improving processing efficiency.

[0021] Optionally, after the second processor writes the image data of the first image at the second resolution to the first memory, and before the second processor reads the image data of the first image at the third resolution from the first memory, the method further includes: the third processor reading the image data of the first image at the second resolution from the first memory; the third processor generating image data of the first image at the third resolution based on the image data of the first image at the second resolution; and the third processor writing the image data of the first image at the third resolution to the first memory.

[0022] Optionally, after the second processor writes the image data of the first image at the second resolution to the first memory, and before the third processor reads the image data of the first image at the second resolution from the first memory, the method further includes: the first processor sending a fifth notification to the third processor, the fifth notification instructing the third processor to read the image data of the first image at the second resolution from the first memory. The first processor can monitor the image data read / write operations of the second processor, and promptly trigger the third processor to read the image data of the first image at the second resolution after the second processor writes the image data of the first image at the second resolution to the first memory, thereby improving efficiency.

[0023] Optionally, after the third processor writes the image data of the first image at the third resolution to the first memory, and before the second processor reads the image data of the first image at the third resolution from the first memory, the method further includes: the first processor sending a sixth notification to the second processor, the sixth notification instructing the second processor to read the image data of the first image at the third resolution from the first memory. The first processor can monitor the image data read / write operations of the third processor, and promptly trigger the second processor to read the image data of the first image at the third resolution after the third processor writes the image data of the first image at the third resolution to the first memory, thereby improving efficiency.

[0024] Optionally, the electronic device also includes a third processor, which generates image data of the first image at a third resolution. The third processor runs an artificial intelligence super-resolution model, which uses the artificial intelligence super-resolution model to perform super-resolution rendering on the image data of the first image at a second resolution, thereby generating image data of the first image at a third resolution.

[0025] Optionally, before the third processor performs super-resolution rendering on the image data of the first image at the second resolution using the artificial intelligence super-resolution model, the method further includes: the first processor sending the first resolution and the second resolution to the third processor; the third processor determining the super-resolution factor of the artificial intelligence super-resolution model based on the first resolution and the second resolution; and the artificial intelligence super-resolution model performing super-resolution rendering on the image data of the first image at the second resolution based on the super-resolution factor. The artificial intelligence super-resolution model may have at least one super-resolution factor, and each super-resolution factor can perform conversions to different resolutions. For example, using image data at the same resolution as input, different super-resolution factors result in different resolutions for the image data output by the artificial intelligence super-resolution model. To ensure that the resolution of the image data output by the artificial intelligence super-resolution model is not less than the first resolution, the first processor may send the first resolution and the second resolution to the third processor. The third processor selects a super-resolution factor not less than the factor difference between the first resolution and the second resolution, thereby ensuring that the resolution of the image data output by the artificial intelligence super-resolution model is not less than the first resolution.

[0026] Optionally, before the third processor performs super-resolution rendering on the image data of the first image at the second resolution using the artificial intelligence super-resolution model, the method further includes: the first processor initializing the artificial intelligence super-resolution model, the initialization being used to determine whether the artificial intelligence super-resolution model is running and whether the artificial intelligence super-resolution model is running normally; the initialization includes runtime detection, model loading, model compilation, and memory configuration, the runtime detection being used to determine whether the artificial intelligence super-resolution model is running, and the model loading, model compilation, and memory configuration being used to determine whether the artificial intelligence super-resolution model is running normally.

[0027] Optionally, before the first processor sends rendering instructions to the second processor, the method further includes: the first processor reducing the resolution of the first image from a first resolution to a second resolution.

[0028] Optionally, the electronic device further includes a third processor, which generates image data of the first image at a third resolution. The third processor has a super-resolution factor, which indicates the difference between the second and third resolutions. The third resolution is the same as the first resolution. The first processor reduces the resolution of the first image from the first resolution to the second resolution by: the first processor reducing the resolution of the first image from the first resolution to the second resolution based on the super-resolution factor. The artificial intelligence super-resolution model may have at least one super-resolution factor, and each super-resolution factor can perform different resolution conversions. For example, with image data at the same resolution as input, different super-resolution factors result in different resolutions for the image data output by the artificial intelligence super-resolution model. Therefore, the first processor can reduce the resolution of the first image based on the super-resolution factor, matching the resolution reduction with a super-resolution factor of the artificial intelligence super-resolution model. Subsequently, the artificial intelligence super-resolution model can be used to improve the resolution, ensuring accuracy.

[0029] Optionally, the third processor is a neural network processor or a digital signal processor.

[0030] Secondly, this application provides a rendering method applied to a second processor in an electronic device. The electronic device runs an application program and also includes a first processor. The application program sends a rendering command to the first processor, instructing the second processor to render a first image based on a first resolution. The method includes: the second processor receiving a rendering instruction sent by the first processor, the rendering instruction instructing the second processor to render the first image; the second processor generating image data of the first image at a second resolution, where the second resolution is not greater than the first resolution, based on the rendering instruction; the second processor writing the image data of the first image at the second resolution into a first memory; the second processor reading image data of the first image at a third resolution, where the third resolution is greater than the second resolution, from the first memory; and the second processor generating the first image based on the image data of the first image at the third resolution. After the second processor writes the generated image data of the first image at the second resolution into the first memory, the second processor can read the image data of the first image at the third resolution from the first memory, eliminating the need for the second processor to generate the image data of the first image at the third resolution. This saves the computing power of the second processor, shortens the rendering time, thereby reducing power consumption and improving rendering smoothness, and solving the problems of severe device overheating and lag. Furthermore, the third resolution is greater than the second resolution, indicating that the second processor can obtain image data with a relatively high resolution. The second processor can then draw the first image based on the image data with the relatively high resolution, thereby improving the image quality of the first image.

[0031] Optionally, before the second processor writes the image data of the first image at the second resolution to the first memory, the method further includes: the second processor writing the image data of the first image at the second resolution to the second memory, the second processor having access to both the second and first memory, and a third processor having access to the first memory; the second processor sending a first notification to the first processor, the first notification indicating that the image data of the first image at the second resolution has been successfully written to the second memory; in response to the second notification sent by the first processor, the second processor reading the image data of the first image at the second resolution from the second memory, the second notification being sent by the first processor after receiving the first notification, the second notification indicating that the second processor writes the image data of the first image at the second resolution to the first memory, the second notification carrying an address pointer to the first memory. The first processor can monitor the image data read / write operations of the second processor to promptly trigger the second processor to write the image data of the first image at the second resolution to the first memory after the second processor writes the image data of the first image at the second resolution to the second memory, improving efficiency. Furthermore, since the second notification sent by the first processor carries an address pointer to the first memory, the second processor can write the image data of the first image at the second resolution to the first memory based on the address pointer of the first memory, improving the accuracy of the write operation.

[0032] Optionally, after the second processor reads the image data of the first image at the third resolution from the first memory, the method further includes: the second processor writing the image data of the first image at the third resolution into the second memory, the second processor having access to both the second memory and the first memory; the image data of the first image at the third resolution is generated by a third processor in the electronic device, the third processor having access to the first memory.

[0033] Optionally, before the second processor reads the image data of the first image at the third resolution from the first memory, the method further includes: the second processor receiving a notification sent by the first processor, the notification instructing the second processor to read the image data of the first image at the third resolution from the first memory, the notification carrying an address pointer of the first memory. The second processor can then read the image data of the first image at the third resolution from the correct address based on the address pointer of the first memory, thereby improving the accuracy of the read image data.

[0034] Thirdly, this application provides an electronic device, which includes a first processor, a second processor, and a memory; wherein the memory is used to store one or more computer program codes, the computer program codes including computer instructions, and when the first processor and the second processor execute the computer instructions, the first processor and the second processor execute the above-described rendering method.

[0035] Fourthly, this application provides a chip system including program code, which, when run on an electronic device, causes a first processor and a second processor in the electronic device to execute the above-described rendering method.

[0036] Fifthly, this application provides a processor, which is a second processor, and the second processor includes a processing unit and a memory; wherein the memory is used to store one or more computer program codes, the computer program codes including computer instructions, and when the second processor executes the computer instructions, the second processor executes the above-described rendering method.

[0037] Sixthly, this application provides a computer storage medium including computer instructions that, when executed on an electronic device, cause a second processor in the electronic device to execute the rendering method described above.

[0038] In a seventh aspect, this application provides a rendering method applied to the rendering processing of a first image by an electronic device. The electronic device runs an application program and includes a first processor, a second processor, and a third processor. The method includes: the first processor receiving a rendering command issued by the application program, the rendering command instructing the rendering of the first image, the first image having a second resolution; the third processor obtaining first image data, the first image data being image data of the first image having a first resolution; the third processor performing super-resolution rendering on the first image data to obtain second image data, the second image data being image data of the first image having a second resolution, the first resolution being smaller than the second resolution; and the second processor generating a first image having a second resolution based on the second image data.

[0039] Optionally, the method further includes: the first processor responding to a rendering command, determining the resolution of the first image as a first resolution, and instructing the third processor to obtain the first image data.

[0040] Optionally, the method further includes: a first processor responding to a rendering command and determining that the resolution of the first image is a second resolution; the first processor reducing the resolution of the first image from the second resolution to the first resolution; and the first processor instructing a third processor to obtain the first image data.

[0041] Optionally, the method further includes: the first processor selecting a scaling factor related to the super-resolution factor from the scaling range based on the super-resolution factor of the third processor, the super-resolution factor being used to indicate the multiplier relationship between the first resolution and the second resolution; the first processor reducing the resolution of the first image from the second resolution to the first resolution includes: the first processor using the scaling factor to reduce the resolution of the first image, so that the resolution of the first image is reduced from the second resolution to the first resolution, the relationship between the scaling factor and the super-resolution factor is that the product of the scaling factor and the super-resolution factor is 1.

[0042] Optionally, the third processor runs an AI super-resolution model, and the third processor performs super-resolution rendering on the first image data to obtain the second image data, including: the third processor uses the AI ​​super-resolution model to perform super-resolution rendering on the first image data.

[0043] Optionally, the method further includes: the first processor initializes the AI ​​super-resolution model, the initialization being used to determine whether the AI ​​super-resolution model is running and whether the AI ​​super-resolution model can run normally; the initialization includes runtime detection, model loading, model compilation, and memory configuration, the runtime detection being used to determine whether the AI ​​super-resolution model is running, and the model loading, model compilation, and memory configuration being used to determine whether the AI ​​super-resolution model can run normally.

[0044] Optionally, when the electronic device performs rendering processing on the first image, it invokes one or more frame buffers. The method further includes: when the first processor determines that the resolution of the first image is a second resolution, the rendering mode of the first image is deferred rendering, and the currently processed frame buffer is not the last frame buffer, it instructs the third processor to obtain the first image data to perform rendering processing on the currently processed frame buffer.

[0045] Optionally, the method further includes: when the first processor determines that the resolution of the first image is the second resolution, the rendering mode of the first image is deferred rendering, and the currently processed frame buffer is the last frame buffer, instructing the second processor to obtain fourth image data to perform rendering processing on the currently processed frame buffer, wherein the fourth image data is image data of the first image having the second resolution.

[0046] Optionally, when the electronic device performs rendering processing on the first image, it invokes one or more frame buffers. The method further includes: a first processor determining the main frame buffer in the rendering processing of the first image; the main frame buffer is the frame buffer that performs the most rendering operations in the rendering processing of the first image by the electronic device; when the first processor determines that the resolution of the first image is a second resolution, the rendering mode of the first image is forward rendering, and the currently processed frame buffer is the main frame buffer, it instructs a third processor to obtain the first image data to perform rendering processing on the currently processed frame buffer.

[0047] Optionally, the method further includes: when the first processor determines that the resolution of the first image is the second resolution, the rendering mode of the first image is forward rendering, and the currently processed frame buffer is not the main frame buffer, the first processor instructs the second processor to obtain fourth image data to perform rendering processing on the currently processed frame buffer, wherein the fourth image data is image data of the first image with the second resolution.

[0048] Optionally, the first processor determines the main frame buffer during the rendering process of the first image by: the first processor determining the main frame buffer based on the frame buffer that performs the most rendering operations during the rendering process of the second image; the rendering process of the second image is performed before the rendering process of the first image.

[0049] Optionally, before determining the main frame buffer during the rendering process of the first image, the method further includes: during the rendering process of the second image, determining the number of drawing instructions executed on each frame buffer; and determining the frame buffer with the most drawing instructions executed as the main frame buffer.

[0050] Optionally, the second image is the previous frame of the first image.

[0051] Optionally, the first processor is a central processing unit, the second processor is a graphics processing unit, and the third processor is a neural network processor or a digital signal processor.

[0052] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0053] Figure 1 A schematic diagram of a rendered image provided in this application; Figure 2 A schematic diagram of another rendered image provided in this application; Figure 3 A schematic diagram of an interface for resolution enhancement settings provided in this application; Figure 4 A schematic diagram illustrating another rendered image provided in this application; Figure 5 A schematic diagram illustrating another rendered image provided in this application; Figure 6 A schematic diagram illustrating the determination of the number of draw calls on different frame buffers provided in this application; Figure 7A schematic diagram illustrating the determination of the number of drawcalls on a different framebuffer provided in this application; Figure 8 A schematic diagram illustrating another rendered image provided in this application; Figure 9 A schematic diagram of a rendering method provided in this application; Figure 10 A flowchart of a rendering method provided in this application; Figure 11 A signaling diagram for the rendering method provided in this application; Figure 12 A schematic diagram illustrating a memory access method provided in this application; Figure 13 Another signaling diagram for the rendering method provided in this application; Figure 14 A schematic diagram illustrating another memory access method provided in this application; Figure 15 Another signaling diagram for the rendering method provided in this application. Detailed Implementation

[0054] The terms "first," "second," and "third," etc., used in this application specification, claims, and drawings are used to distinguish different objects, not to limit a specific order.

[0055] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0056] During user interaction with an electronic device, the device can display frames of images on its screen. Taking a video stream as an example, a video stream can include multiple frames, and the electronic device can display each frame sequentially on the screen, thus displaying the video stream. This image display can be triggered by an application within the electronic device. The application can send rendering commands to the electronic device for different images, and the electronic device responds to these commands to render the images and displays the results.

[0057] In some implementations, each frame corresponds to multiple frame buffers (FBs). Each FB stores the rendering results of a portion of the image elements, such as image data of those elements. This image data is used to draw the corresponding elements. For example, if an image includes elements such as people and trees, the FB can store image data of the people and the trees. The electronic device can then draw based on the image data stored in the FB. In this embodiment, an optional structure of the electronic device and the rendering process based on FBs are described as follows: Figure 1 As shown, an electronic device may include a central processing unit (CPU), a graphics processing unit (GPU), internal memory, and a display screen. Internal memory may also be referred to as RAM.

[0058] exist Figure 1 In a computer, after an application (such as a game or video application) installed on an electronic device is launched, it can display images on the screen. During image display, the application issues rendering commands, which the CPU can intercept. The CPU responds to the rendering commands by creating a corresponding FB (Frame Rendering) in memory for the rendering of the i-th frame. Figure 1 The CPU can create three rendering frames (FBs) for the i-th frame image, denoted as FB0, FB1, and FB2. The CPU can issue rendering instructions to the GPU based on rendering commands, and the GPU responds to these instructions by performing the corresponding rendering. In one example, the GPU responds to the rendering instructions and renders FB1 and FB2 of the i-th frame image. After rendering, FB1 and FB2 store the image data of the i-th frame image, respectively. For example, FB1 can store image data of a portion of the i-th frame image (referred to as image data 1), and FB2 can store image data of another portion of the i-th frame image (referred to as image data 2). When displaying the i-th frame image, the GPU merges (or renders) image data 1 and image data 2 into FB0, which stores the complete image data of the i-th frame image. The GPU reads the image data of the i-th frame image from FB0 and displays the i-th frame image on the screen based on this data.

[0059] However, with the development of display technology, screen refresh rates and image resolutions have advanced to higher levels. For example, screen refresh rates have increased from 60Hz (Hertz) to 90Hz, and then from 90Hz to 120Hz; image resolutions have increased from 720P (1280×720) to 1080P (1920×1080), and then from 1080P to 2K (2560×1440). This has led to a trend towards smoother rendering, more complex scenes, and higher image quality in electronic devices, placing higher demands on their computing power. Even when electronic devices use powerful chips, rendering using CPUs and GPUs can still consume excessive computing resources, and these methods are insufficient to meet the rapidly growing computing demands. Excessive consumption of computing resources during CPU and GPU rendering increases power consumption, causes overheating, and can even lead to performance issues such as lag.

[0060] To address the aforementioned issues, this application provides a rendering method that configures processors other than the CPU and GPU in an electronic device. The rendering process is adjusted from being controlled by the CPU and GPU to being controlled by the CPU, GPU, and other processors. For example, the computing power of the configured processors can be greater than that of the GPU, but their energy consumption can be lower, and their control capabilities can be weaker than those of the CPU and GPU. When the CPU, GPU, and other processors control the rendering process, the other processors can generate high-resolution and / or ultra-high-resolution images, actively utilizing their computing power to complete super-resolution rendering. This satisfies the computing power requirements of high-resolution and / or ultra-high-resolution images, reduces the power consumption of the electronic device, and minimizes overheating. Furthermore, the other processors can share the computational load of the GPU, shortening the rendering time and thus improving rendering smoothness and resolving stuttering issues. Super-resolution rendering is used to upgrade the resolution of image data; this upgrade can be from low resolution to high resolution or ultra-high resolution, or from high resolution to ultra-high resolution.

[0061] In some implementations, electronic devices are equipped with a Neural-network Processing Unit (NPU) to control the rendering process using the CPU, GPU, and NPU. The NPU can perform super-resolution rendering to obtain image data of high-resolution or ultra-high-resolution images with NPU assistance.

[0062] In some implementations, digital signal processors (DSPs) are configured in electronic devices to control the rendering process using CPUs, GPUs, and DSPs. DSPs can perform super-resolution rendering.

[0063] In some implementations, electronic devices are equipped with NPUs and DSPs, with the CPU, GPU, NPU, and DSP controlling the rendering process. The NPU and DSP can perform super-resolution rendering, which can correspond to different resolutions. In one example, the NPU can handle low-resolution to high-resolution conversion, and the DSP can handle low-resolution to ultra-high-resolution upscaling. In another example, the NPU can handle low-resolution to ultra-high-resolution upscaling, and the DSP can handle low-resolution to high-resolution upscaling. Yet another example, the NPU can handle low-resolution to high-resolution upscaling, and the DSP can handle high-resolution to ultra-high-resolution upscaling.

[0064] In some implementations, the NPU and DSP can work together to complete super-resolution rendering. For example, the NPU and DSP can work together to handle the upgrade from low resolution to high resolution; or the NPU and DSP can work together to handle the upgrade from low resolution to ultra-high resolution. In this case, working together means that the NPU and DSP participate in the super-resolution rendering process of a frame of image.

[0065] Low resolution can be 540P (960×540) and 720P, high resolution can be 1080P, and ultra-high resolution can be 2K or even higher than 2K. When electronic devices display images on a screen, the GPU can generate image data for 540P and 720P images. The NPU can perform upscaling from 540P to 1080P, and also upscaling from 720P to 1080P. In order to generate 1080P image data using the NPU, the NPU can also generate image data for 2K or even higher resolution images.

[0066] In this embodiment, one use case of the rendering method is: the image quality output by the electronic device is higher than the image quality required by the application. This can be achieved by the electronic device increasing the resolution of the image required by the application. When the resolution of the image required by the application is low, the electronic device can increase its resolution to high or ultra-high resolution, automatically improving the image quality of the application. Alternatively, when the resolution of the image required by the application is high, the electronic device can increase it to ultra-high resolution, which can also automatically improve the image quality of the application.

[0067] like Figure 2As shown, the application requires a low-resolution image. The electronic device's rendering method can upscale the image from low to high resolution, automatically changing the image resolution to improve image quality. The processing steps involving the CPU, NPU, and GPU include: the CPU intercepts the rendering command from the application; the CPU responds to the rendering command by creating a corresponding rendering frame (FB) in memory; after obtaining the low-resolution image from the rendering command, the CPU sends a rendering instruction to the GPU; the GPU generates low-resolution image data based on the rendering instruction and sends it to the NPU; the NPU performs super-resolution rendering on the low-resolution image data to obtain high-resolution image data, which is then sent to the GPU; after receiving the high-resolution image data, the GPU performs user interface (UI) rendering and post-processing based on it to complete the drawing of one frame of the image. For example, the GPU renders the FB pointed to by the rendering command based on the high-resolution image data, obtaining a rendering result, which may be image data of some elements in the image. After obtaining the rendering results of all FBs, the GPU displays the drawn image on the display screen based on the rendering results.

[0068] In this embodiment, the electronic device may provide a settings interface with a "resolution enhancement" option, which is used to provide resolution selection. Figure 3 A schematic diagram of a settings interface is shown. Figure 3 The settings options offer multiple resolution choices, such as 1080P, 2K, and above 2K, from which users can select one; when the application's resolution is lower than the user-selected resolution, the electronic device performs the above... Figure 2 The rendering method shown. Figure 3 This is just one example; the setting options can also take other forms, such as... Figure 3 As shown in the dashed box, this embodiment does not limit the setting options.

[0069] In addition, electronic devices can set a whitelist that records applications that perform resolution upscaling. After the CPU intercepts the rendering command sent by the application, it determines whether the application is on the whitelist. If the application is on the whitelist, the CPU can trigger the NPU through the GPU to upscale the resolution. The NPU sends the upscaled image data to the GPU, and the GPU performs rendering processing based on the upscaled image data. If the application is not on the whitelist, the CPU triggers the GPU to perform rendering processing. In this case, the resolution of the image data is the same as the resolution sent by the application.

[0070] Another use case for this rendering method is when the image quality output by the electronic device is the same as the image quality required by the application, but the electronic device can first reduce the resolution and then increase it. For example... Figure 4 As shown, the application requires images with high resolution. The processing steps of the CPU, NPU, and GPU include: the CPU intercepts the rendering commands issued by the application; the CPU responds to the rendering commands and creates a corresponding FB (Frame Rendering) in memory for image rendering; after the CPU extracts the image resolution as high resolution from the rendering commands, the CPU reduces the image resolution and then sends rendering commands to the GPU; the GPU generates low-resolution image data based on the rendering commands and sends the low-resolution image data to the NPU; the NPU performs super-resolution rendering on the low-resolution image data to obtain high-resolution image data and sends the high-resolution image data to the GPU; after receiving the high-resolution image data, the GPU performs UI rendering and post-processing based on the high-resolution image data to complete the drawing of one frame of image. The rendering result of the image is stored in the FB, and the GPU displays the drawn image on the display screen.

[0071] After the CPU reduces the image resolution, the NPU can use super-resolution rendering to reconstruct high-resolution image data. The GPU can then use this high-resolution image data to reconstruct a high-resolution image, ensuring that the image rendered by the GPU has the same image quality as the application, thus meeting the application's image quality requirements and improving rendering accuracy. Furthermore, reducing the image resolution by the CPU decreases the amount of data input to the NPU, further improving rendering accuracy and speeding up rendering.

[0072] For example, if the image resolution is 1080P or higher, the CPU can reduce the image resolution and then use the NPU to perform super-resolution rendering on the reduced image data, so that the rendered result corresponds to the original resolution. The amount of resolution reduction depends on the NPU's super-resolution capability. For example, a 2x super-resolution capability would reduce the resolution of a 1080P image by 0.5x. It's important to note that 2x and 0.5x correspond to the side length of the resolution, not the number of pixels in the image.

[0073] for Figure 4 The rendering method shown allows the CPU to decide whether to reduce the image resolution based on the resolution required by the application. For example, in this embodiment, if the application requires an image resolution of 1080P or higher, the CPU reduces the image resolution, and the NPU performs super-resolution rendering on the low-resolution image data. If the application requires an image resolution lower than 1080P, the CPU triggers the GPU to perform rendering processing while maintaining the image resolution unchanged. For example, in... Figure 4 In the rendering method shown, the user clicks the game application on the desktop ( Figure 4 (Game 1) The game application requires a high-resolution image, and the electronic device executes [the necessary actions] after the game application is launched. Figure 4 The rendering method shown.

[0074] Furthermore, when the application requires a high-resolution image, the NPU receives low-resolution image data from the GPU and performs super-resolution rendering on this low-resolution data to obtain ultra-high-resolution image data. The GPU then renders the image based on this ultra-high-resolution data, resulting in an image quality higher than that required by the application. For example, if the application requires a 1080p image, the CPU reduces the resolution to 540p, the GPU generates 540p image data, and the NPU sends 2k image data back to the GPU. The GPU can then render a 2k image, which has a higher image quality than the 1080p image, thus improving the overall image quality for the application.

[0075] Another use case for the rendering method is: during the application's operation, the user can adjust the application's screen quality. The user can adjust the screen quality from low resolution to high resolution or ultra-high resolution, or from high resolution or ultra-high resolution to low resolution. Figure 5 This demonstrates how the user adjusts the image quality from low resolution to high resolution, and the corresponding processing changes after the resolution adjustment. The processing procedure is as follows: 1) The CPU intercepts the rendering commands issued by the application; the CPU responds to the rendering commands and creates the corresponding FB in memory for the rendering processing of the image.

[0076] 2) After the CPU obtains the image resolution as low resolution based on the rendering command, the CPU sends a rendering command to the GPU. The GPU responds to the rendering command, performs rendering processing on the FB (Framework Function) and obtains the rendering result. After the GPU completes the rendering processing of all FBs, it displays the image on the display screen based on the rendering result.

[0077] 3) After the user adjusts the application's screen settings to high resolution, the CPU can obtain the image resolution as high resolution based on the rendering command. The CPU then reduces the image resolution to low resolution and sends rendering commands to the GPU. The GPU generates low-resolution image data based on the rendering commands and sends the low-resolution image data to the NPU. The NPU performs super-resolution rendering on the low-resolution image data to obtain high-resolution image data, which is then sent to the GPU. After receiving the high-resolution image data, the GPU performs UI rendering and post-processing based on the high-resolution image data to complete the drawing of one frame of image. The rendering result of the image is stored in the FB (Frame Renderer), and the GPU displays the drawn image on the display screen.

[0078] In this embodiment, the CPU can specify the FB for super-resolution rendering. When the GPU processes the specified FB, it uses the NPU and / or DSP to complete the super-resolution rendering. For example, it uses the NPU and / or DSP to complete the conversion from low resolution to high resolution and / or ultra-high resolution. For other FBs, the GPU is used for rendering. The GPU no longer needs to distinguish the resolution of the image.

[0079] The FB (Frame Attachment) used for super-resolution rendering can vary depending on the rendering method. Rendering methods include deferred rendering and forward rendering. In deferred rendering, color attachments record not only color data but also the normal vector data and depth data for each pixel. The upsampled normal vector data has a high probability of error during lighting calculations, and the FBs (excluding FB0) are highly correlated (e.g., the rendering result of one FB is bound to the next FB). If super-resolution rendering is performed on one FB, the probability of errors when other FBs use that FB's rendering result is high. Therefore, in deferred rendering, super-resolution rendering can be performed on all FBs (excluding FB0) corresponding to each frame. That is, in deferred rendering, the FBs used for super-resolution rendering are all FBs (excluding FB0) corresponding to a frame. For example, if a frame contains FBs 0, 1, and 2, the FBs used for super-resolution rendering are FB1 and FB2, and FB0 is the last FB issued by the application.

[0080] In forward rendering, the correlation between rendering blocks (FBs) is relatively small, so at least one FB can be super-resolution rendered. In forward rendering, the CPU can determine the FB to be super-resolution rendered based on the number of rendering operations. In one example, the CPU can determine the FB with more than a preset number of rendering operations as the FB to be super-resolution rendered; in another example, the CPU can determine the FB with the most rendering operations as the FB to be super-resolution rendered. The FB with the most rendering operations can be the one with the most draw calls (drawing-related instructions) executed by the GPU. This is because the more draw calls the GPU executes on the FB, the more rendering operations the GPU performs on that FB, consuming more computing resources, generating higher power consumption and heat. Therefore, in this embodiment, the FB with the most rendering operations can be determined as the FB to be super-resolution rendered. The FB with the most rendering operations is the FB with the most draw calls executed. For ease of explanation, the FB that performs super-resolution rendering in forward rendering is called the main FB. The following explanation uses examples to illustrate how to determine the FB with the most draw calls.

[0081] In this embodiment, the CPU can receive rendering commands from the application and, based on these commands, issue corresponding rendering instructions to the GPU so that the GPU can execute the corresponding rendering. As an example, the rendering command may include a glBindFrameBuffer() function and one or more glDrawElement, glDrawArray, glDrawElementInstanced, and glDrawArrayInstanced. Correspondingly, the rendering instructions may also include a glBindFrameBuffer() function and one or more glDrawElement, glDrawArray, glDrawElementInstanced, and glDrawArrayInstanced. The glBindFrameBuffer() function can be used to indicate the currently bound frame buffer (FB), thus binding the corresponding rendering operation to the FB. For example, glBindFrameBuffer(1) can indicate that the currently bound frame buffer is FB1, and the Drawcalls executed by the GPU on FB1 include: glDrawElement, glDrawArray, glDrawElementInstanced, and glDrawArrayInstanced.

[0082] For example, the Nth frame of the image can use FB0, FB1, and FB2 as frame buffers. Combined with... Figure 6This is an example of a rendering command corresponding to FB1. The rendering command issued by the application may include the glBindFrameBuffer(1) function, thereby binding the current rendering operation to FB1. After binding FB1, the rendering operation on that FB1 can be indicated through the glDrawElement instruction. In this example, one glDrawElement instruction can correspond to one Drawcall. In different implementations, there can be multiple glDrawElement instructions executed on FB1, and thus multiple Drawcalls executed on FB1.

[0083] The CPU can initialize counter 1 when executing glBindFrameBuffer(1) according to the rendering command issued by the application. For example, if a corresponding counting frame bit is configured for FB1 in memory, the value of the frame bit can be initialized to 0 by initializing counter 1. Subsequently, each time glDrawElement is executed on FB1, counter 1 is incremented by 1, such as by executing count1++. For example, after executing glDrawElement1-1, the CPU can execute count1++ on counter 1, thereby changing the value of the frame bit storing the number of FB1Drawcalls from 0 to 1, at which point the number of Drawcalls executed on FB1 is 1. By analogy, the CPU can determine that during the rendering of the Nth frame image, the number of Drawcalls executed on FB1 is the current count of counter 1 (for example, this count can be A).

[0084] Similarly, for FB2, the GPU can bind FB2 using the glBindFrameBuffer(2) function. Afterwards, the GPU uses instructions such as glDrawElement, glDrawArray, glDrawElementInstanced, and glDrawArrayInstanced to implement rendering operations on FB2. Similar to FB1, the CPU can also initialize counter 2 when calling glBindFrameBuffer(2) based on the rendering command issued by the application. For example, initialize count2=0. Subsequently, each time glDrawElement is executed on FB2, counter 2 is incremented by 1, such as executing count2++. After completing the rendering process of the image on FB2, the CPU can determine that during the rendering of the Nth frame, the number of Drawcalls executed on FB2 is equal to the current count of counter 2 (for example, this count can be B). See reference... Figure 7After completing the rendering of FB1 and FB2, the number of frames in memory storing the number of FB1 draw calls can be A, and the number of frames storing the number of FB2 draw calls can be B.

[0085] In this example, the CPU can select the FB corresponding to the larger count between A and B as the primary FB. For instance, if A is greater than B, the CPU can determine that FB1 executes more draw calls and therefore designates FB1 as the primary FB. Conversely, if A is less than B, the CPU can determine that FB2 executes more draw calls and therefore designates FB2 as the primary FB.

[0086] In other examples, the CPU can determine the primary FB (Executable File Unit) based on the number of Drawcalls executed that exceeds a preset threshold. The CPU can also select the FB with the number of Drawcalls executed that exceeds the preset threshold by counting, which will not be described in detail here.

[0087] Correspondingly, the rendering process of CPU, GPU, and NPU is as follows: Figure 8 As shown, the CPU triggers the rendering process based on the rendering commands issued by the application, and the CPU creates the rendering flowsheet (FB). Figure 8 The application instructs the CPU to render a high-resolution image. Upon receiving the rendering command, the CPU determines whether the rendering method is forward rendering or deferred rendering. If forward rendering is used, the CPU checks if the current rendering frame (FB) is FB1 (FB1 is the primary FB). If FB1 is FB1, the CPU reduces the image resolution and sends a rendering instruction to the GPU, instructing the GPU to generate low-resolution image data. The GPU then sends this low-resolution image data to the NPU, which performs the super-resolution rendering. If the CPU determines that deferred rendering is used, it checks if the current rendering frame is FB0. If FB1 or FB2 is FB2, the CPU can also reduce the image resolution, instructing the GPU to generate low-resolution image data. The GPU then sends this low-resolution image data to the NPU, which performs the super-resolution rendering. After receiving the high-resolution image data from the NPU, the GPU performs UI rendering and post-processing based on the high-resolution image data to complete the drawing of one frame. The rendered image is stored in the FB, and the GPU displays the drawn image on the screen.

[0088] Figure 8 The two GPUs in the diagram can be the same GPU. The reason for showing two GPUs is to illustrate that different processing methods are used for different rendering frames under different rendering modes.

[0089] In some embodiments, under deferred rendering, when the CPU is currently performing rendering operations on FB1 and FB2, the CPU reduces the image resolution. The GPU can generate low-resolution image data corresponding to FB1 or FB2, and then the NPU performs super-resolution rendering on the low-resolution image data corresponding to the two FBs. In other embodiments, under deferred rendering, the image data of one frame corresponds to the previous FB of FB0. The GPU can generate low-resolution image data of the previous FB of FB0, and the NPU performs super-resolution rendering on the low-resolution image data corresponding to the previous FB of FB0. That is, under deferred rendering, when the FB is any FB other than FB0, the GPU can generate low-resolution image data, and then the NPU performs super-resolution rendering on the low-resolution image; alternatively, when the FB is the previous FB of FB0, the GPU can generate low-resolution image data, and the NPU performs super-resolution rendering on the low-resolution image data.

[0090] The CPU can reduce the image resolution based on the NPU's super-resolution multiplier. The super-resolution multiplier indicates the factor by which the NPU can improve the resolution. For example, if the NPU's super-resolution multiplier is 2x, the CPU can reduce the image resolution by a factor of 2; if the NPU's super-resolution multiplier is 4x, the CPU can reduce the image resolution by a factor of 4. For instance, if the NPU's super-resolution multiplier is 2x, the CPU can reduce the image resolution from 1080P to 540P, and the NPU can then convert 540P to 1080P, completing the super-resolution rendering from 540P to 1080P. If the NPU's super-resolution multiplier is 4x, the CPU can reduce the image resolution from 2K to 540P, and the NPU can then convert 540P to 2K, completing the super-resolution rendering from 540P to 2K; the CPU can also reduce the image resolution from 1080P to 270P, and the NPU can correspondingly convert 207P to 1080P.

[0091] In this embodiment, the corresponding schematic diagram of the rendering method is shown in Figure 9. The application sends rendering commands for each frame of image. The CPU intercepts the rendering commands sent by the application. Based on the rendering commands, the CPU determines that the resolution of the image corresponding to the current rendering command is high resolution. The CPU can reduce the resolution of the image to a low resolution. The CPU sends rendering instructions to the GPU. Based on the rendering instructions, the GPU generates low-resolution image data. The NPU completes the conversion from low resolution to high resolution based on the Artificial Intelligence (AI) super-resolution model to obtain high-resolution image data. The GPU performs UI rendering and post-processing based on the high-resolution image data and displays the image on the display screen.

[0092] Figure 9The flowchart of the rendering method shown is as follows Figure 10 As shown, in Figure 10 In the rendering method shown, an AI super-resolution model runs in the NPU. Using the AI ​​super-resolution model to perform super-resolution rendering can include the following steps: (1) After the application starts, FB1 is determined to be the main FB, and the GPU executes the most Drawcalls on FB1.

[0093] (2) Initialize the AI ​​super-resolution model, including runtime checks, model loading, model compilation, and memory configuration; (3) When the rendering mode is forward rendering, (4) is executed when the currently processed FB is FB1; when the rendering mode is deferred rendering, (4) is executed when the currently processed FB is any FB other than FB0. (4) Reduce the image resolution from high resolution to low resolution; (5) Generate low-resolution image data; (6) Use AI super-resolution models to perform super-resolution rendering on low-resolution image data to obtain high-resolution image data; (7) UI rendering and post-processing based on high-resolution image data to complete the drawing of a frame of image; (8) Display images on the screen; (9) During the operation of the application, data can be collected to obtain a data sample set. The AI ​​super-resolution model can be trained using the data sample set to obtain the AI ​​super-resolution model. Then, the AI ​​super-resolution model can be converted into a model file that can be recognized by electronic devices using conversion tools.

[0094] The following describes the detailed process of the rendering method provided in this embodiment, using the determination of FB1 as the primary FB during the rendering of the (N-1)th frame image and the rendering process of the Nth frame image after determining the primary FB as an example. Figure 11 As shown, the following steps may be included: S101. During the rendering of the (N-1)th frame of the image, the CPU determines FB1 as the main FB. The main FB is the FB that performs the most rendering operations during the rendering process. The FB that performs the most rendering operations can be the FB that performs the most draw calls. In this embodiment, the CPU determines FB1 as the FB that performs the most draw calls.

[0095] One frame of image corresponds to multiple drawing frames (FBs), and each FB executes a different number of draw calls. In this embodiment, all FBs in a frame of image can be counted to obtain the total number of draw calls executed by all FBs in that frame of image. Then, the FB with the most draw calls executed in that frame of image is determined as the primary FB. The primary FB of other images after that frame of image can be the same as the primary FB of that frame of image. For example, taking the currently rendered frame of image N as an example, the CPU can determine the primary FB based on the number of draw calls executed on different FBs in the N-1 frame of image during the rendering process of the previous frame (e.g., the N-1 frame of image). The primary FB of the N-1 frame of image is then determined as the primary FB of the Nth frame of image, the N+1th frame of image, the N+2th frame of image, and so on.

[0096] S102, The CPU intercepts the rendering command for the Nth frame image issued by the application.

[0097] S103, the CPU obtains the initial resolution of the Nth frame image as high resolution and the currently processed FB based on the rendering command.

[0098] The CPU can obtain the width and height of the Nth frame image based on rendering commands. The width and height of an image are fixed at a given resolution. For example, when the image resolution is 720P, the width is 1280 and the height is 720; when the image resolution is 1080P, the width is 1920 and the height is 1080. Therefore, the initial resolution of the Nth frame image can be determined by its width and height. Generally, images with a resolution of 1920×1080 or higher are considered high resolution. In this embodiment, the initial resolution of the image is determined to be high resolution if the width and height of the image are 1920×1080 or higher. The CPU can obtain the currently processed frame identifier (FB), such as FB1, based on rendering commands. The initial resolution of the Nth frame image and the currently processed FB can be obtained from different rendering commands.

[0099] S104. The CPU initializes the AI ​​super-resolution model. The initialization operation determines whether to run the AI ​​super-resolution model in the NPU and, if so, ensures that the AI ​​super-resolution model can run normally.

[0100] The initialization of an AI super-resolution model includes runtime checks, model loading, model compilation, and memory configuration. Runtime checks are to determine whether to run the AI ​​super-resolution model in the NPU, while model loading, model compilation, and memory configuration are to ensure that the AI ​​super-resolution model can run normally.

[0101] In some implementations, runtime checks include checking whether an NPU is configured in the electronic device and checking the application's resolution; if an NPU is configured, the AI ​​super-resolution model is run in the NPU; if the application's resolution is high or ultra-high resolution, the AI ​​super-resolution model is run; if the electronic device does not have an NPU configured or the application's resolution is low, the AI ​​super-resolution model is disabled, and the CPU sends rendering instructions to the GPU, which responds to the rendering instructions to perform rendering.

[0102] Model loading involves converting the AI ​​super-resolution model into a model file that the NPU can recognize. During the initialization phase, the model file is loaded into memory. Model compilation is used to verify that the model file can run successfully. Memory configuration involves allocating memory for the AI ​​super-resolution model. The allocated memory is used to store the input and output data of the AI ​​super-resolution model. In this embodiment, the memory allocated to the AI ​​super-resolution model can be CPU memory, memory managed by the Neural Processing API (such as the SNPE ITensor buffer), or shared memory.

[0103] CPU memory can be the memory allocated to the CPU. Data used by the CPU during operation is written to CPU memory. AI super-resolution models occupy a portion of CPU memory as their own memory. Data exchange between the GPU and NPU can be done through CPU memory. When the GPU renders images, it can write data from CPU memory to GPU memory (memory allocated to the GPU) and then read data from GPU memory for rendering. Shared memory can be memory shared by the CPU, GPU, and NPU. The GPU can directly read data from shared memory for rendering.

[0104] It's worth noting that triggering AI super-resolution model initialization when the CPU determines that the initial resolution of the Nth frame is high is one method of AI super-resolution model initialization. The AI ​​super-resolution model can be initialized at other times. In one example, the CPU initializes the AI ​​super-resolution model after detecting application startup. Some applications perform fewer image rendering operations and have low GPU usage. The CPU can set a whitelist of applications, and upon detecting application startup, determine if the application is on the whitelist. If it is, the CPU initializes the AI ​​super-resolution model. In another example, the CPU initializes the AI ​​super-resolution model when the rendering mode is forward rendering and the currently processed FB is FB1; and when the rendering mode is deferred rendering and the currently processed FB is not FB0.

[0105] S105, The CPU obtains the rendering method from the application's configuration file.

[0106] Rendering methods include deferred rendering and forward rendering. The frame elements (FBs) used for super-resolution rendering differ depending on the rendering method. In deferred rendering, super-resolution rendering can be performed on all FBs except FB0 for each frame. In contrast, the CPU performs super-resolution rendering on the FB for the currently executing rendering command if it determines that the FB is not FB0.

[0107] In forward rendering, the GPU's computing power is concentrated on the main fragment shader (FB) rendering. Within the main FB rendering, the GPU's computing power is concentrated on the calculations of the fragment shader (FS). Therefore, it can be inferred that for super-resolution rendering of the main FB in forward rendering, the main FB is likely the FB that executes the most draw calls. In this embodiment, if the CPU determines FB1 as the main FB during the rendering of frame N-1, then the currently processed FB is FB1, and the CPU performs super-resolution rendering on FB1.

[0108] If the rendering method is forward rendering and the currently processed FB is not FB1, the CPU can send a rendering command to the GPU, and the GPU will render the current FB. If the rendering method is deferred rendering and the currently processed FB is FB0, the CPU can send a rendering command to the GPU, and the GPU will render the current FB.

[0109] S106. If the rendering method is forward rendering and the currently processed FB is FB1, the CPU reduces the initial resolution of the image to a lower resolution based on the super-resolution factor; if the rendering method is deferred rendering and the FB is not FB0 (such as FB1 and FB2), the CPU reduces the initial resolution of the image to a lower resolution based on the super-resolution factor.

[0110] CPUs can reduce image resolution by decreasing the width and height of the image. When reducing image resolution, the CPU can refer to the super-resolution factor of the AI ​​super-resolution model. The AI ​​super-resolution model can convert image data from low resolution to high resolution, where high resolution is greater than low resolution. For example, high resolution can be one of 2, 3, or 4 times the low resolution, meaning the super-resolution factor of the AI ​​super-resolution model is one of 2, 3, or 4 times. For example, a 540P image, after being rendered with 2x super-resolution, becomes a 1080P image; after being rendered with 4x super-resolution, it becomes a 2K image.

[0111] In this embodiment, the CPU can reduce the image resolution based on the super-resolution factor of the AI ​​super-resolution model. The CPU can reduce the image resolution using a scaling factor, which can be achieved by reducing the image's width and height, such as representing the image using width and height: The scaling factor r takes the value of That is, the scaling factor r takes a value between 0 and 1, and the image after reducing the resolution is... The scaling factor used is related to the super-resolution factor of the AI ​​super-resolution model. For example, if the initial resolution of an image is 1080P (1920×1080 in width and height), and the AI ​​super-resolution model has a super-resolution factor of 2, it can convert from 540P to 1080P. A 540P image has a width and height of 960×540, meaning it's a 2x reduction of the 1080P image, corresponding to a scaling factor of 0.5. If the AI ​​super-resolution model has a super-resolution factor of 4, it can convert from 270P to 1080P, a 4x reduction of the 1080P image, corresponding to a scaling factor of 0.25. The scaling factor refers to scaling the image's side length. For example, a scaling factor of 0.5 means scaling both the width and height of the image by 0.5. When both width and height are scaled by 0.5, the image's pixels are scaled by 0.25.

[0112] The super-resolution factor is used to indicate the super-resolution capability of the NPU, indicating the factor by which the NPU improves the resolution. The CPU can determine the super-resolution factor during the initialization of the AI ​​super-resolution model. If the CPU does not determine the super-resolution factor during the initialization process, the CPU can send the high-resolution and low-resolution images to the NPU, that is, reduce the previous resolution and the reduced resolution. The NPU selects the matching super-resolution factor based on the previous resolution and the reduced resolution.

[0113] S107. The CPU instructs the GPU to generate low-resolution image data. In some examples, the NPU's super-resolution multiplier can be a fixed multiplier, and the resolution of the images required by each application in the CPU can also be a fixed multiplier. For example, if the NPU's super-resolution multiplier is 2x and the required image resolution for each application is 1080p, then the CPU can reduce the resolution to 540p. Because the super-resolution multiplier and the image resolution are fixed, the GPU knows that the resolution can be reduced to 540p, so when the CPU instructs the GPU to generate low-resolution image data, the notification does not need to include the low resolution.

[0114] In some examples, the super-resolution multiplier of the NPU can be a fixed multiplier, but the resolution of the images required by each application in the CPU is not fixed. For example, some applications require an image resolution of 1080P, while others require an image resolution of 2K. In this case, the low resolution after the CPU reduces the resolution may be different, and the notification sent by the CPU to the GPU may carry the low resolution.

[0115] In some examples, the super-resolution factor of the NPU can be variable, meaning there are multiple super-resolution factors. The CPU can choose one of these factors to reduce the resolution, and the CPU can send a notification to the GPU with the lower resolution.

[0116] The CPU can also send a notification to the NPU, which can carry the resolution before and after the Nth frame image is reduced, such as step S107'. Step S107' is an optional step, for example, when the super-resolution factor of the NPU is not fixed.

[0117] S107' The CPU notifies the NPU of the high and low resolution of the Nth frame image. The NPU can then determine the resolution changes of the Nth frame image using these parameters. When performing super-resolution rendering using the AI ​​super-resolution model in the NPU, the AI ​​super-resolution module can determine the super-resolution factor to reconstruct image data that matches the high resolution of the Nth frame image. Of course, the NPU can also reconstruct image data with a resolution higher than the Nth frame image.

[0118] S108, GPU generates low-resolution image data, such as low-resolution RGB image data, which includes R channel data, G channel data and B channel data.

[0119] S109, the GPU sends low-resolution image data to the NPU.

[0120] S110' and NPU determine the super-resolution factor of the AI ​​super-resolution model.

[0121] The S110 and NPU utilize an AI super-resolution model to perform super-resolution rendering on low-resolution image data based on the super-resolution factor, thereby obtaining high-resolution image data.

[0122] The super-resolution factor of the AI ​​super-resolution model can be used as the super-resolution factor of the NPU. In some examples, the super-resolution factor of the AI ​​super-resolution model is a fixed factor, omitting step S110'. In some examples, the super-resolution factor of the AI ​​super-resolution model may not be fixed, such as the super-resolution factor of the AI ​​super-resolution model including 2x and 4x. The NPU executes step S110' to determine the currently used super-resolution factor based on the high resolution and low resolution of the Nth frame image notified in step S107'.

[0123] In this embodiment, the AI ​​super-resolution model can be trained using low-resolution image data as input and high-resolution image data as output, using historical low-resolution image data. During training, the AI ​​super-resolution model is validated using high-resolution images rendered by the GPU as a reference. Low resolution and high resolution are relative terms; high resolution can be greater than low resolution. For example, a low resolution of 540P and a high resolution of 1080P; or a low resolution of 720P and a high resolution of 2K.

[0124] AI super-resolution models can be trained offline using frameworks such as TensorFlow and PyTorch. However, some electronic devices do not support these frameworks. For example, mobile devices using frameworks like Qualcomm SNPE and NNAPI (Android Neural Networks API) support different model formats than TensorFlow and PyTorch. Qualcomm SNPE supports the DLC format, while NNAPI supports TensorFlow Lite. If an AI super-resolution model is trained offline using frameworks like TensorFlow and PyTorch, it can be converted to DLC or TensorFlow Lite formats for use on mobile devices.

[0125] After training an AI super-resolution model, it can be updated to adapt to image changes. For example, an electronic device can obtain the super-resolution rendering effect of the AI ​​super-resolution model and adjust its parameters based on this effect. This parameter adjustment can be done offline or online. The device adjusting the parameters can be the electronic device using the AI ​​super-resolution model for super-resolution rendering, or other electronic devices. After the other electronic device completes the parameter adjustment, it sends the results to the electronic device using the AI ​​super-resolution model for super-resolution rendering. For example, a computer adjusts the parameters of the AI ​​super-resolution model, and then sends the adjusted data to the mobile phone. During the application rendering process, the electronic device can collect data samples to obtain a data sample set, which can then be used to adjust the AI ​​super-resolution model.

[0126] S111, NPU sends high-resolution image data to GPU.

[0127] S112, the GPU performs rendering processing on the currently processed FB based on high-resolution image data, and obtains the rendering result, such as the image data of the elements in the Nth frame image.

[0128] S113: After obtaining the rendering results of all frames, the GPU displays the Nth frame image on the display screen based on the rendering results.

[0129] The above rendering method can be used in various scenarios of GPU-rendered images, such as in game applications, home design applications, modeling applications, augmented reality applications, and virtual display applications. It utilizes AI super-resolution models running in the NPU to perform super-resolution rendering, thereby using the computing power of the NPU to share some of the GPU's computing load, resulting in lower power consumption and shorter rendering time for electronic devices.

[0130] In this embodiment, image data interaction between the GPU and NPU can be accomplished using GPU memory and CPU memory. GPU memory can be the GPU's private memory, prohibiting the CPU and NPU from accessing GPU memory. However, the GPU can read image data from GPU memory by calling functions or other means. The image data read by the GPU can be written to CPU memory or used when the GPU is drawing images. Therefore, GPU memory can be used as the GPU's interaction memory and computing memory.

[0131] The CPU can designate a block of CPU memory (such as the first region) as both NPU input memory and NPU output memory. NPU input memory contains the input data written to the NPU for super-resolution rendering, while NPU output memory contains the output data written to the NPU after super-resolution rendering. The NPU input and output memory can be the same region within CPU memory. The CPU can specify the space in CPU memory occupied by the NPU based on the amount of input and output data. Both the GPU and NPU can access CPU memory. The corresponding memory access process is as follows: Figure 12 As shown, the following steps may be included: S200: The CPU designates the first region in the CPU memory as the NPU input memory and the NPU output memory. The NPU input memory and the NPU output memory can be the first region. When inputting data into the NPU, it is used as the NPU input memory, and when outputting data into the NPU, it is used as the NPU output memory. The size of the first region can be determined according to the amount of data processed by the NPU, which will not be elaborated here.

[0132] S201, The CPU notifies the NPU of the pointer address of the first region. The NPU can read image data from the NPU input memory based on the pointer address and write image data to the NPU output memory based on the pointer address.

[0133] S202, the GPU writes low-resolution image data into GPU memory.

[0134] S203, The GPU notifies the CPU that low-resolution image data has been written to GPU memory.

[0135] S204, the CPU notifies the GPU to write low-resolution image data into the first region. The notification sent by the CPU carries a pointer address to the first region. The CPU notifying the GPU of the pointer address of the first region can occur in step S201, i.e., in step S201, the CPU notifies the NPU and GPU of the pointer address of the first region.

[0136] S205, the GPU reads low-resolution image data from GPU memory.

[0137] S206, GPU writes low-resolution image data into the first region based on pointer addresses.

[0138] S207, The CPU instructs the NPU to read low-resolution image data from the first region.

[0139] The S208 and NPU read low-resolution image data from the first region based on pointer addresses.

[0140] S209, NPU writes high-resolution image data to the first region based on pointer addresses.

[0141] After reading low-resolution image data, the NPU inputs it into the AI ​​super-resolution model running within the NPU. The AI ​​super-resolution model performs super-resolution rendering on the low-resolution image data, outputting high-resolution image data. The NPU then writes the high-resolution image data into the first region based on the pointer address.

[0142] S210, The CPU instructs the GPU to read high-resolution image data from the first region.

[0143] S211, GPU reads high-resolution image data from the first region based on pointer addresses.

[0144] S212, the GPU writes high-resolution image data into GPU memory.

[0145] Combination Figure 12 The memory access process shown is as follows: Figure 11 The rendering method shown Figure 13 Another timing diagram of the rendering method is shown, which may include the following steps: S301, During the rendering of the N-1th frame of the image, the CPU determines FB1 as the master FB.

[0146] S302, The CPU intercepts the rendering command for the Nth frame image issued by the application.

[0147] S303, CPU obtains the initial resolution of the Nth frame image as high resolution and the currently processed FB based on the rendering command.

[0148] S304, the CPU performs runtime checks, model loading, and model compilation on the AI ​​super-resolution model.

[0149] S305, the CPU designates the first region in the CPU memory as the NPU input memory and NPU output memory, completing the memory configuration of the AI ​​super-resolution model, and completes the initialization of the AI ​​super-resolution model through steps S304 and S305.

[0150] S306, the CPU notifies the NPU of the pointer address of the first region.

[0151] S307: The CPU obtains the rendering method from the application's configuration file.

[0152] S308. If the rendering method is forward rendering and the currently processed FB is FB1, the CPU reduces the initial resolution of the image to a lower resolution based on the super-resolution factor; if the rendering method is deferred rendering and the FB is not FB0 (such as FB1 and FB2), the CPU reduces the initial resolution of the image to a lower resolution based on the super-resolution factor.

[0153] S309: The CPU instructs the GPU to generate low-resolution image data.

[0154] S309', The CPU notifies the NPU of the high and low resolution of the Nth frame of the image.

[0155] S310 and GPU generate low-resolution image data.

[0156] S311, the GPU writes low-resolution image data into GPU memory.

[0157] S312, GPU notifies CPU that low-resolution image data has been written to GPU memory.

[0158] S313, The CPU instructs the GPU to write low-resolution image data into the first region. The notification sent by the CPU carries a pointer address to the first region.

[0159] S314: The GPU reads low-resolution image data from GPU memory.

[0160] The S315 GPU writes low-resolution image data into the first region based on pointer addresses.

[0161] S316, The CPU instructs the NPU to read low-resolution image data from the first region.

[0162] The S317 NPU reads low-resolution image data from the first region based on pointer addresses.

[0163] S318' and NPU determine the super-resolution factor of the AI ​​super-resolution model.

[0164] The S318 and NPU utilize an AI super-resolution model to perform super-resolution rendering on low-resolution image data based on the super-resolution factor, thereby obtaining high-resolution image data.

[0165] The S319 and NPU write high-resolution image data to the first region based on pointer addresses.

[0166] S320, the CPU instructs the GPU to read high-resolution image data from the first region.

[0167] S321, the GPU reads high-resolution image data from the first region.

[0168] S322, the GPU writes high-resolution image data into GPU memory.

[0169] S323 and GPU perform rendering processing on the currently processed FB based on high-resolution image data to obtain the rendering result, such as the image data of the elements in the Nth frame image.

[0170] After obtaining the rendering results of all frames (FB), the S324 and GPU display the Nth frame image on the display screen based on the rendering results.

[0171] If the CPU designates one region of CPU memory as NPU input memory and another region as NPU output memory, the CPU sends pointer addresses to both the NPU input and output memory regions to the NPU. The NPU reads low-resolution image data based on the NPU input memory pointer address and reads high-resolution image data based on the NPU output memory pointer address. Similarly, the CPU sends these pointer addresses to the GPU, and the GPU writes low-resolution image data based on the NPU input memory pointer address and reads high-resolution image data based on the NPU output memory pointer address.

[0172] However, regardless of whether the image data is low-resolution or high-resolution, the GPU and NPU can interact using GPU memory and CPU memory. This results in the problem of image data being copied between GPU memory and CPU memory, which prolongs the time it takes for the GPU and NPU to acquire data, thereby increasing rendering time.

[0173] To address this issue, in this embodiment, the GPU, CPU, and NPU can interact with each other using shared memory. Shared memory refers to the memory that the GPU, CPU, and NPU can access. Shared memory can serve as an external cache for the GPU, storing at least the low-resolution image data input from the GPU to the NPU, and it can also serve as a processing cache for the NPU, storing at least the high-resolution image data output by the NPU.

[0174] The memory access process between the GPU and NPU using shared memory is as follows: Figure 14 As shown, the following steps may be included: S401. The CPU requests shared memory from the hardware buffer. The CPU can request shared memory from the hardware buffer of Random Access Memory (RAM). One method of shared memory allocation is for the CPU to request storage space for the GPU Shader Storage Buffer Object (SSBO) from the hardware buffer. After obtaining low-resolution image data from the GPU's Fragment Shader or Compute Shader, the low-resolution image data is written to the GPU SSBO. High-resolution image data output by the NPU can also be written to shared memory.

[0175] As a type of memory, the Hardware Buffer has certain requirements regarding its format when accessed by the CPU, GPU, and NPU. For example, when requesting shared memory that is accessible to the Fragment Shader or Compute Shader in the GPU, the Hardware Buffer format can be specified as AHARDWAREBUFFER_FORMAT_BLOB.

[0176] Another method for allocating shared memory is for the CPU to request two blocks of shared memory from the Hardware Buffer. One block stores low-resolution image data obtained by the GPU, serving as input data for the NPU; the other block stores NPU output data, such as high-resolution image data obtained from super-resolution rendering, serving as input data for the GPU during UI rendering. The two shared memory blocks requested by the CPU have the same format, such as AHARDWAREBUFFER_FORMAT_BLOB. The size of the allocated shared memory depends on the image resolution. For example, the allocated shared memory size is width × height × bytes. If the image data type is float, and a float variable is 4 bytes, then the allocated shared memory size would be width × height × 3 × 4, where the width and height depend on the image resolution. For example, for a 1080P image, the allocated shared memory size would be 1920 × 1080 × 3 × 4.

[0177] S402, the CPU notifies the GPU of the pointer address of the shared memory so that the shared memory can be used as storage memory for low-resolution image data.

[0178] S403: The CPU notifies the NPU of the pointer address of the shared memory so that the shared memory can be used as the storage memory for high-resolution image data.

[0179] S404 and GPU write low-resolution image data into shared memory based on pointer addresses in shared memory.

[0180] S405: The CPU instructs the NPU to read low-resolution image data from shared memory.

[0181] The S406 and NPU read low-resolution image data from shared memory based on pointer addresses in shared memory.

[0182] The S407 and NPU write high-resolution image data into shared memory based on pointer addresses in shared memory.

[0183] S408: The CPU instructs the GPU to read high-resolution image data from shared memory.

[0184] The S409 GPU reads high-resolution image data from shared memory based on pointer addresses in shared memory.

[0185] If the CPU requests two shared memory blocks from the Hardware Buffer, one shared memory block is used to store low-resolution image data obtained by the GPU, serving as input data for the NPU; the other shared memory block is used to store NPU output data, such as high-resolution image data obtained from super-resolution rendering, serving as input data for the GPU during UI rendering. The CPU can send pointer addresses of these two shared memory blocks to the GPU and NPU, enabling the GPU and NPU to read and write image data within the shared memory.

[0186] GPUs and NPUs use the above... Figure 14 The memory access process shown enables data sharing using shared memory, achieving zero-copy data sharing between the GPU and NPU, thus improving processing efficiency. Combined with... Figure 14 The memory access process shown is as follows: Figure 11 The rendering method shown Figure 15 Another timing diagram of the rendering method is shown, which may include the following steps: S501, during the rendering of the N-1th frame of the image, the CPU determines FB1 as the main FB.

[0187] S502, CPU intercepts the rendering command for the Nth frame image issued by the application.

[0188] S503 and CPU obtain the initial resolution of the Nth frame image as high resolution and the currently processed FB based on the rendering command.

[0189] S504 and CPU perform runtime checks, model loading, and model compilation on AI super-resolution models.

[0190] S505, the CPU requests shared memory from the Hardware Buffer of RAM. The shared memory serves as storage space for low-resolution image data and high-resolution image data. The memory configuration of the AI ​​super-resolution model is completed by requesting shared memory. The initialization of the AI ​​super-resolution model is completed through steps S504 and S505.

[0191] S506, the CPU notifies the GPU and NPU of the pointer address of shared memory.

[0192] S507: The CPU obtains the rendering method from the application's configuration file.

[0193] S508. If the rendering method is forward rendering and the currently processed FB is FB1, the CPU reduces the initial resolution of the image to a lower resolution based on the super-resolution factor; if the rendering method is deferred rendering and the FB is not FB0 (such as FB1 and FB2), the CPU reduces the initial resolution of the image to a lower resolution based on the super-resolution factor.

[0194] S509: The CPU instructs the GPU to generate low-resolution image data.

[0195] S509', the CPU notifies the NPU of the high and low resolution of the Nth frame of the image.

[0196] S510 and GPU generate low-resolution image data.

[0197] The S511 GPU writes low-resolution image data into shared memory based on pointer addresses in shared memory.

[0198] S512, the CPU instructs the NPU to read low-resolution image data from shared memory.

[0199] The S513 and NPU read low-resolution image data from shared memory based on pointer addresses in shared memory.

[0200] The S514' and NPU determine the super-resolution factor of the AI ​​super-resolution model.

[0201] The S514 and NPU utilize an AI super-resolution model to perform super-resolution rendering on low-resolution image data based on the super-resolution factor, thereby obtaining high-resolution image data.

[0202] The S515 and NPU use pointer addresses in shared memory to write high-resolution image data into shared memory.

[0203] S516: The CPU instructs the GPU to read high-resolution image data from shared memory.

[0204] The S517 GPU reads high-resolution image data from shared memory based on pointer addresses in shared memory.

[0205] The S518 GPU renders the currently processed FB based on high-resolution image data, and obtains the rendering result, such as the image data of the elements in the Nth frame image.

[0206] After obtaining the rendering results of all frames (FBs), the S519 GPU displays the Nth frame image on the display screen based on the rendering results.

[0207] Some embodiments of this application provide an electronic device, which includes a first processor, a second processor, and a memory; wherein the memory is used to store one or more computer program codes, the computer program codes including computer instructions, and when the first processor and the second processor execute the computer instructions, the first processor and the second processor execute the above-described rendering method.

[0208] Some embodiments of this application provide a chip system including program code that, when run on an electronic device, causes a first processor and a second processor in the electronic device to execute the rendering method described above.

[0209] Some embodiments of this application provide a processor, which is a second processor, including a processing unit and a memory; wherein the memory is used to store one or more computer program codes, the computer program codes including computer instructions, and when the second processor executes the computer instructions, the second processor executes the above-described rendering method.

[0210] Some embodiments of this application provide a computer storage medium including computer instructions that, when executed on an electronic device, cause a second processor in the electronic device to perform the rendering method described above.

[0211] This embodiment also provides a control device, which includes one or more processors and a memory. The memory stores one or more computer program codes, which include computer instructions. When the one or more processors execute the computer instructions, the control device performs the above-described method. This control device can be an integrated circuit (IC) or a system-on-a-chip (SoC). The integrated circuit can be a general-purpose integrated circuit, a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC).

[0212] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0213] In the several embodiments provided in this example, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0214] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0215] Furthermore, in each embodiment of this invention, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0216] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0217] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A rendering method, characterized in that, The method is applied to an electronic device running an application program, the electronic device including a first processor and a second processor, and the method includes: The first processor receives a first rendering instruction from the application, the first rendering instruction corresponding to a first resolution and a first frame buffer; The first processor sends a second rendering instruction to the second processor according to the first rendering instruction; The second processor generates image data with a second resolution based on the second rendering instruction, the image data with the second resolution corresponding to the first image, and the second resolution being no greater than the first resolution; The second processor generates the first image based on image data with a third resolution, the image data with the third resolution corresponding to the first image, the third resolution being greater than the second resolution.

2. The method according to claim 1, characterized in that, The application uses forward rendering, and the first frame buffer is at least one of all frame buffers (FBs).

3. The method according to claim 1 or 2, characterized in that, The first frame buffer is FB0.

4. The method according to any one of claims 1-3, characterized in that, The number of drawing instructions corresponding to the second rendering instruction executed by the first frame buffer is greater than a preset threshold.

5. The method according to claim 1, characterized in that, The application uses deferred rendering, and the second rendering instruction corresponds to all frame buffers issued by the application except for the last frame buffer.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The display settings interface includes a first settings option, which is used to improve the resolution. Received an operation applied to the first setting option; Upon receiving the operation applied to the first setting option, the first processor receives a first rendering instruction from the application.

7. The method according to claim 6, characterized in that, Before displaying the settings interface, the method further includes: The first processor receives a third rendering instruction from the application, the third rendering instruction corresponding to a first resolution and a first frame buffer; The first processor sends a fourth rendering instruction to the second processor according to the third rendering instruction; The second processor generates image data with the first resolution based on the fourth rendering instruction, the image data with the first resolution corresponding to the second image; The second processor generates the second image based on the image data having the first resolution.

8. The method according to claim 6 or 7, characterized in that, The received operation applied to the first setting option specifically includes: An operation applied to the first setting option is received during the operation of the application.

9. The method according to any one of claims 1-8, characterized in that, The application is either a game application or a video application.

10. The method according to any one of claims 1-9, characterized in that, The third resolution is greater than the first resolution.

11. The method according to any one of claims 1-10, characterized in that, The second resolution is equal to the first resolution.

12. The method according to any one of claims 1-11, characterized in that, The first processor is a central processing unit, and the second processor is a graphics processing unit.

13. The method according to any one of claims 1-12, characterized in that, The electronic device also includes a third processor, which generates the image data with a third resolution.

14. The method according to claim 13, characterized in that, The third processor is a neural network processor or a digital signal processor.

15. The method according to any one of claims 1-14, characterized in that, The electronic device in question is a mobile phone.

16. The method according to any one of claims 1-15, characterized in that, The method further includes: The second processor writes the image data with the second resolution into the first memory; The second processor reads the image data with the third resolution from the first memory.

17. The method according to claim 16, characterized in that, The electronic device further includes a third processor, and after the second processor writes the image data with the second resolution into the first memory, the method further includes: The third processor reads the image data with the second resolution from the first memory; The third processor generates the image data with the third resolution based on the image data with the second resolution. The third processor writes the image data with the third resolution into the first memory.

18. The method according to claim 16 or 17, characterized in that, Before the second processor writes the image data with the second resolution to the first memory, the method further includes: The second processor writes the image data with the second resolution into the second memory, and the second processor has permission to access the second memory and the first memory; The second processor sends a first notification to the first processor, the first notification indicating that the image data with the second resolution has been successfully written to the second memory; In response to receiving the first notification, the first processor sends a second notification to the second processor, the second notification instructing the second processor to write the image data with the second resolution into the first memory, the second notification carrying an address pointer of the first memory; In response to receiving the second notification, the second processor reads the second image data having the second resolution from the second memory.

19. The method according to claim 17, characterized in that, The electronic device further includes a third processor, and after the second processor writes the image data with the second resolution into the first memory, and before the third processor reads the image data with the second resolution from the first memory, the method further includes: The first processor sends a third notification to the third processor, the third notification instructing the third processor to read the image data with the second resolution from the first memory; After the third processor writes the image data with the third resolution into the first memory, and before the second processor reads the image data with the third resolution from the first memory, the method further includes: The first processor sends a fourth notification to the second processor, the fourth notification being used to instruct the second processor to read the image data with the third resolution from the first memory, the fourth notification carrying an address pointer of the first memory; After the second processor reads the image data with the third resolution from the first memory, the method further includes: The second processor writes the image data with the third resolution into the second memory. The second processor has permission to access the second memory and the first memory. The third processor has permission to access the first memory.

20. The method according to claim 19, characterized in that, After the second processor writes the image data with the second resolution to the first memory, and before the second processor reads the image data with the third resolution from the first memory, the method further includes: The first processor sends a fifth notification to the third processor, the fifth notification being used to instruct the third processor to read the second image data having the second resolution from the first memory; The third processor reads the second image data with the second resolution from the first memory; The third processor generates the image data with the third resolution based on the second image data with the second resolution. The third processor writes the image data with the third resolution into the first memory; The first processor sends a sixth notification to the second processor, the sixth notification being used to instruct the second processor to read the image data with the third resolution from the first memory.

21. The method according to any one of claims 13, 14, 17, 19, and 20, characterized in that, The third processor runs an artificial intelligence super-resolution model. The third processor uses the artificial intelligence super-resolution model to perform super-resolution rendering on the image data of the first image at the second resolution, and generates the image data of the first image at the third resolution.

22. The method according to claim 21, characterized in that, Before the third processor performs super-resolution rendering on the first image using the image data with the second resolution, the method further includes: the first processor initializing the artificial intelligence super-resolution model, wherein the initialization is used to determine the operation of the artificial intelligence super-resolution model and to determine the normal operation of the artificial intelligence super-resolution model; The initialization includes runtime detection, model loading, model compilation, and memory configuration. The runtime detection is used to determine the operation of the AI ​​super-resolution model, and the model loading, model compilation, and memory configuration are used to determine the normal operation of the AI ​​super-resolution model.

23. An electronic device, characterized in that, The electronic device includes: a first processor, a second processor, and a memory; wherein the memory is used to store one or more computer program codes, the computer program codes including computer instructions, and when the first processor and the second processor execute the computer instructions, the first processor and the second processor execute the rendering method as described in any one of claims 1 to 22.

24. A chip system, characterized in that, The chip system includes: a first processor and a second processor; The first processor is configured to: receive a first rendering instruction issued by the application; and send a second rendering instruction to the second processor according to the first rendering instruction; wherein the first rendering instruction corresponds to a first resolution and the first rendering instruction corresponds to a first frame buffer; The second processor is configured to: generate image data with a second resolution based on the second rendering instruction, the image data with the second resolution corresponding to the first image, wherein the second resolution is not greater than the first resolution; The second processor is further configured to: generate the first image based on image data having a third resolution, the image data having a third resolution corresponding to the first image, the third resolution being greater than the second resolution.

25. The chip system according to claim 24, characterized in that, The application uses forward rendering, and the first frame buffer is at least one of the frame buffers (FBs).

26. The chip system according to claim 24 or 25, characterized in that, The first frame buffer is FB0.

27. The chip system according to any one of claims 24-26, characterized in that, The number of drawing instructions corresponding to the rendering indication executed by the first frame buffer is greater than a preset threshold.

28. The chip system according to claim 24, characterized in that, The application uses deferred rendering, and the second rendering instruction corresponds to all frame buffers issued by the application except for the last frame buffer.

29. The chip system according to any one of claims 24-28, characterized in that, The method further includes: The display settings interface includes a first settings option, which is used to improve the resolution. Received an operation applied to the first setting option; Upon receiving the operation applied to the first setting option, the first processor receives a first rendering instruction from the application.

30. The chip system according to claim 29, characterized in that, Before displaying the settings interface, the method further includes: The first processor receives a third rendering instruction from the application, the third rendering instruction corresponding to a first resolution and a first frame buffer; The first processor sends a fourth rendering instruction to the second processor according to the third rendering instruction; The second processor generates image data with the first resolution based on the fourth rendering instruction, the image data with the first resolution corresponding to the second image; The second processor generates the second image based on the image data having the first resolution.

31. The chip system according to claim 29 or 30, characterized in that, The received operation applied to the first setting option specifically includes: An operation applied to the first setting option is received during the operation of the application.

32. The chip system according to any one of claims 24-31, characterized in that, The application is either a game application or a video application.

33. The chip system according to any one of claims 24-32, characterized in that, The third resolution is greater than the first resolution.

34. The chip system according to any one of claims 24-33, characterized in that, The second resolution is equal to the first resolution.

35. The chip system according to any one of claims 24-34, characterized in that, The first processor is a central processing unit, and the second processor is a graphics processing unit.

36. The chip system according to any one of claims 24-35, characterized in that, The electronic device also includes a third processor, which generates the image data with a third resolution.

37. The chip system according to claim 36, characterized in that, The third processor is a neural network processor or a digital signal processor.

38. A computer storage medium, characterized in that, The computer storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the rendering method as described in any one of claims 1 to 22.

Images