Rendering methods and devices

Abstract

This application provides a rendering method and apparatus, wherein a first processor receives a rendering command for a first image issued by an application; a third processor performs rendering processing on the first image to obtain a processing result of the first image; and while the third processor is rendering the first image, a second processor draws a second image. After receiving the rendering command for the first image from the application, the first processor can perform rendering processing on the first image to obtain a processing result of the first image. While the third processor is rendering the first image, the second processor is drawing the second image. Although this delays the drawing of the first image, it allows for earlier drawing by the second processor compared to waiting for the processing result of the first image, thus improving smoothness. Furthermore, the third processor can perform rendering processing on the first image in parallel with the second processor drawing the second image.

Description

[0001] This application claims priority to Chinese Patent Application No. 202111362400.3, filed on November 17, 2021, entitled "Rendering Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing technology, and in particular to a rendering method and apparatus. Background Technology

[0003] With the development of display technology, image resolution has evolved towards higher resolutions, such as from 720P to 1080P, and then from 1080P to 2K. Here, P represents the total number of rows of pixels (e.g., 720P has 720 rows of pixels), and k represents the total number of columns of pixels (e.g., 2K has 2000 columns of pixels). Electronic devices consume excessive computing power when rendering high-resolution or ultra-high-resolution images, making it difficult for them to support the rendering requirements. This reduces frame rate and smoothness. Frame rate is the frequency at which images appear continuously on the display screen, measured in frames. Summary of the Invention

[0004] The rendering method and apparatus provided in this application solve the problem of reduced frame rate and smoothness when electronic devices render images.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] Firstly, this application provides a rendering method applied to the rendering processing of a first image by an electronic device. The electronic device runs an application program and includes a first processor, a second processor, and a third processor. The method includes: the first processor receiving a rendering command for the first image issued by the application program; the third processor performing rendering processing on the first image to obtain a processing result of the first image; and the second processor drawing a second image while the third processor is rendering the first image. After receiving the rendering command for the first image from the application program, the third processor can perform rendering processing on the first image to obtain a processing result of the first image. While the third processor is rendering the first image, the second processor is drawing the second image. Although this delays the drawing of the first image, it allows for earlier drawing by the second processor compared to waiting for the processing result of the first image, thus improving smoothness. Furthermore, the third processor can perform rendering processing on the first image in parallel while the second processor is drawing the second image.

[0007] The second image can be the previous frame of the first image, and the processing result of the first image can be its image data. While the third processor is obtaining the image data of the first image, the second processor can draw the previous frame. Although the drawing of the first image is delayed, it allows the second processor to draw earlier than waiting for the processing of the first image, thus improving smoothness. Furthermore, when drawing the second image, its processing result is already complete, reducing or eliminating the waiting time for the second image's processing result, improving the drawing efficiency of the second processor, and consequently increasing the frame rate displayed on the screen.

[0008] Optionally, the electronic device also includes a display screen. Before the first processor receives the rendering command for the first image issued by the application, the method further includes: the first processor determining a first frame buffer during the processing of the third image; the first frame buffer is the frame buffer among all frame buffers issued by the application where the number of drawing instructions executed is greater than a preset threshold; the third image is the previous frame image of the second image, and the second image is the previous frame image of the first image; the second processor draws the third image to display it on the display screen; the first processor receives the rendering command for the second image issued by the application; the third processor performs rendering processing on the second image to obtain the processing result of the second image; during the rendering processing of the second image by the third processor, the second processor controls the display screen to continue displaying the third image. Although the display of the second image is delayed, the display screen can display the third image related to the application, reducing the occurrence of black screens or blue screens and improving the user experience.

[0009] Optionally, before the third processor renders the first image based on a first resolution, the method further includes: the first processor sending a rendering command to the second processor, the rendering command instructing the second processor to render the first image; the second processor generating image data of the first image at a second resolution, where the second resolution is not greater than the first resolution, based on the rendering command; the second processor writing the image data of the first image at the second resolution into a first region of the first memory; the third processor reading the image data of the first image at the second resolution from the first region; and the third processor rendering the first image to obtain the processing result of the first image including: the third processor generating image data of the first image at a third resolution, where the third resolution is greater than the second resolution, based on the image data of the first image at the second resolution. After the second processor writes the generated image data of the first image at the second resolution into the first region of the first memory, the third processor generates the image data of the first image at the third resolution, saving the computing power of the second processor. This allows the second processor to use more resources when drawing images, improving rendering smoothness and solving the problem of runtime lag. Furthermore, the third resolution is greater than the second resolution, indicating that the third processor can obtain image data with a relatively high resolution. When the second processor draws the first image later, it can draw the first image based on the image data with a relatively high resolution, thereby improving the image quality of the first image.

[0010] Optionally, after the third processor generates image data of the first image at a third resolution based on the image data of the first image at a second resolution, the method further includes: the third processor writing the image data of the first image at the third resolution to a second region of the first memory. The image data of the first image at the second resolution can be written to a first region of the first memory, and the image data of the first image at the third resolution can be written to a second region of the first memory. By storing the image data of the first image at different resolutions in two separate regions, the possibility of data overwriting is reduced.

[0011] Optionally, after the second processor writes the image data of the first image at the second resolution to the first region of the first memory, the method further includes: the second processor writing the supplementary resources of all frame buffers of the first image and the drawing instruction stream of the last frame buffer of the first image to the second memory. The second processor has permission to access the second memory. During the process of the application issuing the rendering command for the first image, the supplementary resources of all frame buffers of the first image and the drawing instruction stream of the last frame buffer of the first image are also issued. This information is used when drawing the first image. Therefore, the second processor writes the supplementary resources of all frame buffers of the first image and the drawing instruction stream of the last frame buffer of the first image to the second memory to ensure that the first image can be drawn accurately. The second processor has permission to access the second memory, while the first processor and the third processor do not, preventing users from modifying the data in the second memory through the first processor and the third processor, thereby improving security.

[0012] Optionally, after the second processor writes the image data of the first image at the second resolution to the first region of the first memory, and before the third processor reads the image data of the first image at the second resolution from the first region, the method further includes: the first processor sending a first notification to the third processor, the first notification instructing the third processor to read the image data of the first image at the second resolution from the first region. The first processor can monitor the operation of the second processor and promptly notify the third processor to retrieve the image data of the first image at the second resolution after the second processor writes the image data of the first image at the second resolution to the first region of the first memory, thereby improving efficiency.

[0013] Optionally, before the first processor sends rendering instructions to the second processor, the method further includes: the first processor allocating first memory from a hardware buffer, the first memory including a first region and a second region; the first processor sending pointer addresses of the first region and the second region to the third and second processors, the first, second, and third processors having access to the first memory; the third and second processors performing low-resolution image data read / write operations in the first region based on the pointer address of the first region, and performing high-resolution image data read / write operations in the second region based on the pointer address of the second region. The first processor can allocate first memory from the hardware buffer, and the second processor does not need to write the image data of the first image at the second resolution to other memory before writing it to the first memory. Similarly, the third processor does not need to write the image data of the first image at the third resolution to other memory before writing it to the first memory. The second and third processors can share data based on the first memory, achieving zero-copy data sharing between the second and third processors and improving processing efficiency.

[0014] Optionally, before the second processor draws the second image, the method further includes: the first processor sending a second notification to the second processor, the second notification being used to instruct the second processor to read image data of the second image at a third resolution from the second region.

[0015] Optionally, during the rendering process of the first image by the third processor, the second processor drawing the second image includes: after the second processor writes the image data of the first image at a second resolution to a first area of ​​the first memory, the second processor reads the image data of the second image at a third resolution from the second area; the second processor reads the supplementary resources of all frame buffers of the second image and the drawing instruction stream of the last frame buffer of the second image from the second memory, and the second processor has permission to access the second memory; the second processor draws the second image based on the image data of the second image at the third resolution, the supplementary resources of all frame buffers of the second image, and the drawing instruction stream of the last frame buffer of the second image. In this embodiment, the second memory can back up the supplementary resources of all frame buffers of the second image and the drawing instruction stream of the last frame buffer of the second image to ensure that the second image can be drawn normally and to ensure the accuracy of the second image.

[0016] Optionally, before the first processor sends rendering instructions to the second processor, the method further includes: the first processor allocating first memory from its own memory, the first memory including a first region and a second region; the first processor sending pointer addresses of the first region and the second region to the third processor, the first processor and the third processor having access to the first memory; the third processor performing low-resolution image data reading in the first region based on the pointer address of the first region, and performing high-resolution image data writing in the second region based on the pointer address of the second region.

[0017] Optionally, after the second processor generates image data of the first image at a second resolution based on rendering instructions, and before the second processor writes the image data of the first image at the second resolution to a first region of the first memory, the method further includes: the second processor writing the image data of the first image at the second resolution to the second memory, the second processor having permission to access the second memory; the second processor sending a third notification to the first processor, the third notification indicating that the image data of the first image at the second resolution has been successfully written to the second memory; in response to receiving the third notification, the first processor sending a fourth notification to the second processor, the fourth notification indicating that the second processor writes the image data of the first image at the second resolution to the first region, the fourth notification carrying an address pointer of the first region; and in response to receiving the fourth notification, the second processor reading the image data of the first image at the second resolution from the second memory.

[0018] In this embodiment, the first processor can monitor the image data read / write operations of the second processor. After the second processor writes the image data of the first image at the second resolution to the second memory, the first processor can promptly trigger the second processor to write the image data of the first image at the second resolution to the first memory, thus improving efficiency. Furthermore, the second notification sent by the first processor carries an address pointer to the first memory. Based on this address pointer, the second processor can write the image data of the first image at the second resolution to the first memory, improving the accuracy of the write operation.

[0019] Optionally, before the second processor draws the second image, the method further includes: the first processor sending a fifth notification to the second processor, the fifth notification instructing the second processor to read image data of the second image at a third resolution from the second region, the fifth notification carrying an address pointer of the second region; in response to the fifth notification, the second processor reading image data of the second image at the third resolution from the second region; the second processor writing the image data of the second image at the third resolution into the second memory, the second processor having permission to access the second memory.

[0020] Optionally, during the rendering process of the first image by the third processor, the second processor drawing the second image includes: the second processor reading the image data of the second image at a third resolution, the supplementary resources of all frame buffers of the second image, and the drawing instruction stream of the last frame buffer of the second image from the second memory, wherein the second processor has access to the second memory; and the second processor drawing the second image based on the image data of the second image at the third resolution, the supplementary resources of all frame buffers of the second image, and the drawing instruction stream of the last frame buffer of the second image. In this embodiment, the second memory can back up the supplementary resources of all frame buffers of the second image and the drawing instruction stream of the last frame buffer of the second image to ensure that the second image can be drawn normally and to ensure the accuracy of the second image.

[0021] Optionally, after the second processor reads the image data of the second image at the third resolution, the method further includes: the second processor sending a sixth notification to the first processor, the sixth notification instructing the second processor to complete reading the image data of the second image at the third resolution; in response to the sixth notification, the first processor waking up a first thread in the third processor, the first thread calling an artificial intelligence super-resolution model, the artificial intelligence super-resolution model performing super-resolution rendering on the image data of the first image at the second resolution to generate image data of the first image at the third resolution. After the second processor completes reading the image data of the second image at the third resolution, the second processor can send a sixth notification to the first processor. Under the action of the sixth notification, the first processor wakes up the first thread in the third processor and begins calling the artificial intelligence super-resolution model to perform super-resolution rendering on the image data of the first image at the second resolution, thereby enabling parallel computation of the image data of the first image while rendering the second image. The first thread can be an artificial intelligence thread running in the third processor, created by the first processor, during which the initialization of the artificial intelligence super-resolution model can be completed.

[0022] Optionally, before the first processor wakes up the first thread in the third processor, the method further includes: the first processor sending a first resolution and a second resolution of the first image to the third processor; the third processor determining a super-resolution factor of the artificial intelligence super-resolution model based on the first and second resolutions of the first image; and the artificial intelligence super-resolution model performing super-resolution rendering on the image data of the first image at the second resolution based on the super-resolution factor. The artificial intelligence super-resolution model may have at least one super-resolution factor, and each super-resolution factor can perform conversions to different resolutions. For example, using image data at the same resolution as input, different super-resolution factors result in different resolutions for the image data output by the artificial intelligence super-resolution model. To ensure that the resolution of the image data output by the artificial intelligence super-resolution model is not less than the first resolution, the first processor may send the first and second resolutions to the third processor. The third processor selects a super-resolution factor not less than the difference in factors between the first and second resolutions, thereby ensuring that the resolution of the image data output by the artificial intelligence super-resolution model is not less than the first resolution.

[0023] Optionally, the method further includes: the first processor initializing the AI ​​super-resolution model, wherein initialization is used to determine whether the AI ​​super-resolution model is running and whether it is operating normally; initialization includes runtime checks, model loading, model compilation, and memory configuration, wherein runtime checks are used to determine whether the AI ​​super-resolution model is running, and model loading, model compilation, and memory configuration are used to determine whether the AI ​​super-resolution model is operating normally. The initialization of the AI ​​super-resolution model by the first processor can be completed during the initialization of the first thread, and the initialization of the first thread and the AI ​​super-resolution model is completed in one initialization process.

[0024] Optionally, after the first processor wakes up the first thread in the third processor, the method further includes: during the process of the first thread calling the artificial intelligence super-resolution model to generate image data of the first image at the third resolution, the third processor monitors the operation of the first thread; after the third processor detects that the first thread has finished calling the artificial intelligence super-resolution model, it puts the first thread to sleep, so as to switch the first thread from the running state to the sleeping state; after the artificial intelligence super-resolution model generates image data of the first image at the third resolution, the first thread ends the call to the artificial intelligence super-resolution model, thereby controlling the first thread to sleep after completing the rendering process of the first image, reducing the occupation of resources.

[0025] Optionally, after the third processor detects that the first thread has finished calling the AI ​​super-resolution model, putting the first thread to sleep includes: after the third processor detects that the first thread has finished calling the AI ​​super-resolution model, controlling the first thread to switch from a running state to a running-completed state; or the third processor detects that the duration of the first thread in the running-completed state is longer than a preset duration, and puts the first thread to sleep, thereby switching the first thread from the running-completed state to the sleep state. The preset duration indicates that the first thread will not perform rendering processing for a period of time. The first thread is idle but still running, which will occupy some resources. In this case, switching the first thread to the sleep state reduces resource consumption.

[0026] Optionally, the third processor monitors the operation of the first thread. The method further includes: if the third processor detects an error in the first thread's call to the AI ​​super-resolution model, it forces the first thread to exit; the third processor sends a notification to the second processor, instructing the second processor to render the first image, thereby reducing the resource consumption of the first thread. When the first thread encounters an error in its call to the AI ​​super-resolution model, the second processor can be triggered promptly to render the first image, shortening the rendering time compared to waiting to resolve the error.

[0027] Optionally, if the third processor detects an error in the first thread's call to the AI ​​super-resolution model and forces the first thread to exit, the third processor may detect that the AI ​​super-resolution model's rendering processing of the first image has timed out and force the first thread to exit.

[0028] Optionally, after the third processor detects an error in the first thread's call to the AI ​​super-resolution model and forces the first thread to exit, the method further includes: after the first processor detects that the first thread's exit meets a preset condition, it wakes up the first thread again. If the first processor detects that the first thread's exit meets a preset condition and wakes up the first thread again, the first thread calls the AI ​​super-resolution model to render other images, thus sharing some of the work of the second processor and improving the rendering efficiency of the second processor.

[0029] Optionally, before the first processor sends rendering instructions to the second processor, the method further includes: the first processor reducing the resolution of the first image from a first resolution to a second resolution.

[0030] Optionally, the third processor has a super-resolution factor, which indicates the difference between the second and third resolutions; the third resolution is the same as the first resolution; the first processor reducing the resolution of the first image from the first resolution to the second resolution includes: the first processor reducing the first resolution to the second resolution based on the super-resolution factor. The artificial intelligence super-resolution model may have at least one super-resolution factor, each super-resolution factor capable of performing different resolution conversions. For example, using image data at the same resolution as input, different super-resolution factors result in different resolutions for the image data output by the artificial intelligence super-resolution model. To ensure that the resolution of the image data output by the artificial intelligence super-resolution model is not less than the first resolution, the first processor may send the first resolution and the second resolution to the third processor. The third processor selects a super-resolution factor not less than the factor difference between the first and second resolutions, thereby ensuring that the resolution of the image data output by the artificial intelligence super-resolution model is not less than the first resolution.

[0031] Optionally, if the application uses forward rendering, the rendering instructions correspond to the first framebuffer, and the number of drawing instructions executed in the first framebuffer exceeds a preset threshold; if the application uses deferred rendering, the rendering instructions correspond to all framebuffers except the last one issued by the application. When the application uses different rendering methods, the rendering instructions sent by the first processor to the second processor target different framebuffers, indicating that the timing of the first processor sending rendering instructions differs depending on the rendering method used by the application, thus achieving framebuffer-based control of rendering instruction sending under different rendering methods.

[0032] Optionally, the first framebuffer is the framebuffer that executes the most drawing instructions among all framebuffers.

[0033] Optionally, before the first processor sends rendering instructions to the second processor, the method further includes: the first processor obtaining the rendering method of the application from the application's configuration file.

[0034] Optionally, the rendering instruction is used to instruct the second processor to render the first image based on a second resolution, which is smaller than the first resolution. The rendering instruction sent by the first processor to the second processor carries the second resolution of the first image, specifying the resolution of the image data generated by the second processor. This prevents the second processor from generating image data that does not match the resolution required by the application, thus improving the accuracy of the first image. Furthermore, since the second resolution is smaller than the first resolution, the data volume is also smaller. Therefore, by specifying a second resolution smaller than the first resolution, the amount of data processed by the second processor is reduced, thereby reducing the power consumption of the electronic device and solving the problem of excessive heat generation.

[0035] Optionally, the second resolution is lower than the first resolution; the third resolution is the same as the first resolution, or the third resolution is higher than the first resolution. When the second resolution is lower than the first resolution and the third resolution is the same as the first resolution, the amount of data processed by the second processor is reduced when generating image data for the first image at the second resolution. However, when the third resolution is the same as the first resolution, it means the second processor can read the image data corresponding to the first resolution and draw the first image at the first resolution, ensuring that the image quality of the drawn first image meets the requirements of the application. If the third resolution is higher than the first resolution, it means the second processor can read image data with a resolution higher than the first resolution, and the image quality of the drawn first image is better than the requirements of the application, thus improving the overall image quality of the first image.

[0036] Optionally, the second resolution is equal to the first resolution. In this embodiment, the second resolution is equal to the first resolution, but the third resolution is greater than the second resolution. Therefore, the third resolution is also greater than the first resolution. When the application requests rendering the first image based on the first resolution, the second processor can render the first image based on the third resolution. The image quality of the rendered first image is better than the image quality requested by the application, thus improving the image quality of the first image.

[0037] Optionally, the third processor is a neural network processor or a digital signal processor.

[0038] Secondly, this application provides a rendering method applied to a second processor in an electronic device. The electronic device also includes a first processor and a third processor. The electronic device runs an application program that sends rendering commands for a first image to the first processor. The method includes: while the third processor is rendering the first image, the second processor is drawing a second image. Although the drawing of the first image is delayed, the second processor can draw earlier than waiting for the processing result of the first image, thus improving smoothness. Furthermore, while the second processor is drawing the second image, the third processor can perform rendering processing on the first image in parallel. The second image can be the previous frame of the first image, and the processing result of the first image can be the image data of the first image. While the third processor is obtaining the image data of the first image, the second processor can draw the previous frame. Although the drawing of the first image is delayed, the second processor can draw earlier than waiting for the processing result of the first image, thus improving smoothness. Moreover, when the second image is drawn, the processing result of the second image has already been completed, reducing or eliminating the time spent waiting for the processing result of the second image, improving the drawing efficiency of the second processor, and thus increasing the frame rate displayed on the screen.

[0039] Optionally, the electronic device also includes a display screen. During the third image processing, the first processor determines a first frame buffer, which is the frame buffer among all frame buffers issued by the application where the number of drawing instructions executed exceeds a preset threshold. The third image is the previous frame of the second image, and the second image is the previous frame of the first image. The application sends a rendering command for the second image to the first processor. Before the second processor draws the second image, the method further includes: the second processor drawing the third image to display it on the display screen; during the rendering process of the second image by the third processor, the second processor controls the display screen to continue displaying the third image. Although the display of the second image is delayed, the display screen can display the third image related to the application, reducing the occurrence of black screens or blue screens and improving the user experience.

[0040] Optionally, the rendering command for the first image is used to instruct the second processor to render the first image based on the first resolution; before the second processor draws the second image, the method further includes: the second processor receiving a rendering instruction sent by the first processor, the rendering instruction being used to instruct the second processor to render the first image; the second processor generating image data of the first image at a second resolution based on the rendering instruction, the second resolution being no greater than the first resolution; the second processor writing the image data of the first image at the second resolution into a first area of ​​the first memory; the second processor reading the image data of the second image at a third resolution from a second area of ​​the first memory, the third resolution being greater than the second resolution, the image data of the second image at the third resolution being used to draw the second image.

[0041] After the second processor writes the generated first image data at the second resolution into the first region of the first memory, the third processor generates the first image data at the third resolution. This saves the second processor's computing power, allowing it to use more resources when rendering the image, improving rendering smoothness and resolving stuttering issues. Furthermore, the third resolution is greater than the second resolution, indicating that the image data obtained by the third processor is relatively high-resolution. Subsequent rendering of the first image by the second processor can be based on this higher-resolution image data, improving the image quality of the first image.

[0042] Optionally, after the second processor writes the image data of the first image at the second resolution to the first region of the first memory, the method further includes: the second processor writing the supplementary resources of all frame buffers of the first image and the drawing instruction stream of the last frame buffer of the first image to the second memory. The second processor has permission to access the second memory. During the process of the application issuing the rendering command for the first image, the supplementary resources of all frame buffers of the first image and the drawing instruction stream of the last frame buffer of the first image are also issued. This information is used when drawing the first image. Therefore, the second processor writes the supplementary resources of all frame buffers of the first image and the drawing instruction stream of the last frame buffer of the first image to the second memory to ensure that the first image can be drawn accurately. The second processor has permission to access the second memory, while the first processor and the third processor do not, preventing users from modifying the data in the second memory through the first processor and the third processor, thereby improving security.

[0043] Optionally, after the second processor reads the image data of the second image at the third resolution from the second region of the first memory, the second processor draws the second image by: the second processor reading the attached resources of all frame buffers of the second image and the drawing instruction stream of the last frame buffer of the second image from the second memory, the second processor having access to the second memory; and the second processor drawing the second image based on the image data of the second image at the third resolution, the attached resources of all frame buffers of the second image, and the drawing instruction stream of the last frame buffer of the second image.

[0044] Optionally, after the second processor reads the image data of the second image at the third resolution from the second region of the first memory, the method further includes: the second processor writing the image data of the second image at the third resolution into the second memory, the second processor having permission to access the second memory; the second processor drawing the second image includes: the second processor reading the image data of the second image at the third resolution, the supplementary resources of all frame buffers of the second image, and the drawing instruction stream of the last frame buffer of the second image from the second memory; the second processor drawing the second image based on the image data of the second image at the third resolution, the supplementary resources of all frame buffers of the second image, and the drawing instruction stream of the last frame buffer of the second image.

[0045] Thirdly, this application provides an electronic device, which includes: a first processor, a second processor, a third processor, and a memory; wherein the memory is used to store one or more computer program codes, the computer program codes including computer instructions, and when the first processor, the second processor, and the third processor execute the computer instructions, the first processor and the second processor execute the above-described rendering method.

[0046] Fourthly, this application provides a chip system including program code, which, when run on an electronic device, causes a first processor, a second processor, and a third processor in the electronic device to execute the above-described rendering method.

[0047] Fifthly, this application provides a processor, which is a second processor, and the second processor includes a processing unit and a memory; wherein the memory is used to store one or more computer program codes, the computer program codes including computer instructions, and when the second processor executes the computer instructions, the second processor executes the above-described rendering method.

[0048] Sixthly, this application provides a computer storage medium including computer instructions that, when executed on an electronic device, cause a second processor in the electronic device to execute the rendering method described above.

[0049] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0050] Figure 1 A schematic diagram of a rendered image provided in this application;

[0051] Figure 2 A schematic diagram of another rendered image provided for this application;

[0052] Figure 3 A schematic diagram illustrating another rendered image provided in this application;

[0053] Figure 4 A schematic diagram illustrating the determination of the number of drawcalls on different frame buffers provided in this application;

[0054] Figure 5 A schematic diagram illustrating the determination of the number of drawcalls on a different framebuffer provided in this application;

[0055] Figure 6 A schematic diagram illustrating another rendered image provided in this application;

[0056] Figure 7 A schematic diagram of a rendering method provided in this application;

[0057] Figure 8 Signaling diagram of a rendering method provided in this application;

[0058] Figure 9 A schematic diagram of the rendering process of the Nth frame image provided in this application;

[0059] Figure 10 A schematic diagram illustrating the instruction stream backup provided in this application;

[0060] Figure 11 A schematic diagram illustrating a memory access method provided in this application;

[0061] Figure 12-1 and Figure 12-2 Signaling diagram for another rendering method provided in this application;

[0062] Figure 13 A schematic diagram illustrating another memory access method provided in this application;

[0063] Figure 14-1 and Figure 14-2 Signaling diagram for another rendering method provided in this application;

[0064] Figure 15 This is a schematic diagram illustrating the relationship between the four states of the AI ​​thread provided in this application. Detailed Implementation

[0065] The terms "first," "second," and "third," etc., used in this application specification, claims, and drawings are used to distinguish different objects, not to limit a specific order.

[0066] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0067] During user interaction with an electronic device, the device can display frames of images on its screen. Taking a video stream as an example, a video stream can include multiple frames, and the electronic device can display each frame sequentially on the screen, thus displaying the video stream. This image display can be triggered by an application within the electronic device. The application can send rendering commands to the electronic device for different images, and the electronic device responds to these commands to render the images and displays the results.

[0068] In some implementations, each frame corresponds to multiple frame buffers (FBs). Each FB stores the rendering results of a portion of the image elements, such as image data of those elements. This image data is used to draw the corresponding elements. For example, if an image includes elements such as people and trees, the FB can store image data of the people and the trees. The electronic device can then draw based on the image data stored in the FB. In this embodiment, an optional structure of the electronic device and the rendering process based on FBs are described as follows: Figure 1As shown, an electronic device may include a central processing unit (CPU), a graphics processing unit (GPU), internal memory, and a display screen. Internal memory may also be referred to as RAM.

[0069] exist Figure 1 In a computer, after an application (such as a game or video application) installed on an electronic device is launched, it can display images on the screen. During image display, the application issues rendering commands, which the CPU can intercept. The CPU responds to the rendering commands by creating a corresponding FB (Frame Rendering) in memory for the rendering of the i-th frame. Figure 1 The CPU can create three rendering frames (FBs) for the i-th frame image, denoted as FB0, FB1, and FB2. The CPU can issue rendering instructions to the GPU based on rendering commands, and the GPU responds to these instructions by performing the corresponding rendering. In one example, the GPU responds to the rendering instructions and renders FB1 and FB2 of the i-th frame image. After rendering, FB1 and FB2 store the image data of the i-th frame image, respectively. For example, FB1 can store image data of a portion of the i-th frame image (referred to as image data 1), and FB2 can store image data of another portion of the i-th frame image (referred to as image data 2). When displaying the i-th frame image, the GPU merges (or renders) image data 1 and image data 2 into FB0, which stores the complete image data of the i-th frame image. The GPU reads the image data of the i-th frame image from FB0 and draws the i-th frame image on the display screen based on this data.

[0070] Neural-network Processing Units (NPUs) can be introduced into electronic devices. The GPU and NPU respond to rendering instructions based on the resolution of the currently rendered image. For example, when the currently rendered image is a low-resolution image, the CPU triggers the GPU to execute rendering instructions; when the currently rendered image is a high-resolution or ultra-high-resolution image, the NPU is introduced. For example, low resolution can be 540P (960×540) and 720P, high resolution can be 1080P, and ultra-high resolution can be 2k or even greater than 2k.

[0071] like Figure 2As shown, the CPU intercepts rendering commands issued by the application; the CPU responds to the rendering commands and creates corresponding rendering frames (FBs) in memory for image rendering. After obtaining a low-resolution image based on the rendering commands, the CPU sends rendering instructions to the GPU. The GPU responds to the rendering instructions, performs rendering processing on the FBs, and obtains the rendering result. After completing the rendering processing of all FBs, the GPU displays the image on the screen based on the rendering result. Alternatively, after the user adjusts the application's screen resolution to high resolution, the CPU can obtain a high-resolution image based on the rendering commands. The CPU then reduces the image resolution to low resolution and sends rendering instructions to the GPU. The GPU generates low-resolution image data based on the rendering instructions and sends the low-resolution image data to the NPU. The NPU performs super-resolution rendering processing on the low-resolution image data to obtain high-resolution image data, which is then sent to the GPU. After receiving the high-resolution image data, the GPU performs user interface (UI) rendering and post-processing based on the high-resolution image data to complete the drawing of a frame. For example, the GPU performs rendering processing on the FB pointed to by the rendering command based on the high-resolution image data to obtain the rendering result, which may be image data of some elements in the image. After obtaining all the rendering results from the FB (Fast Frame), the GPU displays the drawn image on the screen based on the rendering results.

[0072] Although the NPU bears a certain amount of computational load in the CPU, NPU and GPU framework, reducing the load on the CPU and NPU, the NPU may still have excessively long rendering time during the rendering process. For example, when the NPU is processing high-resolution or ultra-high-resolution images, the large amount of data in the high-resolution or ultra-high-resolution images increases the NPU rendering time, thereby reducing the frame rate and smoothness.

[0073] To address the aforementioned issues, this embodiment provides a rendering method. The application triggers the rendering of the Nth frame image, and the GPU obtains the image data of the (N-1)th frame image. Before receiving the rendering instruction for the Nth frame image, the image data of the (N-1)th frame image has already been stored in the corresponding FB. Thus, the GPU can render the (N-1)th frame image based on the obtained image data of the (N-1)th frame image and display the (N-1)th frame image on the display screen. The NPU can then begin processing the Nth frame image to obtain the image data of the Nth frame image.

[0074] If the application triggers the drawing of the (N+1)th frame image, the GPU can draw the Nth frame image based on the image data of the obtained Nth frame image and display the Nth frame image on the display screen. The NPU then begins processing the (N+1)th frame image. Thus, this application provides a rendering method that delays drawing and calculates the rendering result in parallel while drawing the image.

[0075] Although the GPU renders the Nth frame image under the rendering instruction for the (N+1)th frame image, thus delaying the rendering of the Nth frame image, it allows for earlier rendering compared to waiting for the image data of the Nth frame image, improving smoothness. Furthermore, experiments have demonstrated that the rendering method provided in this application can reduce the impact on frame rate, solving the problem of frame rate reduction. For example, in game applications, when the GPU receives the rendering instruction for the N+2th frame image, it can render the N+1th frame image. The image data of the N+1th frame image has already been written into memory by the NPU in advance, reducing the time the GPU waits for the image data of the N+1th frame image, improving GPU rendering efficiency, and thus increasing the frame rate displayed on the screen, solving the problem of high frame rates in games.

[0076] In one example, before the NPU computes image data, the CPU can first reduce the image resolution. The GPU generates low-resolution image data, and the NPU performs super-resolution rendering on this low-resolution image data to obtain high-resolution image data. The super-resolution rendered image data corresponds to the resolution before reduction. Super-resolution rendering refers to increasing the resolution of the image data. For example, if the image resolution is 1080P or higher, the CPU can reduce the image resolution to 540P, the GPU generates 540P image data, and then the NPU performs super-resolution rendering, resulting in 1080P image data. The extent of resolution reduction depends on the NPU's super-resolution capability. For example, if the NPU's super-resolution capability is 2x, the CPU can reduce the resolution of a 1080P image by 0.5x.

[0077] By reducing the image resolution, the CPU can decrease the amount of data input to the NPU, thereby speeding up the NPU's processing. Furthermore, after the NPU completes the super-resolution rendering, the processing result corresponds to the original resolution, ensuring the NPU doesn't alter the image's resolution and improving rendering accuracy. Simultaneously, reducing the CPU's image resolution reduces rendering time and workload. While the NPU is performing super-resolution rendering on frame N+1, the GPU can render frame N. Since the NPU has already completed super-resolution rendering of frame N by the time frame N is rendered, the GPU's waiting time for the rendering result of frame N is reduced. This allows for parallel operation of the NPU and GPU, increasing the frame rate.

[0078] like Figure 3 As shown, when the application triggers the rendering of the Nth frame image, the CPU reduces the resolution of the Nth frame image and instructs the GPU to generate low-resolution image data. The GPU generates low-resolution image data and sends it to the NPU. The NPU performs super-resolution rendering on the low-resolution image data to obtain high-resolution image data. During the NPU super-resolution rendering process, the GPU has already read the high-resolution image data of the (N-1)th frame image from memory. Then, the GPU can perform UI rendering and post-processing based on the high-resolution image data of the (N-1)th frame image to complete the rendering of the (N-1)th frame image. The GPU then displays the rendered (N-1)th frame image on the display screen. The high-resolution image data of the (N-1)th frame image can be generated by the NPU through super-resolution rendering.

[0079] After the NPU completes super-resolution rendering, it writes the high-resolution image data of the Nth frame into memory. When the application triggers the rendering of the N+1th frame, the GPU can read the high-resolution image data of the Nth frame from memory and render the Nth frame. The NPU can perform super-resolution rendering on the low-resolution image data of the N+1th frame.

[0080] In one example, the CPU can specify the FB (Frame Attachment) for super-resolution rendering. The FB used for super-resolution rendering can vary depending on the rendering method. Rendering methods include deferred rendering and forward rendering. In deferred rendering, color attachment records not only color data but also the normal vector data and depth data for each pixel. The upsampled normal vector data has a high probability of error during lighting calculations, and there is a high correlation between all FBs except FB0 (e.g., the rendering result of one FB is bound to the next FB). If super-resolution rendering is performed on one FB, the probability of errors when other FBs use the rendering result of that FB is high. Therefore, in deferred rendering, super-resolution rendering can be performed on all FBs except FB0 corresponding to each frame of the image. That is, in deferred rendering, the FBs used for super-resolution rendering are all FBs except FB0 corresponding to a frame of the image. For example, if all FBs corresponding to a frame of the image include FB0, FB1, and FB2, the FBs used for super-resolution rendering include FB1 and FB2. FB0 is the last FB among all FBs issued by the application.

[0081] In forward rendering, the correlation between rendering blocks (FBs) is relatively small, so at least one FB can be super-resolution rendered. In forward rendering, the CPU can determine the FB to be super-resolution rendered based on the number of rendering operations. In one example, the CPU can determine the FB with more than a preset number of rendering operations as the FB to be super-resolution rendered; in another example, the CPU can determine the FB with the most rendering operations as the FB to be super-resolution rendered. The FB with the most rendering operations can be the one with the most draw calls (drawing-related instructions) executed by the GPU. This is because the more draw calls executed by the GPU on that FB, the more rendering operations the GPU performs on that FB, resulting in more computing resources being consumed by the CPU when rendering that FB, as well as higher power consumption and heat generation. Therefore, in this embodiment, the FB with the most rendering operations can be determined as the FB to be super-resolution rendered. The FB with the most rendering operations is the FB with the most draw calls executed. For ease of explanation, the FB that performs super-resolution rendering in forward rendering is called the main FB. The following explanation uses examples to illustrate how to determine the FB with the most draw calls.

[0082] In this embodiment, the CPU can receive rendering commands from the application and, based on these commands, issue corresponding rendering instructions to one of the GPU, NPU, and DSP, so that one of the GPU, NPU, and DSP can execute the corresponding rendering according to the rendering instructions. As an example, the rendering command may include a glBindFrameBuffer() function, and one or more glDrawElement, glDrawArray, glDrawElementInstanced, and glDrawArrayInstanced. Correspondingly, the rendering instructions may also include a glBindFrameBuffer() function, and one or more glDrawElement, glDrawArray, glDrawElementInstanced, and glDrawArrayInstanced. The glBindFrameBuffer() function can be used to indicate the currently bound frame buffer, thus binding the corresponding rendering operation to the frame buffer. For example, glBindFrameBuffer(1) can indicate that the currently bound frame buffer is FB1, and the Drawcalls executed by the GPU on FB1 include: glDrawElement, glDrawArray, glDrawElementInstanced, and glDrawArrayInstanced.

[0083] For example, the Nth frame of the image can use FB0, FB1, and FB2 as frame buffers. Combined with... Figure 4 This is an example of a rendering command corresponding to FB1. The rendering command issued by the application may include the glBindFrameBuffer(1) function, thereby binding the current rendering operation to FB1. After binding FB1, the rendering operation on that FB1 can be indicated through the glDrawElement instruction. In this example, one glDrawElement instruction can correspond to one Drawcall. In different implementations, there can be multiple glDrawElement instructions executed on FB1, and thus multiple Drawcalls executed on FB1.

[0084] The CPU can initialize counter 1 when executing glBindFrameBuffer(1) according to the rendering command issued by the application. For example, if a corresponding counting frame bit is configured for FB1 in memory, the value of the frame bit can be initialized to 0 by initializing counter 1. Subsequently, each time glDrawElement is executed on FB1, counter 1 is incremented by 1, such as by executing count1++. For example, after executing glDrawElement1-1, the CPU can execute count1++ on counter 1, thereby changing the value of the frame bit storing the number of FB1Drawcalls from 0 to 1, at which point the number of Drawcalls executed on FB1 is 1. By analogy, the CPU can determine that during the rendering of the Nth frame image, the number of Drawcalls executed on FB1 is the current count of counter 1 (for example, this count can be A).

[0085] Similarly, for FB2, the GPU can bind FB2 using the glBindFrameBuffer(2) function. Afterwards, the GPU can use instructions such as glDrawElement, glDrawArray, glDrawElementInstanced, and glDrawArrayInstanced to implement rendering operations on FB2. Similar to FB1, the CPU can also initialize counter 2 when calling glBindFrameBuffer(2) based on the rendering command issued by the application. For example, initialize count2=0. Subsequently, each time glDrawElement is executed on FB2, counter 2 is incremented by 1, such as executing count2++. After completing the rendering process of the image on FB2, the CPU can determine that during the rendering of the Nth frame, the number of Drawcalls executed on FB2 is the current count of counter 2 (for example, this count can be B). See reference... Figure 5After completing the rendering of FB1 and FB2, the number of frames in memory storing the number of FB1 draw calls can be A, and the number of frames storing the number of FB2 draw calls can be B.

[0086] In this example, the CPU can select the FB corresponding to the larger count between A and B as the primary FB. For instance, if A is greater than B, the CPU can determine that FB1 executes more draw calls and therefore designates FB1 as the primary FB. Conversely, if A is less than B, the CPU can determine that FB2 executes more draw calls and therefore designates FB2 as the primary FB.

[0087] In other examples, the CPU can determine the primary FB (Executable File Unit) based on the number of Drawcalls executed that exceeds a preset threshold. The CPU can also select the FB with the number of Drawcalls executed that exceeds the preset threshold by counting, which will not be described in detail here.

[0088] Correspondingly, the rendering process combined with the rendering method is as follows: Figure 6 As shown, Figure 6 The application instructs the CPU to render the Nth frame. Upon receiving the rendering command, the CPU determines whether the rendering method is forward rendering or deferred rendering. If the rendering method is forward rendering, the CPU determines whether the current frame (FB) performing the rendering operation is FB1. If the current FB performing the rendering operation is FB1, the CPU reduces the resolution of the Nth frame and instructs the GPU to generate low-resolution image data. The GPU sends the low-resolution image data to the NPU, which performs super-resolution rendering. During the NPU's super-resolution rendering process, the GPU has already read the high-resolution image data of the (N-1)th frame from memory. Then, the GPU performs UI rendering and post-processing based on the high-resolution image data of the (N-1)th frame to complete the drawing of the (N-1)th frame. The GPU then displays the drawn (N-1)th frame on the display screen.

[0089] After the NPU completes super-resolution rendering, it writes the high-resolution image data of the Nth frame into memory. When the application triggers the rendering of the (N+1)th frame, the GPU can read the high-resolution image data of the Nth frame from memory to render the Nth frame. The NPU can also perform super-resolution rendering on the low-resolution image data of the (N+1)th frame.

[0090] If the CPU determines that the rendering method is deferred rendering, the CPU determines whether the FB currently performing the rendering operation is FB1 or FB2; if the FB currently performing the rendering operation is one of FB1 and FB2, the GPU renders and draws the N-1th frame image while the NPU performs super-resolution rendering on the image data of the Nth frame at low resolution.

[0091] In this embodiment, the CPU can create two threads: a rendering thread and an Artificial Intelligence (AI) thread. The AI ​​thread can run on the NPU and is used to calculate image data, such as performing super-resolution rendering. The rendering thread is used to detect rendering commands issued by the application, perform UI rendering and post-processing using image data, and display images on the screen.

[0092] The process of lazy rendering and parallel computation of rendering results between the rendering thread and the AI ​​thread while rendering the image is as follows: Figure 7 As shown, after the application starts, the rendering thread intercepts the instruction stream to capture rendering commands. Responding to these commands, the rendering thread reduces the resolution of the currently rendered image. The rendering thread then triggers the AI ​​thread to begin calculations on the currently rendered image, determining its image data. Furthermore, the rendering thread can read the image data from the previous frame, which is calculated by the AI ​​thread. The rendering thread then performs UI rendering and post-processing operations to draw and display the previous frame on the screen. The process from detecting application startup to reducing the resolution of the currently rendered image is executed by the CPU, while the process from reading the rendering result of the previous frame to drawing the previous frame is executed by the GPU.

[0093] Taking the current rendered image as frame N (or simply Frame N) as an example, the rendering thread reads the high-resolution image data of the previous frame (Frame N-1) calculated by the AI ​​thread from memory. The rendering thread then sends the low-resolution image data of Frame N to the AI ​​thread and starts the AI ​​thread for calculation. The rendering thread can also cache the OpenGL / Vulkan instruction stream of Frame N and all the auxiliary resources of Frame N's function blocks (FBs). The OpenGL / Vulkan instruction stream of Frame N includes the drawing instruction stream of FB0. If the rendering command issued by the application is an OpenGL command, the rendering thread caches the OpenGL instruction stream of Frame N; if the rendering command issued by the application is a Vulkan command, the rendering thread caches the Vulkan instruction stream of Frame N.

[0094] Subsequently, the rendering thread reads the drawing instruction stream of Frame N-1 and the attached resources of all FBs of Frame N-1 from the cache. Based on the high-resolution image data of Frame N-1, the drawing instruction stream of Frame N-1, and the attached resources of all FBs of Frame N-1, it completes the on-screen display, enabling the rendering thread to draw Frame N-1. Thus, the rendering thread renders Frame N-1 under the rendering command of Frame N, with a one-frame delay before on-screen display. In this way, the rendering thread and the AI ​​thread achieve delayed drawing and parallel computation while drawing the image. The AI ​​thread can also perform super-resolution rendering to speed up the processing speed.

[0095] Figure 7 The rendering method shown is as follows: Figure 8 As shown, in Figure 8 In the rendering method shown, the AI ​​thread runs in the NPU. The AI ​​thread can use the AI ​​super-resolution model to complete super-resolution rendering. Figure 8 Taking the example of an application issuing a rendering command for the Nth frame, where the previous frame is the (N-1)th frame, as an example, where... Figure 8 The rendering method shown may include the following steps:

[0096] S101. The rendering thread intercepts rendering commands issued by the application, responds to the rendering commands, and determines the main rendering flow (FB). The main FB is the FB that performs the most rendering operations during the rendering process, and the FB that performs the most rendering operations can be the FB that executes the most draw calls.

[0097] One frame of image corresponds to multiple drawing frames (FBs), and each FB executes a different number of draw calls. In this embodiment, all FBs in a frame of image can be counted to obtain the number of draw calls executed by all FBs in that frame of image. Then, the FB with the most draw calls executed in that frame of image is determined as the FB for super-resolution rendering (denoted as the main FB). The main FB of other images after that frame of image can be the same as the main FB of that frame of image. For example, taking the currently rendered frame of image N as an example, the CPU can determine the main FB of image N-1 based on the number of draw calls executed on different FBs in the previous frame (such as frame N-1) during the rendering process of the previous frame (such as frame N-1). The main FB of image N-1 is then determined as the main FB of image N, frame N+1, frame N+2, etc.

[0098] In this embodiment, the rendering thread can determine the FB (Frame Builder) to be used for super-resolution rendering before intercepting the rendering commands issued by the application. For example, the rendering thread can determine the main FB based on the number of draw calls executed on different FBs in the (N-1)th frame during the rendering process of the (N-1)th frame image.

[0099] S102. The rendering thread creates an AI thread. During the initialization process of creating the AI ​​thread, the AI ​​super-resolution model can be initialized to ensure that the AI ​​super-resolution model can run normally.

[0100] The initialization of an AI super-resolution model includes model loading, model compilation, and memory configuration. These steps are to ensure that the AI ​​super-resolution model can run normally.

[0101] Model loading involves converting the AI ​​super-resolution model into a model file that can be recognized by the AI ​​thread. During the initialization phase, the model file is loaded into memory. Model compilation is used to verify that the model file can run successfully. Memory configuration involves allocating memory for the AI ​​super-resolution model. The allocated memory is used to store the input and output data of the AI ​​super-resolution model. In this embodiment, the memory allocated to the AI ​​super-resolution model can be CPU memory, memory managed by the Neural Processing API (such as the SNPE ITensor buffer), or shared memory.

[0102] CPU memory refers to the memory allocated to the CPU. Data used during CPU operation is written to CPU memory. The AI ​​super-resolution model occupies a portion of CPU memory as its own memory. Data interaction between the rendering thread and the AI ​​thread requires CPU memory. When the rendering thread draws images, it needs to write data from CPU memory to GPU memory (memory allocated to the GPU) and then read data from GPU memory for rendering. Shared memory refers to memory shared by the rendering thread and the AI ​​thread. The rendering thread and the AI ​​thread can directly read data from shared memory.

[0103] AI threads can be created when the rendering command for the Nth frame is intercepted, when the resolution of the Nth frame is determined to be high, after application startup is detected, or when the currently processed frame (FB) is a FB performing super-resolution rendering. For applications, the CPU can also set a whitelist and determine whether the application is on the whitelist after startup; if so, an AI thread is created. After completing AI thread creation and initialization, the AI ​​thread enters a sleep state, waiting to be awakened by the rendering thread.

[0104] S103. The rendering thread obtains the initial resolution of the Nth frame image as high resolution and the currently processed FB based on the rendering command.

[0105] The rendering thread can obtain the width and height of the Nth frame image based on rendering commands. The width and height of an image are fixed at a given resolution. For example, at 720P resolution, the image width is 1280 and the height is 720; at 1080P resolution, the image width is 1920 and the height is 1080. Therefore, the initial resolution of the Nth frame image can be determined using its width and height. Generally, images with a resolution of 1920×1080 or higher are considered high resolution. The rendering thread can determine that the initial resolution of the image is high resolution if its width and height meet the requirement of 1920×1080 or higher. Based on rendering commands, the rendering thread can obtain the currently processed frame identifier, such as FB1. The initial resolution of the Nth frame image and the currently processed frame identifier can be obtained from different rendering commands.

[0106] S104. The rendering thread obtains the rendering method from the application's configuration file.

[0107] Rendering methods include deferred rendering and forward rendering. The frame elements (FBs) used for super-resolution rendering differ depending on the rendering method. In deferred rendering, super-resolution rendering can be performed on all FBs in each frame except FB0. In other words, in deferred rendering, all FBs in each frame except FB0 can be super-resolution rendered. The CPU performs super-resolution rendering on the FB for which the rendering command is currently being executed if it determines that the FB is not FB0.

[0108] In forward rendering, the GPU's computing power is concentrated in the rendering of the main fragment shader (FB). Within the main FB, the GPU's computing power is concentrated on the calculations of the fragment shader (FS). Therefore, it can be inferred that the main FB in forward rendering is the FB that executes the most draw calls for super-resolution rendering. In this embodiment, if the CPU determines FB1 as the main FB during the rendering of frame N-1, then FB1 is currently being processed, and super-resolution rendering is performed on FB1.

[0109] S105. If the rendering method is forward rendering and the currently processed FB is the primary FB, the rendering thread will reduce the resolution of the Nth frame image from high resolution to low resolution; if the rendering method is deferred rendering and the currently processed FB is not FB0, the rendering thread will reduce the resolution of the Nth frame image from high resolution to low resolution.

[0110] The rendering thread can reduce the resolution of the Nth frame by reducing its width and height. The rendering thread can determine the super-resolution factor of the AI ​​super-resolution model when reducing the resolution of the Nth frame. This AI super-resolution model runs within the AI ​​thread and can convert image data from low resolution to high resolution, where high resolution is greater than low resolution. For example, high resolution can be one of 2x, 3x, or 4x that of low resolution; that is, the super-resolution factor of the AI ​​super-resolution model is one of 2x, 3x, or 4x. For example, a 540P image becomes a 1080P image after 2x super-resolution rendering, and a 2K image after 4x super-resolution rendering.

[0111] In this embodiment, the rendering thread can reduce the resolution of the Nth frame image based on the super-resolution factor of the AI ​​super-resolution model. The CPU can reduce the image resolution using a scaling factor, which can be achieved by reducing the width and height of the Nth frame image, such as by representing the image in terms of width and height: The scaling factor r takes the value of That is, the scaling factor r takes a value between 0 and 1, and the image after reducing the resolution is... The scaling factor used is related to the super-resolution factor of the AI ​​super-resolution model.

[0112] For example, if the NPU's super-resolution ratio is 2x, the CPU can reduce the image resolution by a factor of 2; if the NPU's super-resolution ratio is 4x, the CPU can reduce the image resolution by a factor of 4. For instance, if the image resolution is 1080P (1920×1080 pixels wide and high), and the AI ​​super-resolution model has a super-resolution ratio of 2, it can convert from 540P to 1080P. A 540P image has a width and height of 960×540, meaning it's a 2x reduction of the 1080P image, corresponding to a scaling factor of 0.5. If the AI ​​super-resolution model has a super-resolution ratio of 4, it can convert from 270P to 1080P, reducing the 1080P image by a factor of 4, corresponding to a scaling factor of 0.25. The scaling factor refers to the scaling of the image's side length. For example, a scaling factor of 0.5 means that the width and height of the image are scaled by 0.5 at the same time. When the image is scaled by 0.5 at the same time, the image's pixels are scaled by 0.25.

[0113] S106. The rendering thread generates image data for the Nth frame at a low resolution. At this point, the rendering thread runs on the GPU. For example, the rendering thread generates low-resolution RGB image data, which includes R channel data, G channel data, and B channel data.

[0114] S107. The rendering thread stores the Nth frame image data at low resolution into memory, such as in shared memory accessible to both the rendering thread and the AI ​​thread.

[0115] S108. The rendering thread backs up the resource data (excluding image data) and the OpenGL / Vulkan instruction stream required to draw the Nth frame image. The resource data (excluding image data) can include the attachment resources of all FBs of the Nth frame image. The attachment resources of each FB include color resources and depth resources (also known as color, depth, etc. of the Frame Buffer Attachment). The OpenGL / Vulkan instruction stream required to draw the Nth frame image can include the drawing instruction stream of FB0.

[0116] For example, the complete rendering process of the Nth frame image is as follows: Figure 9 As shown, Figure 9 In the Nth frame of the image, the resources of Frame Buffer Attachment are the attachment resources of all FBs in the Nth frame. The attachment resources of each FB include: color resources and depth resources, etc. Figure 9 In this example, Framebuffer N0 represents FB0 of the Nth frame image, and Attach color and depth represent the color and depth resources of each FB. These resources can be cached in the backup area for backup. In addition to backing up the attachment resources of all FBs, the OpenGL / Vulkan instruction streams required to draw the Nth frame image are also backed up, such as the drawing instruction stream of FB0. The backup of the drawing instruction stream and attachment resources can be performed either by the rendering thread storing the low-resolution image data of the Nth frame image in memory, or by the rendering thread generating the low-resolution image data of the Nth frame image; this embodiment does not impose any limitations.

[0117] The reason for backup is that during the process of responding to the application's rendering command for the Nth frame, the rendering thread draws the (N-1)th frame, and during the process of responding to the application's rendering command for the N+1th frame, the rendering thread draws the Nth frame. If the rendering thread does not back up the relevant information of the Nth frame, it will not be able to obtain the above information at the time of rendering the N+1th frame, which will cause the rendering thread to draw an incorrect Nth frame or fail to draw the Nth frame.

[0118] S109. The rendering thread retrieves the image data of the (N-1)th frame at high resolution from memory.

[0119] The high-resolution image data for the (N-1)th frame is obtained by the AI ​​thread. Optionally, it is derived from the AI ​​super-resolution model through super-resolution rendering, calculated in response to the rendering command for the (N-1)th frame, and the high-resolution image data of the (N-1)th frame is written into memory. The initial resolution of the (N-1)th frame is high-resolution; the AI ​​super-resolution model can reconstruct the high-resolution image data of the (N-1)th frame, improving the accuracy of the rendered (N-1)th frame.

[0120] When using an AI super-resolution model to perform super-resolution rendering on low-resolution image data, the input to the AI ​​super-resolution model is the low-resolution image data. The model completes super-resolution rendering through a multi-layered network architecture, resulting in high-resolution image data. The AI ​​super-resolution model takes low-resolution images as input and high-resolution image data as output. It is trained using historical low-resolution images, and during training, high-resolution images rendered by the GPU are used as reference images to validate the AI ​​super-resolution model. Low resolution and high resolution are relative terms; for example, low resolution can be low, and high resolution can be high. High resolution is greater than low resolution. For instance, low resolution is 540P, and high resolution is 1080P; another example is low resolution 720P, and high resolution 2K.

[0121] In this embodiment, the AI ​​super-resolution model can be trained offline using training frameworks such as TensorFlow and PyTorch. However, some electronic devices do not support training frameworks such as TensorFlow and PyTorch. For example, the model formats supported by mobile phones such as the Qualcomm SNPE framework and NNAPI (Android Neural Networks API) are different from those supported by TensorFlow and PyTorch. The Qualcomm SNPE framework supports the DLC format, and NNAPI supports the TensorFlow Lite format. If an AI super-resolution model is trained offline using training frameworks such as TensorFlow and PyTorch, the AI ​​super-resolution model in TensorFlow or PyTorch format needs to be converted to DLC or TensorFlow Lite format for use on mobile phones and other devices.

[0122] After training an AI super-resolution model, it can be updated to adapt to image changes. For example, an electronic device can obtain the super-resolution rendering effect of the AI ​​super-resolution model and adjust its parameters based on this effect. This parameter adjustment can be done offline or online. The device adjusting the parameters can be the electronic device using the AI ​​super-resolution model for super-resolution rendering, or other electronic devices. After adjusting the parameters on other electronic devices, the data is sent to the electronic device using the AI ​​super-resolution model for super-resolution rendering. For example, a computer adjusts the parameters of the AI ​​super-resolution model, and then sends the adjusted data to the mobile phone. During the application rendering process, the electronic device can collect data samples to obtain a data sample set, which is then used to adjust the AI ​​super-resolution model.

[0123] S110. The rendering thread wakes up the AI ​​thread, switching the AI ​​thread from a dormant state to a running state.

[0124] S111, the AI ​​thread reads the image data of the Nth frame at low resolution from memory.

[0125] S112. The AI ​​thread performs super-resolution rendering on the image data of the Nth frame at low resolution to obtain the image data of the Nth frame at high resolution. The image data of the Nth frame at high resolution is written into memory, and then the AI ​​thread switches from running state to sleeping state.

[0126] In this embodiment, the AI ​​thread can utilize an AI super-resolution model to perform super-resolution rendering on the image data of the Nth frame at low resolution. The AI ​​super-resolution model corresponds to at least one super-resolution factor. After receiving the image data of the Nth frame at low resolution, the AI ​​super-resolution model determines the corresponding super-resolution factor based on the low and high resolution of the Nth frame, completing the upscaling conversion of the image data from low resolution to high resolution. The rendering thread then wakes up the AI ​​thread and notifies the AI ​​thread of the low and high resolution of the Nth frame.

[0127] In some examples, the rendering thread obtains the super-resolution factor of the AI ​​super-resolution model when initializing the AI ​​super-resolution model. The rendering thread reduces the resolution of the image based on the super-resolution factor. When the AI ​​thread performs super-resolution rendering, the rendering thread omits sending the low-resolution and high-resolution images of the Nth frame to the AI ​​thread. This example can be applicable to AI super-resolution models with a super-resolution factor.

[0128] In some embodiments, under deferred rendering, when the currently processed frame (FB) is FB1 or FB2, the rendering thread reduces the image resolution. The rendering thread can generate low-resolution image data corresponding to either FB1 or FB2, and then the AI ​​thread performs super-resolution rendering on the low-resolution image data corresponding to the two FBs. In other embodiments, under deferred rendering, the previous FB of FB0 corresponds to the image data of one frame. The rendering thread can generate low-resolution image data of the previous FB of FB0, and the NPU performs super-resolution rendering on the low-resolution image data corresponding to the previous FB of FB0. That is, under deferred rendering, when the FB is any FB other than FB0, the rendering thread can generate low-resolution image data for each FB other than FB0, and then the AI ​​thread performs super-resolution rendering on the low-resolution image. Alternatively, when the FB is the previous FB of FB0, the rendering thread can generate low-resolution image data only for the previous FB of FB0, and then the AI ​​thread performs super-resolution rendering on the low-resolution image data.

[0129] S113. The rendering thread retrieves the OpenGL / Vulkan instruction stream required for drawing the (N-1)th frame image from the previous backup, responds to the OpenGL / Vulkan instruction stream required for drawing the (N-1)th frame image, and obtains the resource data related to the (N-1)th frame image from the backup. Based on the OpenGL / Vulkan instruction stream required for drawing the (N-1)th frame image, the resource data related to the (N-1)th frame image, and the image data of the (N-1)th frame image at high resolution calculated by the rendering thread, UI rendering and post-processing are performed to draw the (N-1)th frame image.

[0130] The OpenGL / Vulkan instruction streams required for the (N-1)th frame and the resource data associated with the (N-1)th frame are backed up in response to the rendering command for the (N-1)th frame, such as when generating a low-resolution image data backup of the (N-1)th frame, or after storing the low-resolution image data of the (N-1)th frame into memory. When the application issues the rendering command for the Nth frame, the backup associated with the (N-1)th frame is retrieved.

[0131] The rendering thread caches and backs up the OpenGL / Vulkan command streams required to draw the Nth frame image and the resource data associated with the Nth frame image. This cache is used when the application issues the rendering command for the (N+1)th frame image, thus delaying the rendering and display of the Nth frame image. The resource data associated with the Nth frame image includes all the attached resources of all frame elements (FBs) of the Nth frame image.

[0132] like Figure 10As shown, during the execution of the instruction stream for the Nth frame image, the OpenGL / Vulkan instruction stream required to draw the Nth frame image and the resource data associated with the Nth frame image are cached and backed up. The OpenGL / Vulkan instruction stream required to draw the (N-1)th frame image and the resource data associated with the (N-1)th frame image are read from the backup area. Responding to this instruction stream, based on the resource data read from the backup area and the high-resolution image data of the (N-1)th frame image read from memory, the (N-1)th frame image is drawn on the display screen. The backup area can be GPU memory, and the memory can be shared memory.

[0133] The above rendering method can be used in various scenarios of image rendering, such as game applications, home design applications, modeling applications, augmented reality applications, and virtual display applications. It uses a rendering thread and an AI thread to render a single frame of an image, achieving deferred rendering and parallel computation of rendering results while the image is being drawn. This ensures that rendering processing does not affect the rendering thread's drawing, improving smoothness. Furthermore, the AI ​​thread can run in the NPU, using the NPU's computing power to offload some of the GPU's computational load, resulting in lower power consumption and shorter rendering times for electronic devices.

[0134] One point to note: During forward rendering, when the currently processed FB (Frame 1) is the primary FB, the AI ​​thread performs super-resolution rendering. In this case, the low-resolution image data of frame N corresponds to the primary FB, and the rendering result of the primary FB can be obtained based on the high-resolution image data generated by the AI ​​thread. When other FBs are processed during forward rendering, the AI ​​thread will not trigger super-resolution rendering. Instead, the rendering thread processes the other FBs, obtaining their rendering results. If the rendering results of other FBs are not backed up, then when the rendering thread draws frame N-1, it uses the rendering result of the primary FB in frame N-1 and the rendering results of the other FBs in frame N.

[0135] Under deferred rendering, all FBs except FB0 can trigger the AI ​​thread to perform super-resolution rendering. The AI ​​thread can perform super-resolution rendering on all FBs except FB0. Under deferred rendering, the rendering results of all FBs except FB0 can be obtained based on the high-resolution image data generated by the AI ​​thread. In this way, the rendering thread can read the complete high-resolution image data of the N-1 frame image required for drawing the N-1 frame image from memory. When drawing the N-1 frame image, the rendering thread uses the rendering results of the FBs of the N-1 frame image.

[0136] In this embodiment, the interaction of image data between the rendering thread and the AI ​​thread occurs after the image resolution is reduced. After the image resolution is reduced, the process of reading the rendering result of the previous frame and drawing the previous frame is executed by the GPU. The AI ​​thread can be executed by the NPU. The interaction of image data between the GPU and the NPU can be accomplished with the help of GPU memory and CPU memory. GPU memory can be the private memory of the GPU, and the CPU and NPU are prohibited from accessing GPU memory. However, the GPU can read image data from GPU memory by calling functions, etc. The image data read by the GPU can be written to CPU memory or used by the GPU when drawing images. Therefore, GPU memory can be used as the GPU's interaction memory and computing memory.

[0137] The CPU can designate a region of CPU memory as NPU input memory and another region of CPU memory as NPU output memory. NPU input memory contains the input data written to the NPU for super-resolution rendering, while NPU output memory contains the output data written to the NPU after super-resolution rendering is complete. The CPU can specify the size of the NPU input memory based on the amount of input data and the size of the NPU output memory based on the amount of output data. Both the GPU and NPU can access CPU memory. The corresponding memory access process is as follows: Figure 11 As shown, the following steps may be included:

[0138] S200: The CPU uses one storage area in the CPU memory as the NPU input memory and another storage area in the CPU memory as the NPU output memory. The storage area is used as the NPU input memory when inputting data into the NPU and as the NPU output memory when outputting data into the NPU. The size of the two storage areas can be determined according to the amount of data processed by the NPU, which will not be elaborated here.

[0139] S201. The CPU sends the pointer address of the NPU input memory to the NPU, and sends the pointer address of the NPU output memory to the NPU.

[0140] S202, The GPU writes the image data of the Nth frame at low resolution into the GPU memory.

[0141] S203, The GPU notifies the CPU that the image data of the Nth frame at low resolution has been written to the GPU memory.

[0142] S204. The CPU instructs the GPU to write the low-resolution image data of frame N to the NPU input memory and to read the high-resolution image data of frame N-1 from the NPU output memory. During the notification process, the pointer addresses of the NPU input memory and the NPU output memory are sent to the GPU.

[0143] S205, The GPU reads the image data of the Nth frame at low resolution from the GPU memory.

[0144] S206: The GPU writes the image data of the Nth frame at low resolution to the NPU input memory based on the pointer address of the NPU input memory.

[0145] S207: The GPU reads the image data of the (N-1)th frame at high resolution from the NPU output memory based on the pointer address of the NPU output memory.

[0146] S208, the GPU writes the image data of the (N-1)th frame at high resolution into the GPU memory.

[0147] S209. The CPU wakes up the AI ​​thread and notifies the NPU to read the low-resolution image data of the Nth frame from the NPU input memory. After the AI ​​thread is woken up, it can call the AI ​​super-resolution model to perform super-resolution rendering on the low-resolution image data of the Nth frame. Before calling the AI ​​super-resolution model, the AI ​​thread must first obtain the low-resolution image data of the Nth frame. Therefore, one function of the CPU waking up the AI ​​thread is to notify the NPU to read the low-resolution image data of the Nth frame from the NPU input memory, and then the NPU reads the low-resolution image data of the Nth frame and provides it to the AI ​​super-resolution model.

[0148] S210 and NPU read the image data of the Nth frame at low resolution from the NPU input memory based on the pointer address of the NPU input memory.

[0149] S211: The NPU writes the high-resolution image data of the Nth frame to the NPU output memory based on the pointer address of the NPU output memory. When the GPU is drawing the Nth frame, the GPU reads the high-resolution image data of the Nth frame from the NPU output memory and writes it to the GPU memory.

[0150] After reading the low-resolution image data of the Nth frame, the NPU inputs this data into the AI ​​super-resolution model running within the NPU. The AI ​​super-resolution model performs super-resolution rendering on the low-resolution image data of the Nth frame, outputting the high-resolution image data of the Nth frame. The NPU then writes the high-resolution image data of the Nth frame into its output memory based on the pointer address of the NPU output memory.

[0151] Combination Figure 11 The memory access process shown is as follows: Figure 8 The rendering method obtains Figure 12-1 and Figure 12-2 Another flow of the rendering method shown is as follows: Figure 12-1 and Figure 12-2 The diagram illustrates the processing flow of the GPU, NPU, and CPU after the application issues rendering commands for the (N-1)th and Nth frames, following the determination that FB1 is the primary FB in the (N-2)th frame. The GPU and CPU run rendering threads, while the NPU runs an AI thread. This process may include the following steps:

[0152] S301, During the rendering of the N-2th frame of the image, the CPU determines FB1 as the main FB.

[0153] S302, The CPU intercepts the rendering command for the (N-1)th frame image issued by the application.

[0154] S303, CPU creates AI thread. During the AI ​​thread creation and initialization process, the AI ​​super-resolution model can be initialized to ensure that the AI ​​super-resolution model can run normally.

[0155] S304, the CPU designates the first region in the CPU memory as NPU input memory and the second region as NPU output memory.

[0156] S305, the CPU notifies the NPU of the pointer addresses of the first and second regions.

[0157] S306: The CPU obtains the initial resolution of the (N-1)th frame image as high resolution and the current processing frame (FB) based on the rendering command.

[0158] S307: The CPU obtains the rendering method from the application's configuration file.

[0159] S308. If the rendering method is forward rendering and FB is FB1, the CPU reduces the initial resolution of the image to a lower resolution based on the super-resolution factor; if the rendering method is deferred rendering and FB is not FB0 (such as FB1 and FB2), the CPU reduces the initial resolution of the image to a lower resolution based on the super-resolution factor.

[0160] S309, The CPU instructs the GPU to generate image data for the (N-1)th frame at low resolution.

[0161] S309', The CPU notifies the NPU of the high and low resolution of the N-1th frame of the image.

[0162] The S310 GPU generates image data for the (N-1)th frame at low resolution.

[0163] S311, the GPU writes the image data of the (N-1)th frame at low resolution into the GPU memory.

[0164] S312, GPU notifies CPU that the image data of frame N-1 at low resolution has been written to GPU memory.

[0165] S313, the CPU instructs the GPU to write the low-resolution image data of frame N-1 into the first region and read the high-resolution image data of frame N-2 from the second region. The notification sent by the CPU carries pointer addresses to the first and second regions.

[0166] S314, The GPU reads the image data of the (N-1)th frame at low resolution from the GPU memory.

[0167] S315 and GPU write the image data of the (N-1)th frame at low resolution into the first region based on the pointer address of the first region.

[0168] S316, The GPU backs up the FB0 drawing instruction stream of the N-1th frame image and the ancillary resources of all FBs of the N-1th frame image in GPU memory.

[0169] S317: The GPU reads the image data of the N-2th frame at high resolution from the second region based on the pointer address of the second region.

[0170] S318, GPU writes the image data of the (N-2)th frame at high resolution to GPU memory.

[0171] S319, The GPU reads the image data of the (N-2)th frame at high resolution, the FB0 drawing instruction stream of the (N-2)th frame, and the ancillary resources of all FBs of the (N-2)th frame from the GPU memory.

[0172] The data of the N-2th frame image obtained by the S320 and GPU is empty, and the GPU keeps the content displayed on the screen unchanged.

[0173] S321, The GPU notifies the CPU that it has read the image data of the (N-2)th frame at high resolution.

[0174] S322: The CPU wakes up the AI ​​thread in the NPU and notifies the NPU to read the image data of the (N-1)th frame at low resolution from the first region. The NPU can then use the AI ​​thread to call the AI ​​super-resolution model.

[0175] S323 and NPU read the image data of the (N-1)th frame at low resolution from the first region based on the pointer address of the first region.

[0176] S324' and NPU determine the super-resolution factor of the AI ​​super-resolution model.

[0177] The S324 and NPU utilize an AI super-resolution model to perform super-resolution rendering on the image data of the N-1th frame at low resolution based on the super-resolution factor, thereby obtaining the image data of the N-1th frame at high resolution.

[0178] Based on the pointer address of the second region, the S325 and NPU write the image data of the (N-1)th frame at high resolution into the second region.

[0179] S326. The CPU intercepts the rendering command for the Nth frame image issued by the application.

[0180] S327: The CPU obtains the initial resolution of the Nth frame image as high resolution and the currently processed FB based on the rendering command.

[0181] S328. If the rendering method is forward rendering and FB is FB1, the CPU reduces the initial resolution of the image to a lower resolution based on the super-resolution factor; if the rendering method is deferred rendering and FB is not FB0 (such as FB1 and FB2), the CPU reduces the initial resolution of the image to a lower resolution based on the super-resolution factor.

[0182] S329. The CPU instructs the GPU to generate image data for the Nth frame at a low resolution.

[0183] S329', The CPU notifies the NPU of the high and low resolution of the Nth frame of the image.

[0184] The S330 GPU generates image data for the Nth frame at low resolution.

[0185] S331, The GPU writes the image data of the Nth frame at low resolution into the GPU memory.

[0186] S332, GPU notifies CPU that the image data of frame N at low resolution has been written to GPU memory.

[0187] S333: The CPU instructs the GPU to write the low-resolution image data of frame N to the first region and read the high-resolution image data of frame N-1 from the second region. The notification sent by the CPU carries pointer addresses to the first and second regions.

[0188] S334: The GPU reads the image data of the Nth frame at low resolution from the GPU memory.

[0189] S335 and GPU write the image data of the Nth frame at low resolution into the first region based on the pointer address of the first region.

[0190] S336, The GPU backs up the FB0 drawing instruction stream of the Nth frame image and the ancillary resources of all FBs of the Nth frame image in GPU memory.

[0191] S337 and GPU read the image data of the (N-1)th frame at high resolution from the second region based on the pointer address of the second region.

[0192] S338: The GPU writes the image data of the (N-1)th frame at high resolution to the GPU memory.

[0193] S339, The GPU reads the image data of the (N-1)th frame at high resolution, the FB0 drawing instruction stream of the (N-1)th frame, and the ancillary resources of all FBs of the (N-1)th frame from the GPU memory.

[0194] The S340 and GPU render the N-1th frame image based on the high-resolution image data of the N-1th frame image, the FB0 drawing instruction stream of the N-1th frame image, and the ancillary resources of all FBs of the N-1th frame image.

[0195] S341, The GPU notifies the CPU that it has read the image data of the (N-1)th frame at high resolution.

[0196] S342, the CPU wakes up the AI ​​thread in the NPU and notifies the NPU to read the image data of the Nth frame at low resolution from the first region. The NPU can then use the AI ​​thread to call the AI ​​super-resolution model.

[0197] S343 and NPU read the image data of the Nth frame at low resolution from the first region based on the pointer address of the first region.

[0198] S344' and NPU determine the super-resolution factor of the AI ​​super-resolution model.

[0199] S344 and NPU utilize an AI super-resolution model to perform super-resolution rendering on the image data of the Nth frame at low resolution based on the super-resolution factor, thereby obtaining the image data of the Nth frame at high resolution.

[0200] Based on the pointer address of the second region, the S345 and NPU write the image data of the Nth frame at high resolution into the second region.

[0201] From the above Figure 12-1 and Figure 12-2 It is known that during the rendering of the N-2th frame image, the CPU determines FB1 as the main FB. After receiving the rendering command for the N-1th frame image, the CPU, GPU, and NPU implement the rendering method provided in this application. When responding to the rendering command for the N-1th frame image, the N-2th frame image is continuously displayed. When responding to the rendering command for the Nth frame image, the N-1th frame image is displayed. Before responding to the rendering command for the Nth frame image, the image data of the N-1th frame image at high resolution has been calculated, thereby improving the rendering speed while delaying the display of one frame.

[0202] In the above Figure 11 In the memory access process shown, whether it is low-resolution or high-resolution image data, the GPU and NPU can interact with each other using GPU memory and CPU memory. There is a problem of image data being copied between GPU memory and CPU memory. Copying image data between GPU memory and CPU memory can prolong the time for the GPU and NPU to obtain data, thereby increasing the rendering time.

[0203] To address this issue, in this embodiment, the GPU, CPU, and NPU can interact with each other using shared memory, where shared memory refers to the memory accessible to the GPU, CPU, and NPU. Shared memory can serve as an external cache for the GPU, storing at least the low-resolution image data input from the GPU to the NPU, and it can also serve as a processing cache for the NPU, storing at least the high-resolution image data output by the NPU.

[0204] The memory access process between the GPU and NPU using shared memory is as follows: Figure 13 As shown, the following steps may be included:

[0205] S401: The CPU requests shared memory from the hardware buffer. The shared memory consists of a first region and a second region. The first region is used to store low-resolution image data, and the second region is used to store high-resolution image data.

[0206] The CPU can request shared memory from the Hardware Buffer of Random Access Memory (RAM). One method of shared memory allocation is for the CPU to request storage space for a GPU ShaderStorage Buffer Object (SSBO) from the Hardware Buffer. After the GPU's Fragment Shader or Compute Shader obtains low-resolution image data, the low-resolution image data is written to the GPU SSBO. High-resolution image data output by the NPU can also be written to shared memory.

[0207] As a type of memory, the Hardware Buffer has certain requirements regarding its format when accessed by the CPU, GPU, and NPU. For example, when requesting shared memory that is accessible to the Fragment Shader or Compute Shader in the GPU, the Hardware Buffer format can be specified as AHARDWAREBUFFER_FORMAT_BLOB.

[0208] Another method of allocating shared memory is for the CPU to request two blocks of shared memory from the Hardware Buffer. One block stores low-resolution image data obtained by the GPU, serving as input data for the NPU; the other block stores NPU output data, such as high-resolution image data obtained from super-resolution rendering, serving as input data for the GPU during UI rendering. The two shared memory blocks requested by the CPU have the same format, such as AHARDWAREBUFFER_FORMAT_BLOB. The size of the allocated shared memory depends on the image resolution, such as width × height × bytes. For example, if the image data type is float and a float variable is 4 bytes, then the allocated shared memory size would be width × height × 3 × 4, where width and height depend on the image resolution. For instance, for a 1080P image, the allocated shared memory size would be 1920 × 1080 × 3 × 4.

[0209] S402, the CPU notifies the GPU of the pointer addresses of the first and second regions.

[0210] S403, the CPU notifies the NPU of the pointer addresses of the first and second regions.

[0211] S404: Based on the pointer address of the first region, the GPU writes the image data of the Nth frame at low resolution into the first region. After the GPU completes the image data writing, the CPU can synchronously know this, and the GPU does not need to send a notification to the CPU.

[0212] S405 and GPU read the image data of the N-1th frame at low resolution from the second region based on the pointer address of the second region.

[0213] S406, The CPU instructs the NPU to read the image data of the Nth frame at low resolution from the first region.

[0214] S407 and NPU read the image data of the Nth frame at low resolution from the first region based on the pointer address of the first region.

[0215] The S408 and NPU write the high-resolution image data of the Nth frame to the second region based on the pointer address of the second region. The CPU can synchronously detect the completion of the image data writing by the NPU, and the NPU may not need to send a notification to the CPU.

[0216] If shared memory uses a single region to store both low-resolution and high-resolution image data, then the high-resolution image data is read first, followed by the low-resolution image data. If the CPU requests two shared memory blocks from the Hardware Buffer, one block stores the low-resolution image data obtained by the GPU, serving as input data for the NPU; the other block stores the NPU's output data, such as high-resolution image data obtained from super-resolution rendering, serving as input data for the GPU during UI rendering. The CPU can send pointers to these two shared memory blocks to the GPU and NPU, enabling them to read and write image data within the shared memory. The GPU and NPU use the above... Figure 13 The memory access process shown can utilize shared memory to achieve data sharing, enabling zero-copy data sharing between the GPU and NPU, thus improving processing efficiency.

[0217] Combination Figure 13 The memory access process shown is as follows: Figure 8 The rendering method shown yields... Figure 14-1 and Figure 14-2 Another flow of the rendering method shown is as follows: Figure 14-1 and Figure 14-2 This illustrates the processing flow of the GPU, NPU, and CPU after the application issues rendering commands for the (N-1)th and Nth frames, following the determination that FB1 is the primary FB in the (N-2)th frame. The GPU and CPU run rendering threads, while the NPU runs an AI thread. This process is consistent with the above. Figure 12-1 and Figure 12-2 The difference lies in that image data is stored in shared memory, while the FB0 drawing instruction stream and auxiliary resources are backed up in GPU memory, which may include the following steps:

[0218] S501, during the rendering of the N-2th frame of the image, the CPU determines FB1 as the primary FB.

[0219] S502, CPU intercepts the rendering command for the (N-1)th frame image issued by the application.

[0220] S503 and CPU create AI threads. During the initialization process of AI thread creation, the AI ​​super-resolution model can be initialized to ensure that the AI ​​super-resolution model can run normally.

[0221] S504, CPU requests shared memory, which includes a first region and a second region. The first region is used as storage space for low-resolution image data, and the second region is used as storage space for high-resolution image data.

[0222] S505, CPU informs NPU and GPU of the pointer addresses of the first and second regions.

[0223] S506, CPU obtains the initial resolution of the N-1th frame image as high resolution and the current processing frame (FB) based on the rendering command.

[0224] S507: The CPU obtains the rendering method from the application's configuration file.

[0225] S508. If the rendering method is forward rendering and FB is FB1, the CPU reduces the initial resolution of the image to a lower resolution based on the super-resolution factor; if the rendering method is deferred rendering and FB is not FB0 (such as FB1 and FB2), the CPU reduces the initial resolution of the image to a lower resolution based on the super-resolution factor.

[0226] S509, The CPU instructs the GPU to generate image data for the (N-1)th frame at a low resolution.

[0227] S509', the CPU notifies the NPU of the high and low resolution of the N-1th frame of the image.

[0228] The S510 GPU generates image data for the (N-1)th frame at low resolution.

[0229] S511 and GPU write the image data of the (N-1)th frame at low resolution into the first region based on the pointer address of the first region.

[0230] S512, the GPU backs up the FB0 drawing instruction stream of the N-1th frame image and all the FB auxiliary resources of the N-1th frame image in GPU memory.

[0231] S513, The CPU instructs the GPU to read the image data of the (N-2)th frame from the second region at high resolution.

[0232] The S514 GPU reads the image data of the (N-2)th frame at high resolution from the second region based on the pointer address of the second region.

[0233] S515, the GPU reads the FB0 drawing instruction stream of the N-2th frame image and the ancillary resources of all FBs of the N-2th frame image from the GPU memory.

[0234] S516, the data of the N-2th frame image obtained by the GPU is empty, and the GPU keeps the content displayed on the screen unchanged.

[0235] S517, GPU notifies CPU that it has read the image data of frame N-2 at high resolution.

[0236] S518: The CPU wakes up the AI ​​thread in the NPU and instructs the NPU to read the low-resolution image data of the (N-1)th frame from shared memory. The NPU can then use the AI ​​thread to call the AI ​​super-resolution model.

[0237] S519 and NPU read the image data of the (N-1)th frame at low resolution from the first region based on the pointer address of the first region.

[0238] The S520' and NPU determine the super-resolution factor of the AI ​​super-resolution model.

[0239] The S520 and NPU utilize an AI super-resolution model to perform super-resolution rendering on the image data of the N-1th frame at low resolution based on the super-resolution factor, thereby obtaining the image data of the N-1th frame at high resolution.

[0240] Based on the pointer address of the second region, the S521 and NPU write the image data of the (N-1)th frame at high resolution into the second region.

[0241] S522, CPU intercepts the rendering command for the Nth frame image issued by the application.

[0242] S523: The CPU obtains the initial resolution of the Nth frame image as high resolution and the currently processed FB based on the rendering command.

[0243] S524. If the rendering method is forward rendering and FB is FB1, the CPU reduces the initial resolution of the image to a lower resolution based on the super-resolution factor; if the rendering method is deferred rendering and FB is not FB0 (such as FB1 and FB2), the CPU reduces the initial resolution of the image to a lower resolution based on the super-resolution factor.

[0244] S525, the CPU instructs the GPU to generate image data for the Nth frame at a low resolution.

[0245] S525', the CPU notifies the NPU of the high and low resolution of the Nth frame of the image.

[0246] S526, GPU generates image data of the Nth frame at low resolution.

[0247] S527: The GPU writes the image data of the Nth frame at low resolution into the first region based on the pointer address of the first region.

[0248] S528, The GPU backs up the FB0 drawing instruction stream of the Nth frame image and the ancillary resources of all FBs of the Nth frame image in GPU memory.

[0249] S529, The CPU instructs the GPU to read the image data of the (N-1)th frame from the second region at high resolution.

[0250] The S530 GPU reads the image data of the (N-1)th frame at high resolution from the second region based on the pointer address of the second region.

[0251] S531, the GPU reads the FB0 drawing instruction stream of the N-1th frame image and the ancillary resources of all FBs of the N-1th frame image from the GPU memory.

[0252] The S532 GPU renders the N-1 frame image based on the high-resolution image data of the N-1 frame image, the FB0 drawing instruction stream of the N-1 frame image, and the supplementary resources of all FBs of the N-1 frame image. It switches from displaying the N-2 frame image to displaying the N-1 frame image, but the application issues the rendering command for the N frame image.

[0253] S533, GPU notifies CPU that it has read the image data of frame N-1 at high resolution.

[0254] S534: The CPU wakes up the AI ​​thread in the NPU and notifies the NPU to read the image data of the Nth frame at low resolution.

[0255] The S535 and NPU read the image data of the Nth frame at low resolution from the first region based on the pointer address of the first region.

[0256] The S536' and NPU determine the super-resolution factor of the AI ​​super-resolution model.

[0257] The S536 and NPU utilize an AI super-resolution model to perform super-resolution rendering on the image data of the Nth frame at low resolution based on the super-resolution factor, thereby obtaining the image data of the Nth frame at high resolution.

[0258] Based on the pointer address of the second region, the S537 and NPU write the image data of the Nth frame at high resolution into the second region.

[0259] Furthermore, the AI ​​thread provided in this embodiment includes four states: sleep state, running state, running completed state, and running error state. The relationship between these four states is as follows: Figure 15 As shown, when the AI ​​thread is in a sleeping state, the rendering thread can directly wake up the AI ​​thread and put it into running state. While the AI ​​thread is running, the rendering thread waits for it to complete. If the AI ​​thread times out, it is forced to exit, switching to an error state. If the AI ​​thread does not time out, it switches to a completed state. If the AI ​​thread is in an error state, it is forced to exit. After the AI ​​thread is forcibly exited, subsequent images can be rendered using the rendering thread.

[0260] After the AI ​​thread switches to the completed state, the rendering thread can continue to monitor the AI ​​thread. If the duration of the AI ​​thread in the completed state exceeds a preset duration, the AI ​​thread is switched from the completed state to the sleep state. The preset duration indicates that the AI ​​thread will not perform rendering processing for a period of time. An idle but still running AI thread will consume some resources. In this case, switching the AI ​​thread to the sleep state reduces resource consumption. The value of the preset duration is not limited in this embodiment. After the AI ​​thread exits for a certain period or the rendering thread renders x frames of image, the rendering thread can wake up the AI ​​thread again to continue rendering processing. If the AI ​​thread's rendering processing still times out after being woken up again, the rendering thread can disable the AI ​​thread or wait for the next application to start before waking up the AI ​​thread.

[0261] Some embodiments of this application provide an electronic device, which includes a first processor, a second processor, a third processor, and a memory; wherein the memory is used to store one or more computer program codes, the computer program codes including computer instructions, and when the first processor, the second processor, and the third processor execute the computer instructions, the first processor, the second processor, and the third processor execute the above-described rendering method.

[0262] Some embodiments of this application provide a chip system including program code that, when run on an electronic device, causes a first processor, a second processor, and a third processor in the electronic device to execute the above-described rendering method.

[0263] Some embodiments of this application provide a processor, which is a second processor, including a processing unit and a memory; wherein the memory is used to store one or more computer program codes, the computer program codes including computer instructions, and when the second processor executes the computer instructions, the second processor executes the above-described rendering method.

[0264] Some embodiments of this application provide a computer storage medium including computer instructions that, when executed on an electronic device, cause a second processor in the electronic device to perform the rendering method described above.

[0265] This embodiment also provides a control device, which includes one or more processors and a memory. The memory stores one or more computer program codes, which include computer instructions. When the one or more processors execute the computer instructions, the control device performs the above-described method. This control device can be an integrated circuit (IC) or a system-on-a-chip (SoC). The integrated circuit can be a general-purpose integrated circuit, a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC).

[0266] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0267] In the several embodiments provided in this example, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0268] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0269] Furthermore, in each embodiment of this invention, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0270] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0271] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A rendering method, characterized in that, A method for rendering a first image on an electronic device, the electronic device running an application program, the electronic device including a first processor, a second processor, and a third processor, the method comprising: The first processor receives a rendering command for the first image issued by the application. During the process of the third processor rendering the first image data of the first image to obtain the second image data of the first image, the second processor draws the second image based on the second image data of the second image. The second image is the previous frame image of the first image. The second image data of the second image is obtained by the third processor rendering the first image data of the second image. The first image data is image data at a second resolution, and the second image data is image data at a third resolution, which is greater than the second resolution.

2. The method according to claim 1, characterized in that, The electronic device further includes a display screen, and before the first processor receives the rendering command for the first image issued by the application, the method further includes: The first processor determines the first frame buffer during the third image processing process. The first frame buffer is the frame buffer in which the number of drawing instructions executed is greater than a preset threshold among all frame buffers issued by the application. The third image is the previous frame image of the second image. The second processor draws the third image to display the third image on the display screen; The first processor receives a rendering command for the second image issued by the application. The third processor renders the first image data of the second image to obtain the second image data of the second image. During the rendering process of the first image data of the second image by the third processor, the second processor controls the display screen to continue displaying the third image.

3. The method according to claim 1 or 2, characterized in that, The rendering command for the first image is used to instruct the second processor to render the first image based on the first resolution; Before the third processor renders the first image data of the first image to obtain the second image data of the first image, the method further includes: The first processor sends a rendering instruction to the second processor, the rendering instruction being used to instruct the second processor to render the first image; The second processor generates first image data of the first image based on the rendering instructions. The first image data is image data at a second resolution, which is no greater than the first resolution. The second processor writes the first image data of the first image into the first area of ​​the first memory; The third processor reads the first image data of the first image from the first region; The rendering process of the first image data of the first image by the third processor includes: the third processor generating second image data of the first image based on the first image data of the first image, wherein the second image data is image data at a third resolution.

4. The method according to claim 3, characterized in that, After the third processor generates second image data of the first image based on the first image data of the first image, the method further includes: the third processor writing the second image data of the first image into a second area of ​​the first memory.

5. The method according to claim 3, characterized in that, After the second processor writes the first image data of the first image into the first region of the first memory, the method further includes: The second processor writes the supplementary resources of all frame buffers of the first image and the drawing instruction stream of the last frame buffer of the first image into the second memory, and the second processor has permission to access the second memory.

6. The method according to claim 3, characterized in that, After the second processor writes the first image data of the first image to a first region of the first memory, and before the third processor reads the first image data of the first image from the first region, the method further includes: The first processor sends a first notification to the third processor, the first notification being used to instruct the third processor to read the first image data of the first image from the first region.

7. The method according to claim 3, characterized in that, Before the first processor sends rendering instructions to the second processor, the method further includes: The first processor allocates the first memory from the hardware buffer, and the first memory includes the first region and the second region; The first processor sends the pointer address of the first region and the pointer address of the second region to the third processor and the second processor. The first processor, the second processor and the third processor have the right to access the first memory. The third processor and the second processor perform low-resolution image data reading and writing in the first region based on the pointer address of the first region, and perform high-resolution image data reading and writing in the second region based on the pointer address of the second region.

8. The method according to claim 3, characterized in that, Before the second processor draws the second image based on the second image data of the second image, the method further includes: The first processor sends a second notification to the second processor, the second notification being used to instruct the second processor to read the second image data of the second image from the second region.

9. The method according to claim 3, characterized in that, In the process of the third processor rendering the first image data of the first image to obtain the second image data of the first image, the second processor drawing the second image based on the second image data of the second image includes: After the second processor writes the first image data of the first image into the first area of ​​the first memory, the second processor reads the second image data of the second image from the second area; The second processor reads the supplementary resources of all frame buffers of the second image and the drawing instruction stream of the last frame buffer of the second image from the second memory, and the second processor has permission to access the second memory; The second processor draws the second image based on the second image data of the second image, the supplementary resources of all frame buffers of the second image, and the drawing instruction stream of the last frame buffer of the second image.

10. The method according to claim 3, characterized in that, Before the first processor sends rendering instructions to the second processor, the method further includes: The first processor allocates the first memory from the memory of the first processor, the first memory including the first region and the second region; The first processor sends the pointer address of the first region and the pointer address of the second region to the third processor. The first processor and the third processor have the right to access the first memory. The third processor performs low-resolution image data reading in the first region based on the pointer address of the first region, and performs high-resolution image data writing in the second region based on the pointer address of the second region.

11. The method according to claim 10, characterized in that, After the second processor generates the first image data of the first image based on the rendering instructions, and before the second processor writes the first image data of the first image into the first region of the first memory, the method further includes: The second processor writes the first image data of the first image into the second memory, and the second processor has permission to access the second memory; The second processor sends a third notification to the first processor, the third notification indicating that the first image data of the first image has been successfully written to the second memory; In response to receiving the third notification, the first processor sends a fourth notification to the second processor, the fourth notification being used to instruct the second processor to write the first image data of the first image into the first region, the fourth notification carrying an address pointer of the first region; In response to receiving the fourth notification, the second processor reads the first image data of the first image from the second memory.

12. The method according to claim 3, characterized in that, Before the second processor draws the second image based on the second image data of the second image, the method further includes: The first processor sends a fifth notification to the second processor, the fifth notification being used to instruct the second processor to read the second image data of the second image from the second region, the fifth notification carrying an address pointer of the second region; In response to the fifth notification, the second processor reads the second image data of the second image from the second region; The second processor writes the second image data of the second image into the second memory, and the second processor has permission to access the second memory.

13. The method according to claim 3, characterized in that, In the process of the third processor rendering the first image data of the first image to obtain the second image data of the first image, the second processor drawing the second image based on the second image data of the second image includes: The second processor reads the second image data of the second image, the supplementary resources of all frame buffers of the second image, and the drawing instruction stream of the last frame buffer of the second image from the second memory. The second processor has permission to access the second memory. The second processor draws the second image based on the second image data of the second image, the supplementary resources of all frame buffers of the second image, and the drawing instruction stream of the last frame buffer of the second image.

14. The method according to claim 9, characterized in that, After the second processor reads the second image data of the second image, the method further includes: The second processor sends a sixth notification to the first processor, the sixth notification being used to instruct the second processor to complete reading the second image data of the second image; In response to the sixth notification, the first processor wakes up the first thread in the third processor. The first thread can call the artificial intelligence super-resolution model, which is used to perform super-resolution rendering on the first image data of the first image to generate the second image data of the first image.

15. The method according to claim 14, characterized in that, Before the first processor wakes up the first thread in the third processor, the method further includes: The first processor sends the first resolution and the second resolution of the first image to the third processor; The third processor determines the super-resolution factor of the artificial intelligence super-resolution model based on the first resolution and the second resolution of the first image, and the artificial intelligence super-resolution model performs super-resolution rendering on the first image data of the first image based on the super-resolution factor.

16. The method according to claim 14, characterized in that, The method further includes: the first processor initializing the artificial intelligence super-resolution model, the initialization being used to determine the operation of the artificial intelligence super-resolution model and to determine the normal operation of the artificial intelligence super-resolution model; The initialization includes runtime checks, model loading, model compilation, and memory configuration. The runtime checks are used to determine the operation of the AI ​​super-resolution model, and the model loading, model compilation, and memory configuration are used to determine the normal operation of the AI ​​super-resolution model.

17. The method according to claim 10, characterized in that, After the first processor wakes up the first thread in the third processor, the method further includes: during the process of the first thread calling the artificial intelligence super-resolution model to generate image data of the first image at the third resolution, the third processor monitors the operation of the first thread; After the third processor detects that the first thread has ended the call to the artificial intelligence super-resolution model, it puts the first thread to sleep to switch the first thread from the running state to the sleep state. After the artificial intelligence super-resolution model generates the second image data of the first image, the first thread ends the call to the artificial intelligence super-resolution model.

18. The method according to claim 17, characterized in that, After the third processor detects that the first thread has finished calling the artificial intelligence super-resolution model, it puts the first thread into a sleep state, including: After the third processor detects that the first thread has finished calling the artificial intelligence super-resolution model, it controls the first thread to switch from the running state to the running completed state. The third processor detects that the duration of the first thread in the completed state is longer than a preset duration, and puts the first thread into sleep mode to switch the first thread from the completed state to the sleep state.

19. The method according to claim 17, characterized in that, The method further includes: when the third processor detects an error in the first thread's call to the artificial intelligence super-resolution model, the third processor forces the first thread to exit. The third processor sends a notification to the second processor, the notification instructing the second processor to perform rendering processing on the first image.

20. The method according to claim 19, characterized in that, The third processor detects an error in the first thread's call to the AI ​​super-resolution model and forces the first thread to exit, including: The third processor detects that the rendering process of the first image by the artificial intelligence super-resolution model has timed out, and forces the first thread to exit.

21. The method according to claim 19, characterized in that, After the third processor detects that the first thread's call to the artificial intelligence super-resolution model has failed and forces the first thread to exit, the method further includes: after the first processor detects that the exit of the first thread meets a preset condition, it wakes up the first thread again.

22. The method according to claim 3, characterized in that, Before the first processor sends the rendering instruction to the second processor, the method further includes: The first processor reduces the resolution of the first image from the first resolution to the second resolution.

23. The method according to claim 22, characterized in that, The third processor has a super-resolution factor, which is used to indicate the difference between the second resolution and the third resolution; The third resolution is the same as the first resolution; The first processor reduces the resolution of the first image from the first resolution to the second resolution by: the first processor reducing the first resolution to the second resolution based on the super-resolution factor.

24. The method according to claim 3, characterized in that, If the rendering method of the application is forward rendering, the rendering instruction corresponds to the first frame buffer, and the number of drawing instructions executed by the first frame buffer is greater than a preset threshold. If the application uses deferred rendering, the rendering instruction corresponds to all framebuffers except the last one issued by the application.

25. The method according to claim 24, characterized in that, The first frame buffer is the frame buffer that executes the most drawing instructions among all the frame buffers.

26. The method according to claim 24, characterized in that, Before the first processor sends rendering instructions to the second processor, the method further includes: The first processor obtains the rendering method of the application from the application's configuration file.

27. The method according to claim 3, characterized in that, The rendering instruction is used to instruct the second processor to render the first image based on the second resolution, where the second resolution is smaller than the first resolution.

28. The method according to claim 3, characterized in that, The second resolution is smaller than the first resolution; The third resolution is the same as the first resolution, or the third resolution is greater than the first resolution.

29. The method according to claim 3, characterized in that, The second resolution is equal to the first resolution.

30. The method according to claim 1 or 2, characterized in that, The third processor is a neural network processor or a digital signal processor.

31. A rendering method, characterized in that, In a second processor applied to an electronic device, the electronic device further includes a first processor and a third processor, the electronic device running an application program that sends rendering commands for a first image to the first processor; the method includes: During the process of the third processor rendering the first image data of the first image to obtain the second image data of the first image, the second processor draws the second image based on the second image data of the second image. The second image is the previous frame image of the first image. The second image data of the second image is obtained by the third processor rendering the first image data of the second image. The first image data is image data at a second resolution, and the second image data is image data at a third resolution, which is greater than the second resolution.

32. The method according to claim 31, characterized in that, The electronic device further includes a display screen. The first processor determines a first frame buffer during the third image processing. The first frame buffer is the frame buffer in which the number of drawing instructions executed is greater than a preset threshold among all frame buffers issued by the application. The third image is the previous frame image of the second image. The application sends a rendering command for the second image to the first processor. Before the second processor draws the second image, the method further includes: The second processor draws the third image to display the third image on the display screen; During the rendering process of the second image by the third processor, the second processor controls the display screen to continue displaying the third image.

33. The method according to claim 31 or 32, characterized in that, The rendering command for the first image is used to instruct the second processor to render the first image based on a first resolution; before the second processor draws the second image based on the processing result of the rendering processing already completed on the second image, the method further includes: The second processor receives a rendering instruction sent by the first processor, the rendering instruction being used to instruct the second processor to render the first image; The second processor generates first image data of the first image based on the rendering instructions. The first image data is image data at a second resolution, which is no greater than the first resolution. The second processor writes the first image data of the first image at the second resolution into the first area of ​​the first memory; The second processor reads second image data of the second image from a second region of the first memory, the second image data being image data at a third resolution.

34. The method according to claim 33, characterized in that, After the second processor writes the first image data of the first image into the first region of the first memory, the method further includes: The second processor writes the supplementary resources of all frame buffers of the first image and the drawing instruction stream of the last frame buffer of the first image into the second memory, and the second processor has permission to access the second memory.

35. The method according to claim 33, characterized in that, After the second processor reads the second image data of the second image from the second region of the first memory, the second processor draws the second image by: The second processor reads the supplementary resources of all frame buffers of the second image and the drawing instruction stream of the last frame buffer of the second image from the second memory, and the second processor has permission to access the second memory; The second processor draws the second image based on the second image data of the second image, the supplementary resources of all frame buffers of the second image, and the drawing instruction stream of the last frame buffer of the second image.

36. The method according to claim 33, characterized in that, After the second processor reads the second image data of the second image from the second region of the first memory, the method further includes: The second processor writes the second image data of the second image into the second memory, and the second processor has permission to access the second memory; The second processor draws the second image by: the second processor reading the second image data of the second image, the supplementary resources of all frame buffers of the second image, and the drawing instruction stream of the last frame buffer of the second image from the second memory; The second processor draws the second image based on the second image data of the second image, the supplementary resources of all frame buffers of the second image, and the drawing instruction stream of the last frame buffer of the second image.

37. An electronic device, characterized in that, The electronic device includes: a first processor, a second processor, a third processor, and a memory; wherein the memory is used to store one or more computer program codes, the computer program codes including computer instructions, and when the first processor, the second processor, and the third processor execute the computer instructions, the first processor and the second processor execute the rendering method as described in any one of claims 1 to 30.

38. A chip system, characterized in that, The chip system includes program code that, when run on an electronic device, causes a first processor, a second processor, and a third processor in the electronic device to execute the rendering method as described in any one of claims 1 to 30.

39. A processor, characterized in that, The processor is a second processor, which includes a processing unit and a memory; wherein the memory is used to store one or more computer program codes, the computer program codes including computer instructions, and when the second processor executes the computer instructions, the second processor executes the rendering method as described in any one of claims 31 to 36.

40. A computer storage medium, characterized in that, The computer storage medium includes computer instructions that, when executed on an electronic device, cause a second processor in the electronic device to perform the rendering method as described in any one of claims 31 to 36.

Images