Image rendering method and related apparatus

Abstract

Provided are an image rendering method and a related apparatus, which are applied to a terminal device. The method comprises: receiving an m-th set of image data to be rendered that is sent by an application, wherein the image data comprises a plurality of pixel values used for generating an image, m≥1, and m is an integer; performing rendering on the basis of all the pixel values in the m-th set of image data, so as to generate an m-th image frame; receiving an (m+1)-th set of image data to be rendered that is sent by the application; and performing rendering on the basis of static pixels of the m-th image frame and at least some pixel values in the (m+1)-th set of image data, so as to generate an (m+1)-th image frame, wherein the static pixels of the m-th image frame are obtained on the basis of the m-th set of image data. Among a plurality of image frames to be rendered, some pixel values of a previous image frame of an image are reused for rendering and generating at least one image frame, thereby facilitating a reduction in the power consumption of shading calculation of a terminal device, and reducing the occurrence of phenomena such as heating and lagging, and thus the use experience for a user can be improved.

Description

Image rendering method and related device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 29, 2023, with application number 202311102975.0 and application name “A method for rendering an image and related devices”, and claims priority to the Chinese patent application filed with the Patent Office of China on September 8, 2023, with application number 202311163041.8 and application name “A method for rendering an image and related devices”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to an image rendering method and related devices. Background Art

[0003] As screen resolution and refresh rates increase, the demand for shading calculations increases dramatically.

[0004] In current image rendering methods, shading calculations consume high power. In the field of mobile rendering, this is limited by the computing power, bandwidth, and heat dissipation of the mobile terminal. The mobile terminal will produce heating, lag, and other phenomena, which will cause some applications (such as game applications) to limit frequency and frame, thereby affecting the user experience.

[0005] Therefore, how to reduce the power consumption of shading calculations on mobile terminals has become a technical problem that needs to be solved urgently.

[0006] Summary of the Invention

[0007] The present application provides an image rendering method and related devices to reduce the power consumption of shading calculations on mobile terminals.

[0008] In the first aspect, the present application provides an image rendering method, which can be executed by a mobile terminal device, or the method can also be executed by a component configured in the mobile terminal device (such as a chip, a chip system, etc.), or it can also be implemented by a logic module or software that can realize all or part of the functions of the mobile terminal device. The present application does not limit this.

[0009] Exemplarily, the method includes: receiving an mth group of image data to be rendered sent by an application, the mth group of image data including a plurality of pixel values ​​for generating an image, m≥1, and m is an integer; rendering to generate an mth frame of image based on the mth group of image data; wherein, in the case where m mod N=1, the mth frame of image is rendered based on all pixel values ​​in the mth group of image data; and in the case where m mod N≠1, the mth frame of image is rendered based on static pixels of an i-th frame of image and at least part of pixel values ​​in the mth group of image data, and the static pixels of the i-th frame of image are obtained based on the i-th group of image data; wherein N is a preset integer, N≥2,

[0010] Among them, "mod" is the modulus operator, which is used to find the remainder when two numbers are divided; (m / N) rounded up.

[0011] Based on the above scheme, N frames of images are taken as a group, and the first frame of the N frames is obtained based on the currently known rendering process, that is, the first frame of the N frames can be rendered based on all pixel values ​​in the image data of the first frame, and at least one frame of the other images in the N frames except the first frame can be rendered based on the static pixels of a frame before the image in this group of images, and at least part of the pixel values ​​in the image data of the image. In this way, the rendering generation of at least one frame of the image in each group of images reuses part of the pixel values ​​of other images in the group, which helps to reduce the power consumption of the shading calculation on the mobile terminal, reduce the occurrence of heat, freezes and other phenomena, and thus improve the user experience.

[0012] In combination with the first aspect, in some possible implementations, the method further includes: intercepting one or more rendering instructions corresponding to the m-th frame image.

[0013] In combination with the first aspect, in some possible implementations, before rendering to generate the mth frame image, the method also includes: determining the static pixels and dynamic pixels of the mth frame image based on the characteristics of one or more rendering instructions (Drawcall) corresponding to the mth frame image.

[0014] In combination with the first aspect, in some possible implementations, after determining the static pixels and dynamic pixels of the mth frame image, the method further includes: marking the static pixels and the dynamic pixels based on an alpha (A) channel or a stencil buffer; wherein the static pixels and the dynamic pixels both include RGBA channels.

[0015] In conjunction with the first aspect, in some possible implementations, the feature of the rendering instruction includes one or more of the following: a program label, a program uniform name, or a program index buffer name.

[0016] In combination with the first aspect, in some possible implementations, determining the static pixels and dynamic pixels of the m-th frame image based on the characteristics of one or more rendering instructions corresponding to the m-th frame image includes: determining the static objects and dynamic objects included in the m-th frame image based on the characteristics of each rendering instruction in the one or more rendering instructions; determining the static pixels based on the static objects; and determining the dynamic pixels based on the dynamic objects.

[0017] In combination with the first aspect, in some possible implementations, the static pixels and dynamic pixels of the m-th frame image are determined based on the characteristics of one or more rendering instructions corresponding to the m-th frame image, including: determining the static objects and dynamic objects included in the m-th frame image based on the characteristics of each rendering instruction in the one or more rendering instructions to obtain a motion vector set of the m-th frame image; determining the static pixels and the dynamic pixels based on the motion vector set, the value of the motion vector corresponding to the static object is different from the value of the motion vector corresponding to the dynamic object.

[0018] In combination with the first aspect, in some possible implementations, when m mod N≠1, and after determining the static objects included in the m-th frame image, the method further includes: reducing the shading rate of the static objects in the m-th frame image, and rendering the m-th frame image is based on the updated shading rate.

[0019] Reducing the shading rate of static objects can further reduce the power consumption of shading calculations on mobile devices, reduce the occurrence of phenomena such as heating and lag, and thus improve the user experience.

[0020] In combination with the first aspect, in some possible implementations, when m mod N≠1, rendering and generating the mth frame image based on the mth group of image data includes: making correspondence between the i-th frame image and the screen space coordinates of the m-th frame image; when the pixels of the first screen space corresponding to the i-th frame image and the m-th frame image are both static pixels, reusing the pixels of the first screen space of the i-th frame image to render the first screen space of the m-th frame image; or, when the pixels of the second screen space corresponding to the i-th frame image and the m-th frame image are both dynamic pixels, or when the pixels of the second screen space of the i-th frame image and the m-th frame image are different types of pixels, rendering the second screen space of the m-th frame image based on the pixels of the second screen space of the m-th frame image.

[0021] On the second aspect, the present application provides an image rendering method, which can be executed by a mobile terminal device, or the method can also be executed by a component configured in the mobile terminal device (such as a chip, a chip system, etc.), or it can also be implemented by a logic module or software that can realize all or part of the functions of the mobile terminal device. This application does not limit this.

[0022] Exemplarily, the method includes: receiving the mth group of image data to be rendered sent by an application, the image data including multiple pixel values ​​for generating an image, m≥1, and m is an integer; rendering to generate the mth frame image based on all pixel values ​​in the mth group of image data; receiving the m+1th group of image data to be rendered sent by the application; rendering to generate the m+1th frame image based on static pixels of the mth frame image and at least part of the pixel values ​​in the m+1th group of image data, wherein the static pixels of the mth frame image are obtained based on the mth group of image data.

[0023] Based on the above scheme, among the multiple frames of images to be rendered, the rendering generation of at least one frame of image reuses part of the pixel values ​​of the previous frame of image, which helps to reduce the power consumption of the shading calculation of the mobile terminal device, reduce the occurrence of heat, freezes and other phenomena, and thus improve the user experience.

[0024] In combination with the second aspect, in some possible implementations, before rendering to generate the m+1th frame image, the method also includes: determining the static pixels and dynamic pixels of the yth frame image based on the characteristics of one or more rendering instructions corresponding to the yth frame image, y∈[m, m+1].

[0025] In combination with the second aspect, in some possible implementations, the static pixels and dynamic pixels of the yth frame image are determined based on the characteristics of one or more rendering instructions corresponding to the yth frame image, including: determining the static objects and dynamic objects included in the yth frame image based on the characteristics of each rendering instruction in the one or more rendering instructions; determining the static pixels of the yth frame image based on the static objects; and determining the dynamic pixels of the yth frame image based on the dynamic objects.

[0026] In combination with the second aspect, in some possible implementations, after determining the static pixels and dynamic pixels of the yth frame image, the method further includes: marking the static pixels and the dynamic pixels based on the A channel or the template buffer; wherein the static pixels and the dynamic pixels both include RGBA channels.

[0027] In combination with the second aspect, in some possible implementations, the static pixels and dynamic pixels of the yth frame image are determined based on the characteristics of one or more rendering instructions corresponding to the yth frame image, including: determining the static objects and dynamic objects included in the yth frame image based on the characteristics of each rendering instruction in the one or more rendering instructions to obtain a motion vector set of the yth frame image; based on the motion vector set, determining the static pixels and dynamic pixels of the yth frame image, the value of the motion vector corresponding to the static object of the yth frame image is different from the value of the motion vector corresponding to the dynamic object of the yth frame image.

[0028] In combination with the second aspect, in some possible implementations, after determining the static object included in the m+1th frame image, the method also includes: reducing the shading rate of the static object in the m+1th frame image, and rendering the m+1th frame image is based on the updated shading rate.

[0029] Reducing the shading rate of static objects can further reduce the power consumption of shading calculations on mobile devices, reduce the occurrence of phenomena such as heating and lag, and thus improve the user experience.

[0030] In conjunction with the second aspect, in some possible implementations, the characteristics of the rendering instruction include one or more of the following: a program alias, a program variable name, or a program index buffer name.

[0031] In combination with the second aspect, in some possible implementations, the method further includes: intercepting one or more rendering instructions corresponding to the yth frame image.

[0032] In combination with the second aspect, in some possible implementations, rendering and generating the m+1th frame image based on the static pixels of the mth frame image and at least part of the pixel values ​​in the m+1th group of image data includes: making correspondence between the mth frame image and the screen space coordinates of the m+1th frame image; when the pixels of the first screen space corresponding to the m-frame image and the m+1th frame image are both static pixels, reusing the pixels of the first screen space of the m-frame image to render the first screen space of the m-frame image; or, when the pixels of the second screen space corresponding to the m-frame image and the m+1th frame image are both dynamic pixels, or when the pixels of the second screen space of the m-frame image and the m+1th frame image are different types of pixels, rendering the second screen space of the m+1th frame image based on the pixels of the second screen space of the m+1th frame image.

[0033] In combination with the second aspect, in some possible implementations, the method further includes: receiving the m+nth group of image data to be rendered sent by the application, where n>1 and n is an integer; rendering to generate the m+nth frame image based on the static pixels of the m+n-1th frame image and at least part of the pixel values ​​in the m+nth group of image data, where the static pixels of the m+n-1th frame image are obtained based on the m+n-1th group of image data.

[0034] In combination with the first aspect and the second aspect, in some possible implementations, receiving the mth group of image data to be rendered sent by an application includes: receiving the mth group of image data to be rendered from a server of a first application, where the first application is any application among one or more preset applications.

[0035] It is understood that one or more applications may be pre-installed on the terminal device, including, but not limited to, a pre-stored whitelist of applications for the image rendering method provided herein, wherein the application whitelist is not limited to being stored in a list format. The first application may be any application in the application whitelist. When the first application is running on the terminal device, the terminal device may receive the mth set of image data from the server of the first application.

[0036] In a third aspect, the present application provides a terminal device, which can be used to implement the method of the first aspect and any possible implementation of the first aspect, or to implement the method of the second aspect and any possible implementation of the second aspect. The terminal device includes corresponding modules for executing the above methods. The modules included in the terminal device can be implemented in software and / or hardware.

[0037] In a fourth aspect, the present application provides a terminal device comprising at least one processor and at least one communication interface. The processor is coupled to the communication interface and can be configured to execute a program to implement the image rendering method of the first aspect and any possible implementation of the first aspect, or to implement the image rendering method of the second aspect and any possible implementation of the second aspect.

[0038] Optionally, the terminal device further includes a memory, and the processor is coupled to the memory.

[0039] In a fifth aspect, the present application provides a chip system comprising at least one processor for supporting the implementation of the functions involved in the above-mentioned first aspect and any possible implementation method of the first aspect, or for supporting the implementation of the functions involved in the above-mentioned second aspect and any possible implementation method of the second aspect, for example, processing the data involved in the above-mentioned method, etc.

[0040] In one possible design, the chip system also includes a memory, which is used to store program instructions and data. The memory is located inside or outside the processor.

[0041] The chip system can be composed of chips, or can include chips and other discrete devices.

[0042] In the sixth aspect, a readable storage medium is provided, on which a program (also referred to as code, or instruction) is stored. When the program is run, the method in the above-mentioned first aspect and any possible implementation of the first aspect is executed, or the method in the above-mentioned second aspect and any possible implementation of the second aspect is executed.

[0043] In the seventh aspect, a program product is provided, which includes: a program (also referred to as code, or instructions), which, when run, causes the method of the above-mentioned first aspect and any possible implementation of the first aspect to be executed, or causes the method of the above-mentioned second aspect and any possible implementation of the second aspect to be executed.

[0044] It should be understood that the third to seventh aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] FIG1 is a schematic structural diagram of a terminal device applicable to the image rendering method provided in an embodiment of the present application;

[0046] FIG2 is a structural block diagram of software of a terminal device applicable to the image rendering method provided in an embodiment of the present application;

[0047] FIG3 is another structural block diagram of software of a terminal device applicable to the image rendering method provided in an embodiment of the present application;

[0048] FIG4 is a schematic flow chart of an image rendering method provided in an embodiment of the present application;

[0049] FIG5 is a schematic flow chart of triggering the execution of the image rendering method provided by the present application according to an embodiment of the present application;

[0050] FIG6 is a schematic diagram of a static-dynamic separation mask provided in an embodiment of the present application;

[0051] FIG7 is a schematic flow chart of generating a static-dynamic separation mask according to an embodiment of the present application;

[0052] FIG8 is a schematic flow chart of whether to change the coloring rate according to an embodiment of the present application;

[0053] FIG9 is a structural block diagram of the software and hardware of a terminal device applicable to the image rendering method provided in an embodiment of the present application;

[0054] FIG10 is a schematic flow chart of an image rendering method provided in an embodiment of the present application implemented on an operating system;

[0055] FIG11 is another structural block diagram of the software and hardware of a terminal device applicable to the image rendering method provided in an embodiment of the present application;

[0056] FIG12 is another schematic flowchart of the image rendering method provided in an embodiment of the present application;

[0057] FIG13 is a schematic diagram of data involved in an image rendering method provided in an embodiment of the present application;

[0058] FIG14 is another schematic flowchart of the image rendering method provided in an embodiment of the present application;

[0059] FIG15 is another schematic diagram of data involved in an image rendering method provided in an embodiment of the present application;

[0060] FIG16 is a schematic flowchart of another image rendering method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solution in this application will be described below with reference to the accompanying drawings.

[0062] First, in the embodiments of the present application, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a device, system, product or equipment that includes a series of modules, modules or units is not necessarily limited to those modules, modules or units clearly listed, but may include other modules, modules or units that are not clearly listed or inherent to these devices, systems, products or equipment.

[0063] Second, in this application, "when...", "in the case of...", "if" and "if" all mean that the device will take corresponding actions under certain objective circumstances, and do not limit the time. It does not require that the device must perform a judgment action when it is implemented, nor does it mean that there are other limitations.

[0064] The following is a brief description of the terms involved in this application.

[0065] 1. Static objects: objects that move only with changes in viewing angle.

[0066] 2. Dynamic objects: objects that not only move with changes in viewing angle, but also move in their own geometry.

[0067] 3. Static pixels: pixels corresponding to static objects.

[0068] 4. Dynamic pixels: pixels corresponding to dynamic objects.

[0069] 5. RGBA: This is a color space that represents the red (R), green (G), blue (B), and A channels. The RGB channels can be used to determine the actual color displayed by the pixel, and the A channel can be used to determine the transparency of the pixel.

[0070] 6. Screen space: In the rendering field, it can refer to the pixel coordinate space of a two-dimensional (2D) image generated during three-dimensional (3D) rendering. It is the 3D projection result of the geometry in camera space and consists of a two-dimensional array of pixels, each of which has color channels such as RGBA.

[0071] 7. Main scene: In the rendering of game graphics, it can be divided into the main game scene drawing and the game user interface (UI) drawing. The main game scene can include game terrain, vegetation, buildings, characters, special effects, etc.

[0072] 8. Open Graphics Library (OpenGL): Also known as "Open Graphics Library", it is a cross-language, cross-platform application programming interface (API) for rendering 2D and 3D vector graphics.

[0073] 9. Vulkan: is a cross-platform 2D and 3D graphics API.

[0074] As screen resolution and refresh rates increase, the demand for shading calculations increases dramatically.

[0075] In current image rendering methods, shading calculations consume high power. In the field of mobile rendering, this is limited by the computing power, bandwidth, and heat dissipation of the mobile terminal. The mobile terminal will produce heating, lag, and other phenomena, which will cause some applications (such as game applications) to be limited in frequency and frame, thus affecting the user experience.

[0076] In order to solve the above problems, the present application provides an image rendering method and related devices, with N (N≥2) frames of images as a group, and the first frame of the N frames is obtained based on the currently known rendering process, that is, the first frame of the N frames can be rendered based on all pixel values ​​in the image data of the first frame, and at least one frame of the other images in the N frames except the first frame can be rendered based on the static pixels of a frame before the image in this group of images, and at least part of the pixel values ​​in the image data of the image. In this way, the rendering generation of at least one frame of the image in each group of images reuses part of the pixel values ​​of other images in the group, which helps to reduce the power consumption of the shading calculation on the mobile terminal, reduce the occurrence of heat, freezes and other phenomena, and thus improve the user experience.

[0077] Before describing in detail the image rendering method provided in the embodiment of the present application, an exemplary description of a terminal device applicable to the embodiment of the present application is first given in conjunction with Figures 1, 2 and 3.

[0078] The image rendering method provided in the embodiments of the present application can be applied to mobile terminal devices, including but not limited to smartphones, tablet computers, wearable devices, vehicle-mounted devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, virtual reality (VR) devices (such as VR glasses), augmented reality (AR) devices, projectors, etc. The embodiments of the present application do not impose any restrictions on the specific type of device.

[0079] In addition, the methods described in the embodiments of the present application can support operating systems such as the Android operating system (Android OS), Harmony OS, OpenHarmony OS, iOS, Windows OS, and lightweight operating systems (such as LiteOS). The embodiments of the present application do not impose any limitations on this.

[0080] FIG1 is a schematic structural diagram of a terminal device applicable to the image rendering method provided in an embodiment of the present application.

[0081] For example, FIG1 shows a schematic structural diagram of a terminal device 100. As shown in FIG1 , the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L and a bone conduction sensor 180M, etc.

[0082] The processor 110 may include one or more processing units. For example, the processor 110 may include one or more of an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0083] The application processor outputs sound signals through the audio module 170 (such as the speaker 170A, etc.), or displays images or videos through the display screen 194.

[0084] The controller can be the nerve center and command center of the terminal device 100. The controller can generate operation control signals according to instruction operation codes and timing signals to complete the control of instruction fetching and execution.

[0085] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0086] The processor 110 can perform different operations to implement different functions by executing instructions. The instructions can be, for example, instructions pre-stored in the memory before the device leaves the factory, or instructions read from the APP after the user installs a new APP during use. This embodiment of the application does not impose any restrictions on this.

[0087] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface.

[0088] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transmit data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other devices, such as augmented reality (AR) devices, etc. It will be understood that the interface connection relationship between the modules illustrated in this application is only a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0089] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the terminal device 100. While charging the battery 142, the charging management module 140 can also provide power to the terminal device 100 via the power management module 141.

[0090] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0091] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0092] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0093] The mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G applied on the terminal device 100.

[0094] The wireless communication module 160 can provide wireless communication solutions applied to the terminal device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC) technology, infrared (IR) technology, etc.

[0095] In some embodiments, antenna 1 of terminal device 100 is coupled to mobile communication module 150 , and antenna 2 is coupled to wireless communication module 160 , so that terminal device 100 can communicate with the network and other devices through wireless communication technology.

[0096] The terminal device 100 can implement display functions through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0097] Display screen 194, also referred to as a screen, can be used to display images, videos, and the like. It should be understood that display screen 194 may also include additional components, such as a backlight panel and a driver circuit. The backlight panel can be used to provide a light source, and the display panel emits light based on the light provided by the backlight panel. The driver circuit can be used to control the light transmission or opacity of the liquid crystal layer.

[0098] The terminal device 100 can realize the shooting function through the ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0099] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0100] The internal memory 121 can be used to store device executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the terminal device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, etc.

[0101] The terminal device 100 can implement audio functions such as music playback and recording through the audio module 170, such as the speaker 170A, the receiver 170B, the microphone 170C and the headphone jack 170D, and the application processor.

[0102] It should be understood that the structure illustrated in this application does not constitute a specific limitation on the terminal device 100. In other embodiments, the terminal device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0103] The software system of the terminal device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The layered architecture divides the software system of the terminal device 100 into several layers, each with a clear role and division of labor, and the layers communicate with each other through software interfaces. This application uses operating systems such as OpenHarmony OS and Android with layered architectures as examples to illustrate the software structure of the terminal device 100, but this does not limit the type of operating system of the terminal device used in the image rendering method provided in the embodiments of this application.

[0104] FIG2 is a structural block diagram of software of a terminal device applicable to the image rendering method provided in an embodiment of the present application.

[0105] As shown in Figure 2, in some embodiments, the operating system is divided into four layers, namely, the application layer, the framework layer, the system service layer, and the kernel layer, from top to bottom. In some embodiments, the terminal device 100 also includes hardware, such as a GPU, a central processing unit (CPU), a display screen, etc.

[0106] The application layer includes a range of applications. As shown in Figure 2, the application layer can include both system applications and third-party applications. System applications can be understood as applications that come with the system, such as the camera, gallery, calendar, call, map, Bluetooth, video, and music applications. Third-party applications can be understood as non-system applications that can be downloaded and installed from app stores or app markets, such as some game applications.

[0107] Not shown in Figure 2, the framework layer can include a UI framework, a user program framework, and a component framework. The framework layer provides operating system applications with a user program framework and component framework in multiple languages, such as C, C++, and JavaScript (JS), as well as APIs for various software and hardware services. It also provides devices with a UI framework in multiple languages, such as C, C++, and JS. The APIs supported by different devices are related to the degree of componentization of the system.

[0108] In an embodiment of the present application, as shown in Figure 2, the framework layer may also include an interception module, a motion-static recognition module, a pixel-level motion-static separation module, a rendering rate update module, a rendering multiplexing module, a reprojection module and an image output module, etc.

[0109] Among them, the interception module can be used to intercept the rendering instructions and image data of the image, that is, the framework layer of the operating system of the terminal device in this application can intercept the graphics API call and cache the relevant rendering instructions and image data. The static and dynamic recognition module can be used to identify whether the rendering instruction is a static rendering instruction or a dynamic rendering instruction based on the characteristics of the rendering instruction of the image, thereby determining the static objects and dynamic objects in the image. The pixel-level static and dynamic separation module can determine and distinguish the static pixels and dynamic pixels of the image based on the static objects and dynamic objects in the image determined by the static and dynamic recognition module, that is, it can be understood as separating the static pixels and dynamic pixels of the image. The rendering rate update module can be used to update the shading rate during rendering, for example, using the variable rate shading (VRS) mechanism to adjust the rendering rate, which can include but is not limited to reducing the shading rate for static objects and maintaining the original shading rate for dynamic objects. The rendering multiplexing module can be used to reuse some pixels in a frame image before the m-th frame image during the rendering of the m-th frame image. The reprojection module can be used to match the screen space coordinates of the mth frame image with the screen space coordinates of a frame image before the mth frame image. The image output module can output the rendered image.

[0110] The system service layer is the core capability set of the operating system, providing various system services to applications through the framework layer. Not shown in Figure 2, the system service layer can include the system basic capability subsystem set, the basic software service subsystem set, the enhanced software service subsystem set, and the hardware service subsystem set.

[0111] Among them, the system basic capability subsystem set can be composed of subsystems such as distributed soft bus and distributed data management. The system basic capability subsystem set can provide basic capabilities for the operation, scheduling, migration and other operations of distributed applications on multiple devices.

[0112] The basic software service subsystem set can be composed of event notification, telephone, multimedia and other subsystems, and can provide public and general software services.

[0113] The enhanced software service subsystem set can be composed of subsystems such as smart screen proprietary business, wearable proprietary business, and Internet of Things (IoT) proprietary business, and can provide differentiated capability-enhanced software services for different devices.

[0114] The hardware service subsystem set can be composed of subsystems such as location services, biometric recognition, wearable proprietary hardware services, IoT proprietary hardware services, etc., and can provide hardware services.

[0115] It should be noted that, depending on the deployment environment of different device forms, the basic software service subsystem set, the enhanced software service subsystem set, and the hardware service subsystem set can be tailored at the subsystem granularity, and each subsystem can be tailored at the functional granularity. This embodiment of the application does not impose any limitation on this.

[0116] Not shown in Figure 2, the kernel layer includes a kernel subsystem and a driver subsystem. The operating system can adopt a multi-kernel design. Based on the kernel subsystem, the operating system supports the selection of appropriate OS kernels for different resource-constrained devices. The kernel abstraction layer (KAL) shields the differences between multiple kernels and provides basic kernel capabilities to upper layers, including process / thread management, memory management, file system, network management, and peripheral management. The driver framework in the driver subsystem is the foundation of the operating system's open hardware ecosystem, providing unified peripheral access capabilities and a driver development and management framework.

[0117] FIG3 is another structural block diagram of software of a terminal device applicable to the image rendering method provided in an embodiment of the present application.

[0118] As shown in FIG3 , in some embodiments, the Android operating system is divided from top to bottom into an application layer, an application framework layer, an Android runtime (Android runtime), a system library, and a kernel layer.

[0119] In some embodiments, the terminal device 100 also includes hardware, such as a GPU, a CPU, a display screen, etc.

[0120] The application layer can include a range of applications. As shown in Figure 3, the application layer can include system applications and third-party applications. System applications can be understood as applications that come with the system, such as camera, gallery, calendar, call, map, Bluetooth, video, and music applications. Third-party applications can be understood as non-system applications that can be downloaded and installed from the app store.

[0121] The application framework layer provides an API and programming framework for applications in the application layer. The application framework layer includes several predefined functions. As shown in Figure 3, the application framework layer can include an interception module, a motion / still identification module, a pixel-level motion / still separation module, a rendering rate update module, a rendering multiplexing module, a reprojection module, and an image output module. A detailed description of the interception module, motion / still identification module, pixel-level motion / still separation module, rendering rate update module, rendering multiplexing module, reprojection module, and image output module can be found in Figure 2 and will not be repeated here for the sake of brevity.

[0122] 3 , the Android runtime may include a core library and a virtual machine. The Android runtime is responsible for scheduling and management of the Android operating system.

[0123] The core library can contain two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core library of the Android operating system.

[0124] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0125] Not shown in FIG3 , the system library may include multiple functional modules, such as a status monitoring service, a surface manager, media libraries, a 3D graphics processing library (e.g., OpenGL ES), and a 2D graphics engine (e.g., SGL).

[0126] The kernel layer is the layer between hardware and software. Not shown in Figure 3, the kernel layer may include power management services, sensor services (also known as sensor drivers), display services (also known as display drivers), camera drivers, audio drivers, etc. This embodiment of the application does not impose any restrictions on this.

[0127] It should be understood that FIG1 , FIG2 and FIG3 are merely examples and should not limit the embodiments of the present application in any way.

[0128] FIG4 is a schematic flowchart of an image rendering method provided in an embodiment of the present application.

[0129] As shown in Figure 4, method 400 may include step 410 and step 420. The steps of method 400 may be performed by a terminal device on a mobile terminal (which may be referred to as a mobile terminal), or the method 400 may be performed by a component configured in the terminal device (such as a chip, a chip system, etc.), or may be implemented by a logic module or software that can realize all or part of the functions of the terminal device, which is not limited in this embodiment of the present application. The terminal device may have a structure as shown in Figure 1, Figure 2 or Figure 3, which is not limited in this embodiment of the present application. The steps in Figure 4 are described in detail below.

[0130] In step 410 , the mth group of image data to be rendered sent by the application is received.

[0131] The mth group of image data includes a plurality of pixel values ​​used to generate an image, m≥1, and m is an integer.

[0132] The terminal device may receive the mth group of image data to be rendered sent by the application.

[0133] In a possible implementation, receiving the mth group of image data to be rendered sent by an application includes: receiving the mth group of image data to be rendered of a first application, where the first application is any application of one or more preset applications.

[0134] It is understandable that one or more applications may be preset on the terminal device, for example, including but not limited to a whitelist of applications pre-stored on the terminal device for use with the image rendering method provided in this application, and the whitelist of applications is not limited to being stored in the form of a list. The first application may be any application in the whitelist of applications. In an embodiment of the present application, the first application may be a system application or a third-party application. The first application may include but is not limited to game applications, VR applications, AR applications, etc.

[0135] The terminal device may obtain the mth set of image data to be rendered by the first application based on the installation path and / or resource path of the first application. In one possible implementation, as an example and not a limitation, when the first application is running on the terminal device, the terminal device may receive the mth set of image data from the server of the first application.

[0136] FIG5 is a schematic flowchart of triggering the execution of the image rendering method provided by the present application, provided in an embodiment of the present application.

[0137] As shown in Figure 5, when an application is started and running on a terminal device, the terminal device can determine whether the application is an application in the application whitelist. If the application is an application in the above-mentioned pre-stored application whitelist for the image rendering method provided by this application, the terminal device can execute the image rendering method provided by this application; if the application is not an application in the above-mentioned pre-stored application whitelist for the image rendering method provided by this application, the terminal device can execute a known image rendering method.

[0138] In step 420 , the mth frame image is generated by rendering based on the mth set of image data.

[0139] Wherein, when m mod N = 1, the m-th frame image is rendered based on all pixel values ​​in the m-th group of image data; when m mod N ≠ 1, the m-th frame image is rendered based on static pixels of the ith frame image and at least part of the pixel values ​​in the m-th group of image data, and the static pixels of the ith frame image are obtained based on the ith group of image data; Wherein, N is a preset integer, N ≥ 2,

[0140] As you can understand, "mod" is the modulus operator, which is used to find the remainder when two numbers are divided; (m / N) rounded up.

[0141] When the mth group of image data of the first application is received, the terminal device may render and generate the mth frame of image based on the mth group of image data.

[0142] When m mod N = 1, the terminal device can execute a currently known image rendering method based on the mth group of image data, that is, the terminal device can render based on all pixel values ​​in the mth group of image data to obtain the mth frame image; when m mod N ≠ 1, the terminal device can execute the image rendering method provided in the embodiment of the present application based on the mth group of image data, that is, the terminal device can render based on static pixels of the i-th frame image and at least part of the pixel values ​​in the mth group of image data to obtain the mth frame image, where N is a preset positive integer. In other words, the rendering generation of the mth frame reuses part of the pixel values ​​of a previous frame image, which helps to reduce the power consumption of shading calculations on the mobile terminal, reduce the occurrence of heat, freezes and other phenomena, and thus improve the user experience.

[0143] For example, take N=2 (which can be understood as every two frames of image as a group), in the case of m=5, 5mod 2=1, and the terminal device can render the fifth frame of image based on all pixel values ​​in the fifth group of image data; in the case of m=6, 6mod 2=0≠1, i=5 (according to The terminal device may render the sixth frame image based on the static pixels of the fifth frame image and at least part of the pixel values ​​in the sixth group of image data.

[0144] It can be understood that, when N=2, the image of the 2x-1th frame can be rendered based on all pixel values ​​in its image data; and the image of the 2xth frame can be rendered based on static pixels of the image of the 2x-1th frame and at least part of the pixel values ​​in the image data of the 2xth frame, where x is a positive integer.

[0145] Another example, taking N=3 (which can be understood as each three frames of image as a group) as an example, in the case of m=4, 4mod3=1, the terminal device can render the fourth frame of image based on all pixel values ​​in the fourth group of image data; in the case of m=5, 5mod3=2≠1, i=4 (according to The terminal device may render the fifth frame image based on the static pixels of the fourth frame image and at least part of the pixel values ​​in the fifth set of image data; in the case of m=6, 6mod 3=0≠1, i=4 or 5 (according to The terminal device may render the sixth frame image based on the static pixels of the fourth frame image or the fifth frame image and at least part of the pixel values ​​in the sixth group of image data.

[0146] It is understood that, in actual application, every N frames of images constitute a group, and among these N frames of images, at least one frame of image may be rendered based on the static pixels of any preceding frame of image in the group and at least a portion of the pixel values ​​in the image data of the at least one frame of image. As an example and not a limitation, for example, N=3, in the second group of images (including the 4th, 5th, and 6th frames of image), the 4th frame of image may be rendered based on all the pixel values ​​in the 4th group of image data; the 5th frame of image may be rendered based on all the pixel values ​​in the 5th group of image data; and the 6th frame of image may be rendered based on the static pixels of the 5th frame of image and at least a portion of the pixel values ​​in the 6th group of image data. For another example, the 4th frame of image may be rendered based on all the pixel values ​​in the 4th group of image data; the 5th frame of image may be rendered based on the static pixels of the 4th frame of image and at least a portion of the pixel values ​​in the 5th group of image data; and the 6th frame of image may be rendered based on all the pixel values ​​in the 6th group of image data.

[0147] In a possible implementation, the method 400 further includes: intercepting one or more rendering instructions corresponding to the m-th frame image.

[0148] It should be understood that applications need to send rendering instructions to the GPU or CPU. Interception can be achieved by intercepting the rendering instructions before they reach the GPU or CPU. Since rendering instructions must be transmitted through the operating system, technically, the application can also send the rendering instructions to the operating system, which then processes them and sends them to the GPU or CPU.

[0149] It can be understood that each frame of image may include one or more objects (static objects or dynamic objects), and each rendering instruction is used to render one object. Therefore, each frame of image may correspond to one or more rendering instructions.

[0150] The terminal device pre-stores an application whitelist applied to the image rendering method provided in this application. After the first application in the application whitelist is started and run on the terminal device, the terminal device can use an interception module as shown in Figure 2 or Figure 3 to intercept one or more rendering instructions corresponding to the mth frame image, as well as the mth group of image data corresponding to the mth frame image.

[0151] In a possible implementation, before rendering to generate the mth frame image, the method 400 further includes: determining static pixels and dynamic pixels of the mth frame image based on features of one or more rendering instructions corresponding to the mth frame image.

[0152] After intercepting one or more rendering instructions corresponding to the m-th frame image, the terminal device can use the motion-static recognition module and pixel-level motion-static separation module as shown in Figure 2 or Figure 3 to determine the static pixels and dynamic pixels of the m-th frame image based on the characteristics of one or more rendering instructions corresponding to the m-th frame image.

[0153] In a possible implementation, the feature of the rendering instruction includes one or more of the following: a program alias, a program variable name, or a program index buffer name.

[0154] It can be understood that the rendering instruction may include one or more attributes. For example, a program alias, a program variable name, or a program index buffer name may be the name of the attribute included in the rendering instruction. The name of each attribute may correspond to a corresponding attribute value. The characteristics of the rendering instruction may include the name of the attribute included in the rendering instruction and the attribute value corresponding to the attribute name.

[0155] This application provides Implementation A and Implementation B to implement "determining static pixels and dynamic pixels of the mth frame image based on the characteristics of one or more rendering instructions corresponding to the mth frame image." Implementation A and Implementation B are described in detail below.

[0156] Implementation method A: Determine the static pixels and dynamic pixels of the m-th frame image based on the characteristics of one or more rendering instructions corresponding to the m-th frame image, including: determining the static objects and dynamic objects included in the m-th frame image based on the characteristics of each rendering instruction in the one or more rendering instructions; determining the static pixels based on the static objects; and determining the dynamic pixels based on the dynamic objects.

[0157] The attribute values ​​of some attributes can be preset (including but not limited to being set or limited in the form of a list). When a rendering instruction includes any one of these preset attribute values, the terminal device can determine that the rendering instruction is a dynamic rendering instruction, that is, in the m-th frame image, the object rendered by the rendering instruction is a dynamic object; when a rendering instruction does not include all of these preset attribute values, the terminal device can determine that the rendering instruction is a static rendering instruction, that is, in the m-th frame image, the object rendered by the rendering instruction is a static object.

[0158] As an example but not a limitation, the one or more rendering instructions corresponding to the m-th frame image include rendering instruction 1, rendering instruction 2, and rendering instruction 3, etc., where, for example, the attribute value of the program index buffer name attribute of rendering instruction 1 is "character index buffer (used to drive character movement or character skeletal movement)", and the terminal device can determine that rendering instruction 1 is a dynamic rendering instruction based on this feature of rendering instruction 1, that is, in the m-th frame image, the object rendered by rendering instruction 1 is a dynamic object; for another example, the attribute value of the program variable name attribute of rendering instruction 2 is "character colour (used to color the character)", and the terminal device can determine that rendering instruction 2 is a dynamic rendering instruction based on this feature of rendering instruction 2, that is, in the m-th frame image, the object rendered by rendering instruction 2 is a dynamic object.

[0159] That is to say, when one or more rendering instructions corresponding to the m-th frame image are intercepted, the terminal device can use the dynamic and static recognition module shown in Figure 2 or Figure 3 to analyze each rendering instruction based on the preset attribute values ​​of these attributes, and determine whether the attribute value of the rendering instruction includes any one of the preset attribute values ​​of these attributes. If included, it is determined that the object rendered by the rendering instruction in the m-th frame image is a dynamic object; otherwise, it is determined that the object rendered by the rendering instruction in the m-th frame image is a static object.

[0160] After determining the static objects and dynamic objects included in the m-th frame image, the terminal device can determine the static pixels based on the static objects, and determine the dynamic pixels based on the dynamic objects. For example, based on the pixel-level static-dynamic separation module shown in Figure 2 or Figure 3, the terminal device can determine the pixel area corresponding to the static objects included in the m-th frame image as the static pixel area, and identify these static pixels with a first identification value; determine the pixel area corresponding to the dynamic objects included in the m-th frame image as the dynamic pixel area, and identify these dynamic pixels with a second identification value, thereby distinguishing the static pixels from the dynamic pixels.

[0161] In a possible implementation, after determining the static pixels and dynamic pixels of the mth frame image, the method 400 further includes: marking the static pixels and the dynamic pixels based on the A channel or the template buffer; wherein the static pixels and the dynamic pixels both include RGBA channels.

[0162] As an example and not a limitation, after determining the static pixels and static pixels of the mth frame image, the first identification value for identifying the static pixels and the second identification value for identifying the dynamic pixels can be written into the template buffer or the A channel in RGBA to generate a mask (Mask) for separating static and dynamic pixels, and the Mask can be saved to facilitate subsequent calculations and analysis based on the Mask.

[0163] FIG6 is a schematic diagram of a static-dynamic separation mask provided in an embodiment of the present application.

[0164] As an example but not limitation, take "0" as an example of the first identification value and "1" as an example of the second identification value, as shown in FIG6 , the pixel point with a value of "0" is a static pixel, and the pixel point with a value of "1" is a dynamic pixel.

[0165] In order to better understand the implementation method A, the generation of the static and dynamic separation mask is further described below with reference to FIG. 7 .

[0166] FIG7 is a schematic flowchart of generating a static-dynamic separation mask provided in an embodiment of the present application.

[0167] Step 701: intercept a rendering instruction of a first application.

[0168] The first application is any one of one or more preset applications. The terminal device can intercept the rendering instructions of the first application. Each frame of image can include one or more objects (static objects or dynamic objects), and each rendering instruction is used to render one object. Therefore, each frame of image can correspond to one or more rendering instructions. For example, the terminal device can intercept one or more rendering instructions corresponding to the mth frame of image.

[0169] Step 702: Record the characteristics of the rendering instruction.

[0170] The terminal device may record the characteristics of each rendering instruction in the one or more rendering instructions corresponding to the m-th frame image. The characteristics of the rendering instruction include one or more of the following: a program alias, a program variable name, or a program index buffer name.

[0171] Step 703: Classify dynamic and static rendering instructions according to the characteristics of the rendering instructions.

[0172] The terminal device can classify the one or more rendering instructions corresponding to the m-th frame image into dynamic and static rendering instructions based on the characteristics of each rendering instruction in the one or more rendering instructions corresponding to the m-th frame image, that is, determine whether each rendering instruction in the one or more rendering instructions corresponding to the m-th frame image is a static rendering instruction or a dynamic rendering instruction.

[0173] Step 704: Generate a dynamic and static separation mask.

[0174] After determining whether each rendering instruction in one or more rendering fingers corresponding to the m-th frame image is a static rendering instruction or a dynamic rendering instruction, the terminal device will know which objects are static and which objects are dynamic in the m-th frame image, so that it can further determine static pixels based on static objects and dynamic pixels based on dynamic objects, and then generate a static-dynamic separation mask based on these static pixels and dynamic pixels.

[0175] Implementation method B: Based on the characteristics of one or more rendering instructions corresponding to the m-th frame image, determine the static pixels and dynamic pixels of the m-th frame image, including: based on the characteristics of each rendering instruction in the one or more rendering instructions, determine the static objects and dynamic objects included in the m-th frame image to obtain a motion vector set of the m-th frame image; based on the motion vector set, determine the static pixels and the dynamic pixels, the value of the motion vector corresponding to the static object is different from the value of the motion vector corresponding to the dynamic object.

[0176] In the implementation method B, when one or more rendering instructions corresponding to the m-th frame image are intercepted, the terminal device can also use the static and dynamic recognition module shown in Figure 2 or Figure 3 to analyze each rendering instruction based on the attribute value of the preset attribute, and determine whether the attribute value of the rendering instruction includes any one of the attribute values ​​of the preset attribute. In the case that the attribute value of the rendering instruction includes any one of the attribute values ​​of the preset attribute, it is determined that in the m-th frame image, the object rendered by the rendering instruction is a dynamic object; otherwise, it is determined that in the m-th frame image, the object rendered by the rendering instruction is a static object. For a detailed description of "determining the static objects and dynamic objects included in the m-th frame image based on the characteristics of one or more rendering instructions corresponding to the m-th frame image", please refer to the relevant description in the above implementation method A. For the sake of brevity, it will not be repeated here.

[0177] After determining the static objects and dynamic objects included in the m-th frame image, the terminal device can obtain a motion vector set for the m-th frame image based on the static objects and the dynamic objects. In the motion vector set for the m-th frame image, the values ​​of the motion vectors corresponding to the static objects are different from the values ​​of the motion vectors corresponding to the dynamic objects. For example, and not by way of limitation, the value of the motion vector corresponding to the static object is zero, while the value of the motion vector corresponding to the dynamic object is non-zero. Thus, the terminal device can determine the static pixels corresponding to the static objects and the dynamic pixels corresponding to the dynamic objects based on the motion vector set for the m-th frame image.

[0178] In one possible implementation, when m mod N≠1, and after determining the static objects included in the m-th frame image, method 400 further includes: reducing the shading rate of the static objects in the m-th frame image, and rendering the m-th frame image is based on the updated shading rate.

[0179] In the case where m mod N ≠ 1, and after determining the static objects included in the m-th frame image, the terminal device can use the rendering rate update module as shown in Figure 2 or Figure 3 to update the shading rate when rendering the m-th frame image. For example, the terminal device can reduce the shading rate of static objects in the m-th frame image, while maintaining the original shading rate for dynamic objects. This is not limited in the embodiments of the present application. Reducing the shading rate of static objects can further reduce the power consumption of the mobile terminal's shading calculation, reduce the occurrence of phenomena such as heating and freezing, and thus improve the user experience.

[0180] It can be understood that when m mod N = 1, the terminal device does not need to use the rendering rate update module as shown in Figure 2 or Figure 3 to update the shading rate when rendering the mth frame image. The terminal device can render the mth frame image based on the currently known image rendering method.

[0181] For example, taking N=2 as an example, when m is 2x, after determining the static objects and dynamic objects included in the m-th frame image, the terminal device can reduce the shading rate of the static objects in the m-th frame image, while maintaining the shading rate of the dynamic objects in the m-th frame image; when m is 2x-1, the terminal device can render the m-th frame image based on the currently known image rendering method.

[0182] FIG8 is a schematic flowchart of whether to update the shading rate provided in an embodiment of the present application.

[0183] Step 801: identifying dynamic and static rendering instructions.

[0184] In the case where m mod N≠1, the terminal device can perform dynamic and static rendering instruction identification on each of the one or more rendering instructions corresponding to the m-th frame image. For the convenience of description, in the embodiment of the present application, any one of the one or more rendering instructions corresponding to the m-th frame image is recorded as a "first rendering instruction". The terminal device can perform dynamic and static rendering instruction identification on the first rendering instruction based on the characteristics of the first rendering instruction, that is, determine whether the first rendering instruction is a static rendering instruction or a dynamic rendering instruction. The first rendering instruction is a static rendering instruction, that is, in the m-th frame image, the object rendered by the rendering instruction is a static object; the first rendering instruction is a dynamic rendering instruction, that is, in the m-th frame image, the object rendered by the rendering instruction is a dynamic object.

[0185] Step 802: Determine whether it is a static rendering instruction.

[0186] After performing dynamic and static rendering instruction identification on the first instruction, the terminal device can determine whether the first rendering instruction is a static rendering instruction. If the first rendering instruction is a static rendering instruction, the terminal device can execute step 703; if the first rendering instruction is a dynamic rendering instruction, the terminal device can execute step 804.

[0187] Step 803: Reduce the shading rate.

[0188] In the case that the first rendering instruction is a static rendering instruction, the terminal device may reduce the shading rate of the static object rendered by the first rendering instruction.

[0189] Step 804: The shading rate remains unchanged.

[0190] In the case where the first rendering instruction is a dynamic rendering instruction, the terminal device may maintain a constant shading rate of the dynamic object rendered by the first rendering instruction.

[0191] Table 1 is a reference table of various coloring rates provided in the examples of this application.

[0192] Table 1

[0193] Among the various shading rates shown in Table 1, the shading rate 1×1 corresponding to number 1 is the largest, and the shading rate 4×4 corresponding to number 7 is the smallest. For example, if the shading rate of a static object rendered by a first rendering instruction is the shading rate 1×2 corresponding to number 2, and the terminal device determines that the first rendering instruction is a static rendering instruction, the terminal device may reduce the shading rate of the static object rendered by the first rendering instruction from the shading rate 1×2 corresponding to number 2 to a shading rate corresponding to any number from 3 to 7.

[0194] It is understandable that in actual application scenarios, the various shading rates may include but are not limited to those shown in Table 1 above.

[0195] In a possible implementation, when m mod N≠1, rendering and generating an m-th frame image based on the m-th group of image data includes: making correspondence between the i-th frame image and the screen space coordinates of the m-th frame image; when the pixels of the first screen space corresponding to the i-th frame image and the m-th frame image are both static pixels, reusing the pixels of the first screen space of the i-th frame image to render the first screen space of the m-th frame image; or, when the pixels of the second screen space corresponding to the i-th frame image and the m-th frame image are both dynamic pixels, or when the pixels of the second screen space of the i-th frame image and the m-th frame image are different types of pixels, rendering the second screen space of the m-th frame image based on the pixels of the second screen space of the m-th frame image.

[0196] The terminal device may use a reprojection method to establish a one-to-one correspondence between the screen space coordinates of the i-th frame image and the m-th frame image. As an example and not a limitation, when m mod N ≠ 1, the terminal device may use a reprojection module as shown in FIG. 2 or 3 to establish a correspondence between the screen space coordinates of the i-th frame image and the m-th frame image.

[0197] In more detail, the terminal device's operating system graphical interface hook layer can be used to obtain the view projection transformation (VP) matrix when rendering each frame of the main scene (i.e., image). For example, the VP matrix is ​​not limited to a 4×4 matrix. The terminal device can also obtain image data for each frame of the image, including but not limited to a pixel matrix, depth information (i.e., depth value), and pixel coordinates (used to represent the position of the pixel point). The terminal device can, based on the depth information, pixel coordinates, and inverse matrix of the VP matrix of the i-th frame image, and the VP matrix of the m-th frame image, make a one-to-one correspondence between the i-th frame image and the screen space coordinates of the m-th frame image. As an example and not a limitation, the depth information, pixel coordinates, and W value of the i-th frame image are combined to obtain a matrix. For example, the depth information is a 1×1 matrix, the pixel coordinates are a 2×1 matrix, and the W value is a 1×1 matrix (e.g., the W value is 1.0). Then, the depth information, pixel coordinates, and W value of the i-th frame image are combined to obtain a 4×1 matrix. For ease of description, this matrix is ​​referred to as a "combined matrix." For another example, the VP matrices of the i-th frame image and the m-th frame image are both 4×4 matrices. Then, the combination matrix × the inverse matrix of the VP matrix of the i-th frame image × the VP matrix of the m-th frame image can establish a one-to-one correspondence between the screen space coordinates of the i-th frame image and the m-th frame image, that is, a one-to-one correspondence can be established between the pixels of the i-th frame image and the pixels of the m-th frame image.

[0198] After establishing a one-to-one correspondence between the pixels of the i-th frame image and the pixels of the m-th frame image, the terminal device can determine which part of the static pixels in the i-th frame image can be reused by the m-th frame image based on the static and dynamic separation mask of the i-th frame image and the static and dynamic separation mask of the m-th frame image.

[0199] Table 2 is a schematic table of pixel reuse rules provided in an embodiment of the present application.

[0200] Table 2

[0201] As shown in Table 2, based on the static and dynamic separation mask of the i-th frame image and the static and dynamic separation mask of the m-th frame image, when the pixels of the i-th group of image data in the first screen space and the pixels of the m-th group of image data in the first screen space are both static pixels, that is, when the pixels of the first screen space corresponding to the i-th frame image and the m-th frame image are both static pixels, the terminal device can utilize the rendering multiplexing module shown in FIG2 or FIG3 to reuse the static pixels of the first screen space of the i-th frame image to render the first screen space of the m-th frame image. In other words, when the pixels of the i-th group of image data in the first screen space and the pixels of the m-th group of image data in the first screen space are both static pixels, the pixels used in rendering the first screen space of the m-th frame image are derived from the i-th group of image data. In this way, the pixels of the i-th group of image data in the first screen space are reused, which helps reduce the power consumption of the mobile terminal's shading calculations, reduces the occurrence of heat generation, lag, and other phenomena, and thus improves the user experience.

[0202] Based on the static and dynamic separation mask of the i-th frame image and the static and dynamic separation mask of the m-th frame image, when the pixels of the i-th group of image data in the second screen space and the pixels of the m-th group of image data in the second screen space are both dynamic pixels, that is, when the pixels of the second screen space corresponding to the i-th frame image and the m-th frame image are both dynamic pixels, the terminal device can render the second screen space of the m-th frame image based on the pixels of the m-th group of image data in the second screen space. In other words, when the pixels of the i-th group of image data in the second screen space and the pixels of the m-th group of image data in the second screen space are both dynamic pixels, the pixels when rendering the second screen space of the m-th frame image are derived from the m-th group of image data, and the pixels of the i-th group of image data in the second screen space are not reused.

[0203] Based on the static-dynamic separation mask of the i-th frame image and the static-dynamic separation mask of the m-th frame image, if the type of pixels in the second screen space of the i-th group of image data is different from the type of pixels in the second screen space of the m-th group of image data, that is, if the second screen space of the i-th frame image and the m-th frame image are different types of pixels, for example, if the pixels in the second screen space of the i-th group of image data are static pixels and the pixels in the second screen space of the m-th group of image data are dynamic pixels, or if the pixels in the second screen space of the i-th group of image data are dynamic pixels and the pixels in the second screen space of the m-th group of image data are static pixels, then the terminal device may render the second screen space of the m-th frame image based on the pixels in the second screen space of the m-th group of image data. That is, if the type of pixels in the second screen space of the i-th group of image data is different from the type of pixels in the second screen space of the m-th group of image data, the pixels in the second screen space of the m-th frame image are rendered from the m-th group of image data, and the pixels in the second screen space of the i-th group of image data are not reused.

[0204] It can be understood that the first screen space is a screen space in which the pixels of the i-th frame image and the m-th frame image are both static, and the second screen space is a screen space in which the pixels of the i-th frame image and the m-th frame image are both dynamic, or the types of pixels of the i-th frame image and the m-th frame image are different.

[0205] Based on the above scheme, N (N≥2) frames of images are taken as a group, and the first frame of the N frames is obtained based on the currently known rendering process, that is, the first frame of the N frames can be rendered based on all pixel values ​​in the image data of the first frame, and at least one frame of the N frames other than the first frame can be rendered based on the static pixels of a frame before the image in this group of images, and at least part of the pixel values ​​in the image data of the image. In this way, the rendering generation of at least one frame of image in each group of images reuses part of the pixel values ​​of other images in the group, which helps to reduce the power consumption of the shading calculation on the mobile terminal, reduce the occurrence of heat, freezes and other phenomena, and thus improve the user experience.

[0206] In order to better understand the image rendering method provided in the embodiment of the present application, the software and hardware structure of the terminal device suitable for the image rendering method provided in the embodiment of the present application is described again in conjunction with Figure 9 below.

[0207] FIG9 is another structural block diagram of software of a terminal device applicable to the image rendering method provided in an embodiment of the present application.

[0208] As shown in Figure 9, the application layer of the terminal device can be deployed with multiple applications suitable for the image rendering method provided in the embodiment of the present application, for example, game application one, game application two, and game application three, etc. The application layer can also be deployed with a game engine, for example, the game engine can include but is not limited to the Unity engine.

[0209] The framework layer of the terminal device can be deployed with an interception module, a static and motion recognition module, a pixel-level static and motion separation module, a rendering rate update module, a rendering multiplexing module, a reprojection module and an image output module, etc. For a detailed description of these modules, please refer to the relevant description in Figure 2 above. For the sake of brevity, they will not be repeated here. In one possible implementation method, the terminal device can be deployed with a graphics interface Hook layer, and the graphics interface Hook layer can be deployed with an interception module, a static and motion recognition module, a pixel-level static and motion separation module, a rendering rate update module, a rendering multiplexing module, a reprojection module and an image output module, etc. The graphics interface Hook layer can intercept and process rendering instructions, and can also send the processed rendering instructions to the GPU or CPU.

[0210] The driver layer of the terminal device may be deployed with a graphics driver. For example, the graphics driver may include but is not limited to OpenGL (OpenGL ES) for embedded systems (ES).

[0211] The hardware layer of the terminal device can be deployed with a GPU and a CPU. The GPU can include, but is not limited to, an Arm Mali processor or a Qualcomm Adreno processor. The CPU can include, but is not limited to, a Kirin 9000 processor or a Qualcomm processor.

[0212] In this application, the terminal device can intercept graphics API calls at the operating system framework layer, cache related instructions and associated data. Taking N=2 as an example, the 2x-1 frame image can maintain the original rendering process and only separate dynamic objects and static objects; for the rendering process of the 2x frame image, the characteristics of the instruction stream can be combined to identify whether the DrawCall is dynamic or static. For the DrawCall of dynamic objects, the original shading rate is maintained, and for the DrawCall of static objects, the shading rate is reduced. In the post-processing part of the rendering, a static object rendering reuse solution is added to reuse the static pixels of the 2x-1 frame image into the 2x frame image, and finally the result image is output.

[0213] In some examples, Drawcall may be a draw call interface, which may refer to a process in which a CPU sends a drawing instruction to a GPU.

[0214] FIG10 is a schematic flowchart of the image rendering method provided in an embodiment of the present application implemented on an operating system.

[0215] It is understood that the software code (e.g., program) for implementing the image rendering method provided in the embodiment of the present application can be deployed in the instruction reassembly layer of the operating system of the terminal device. As shown in Figure 10, in the embodiment of the present application, the terminal device can intercept the rendering instruction issued by the first application at the instruction reassembly layer. The reassembly layer may include the interception module, the motion and static recognition module, the pixel-level motion and static separation module, the rendering rate update module, the rendering multiplexing module, and the reprojection module shown in Figure 2 or Figure 3.

[0216] Step 1001: The rendering interface of the first application is called.

[0217] The first application is any one of one or more preset applications suitable for the image rendering method provided in this application. When the rendering interface of the first application is called, the terminal device can be triggered to execute the image rendering method provided in this application.

[0218] Step 1002: intercept rendering instructions.

[0219] The terminal device may intercept one or more rendering instructions corresponding to the mth frame image of the first application.

[0220] Step 1003: Identify dynamic objects and static objects.

[0221] The terminal device may identify the dynamic objects and static objects included in the m-th frame image based on the features of each rendering instruction in the one or more rendering instructions corresponding to the m-th frame image.

[0222] It can be understood that, in the image rendering method provided in the present application, no matter m satisfies m mod N = 1 or m mod N ≠ 1, steps 1002 and 1003 need to be executed for the mth frame image of the first application.

[0223] Step 1004, original drawing process.

[0224] In the case where m mod N=1, after intercepting one or more rendering instructions corresponding to the m-th frame image, the terminal device may execute an original drawing process in a known image rendering method based on the known image rendering method.

[0225] Step 1005: Update the shading rate.

[0226] In the case that m mod N≠1, after executing step 1003, the terminal device may reduce the shading rate of static objects and maintain the shading rate of dynamic objects.

[0227] Step 1006, reprojection.

[0228] When the mth frame image reuses some pixels in the ith frame image, the terminal device can perform reprojection based on the mth group of image data and the ith group of image data to correspond the ith frame image to the screen space coordinates of the mth frame image.

[0229] Step 1007: Separate static and dynamic pixels.

[0230] Regardless of whether m satisfies m mod N = 1 or m mod N ≠ 1, for the mth frame image of the first application, after executing step 1003, the terminal device can obtain static pixels and dynamic pixels based on the static objects and dynamic objects in the mth frame image to distinguish the static pixels from the dynamic pixels of the mth frame image.

[0231] Step 1008: Static pixel rendering and reuse.

[0232] In the case where m mod N≠1, the terminal device may reuse some static pixels in the i-th frame image when rendering and generating the m-th frame image based on the pixel reuse rule in Table 2 above.

[0233] FIG11 is another structural block diagram of the software and hardware of a terminal device applicable to the image rendering method provided in an embodiment of the present application.

[0234] As shown in Figure 11, applications suitable for the image rendering method provided in the embodiment of the present application may include but are not limited to game applications, VR applications or AR applications, etc., and may also include other applications. A terminal device suitable for the image rendering method provided in the embodiment of the present application may include an instruction reset layer, which can be used to obtain calls to the rendering API, cache rendering instruction streams and data related to the rendering instruction streams (for example, including but not limited to image data). A terminal device suitable for the image rendering method provided in the embodiment of the present application may include a module for executing the original drawing process, and may also include modules provided in this application (for example, including but not limited to the motion and static recognition module shown in Figure 2 or Figure 3, a pixel-level motion and static separation module, a rendering rate update module, a rendering multiplexing module and a reprojection module, etc.).

[0235] In order to facilitate a better understanding of the image rendering method provided by the present application, the image rendering method provided by the present application is further described below with reference to FIG. 12 to FIG. 15 .

[0236] FIG12 is another schematic flowchart of the image rendering method provided in an embodiment of the present application.

[0237] As shown in Figure 12, taking N=2 as an example, the terminal device processes the 2x-1th set of image data for the first application differently from the 2xth set of image data for the first application. For example, when rendering and generating the 2x-1th frame of image, the terminal device may execute steps 1201 and 1202; when rendering and generating the 2xth frame of image, the terminal device may execute steps 1203 through 1208. The following describes the rendering process for the adjacent 2x-1th and 2xth frames of image.

[0238] Step 1201: Identify dynamic and static rendering instructions for the 2x-1th frame image.

[0239] The terminal device can obtain the 2x-1th group of image data of the first application and the rendering instruction corresponding to the 2x-1th frame image, and then the terminal device can determine whether the rendering instruction is a dynamic rendering instruction or a static rendering instruction based on the rendering instruction corresponding to the 2x-1th frame image.

[0240] Step 1202: Normal rendering process.

[0241] The terminal device can perform a normal rendering process based on the 2x-1th group of image data to render and generate the 2x-1th frame image.

[0242] Step 1203: Generate a motion and still separation mask for the 2x-1th frame image.

[0243] Before rendering and generating the 2xth frame image, the terminal device may generate a static and motion separation mask for the 2x-1th frame image after executing step 1201. For a detailed description of generating the static and motion separation mask for the 2x-1th frame image, please refer to the relevant description in the above implementation method A. For the sake of brevity, it is not repeated here.

[0244] Step 1204: Identify dynamic and static rendering instructions for the 2xth frame image.

[0245] The terminal device can obtain the 2xth group of image data of the first application and the rendering instruction corresponding to the 2xth frame image, and then the terminal device can determine whether the rendering instruction is a dynamic rendering instruction or a static rendering instruction based on the rendering instruction corresponding to the 2xth frame image.

[0246] Step 1205: Generate a static and dynamic separation mask for the 2xth frame image.

[0247] Before rendering and generating the 2xth frame image, the terminal device may generate a static and dynamic separation mask for the 2xth frame image after executing step 1204. A detailed description of generating the static and dynamic separation mask for the 2xth frame image can be found in the relevant description of implementation method A above, and for the sake of brevity, it is not repeated here.

[0248] Step 1206: Reduce the shading rate of static objects.

[0249] After the terminal device executes step 1204, the terminal device determines which part of the 2xth frame image is a static object and which part is a dynamic object. Therefore, the terminal device can reduce the shading rate of static objects while maintaining the shading rate of dynamic objects.

[0250] Step 1207: Reprojection.

[0251] The terminal device can use the reprojection method to make a one-to-one correspondence between the screen space coordinates of the odd frame image and the even frame image based on the reprojection method, so as to facilitate the subsequent determination of which part of the pixels of the odd frame image can be reused by the even frame image.

[0252] Step 1208: Static pixel shading reuse.

[0253] After executing steps 1203, 1205, 1206, and 1207, the terminal device can render the 2xth frame image. In the process of rendering and generating the 2xth frame image, some static pixels in the 2x-1th frame image are reused, and the rendering rate of the static objects in the 2xth frame image is based on the shading rate reduced in step 1206.

[0254] It is understandable that in the embodiment of the present application, there is no limitation on the order of step 1203, step 1205, step 1206 and step 1207.

[0255] FIG13 is a schematic diagram of data involved in an image rendering method provided in an embodiment of the present application.

[0256] As shown in FIG13 , again using N=2 as an example, the terminal device processes the 2x-1th set of image data for the first application differently from the 2xth set of image data for the first application. Taking the image rendering method process shown in FIG12 as an example, during the rendering process to generate the 2xth frame image, reprojection and static pixel shading multiplexing are required. Reprojection requires the use of a portion of the image data for the 2x-1th frame image and a portion of the image data for the 2xth frame image. Image data may include, but is not limited to, the image's pixel matrix, depth information, pixel coordinates, and VP matrix. For example, reprojection requires the use of the depth information, pixel coordinates, and VP matrix for the 2x-1th frame image, as well as the VP matrix for the 2xth frame image. Static pixel shading multiplexing requires the use of the static and dynamic separation mask for the pixels in the 2x-1th frame image and the image data for the 2x-1th frame image, as well as the static and dynamic separation mask for the pixels in the 2xth frame image and the image data for the 2xth frame image.

[0257] FIG14 is another schematic flowchart of the image rendering method provided in an embodiment of the present application.

[0258] Different from the image rendering method process shown in FIG12 , the image rendering method process shown in FIG14 utilizes more native rendering data of the game application itself.

[0259] As shown in FIG. 14 , still taking N=2 as an example, the terminal device processes the 2x-1th group of image data of the first application differently from the terminal device processes the 2xth group of image data of the first application.

[0260] In the image rendering method flow shown in FIG14 , the processing process of the 2x-1th group of image data can refer to step 1201 and step 1202 in FIG12 , which will not be repeated here for the sake of brevity.

[0261] In the image rendering method flow shown in FIG14 , the process of processing the 2x-1th group of image data may include steps 1401 to 1403 .

[0262] Step 1401: identifying dynamic and static rendering instructions for the 2xth frame image.

[0263] For a detailed description of step 1401, please refer to the relevant description of step 1204 in Figure 12 above. For the sake of brevity, it will not be repeated here.

[0264] Step 1402: Reduce the shading rate of static objects.

[0265] For a detailed description of step 1402, please refer to the relevant description of step 1206 in Figure 12 above. For the sake of brevity, it will not be repeated here.

[0266] Step 1403, execution steps of the updated TAA module.

[0267] It is understandable that in this implementation, the updated TAA module can replace the currently known native TAA module. The updated TAA module adds the ability of static pixel shading multiplexing on the basis of the native TAA module. The terminal device can perform static pixel shading multiplexing based on the updated TAA module after executing the steps required by the native TAA module (including but not limited to softening the edges of objects) to render and generate the 2x frame image.

[0268] FIG15 is another schematic diagram of data involved in an image rendering method provided in an embodiment of the present application.

[0269] As shown in Figure 15, taking N=2 as an example, the terminal device's processing process for the 2x-1th group of image data of the first application is different from the processing process for the 2xth group of image data of the first application. In the process of rendering and generating the 2xth frame image, static pixel shading multiplexing is required.

[0270] As shown in FIG15 , since some game applications are already able to distinguish between dynamic objects and static objects during the native rendering process, after the dynamic and static separation, a dynamic and static separation mask is no longer generated. Instead, static pixels and dynamic pixels can be distinguished based on the motion vector set. After executing the execution steps required by the native TAA module, the shading rate can be controlled and static pixel shading can be reused. For example, the image data of the 2x-1 frame image and the motion vector set of the 2x-1 frame image, as well as the image data of the 2x frame image and the motion vector set of the 2x frame image, can be input into the native TAA module. After the native TAA module completes anti-aliasing, the terminal device can use the motion vector set of the 2x-1 frame image and the motion vector set of the 2x frame image to distinguish between static pixels and dynamic pixels. Based on the pixel reuse rules in Table 2 above, when rendering and generating the 2x frame image, some static pixels in the 2x-1 frame image of the previous frame are reused, and the 2x frame image is finally output.

[0271] In some examples, the screen displayed by an electronic device such as a mobile phone, such as a photo, a video, or a game screen, etc. Taking a photo as an example, each pixel of the photo may include four RBGA channels. Among them, the three RGB channels can be used to determine the color actually displayed by the pixel, and the A channel is used to determine the transparency of the pixel. For photos with the same A channel value, for example, the A channel value is 1, or the A channel value of the screen displayed by the mobile phone defaults to 1. In some scenarios where the A channel is not enabled, the A channel can be used to record other information, for example, it can be marked whether the pixel is dynamic or static.

[0272] FIG16 is a schematic flowchart of another image rendering method provided in an embodiment of the present application.

[0273] As shown in Figure 16, method 1600 may include steps 1610 to 1640. The steps of method 1600 may be performed by a terminal device at a mobile terminal (which may be referred to as a mobile terminal), or the method 1600 may be performed by a component configured in the terminal device (such as a chip, a chip system, etc.), or may be implemented by a logic module or software that can realize all or part of the functions of the terminal device, which is not limited in this embodiment of the present application. The terminal device may have a structure as shown in Figure 1, Figure 2 or Figure 3, which is not limited in this embodiment of the present application. The steps in Figure 16 are described in detail below.

[0274] In step 1610 , the mth group of image data to be rendered sent by the application is received.

[0275] The image data includes a plurality of pixel values ​​used to generate an image, where m≥1, and m is an integer. In some examples, the image data includes, but is not limited to, a pixel matrix, depth information, and pixel coordinates.

[0276] The terminal device may receive the mth group of image data to be rendered sent by the application.

[0277] In one possible implementation, receiving the mth set of image data to be rendered sent by an application includes receiving the mth set of image data to be rendered from a first application, where the first application is any one of one or more preset applications. A detailed description can be found in the description of step 410 in method 400 above, and for the sake of brevity, is not repeated here.

[0278] In step 1620 , the mth frame image is generated by rendering based on all pixel values ​​in the mth group of image data.

[0279] The terminal device can execute a currently known image rendering method based on the mth group of image data, that is, the terminal device can render based on all pixel values ​​in the mth group of image data to obtain the mth frame image.

[0280] In step 1630 , the (m+1)th group of image data to be rendered sent by the application is received.

[0281] After rendering and generating the mth frame image, the terminal device can receive image data of the next frame image of the mth frame image sent by the application, that is, the terminal device can receive the m+1th group of image data to be rendered sent by the application.

[0282] In step 1640 , an m+1th frame image is generated by rendering based on the static pixels of the mth frame image and at least part of the pixel values ​​in the m+1th group of image data.

[0283] The static pixels of the m-th frame image are obtained based on the m-th group of image data.

[0284] The terminal device may determine, based on the static pixels based on the m-th frame image, the m-th group of image data, and the m+1-th group of image data, whether the m+1-th frame image can reuse some of the static pixels in the m-th frame image during the rendering and generation process. If the m+1-th frame image can reuse some of the static pixels in the m-th frame image during the rendering and generation process, the terminal device may render and generate the m+1-th frame image based on the static pixels based on the m-th frame image and at least some of the pixel values ​​in the m+1-th group of image data.

[0285] In one possible implementation, before rendering to generate the m+1th frame image, the method 1600 further includes: determining the static pixels and dynamic pixels of the yth frame image based on the characteristics of one or more rendering instructions corresponding to the yth frame image, y∈[m, m+1].

[0286] That is, before rendering and generating the (m+1)th frame image, the terminal device can determine the static pixels and dynamic pixels of the (m)th frame image, and can also determine the static pixels and dynamic pixels of the (m+1)th frame image.

[0287] The terminal device can use the static-dynamic recognition module and the pixel-level static-dynamic separation module as shown in Figure 2 or Figure 3 to determine the static pixels and dynamic pixels of the m-th frame image based on the characteristics of one or more rendering instructions corresponding to the m-th frame image. Without loss of generality, the terminal device can also use the static-dynamic recognition module and the pixel-level static-dynamic separation module as shown in Figure 2 or Figure 3 to determine the static pixels and dynamic pixels of the m+1-th frame image based on the characteristics of one or more rendering instructions corresponding to the m+1-th frame image.

[0288] In one possible implementation, the characteristics of the rendering instruction include one or more of the following: a program alias, a program variable name, or a program index buffer name. A detailed description can be found in the description of the characteristics of the rendering instruction in method 400 above, which will not be repeated here for brevity.

[0289] This application provides Implementation Methods 1 and 2 to implement "determining static pixels and dynamic pixels of the mth frame image based on the characteristics of one or more rendering instructions corresponding to the yth frame image." Implementation Methods 1 and 2 are described in detail below.

[0290] Implementation method 1: Determine the static pixels and dynamic pixels of the yth frame image based on the characteristics of one or more rendering instructions corresponding to the yth frame image, including: determining the static objects and dynamic objects included in the yth frame image based on the characteristics of each rendering instruction in the one or more rendering instructions; determining the static pixels of the yth frame image based on the static objects; and determining the dynamic pixels of the yth frame image based on the dynamic objects.

[0291] For a detailed description of implementation method 1, please refer to the relevant description of implementation method A in method 400 above. For the sake of brevity, it will not be repeated here.

[0292] In one possible implementation, after determining the static pixels and dynamic pixels of the yth frame image, the method 1600 further includes: marking the static pixels and the dynamic pixels based on the A channel or the template buffer; wherein the static pixels and the dynamic pixels both include RGBA channels.

[0293] For a detailed description of marking the static pixels and dynamic pixels of the yth frame image, please refer to the relevant description of marking the static pixels and dynamic pixels of the mth frame image in the above method 400. For the sake of brevity, it is not repeated here.

[0294] Implementation method two: Determine the static pixels and dynamic pixels of the yth frame image based on the characteristics of one or more rendering instructions corresponding to the yth frame image, including: determining the static objects and dynamic objects included in the yth frame image based on the characteristics of each rendering instruction in the one or more rendering instructions to obtain a motion vector set of the yth frame image; determine the static pixels and dynamic pixels of the yth frame image based on the motion vector set, and the value of the motion vector corresponding to the static object of the yth frame image is different from the value of the motion vector corresponding to the dynamic object of the yth frame image.

[0295] For a detailed description of implementation method 2, please refer to the relevant description of implementation method B in method 400 above. For the sake of brevity, it will not be repeated here.

[0296] In a possible implementation, the method 1600 further includes: intercepting one or more rendering instructions corresponding to the yth frame image.

[0297] For a detailed description of intercepting one or more rendering instructions corresponding to the y-th frame image, please refer to the relevant description of intercepting one or more rendering instructions corresponding to the m-th frame image in the above method 400. For the sake of brevity, it is not repeated here.

[0298] In one possible implementation, after determining the static object included in the m+1th frame image, the method further includes: reducing the shading rate of the static object in the m+1th frame image, and rendering the m+1th frame image is based on the updated shading rate.

[0299] For a detailed description of reducing the shading rate of the static object in the (m+1)th frame image, please refer to the above method 400, in the case where m mod N≠1, and after determining the static object included in the (m)th frame image, the relevant description of reducing the shading rate of the static object in the (m)th frame image. For the sake of brevity, it is not repeated here.

[0300] In one possible implementation, rendering and generating the m+1th frame image based on the static pixels of the m-th frame image and at least part of the pixel values ​​in the m+1th group of image data includes: making correspondence between the m-th frame image and the screen space coordinates of the m+1th frame image; when the pixels of the first screen space corresponding to the m-th frame image and the m+1th frame image are both static pixels, reusing the pixels of the first screen space of the m-th frame image to render the first screen space of the m-th frame image; or, when the pixels of the second screen space corresponding to the m-th frame image and the m+1th frame image are both dynamic pixels, or when the pixels of the second screen space of the m-th frame image and the m+1th frame image are different types of pixels, rendering the second screen space of the m+1th frame image based on the pixels of the second screen space of the m+1th frame image.

[0301] For a detailed description of "rendering and generating the m+1th frame image based on the static pixels of the mth frame image and at least part of the pixel values ​​in the m+1th group of image data", please refer to the relevant description of "rendering and generating the mth frame image based on the mth group of image data when m mod N≠1" in the above method 400. For the sake of brevity, it will not be repeated here.

[0302] In one possible implementation, the method 1600 further includes: receiving the m+nth group of image data to be rendered sent by the application, where n>1 and x is an integer; rendering and generating the m+nth frame image based on the static pixels of the m+n-1th frame image and at least part of the pixel values ​​in the m+nth group of image data, where the static pixels of the m+n-1th frame image are obtained based on the m+n-1th group of image data.

[0303] In one example, for example, if n=2, then m+n is m+2. After rendering and generating the m+1th frame image, the terminal device may also receive image data of the next frame image of the m+1th frame image sent by the application, that is, the terminal device may receive the m+2th group of image data to be rendered sent by the application. The terminal device may render and generate the m+2th frame image based on the static pixels based on the m-th frame image and at least part of the pixel values ​​in the m+2th group of image data; or the terminal device may render and generate the m+2th frame image based on the static pixels based on the m+1th (based on m+2-1) frame image and at least part of the pixel values ​​in the m+2th group of image data.

[0304] In another example, for example, if n=5, then m+n is m+5. After rendering and generating the m+1th frame image, the terminal device may receive the m+5th set of image data to be rendered sent by the application. The terminal device may render and generate the m+5th frame image based on the static pixels based on the mth frame image and at least part of the pixel values ​​in the m+5th set of image data; or the terminal device may render and generate the m+5th frame image based on the static pixels based on the m+4th (based on m+5-1) frame image and at least part of the pixel values ​​in the m+5th set of image data.

[0305] Based on the above scheme, among the multiple frames of images to be rendered, the rendering generation of at least one frame of image reuses part of the pixel values ​​of the previous frame of image, which helps to reduce the power consumption of the shading calculation of the mobile terminal device, reduce the occurrence of heat, freezes and other phenomena, and thus improve the user experience.

[0306] In some examples, by capturing a screen shot of an application (eg, a game application) while it is running and zooming in and comparing non-overlapping portions of two consecutive motion frames, it is possible to observe a difference in shading rates between the two consecutive motion frames.

[0307] In some examples, RenderDoc is used to open the corresponding application (such as a game application), capture the instruction stream and shader, analyze the graphics library (GL) instructions or Vulkan instructions, view the motion frames for dynamic and static separation, reduce the shading rate of static objects in the motion frames, and increase rendering instructions for rendering reuse.

[0308] The present application also provides a terminal device including corresponding modules for executing the steps of any of the embodiments described in Figures 4, 5, 7, 8, 10, and 12 to 16. The modules included in the terminal device can be implemented in software and / or hardware.

[0309] An embodiment of the present application also provides a terminal device, which includes a memory and a processor, wherein the memory is used to store programs, and the processor is used to call and execute programs, so that the terminal device performs the steps of any one of the embodiments described in Figures 4, 5, 7, 8, 10 and 12 to 16 above.

[0310] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the steps of any one of the embodiments described in Figures 4, 5, 7, 8, 10 and 12 to 16 above.

[0311] In one possible design, the chip system further includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0312] The chip system can be composed of chips, or can include chips and other discrete devices.

[0313] An embodiment of the present application also provides a readable storage medium on which a program is stored. When the program is executed by a terminal device, the terminal device executes the steps of any one of the embodiments described in Figures 4, 5, 7, 8, 10 and 12 to 16 above.

[0314] An embodiment of the present application also provides a program product, including a program, which, when executed by a terminal device, enables the terminal device to execute the steps of any one of the embodiments described in Figures 4, 5, 7, 8, 10, and 12 to 16 above.

[0315] It should be understood that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above-mentioned method embodiment can be completed by hardware integrated logic circuits in the processor or by software instructions. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above-mentioned method.

[0316] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories.

[0317] The terms "unit", "module", etc. used in this specification may be used to represent device-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution.

[0318] Those skilled in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. In the several embodiments provided in this application, it should be understood that the disclosed devices, equipment, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not implemented. In addition, the coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, and can be electrical, mechanical, or other forms.

[0319] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.

[0320] In addition, the functional modules in the various embodiments of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more units may be integrated into one module.

[0321] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0322] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image rendering method, characterized in that: Applied to a terminal device, the method comprises: receiving an mth group of image data to be rendered sent by an application, wherein the image data includes a plurality of pixel values ​​used to generate an image, where m≥1, and m is an integer; Rendering and generating an m-th frame of image based on all pixel values ​​in the m-th group of image data; Receiving the (m+1)th group of image data to be rendered sent by the application; The m+1th frame image is generated by rendering based on static pixels of the mth frame image and at least part of pixel values ​​in the m+1th group of image data, wherein the static pixels of the mth frame image are obtained based on the mth group of image data.

2. The method according to claim 1, characterized in that Before rendering to generate the (m+1)th frame image, the method further includes: Based on the features of one or more rendering instructions corresponding to the yth frame image, the static pixels and dynamic pixels of the yth frame image are determined, y∈[m, m+1].

3. The method according to claim 2, characterized in that The determining of static pixels and dynamic pixels of the yth frame image based on features of one or more rendering instructions corresponding to the yth frame image includes: Determine static objects and dynamic objects included in the yth frame image based on a feature of each rendering instruction in the one or more rendering instructions; Determine the static pixels of the yth frame image based on the static object; The dynamic pixels of the yth frame image are determined based on the dynamic object.

4. The method according to claim 3, characterized in that After determining the static pixels and dynamic pixels of the yth frame image, the method further includes: The static pixels and the dynamic pixels are marked based on an A channel or a template buffer; wherein the static pixels and the dynamic pixels both include RGBA channels.

5. The method according to claim 2, characterized in that The determining of static pixels and dynamic pixels of the yth frame image based on features of one or more rendering instructions corresponding to the yth frame image includes: Determine static objects and dynamic objects included in the yth frame image based on the characteristics of each rendering instruction in the one or more rendering instructions to obtain a motion vector set of the yth frame image; Based on the motion vector set, the static pixels and dynamic pixels of the yth frame image are determined, and the value of the motion vector corresponding to the static object of the yth frame image is different from the value of the motion vector corresponding to the dynamic object of the yth frame image.

6. The method according to any one of claims 3 to 5, characterized in that After determining the static object included in the m+1th frame image, the method further includes: The shading rate of the static object in the (m+1)th frame image is reduced, and the (m+1)th frame image is rendered based on the updated shading rate.

7. The method according to any one of claims 2 to 6, characterized in that The characteristics of the rendering instruction include one or more of the following: A program alias, program variable name, or program index buffer name.

8. The method according to any one of claims 2 to 7, characterized in that The method further comprises: One or more rendering instructions corresponding to the y-th frame image are intercepted.

9. The method according to any one of claims 1 to 8, characterized in that The rendering based on the static pixels of the mth frame image and at least part of the pixel values ​​in the m+1th group of image data to generate the m+1th frame image includes: Matching the screen space coordinates of the m-th frame image with the m+1-th frame image; In a case where the pixels of the first screen space corresponding to the m-th frame image and the m+1-th frame image are both static pixels, the pixels of the first screen space of the m-th frame image are reused to render the first screen space of the m-th frame image; or When the pixels of the second screen space corresponding to the m-th frame image and the m+1-th frame image are both dynamic pixels, or when the pixels of the second screen space of the m-th frame image and the m+1-th frame image are different types, the second screen space of the m+1-th frame image is rendered based on the pixels of the second screen space of the m+1-th frame image.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: Receive the m+nth group of image data to be rendered sent by the application, where n>1 and n is an integer; Based on the static pixels of the mth frame image or the m+n-1th frame image, and at least part of the m+nth group of image data The pixel value rendering generates the m+nth frame image, and the static pixels of the m+n-1th frame image are obtained based on the m+n-1th group of image data.

11. An image rendering method, characterized in that: Applied to a terminal device, the method comprises: receiving an mth group of image data to be rendered sent by an application, wherein the mth group of image data includes a plurality of pixel values ​​used to generate an image, where m≥1, and m is an integer; Based on the mth group of image data, the mth frame image is generated by rendering; wherein, In the case where m mod N=1, the m-th frame image is rendered based on all pixel values ​​in the m-th group of image data; In the case where m mod N≠1, the m-th frame image is rendered based on static pixels of the ith frame image and at least part of pixel values ​​in the m-th group of image data, and the static pixels of the ith frame image are obtained based on the ith group of image data; Wherein, N is a preset integer, N≥2, 12. The method according to claim 11, characterized in that Before rendering to generate the m-th frame image, the method further includes: Based on features of one or more rendering instructions corresponding to the m-th frame image, static pixels and dynamic pixels of the m-th frame image are determined.

13. The method according to claim 12, characterized in that The determining of static pixels and dynamic pixels of the m-th frame image based on features of one or more rendering instructions corresponding to the m-th frame image includes: Determine static objects and dynamic objects included in the m-th frame image based on a feature of each rendering instruction in the one or more rendering instructions; determining the static pixel based on the static object; The dynamic pixels are determined based on the dynamic object.

14. The method according to claim 13, characterized in that After determining the static pixels and dynamic pixels of the m-th frame image, the method further includes: The static pixels and the dynamic pixels are marked based on an A channel or a template buffer; wherein the static pixels and the dynamic pixels both include RGBA channels.

15. The method according to claim 12, characterized in that The determining of static pixels and dynamic pixels of the m-th frame image based on features of one or more rendering instructions corresponding to the m-th frame image includes: Determine static objects and dynamic objects included in the m-th frame image based on a feature of each rendering instruction in the one or more rendering instructions to obtain a motion vector set of the m-th frame image; The static pixel and the dynamic pixel are determined based on the motion vector set, and the value of the motion vector corresponding to the static object is different from the value of the motion vector corresponding to the dynamic object.

16. The method according to any one of claims 13 to 15, characterized in that In the case where m mod N≠1, and after determining the static object included in the m-th frame image, the method further includes: The shading rate of the static object in the m-th frame image is reduced, and the rendering of the m-th frame image is performed based on the updated shading rate.

17. The method according to any one of claims 12 to 16, characterized in that The characteristics of the rendering instruction include one or more of the following: A program alias, program variable name, or program index buffer name.

18. The method according to any one of claims 12 to 17, characterized in that The method further comprises: One or more rendering instructions corresponding to the m-th frame image are intercepted.

19. The method according to any one of claims 11 to 18, characterized in that In the case where m mod N≠1, rendering and generating the m-th frame image based on the m-th group of image data comprises: Matching the screen space coordinates of the i-th frame image with the m-th frame image; In a case where the pixels of the first screen space corresponding to the i-th frame image and the m-th frame image are both static pixels, the pixels of the first screen space of the i-th frame image are reused to render the first screen space of the m-th frame image; or When the pixels of the second screen space corresponding to the i-th frame image and the m-th frame image are both dynamic pixels, or when the pixels of the second screen space of the i-th frame image and the m-th frame image are different types, the second screen space of the m-th frame image is rendered based on the pixels of the second screen space of the m-th frame image.

20. A terminal device, characterized in that: comprising a processor and a memory, wherein: The memory is used to store programs; The processor is used to call the program so that the terminal device executes the method according to any one of claims 1 to 19.

21. A readable storage medium having a program stored thereon, characterized in that: When the program is executed, the terminal device executes the method according to any one of claims 1 to 19.

22. A program product, characterized in that The method comprises a program, which, when executed, causes a terminal device to execute the method according to any one of claims 1 to 19.

Images