Image rendering method and related device

Abstract

Provided are an image rendering method and a related device, applied to a terminal device, the method comprising: receiving an mth group of image data to be rendered sent by an application, the image data comprising a plurality of pixel values for generating an image, m > = 1, m being an integer; rendering and generating an mth frame of image based on all pixel values in the mth group of image data; receiving to-be-rendered (m + 1) th group of image data sent by the application; and rendering and generating an (m + 1) th frame of image based on the static pixels of the mth frame of image and at least part of pixel values in the (m + 1) th group of image data, wherein the static pixels of the mth frame of image are obtained based on the mth group of image data. In the multi-frame image to be rendered, the rendering generation of at least one frame of image multiplexes the partial pixel value of the previous frame of image of the frame of image, so that the power consumption of the coloring calculation of the terminal equipment is reduced, the phenomena of heating, lagging and the like are reduced, and the use experience of a user can be further improved.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 29, 2023, with application number 202311102975.0 and application name “An image rendering method and related device”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to an image rendering method and related devices. Background Art

[0003] As screen resolution and refresh rates increase, the demand for shading calculations increases dramatically.

[0004] In the current image rendering method, the power consumption of shading calculation is relatively high. In the field of mobile rendering, it is limited by the computing power, bandwidth and heat dissipation of the mobile terminal. The mobile terminal will produce heating, lag and other phenomena, which will cause some applications (such as game applications) to limit frequency and frame, thus affecting the user experience.

[0005] Therefore, how to reduce the power consumption of shading calculations on mobile terminals has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The present application provides an image rendering method and related devices, in order to reduce the power consumption of shading calculations on a mobile terminal.

[0007] In the first aspect, the present application provides an image rendering method, which can be executed by a mobile terminal device, or the method can also be executed by a component (such as a chip, a chip system, etc.) configured in the mobile terminal device, or it can also be implemented by a logic module or software that can realize all or part of the functions of the mobile terminal device, and the present application does not limit this.

[0008] Exemplarily, the method includes: receiving an mth group of image data to be rendered sent by an application, the mth group of image data including a plurality of pixel values ​​for generating an image, m≥1, and m is an integer; rendering and generating an mth frame of image based on the mth group of image data; wherein, in the case of m mod N=1, the mth frame of image is rendered based on all pixel values ​​in the mth group of image data; in the case of m mod N≠1, the mth frame of image is rendered based on static pixels of an i-th frame of image and at least part of pixel values ​​in the mth group of image data, and the static pixels of the i-th frame of image are obtained based on the i-th group of image data; wherein N is a preset integer, N≥2,

[0009]

[0010] Among them, "mod" is the modulus operator, which is used to find the remainder when two numbers are divided; (m / N) rounded up.

[0011] Based on the above scheme, N frames of images are taken as a group, and the first frame of the N frames of images is obtained based on the currently known rendering process, that is, the first frame of the N frames of images can be rendered based on all pixel values ​​in the image data of the first frame of image, and at least one frame of the N frames of images except the first frame of image can be rendered based on static pixels of a frame of image before the image in the group of images, and at least part of the pixel values ​​in the image data of the image. In this way, the rendering generation of at least one frame of image in each group of images reuses part of the pixel values ​​of other images in the group, which helps to reduce the power consumption of shading calculations on the mobile terminal, reduce the occurrence of heat, freezes and other phenomena, and thus improve the user experience.

[0012] In combination with the first aspect, in some possible implementations, the method further includes: intercepting one or more rendering instructions corresponding to the m-th frame image.

[0013] In combination with the first aspect, in some possible implementations, before rendering to generate the mth frame image, the method also includes: determining the static pixels and dynamic pixels of the mth frame image based on the characteristics of one or more rendering instructions (Drawcall) corresponding to the mth frame image.

[0014] In combination with the first aspect, in some possible implementations, after determining the static pixels and dynamic pixels of the mth frame image, the method also includes: marking the static pixels and the dynamic pixels based on an alpha (A) channel or a stencil buffer; wherein the static pixels and the dynamic pixels both include RGBA channels.

[0015] In combination with the first aspect, in some possible implementations, the feature of the rendering instruction includes one or more of the following: a program label, a program uniform name, or a program index buffer name.

[0016] In combination with the first aspect, in some possible implementations, the static pixels and dynamic pixels of the mth frame image are determined based on the characteristics of one or more rendering instructions corresponding to the mth frame image, including: determining the static objects and dynamic objects included in the mth frame image based on the characteristics of each rendering instruction in the one or more rendering instructions; determining the static pixels based on the static objects; and determining the dynamic pixels based on the dynamic objects.

[0017] In combination with the first aspect, in some possible implementations, based on the characteristics of one or more rendering instructions corresponding to the m-th frame image, the static pixels and dynamic pixels of the m-th frame image are determined, including: based on the characteristics of each rendering instruction in the one or more rendering instructions, the static objects and dynamic objects included in the m-th frame image are determined to obtain a motion vector set of the m-th frame image; based on the motion vector set, the static pixels and the dynamic pixels are determined, and the value of the motion vector corresponding to the static object is different from the value of the motion vector corresponding to the dynamic object.

[0018] In combination with the first aspect, in some possible implementations, when m mod N≠1, and after determining the static objects included in the m-th frame image, the method also includes: reducing the shading rate of the static objects in the m-th frame image, and rendering the m-th frame image is based on the updated shading rate.

[0019] Reducing the shading rate of static objects can further reduce the power consumption of shading calculations on mobile devices, reduce the occurrence of heating, lags and other phenomena, and thus improve the user experience.

[0020] In combination with the first aspect, in some possible implementations, when m mod N≠1, rendering and generating an m-th frame image based on the m-th group of image data includes: making correspondence between the i-th frame image and the screen space coordinates of the m-th frame image; when the pixels of the first screen space corresponding to the i-th frame image and the m-th frame image are both static pixels, reusing the pixels of the first screen space of the i-th frame image to render the first screen space of the m-th frame image; or, when the pixels of the second screen space corresponding to the i-th frame image and the m-th frame image are both dynamic pixels, or when the pixels of the second screen space of the i-th frame image and the m-th frame image are different types of pixels, rendering the second screen space of the m-th frame image based on the pixels of the second screen space of the m-th frame image.

[0021] On the second aspect, the present application provides an image rendering method, which can be executed by a mobile terminal device, or the method can also be executed by a component (such as a chip, a chip system, etc.) configured in the mobile terminal device, or it can also be implemented by a logic module or software that can realize all or part of the functions of the mobile terminal device, and the present application does not limit this.

[0022] Exemplarily, the method includes: receiving the mth group of image data to be rendered sent by an application, the image data including multiple pixel values ​​for generating an image, m≥1, and m is an integer; rendering to generate an mth frame image based on all pixel values ​​in the mth group of image data; receiving the m+1th group of image data to be rendered sent by the application; rendering to generate an m+1th frame image based on static pixels of the mth frame image and at least part of the pixel values ​​in the m+1th group of image data, the static pixels of the mth frame image being obtained based on the mth group of image data.

[0023] Based on the above scheme, among the multiple frames of images to be rendered, the rendering generation of at least one frame of image reuses part of the pixel values ​​of the previous frame of image, which helps to reduce the power consumption of shading calculation of mobile terminal devices, reduce the occurrence of heat, freezes and other phenomena, and thus improve the user experience.

[0024] In combination with the second aspect, in some possible implementations, before rendering to generate the m+1th frame image, the method also includes: determining the static pixels and dynamic pixels of the yth frame image based on the characteristics of one or more rendering instructions corresponding to the yth frame image, y∈[m, m+1].

[0025] In combination with the second aspect, in some possible implementations, the static pixels and dynamic pixels of the yth frame image are determined based on the characteristics of one or more rendering instructions corresponding to the yth frame image, including: determining the static objects and dynamic objects included in the yth frame image based on the characteristics of each rendering instruction in the one or more rendering instructions; determining the static pixels of the yth frame image based on the static objects; and determining the dynamic pixels of the yth frame image based on the dynamic objects.

[0026] In combination with the second aspect, in some possible implementations, after determining the static pixels and dynamic pixels of the yth frame image, the method also includes: marking the static pixels and the dynamic pixels based on the A channel or the template buffer; wherein the static pixels and the dynamic pixels both include RGBA channels.

[0027] In combination with the second aspect, in some possible implementations, the static pixels and dynamic pixels of the yth frame image are determined based on the characteristics of one or more rendering instructions corresponding to the yth frame image, including: determining the static objects and dynamic objects included in the yth frame image based on the characteristics of each rendering instruction in the one or more rendering instructions to obtain a motion vector set of the yth frame image; determining the static pixels and dynamic pixels of the yth frame image based on the motion vector set, the value of the motion vector corresponding to the static object of the yth frame image is different from the value of the motion vector corresponding to the dynamic object of the yth frame image.

[0028] In combination with the second aspect, in some possible implementations, after determining the static objects included in the m+1th frame image, the method also includes: reducing the shading rate of the static objects in the m+1th frame image, and rendering the m+1th frame image is performed based on the updated shading rate.

[0029] Reducing the shading rate of static objects can further reduce the power consumption of shading calculations on mobile devices, reduce the occurrence of heating, lags and other phenomena, and thus improve the user experience.

[0030] In conjunction with the second aspect, in some possible implementations, the feature of the rendering instruction includes one or more of the following: a program alias, a program variable name, or a program index buffer name.

[0031] In combination with the second aspect, in some possible implementations, the method further includes: intercepting one or more rendering instructions corresponding to the yth frame image.

[0032] In combination with the second aspect, in some possible implementations, rendering and generating the m+1th frame image based on static pixels of the mth frame image and at least part of the pixel values ​​in the m+1th group of image data includes: making correspondence between the mth frame image and the screen space coordinates of the m+1th frame image; when the pixels of the first screen space corresponding to the m+1th frame image and the m+1th frame image are both static pixels, reusing the pixels of the first screen space of the m+1th frame image to render the first screen space of the m+1th frame image; or, when the pixels of the second screen space corresponding to the m+1th frame image and the m+1th frame image are both dynamic pixels, or when the pixels of the m+1th frame image and the second screen space of the m+1th frame image are different types of pixels, rendering the second screen space of the m+1th frame image based on the pixels of the second screen space of the m+1th frame image.

[0033] In combination with the second aspect, in some possible implementations, the method also includes: receiving the m+nth group of image data to be rendered sent by an application, where n>1 and n is an integer; rendering to generate the m+nth frame image based on static pixels of the m+n-1th frame image and at least part of the pixel values ​​in the m+nth group of image data, wherein the static pixels of the m+n-1th frame image are obtained based on the m+n-1th group of image data.

[0034] In combination with the first aspect and the second aspect, in some possible implementations, receiving the mth group of image data to be rendered sent by an application includes: receiving the mth group of image data to be rendered from a server of a first application, where the first application is any application of one or more preset applications.

[0035] It is understandable that one or more applications may be preset on the terminal device, for example, including but not limited to a whitelist of applications pre-stored in the terminal device for use in the image rendering method provided in the present application, and the whitelist of applications is not limited to being stored in the form of a list. The first application may be any application in the whitelist of applications. When the first application is running on the terminal device, the terminal device may receive the mth group of image data from the server of the first application.

[0036] In a third aspect, the present application provides a terminal device, which can be used to implement the method in the first aspect and any possible implementation of the first aspect, or to implement the method in the second aspect and any possible implementation of the second aspect. The terminal device includes a corresponding module for executing the above method. The module included in the terminal device can be implemented by software and / or hardware.

[0037] In a fourth aspect, the present application provides a terminal device, the terminal device comprising at least one processor and at least one communication interface. The processor is coupled to the communication interface and can be used to execute a program to implement the image rendering method in the first aspect and any possible implementation of the first aspect, or to implement the image rendering method in the second aspect and any possible implementation of the second aspect.

[0038] Optionally, the terminal device further includes a memory, and the processor is coupled to the memory.

[0039] In a fifth aspect, the present application provides a chip system comprising at least one processor for supporting the implementation of the functions involved in the above-mentioned first aspect and any possible implementation method of the first aspect, or for supporting the implementation of the functions involved in the above-mentioned second aspect and any possible implementation method of the second aspect, for example, processing the data involved in the above-mentioned method, etc.

[0040] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0041] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0042] In a sixth aspect, a readable storage medium is provided, on which a program (also referred to as code, or instruction) is stored. When the program is run, the method in the above-mentioned first aspect and any possible implementation of the first aspect is executed, or the method in the above-mentioned second aspect and any possible implementation of the second aspect is executed.

[0043] In the seventh aspect, a program product is provided, comprising: a program (also referred to as code, or instruction), wherein when the program is run, the method in the above-mentioned first aspect and any possible implementation of the first aspect is executed, or the method in the above-mentioned second aspect and any possible implementation of the second aspect is executed.

[0044] It should be understood that the third to seventh aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a structural schematic diagram of a terminal device applicable to the image rendering method provided in the embodiment of the present application;

[0046] Figure 2 It is a structural block diagram of software of a terminal device applicable to the image rendering method provided in the embodiment of the present application;

[0047] Figure 3 It is another structural block diagram of software of a terminal device applicable to the image rendering method provided in an embodiment of the present application;

[0048] Figure 4 is a schematic flow chart of an image rendering method provided in an embodiment of the present application;

[0049] Figure 5 is a schematic flow chart of a method for triggering the execution of an image rendering method provided by the present application, provided in an embodiment of the present application;

[0050] Figure 6 is a schematic diagram of a static-dynamic separation mask (Mask) provided in an embodiment of the present application;

[0051] Figure 7 It is a schematic flow chart of generating a dynamic and static separation Mask provided in an embodiment of the present application;

[0052] Figure 8 It is a schematic flow chart of whether to change the coloring rate provided in an embodiment of the present application;

[0053] Fig. 9 It is a structural block diagram of the software and hardware of a terminal device applicable to the image rendering method provided in an embodiment of the present application;

[0054] Fig.10 is a schematic flow chart of the image rendering method provided in the embodiment of the present application implemented on an operating system;

[0055] Fig.11It is another structural block diagram of the software and hardware of a terminal device applicable to the image rendering method provided in the embodiment of the present application;

[0056] Fig.12 is another schematic flow chart of the image rendering method provided in an embodiment of the present application;

[0057] Fig.13 is a schematic diagram of data involved in an image rendering method provided in an embodiment of the present application;

[0058] Fig.14 is another schematic flow chart of the image rendering method provided in an embodiment of the present application;

[0059] Fig.15 is another schematic diagram of data involved in an image rendering method provided in an embodiment of the present application;

[0060] Fig.16 It is a schematic flowchart of another image rendering method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0062] First, in the embodiments of the present application, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a device, system, product or equipment comprising a series of modules, modules or units is not necessarily limited to those modules, modules or units clearly listed, but may include other modules, modules or units that are not clearly listed or inherent to these devices, systems, products or equipment.

[0063] Second, in this application, "when...", "in the case of...", "if" and "if" all mean that the device will take corresponding actions under certain objective circumstances, but do not limit the time, nor do they require that the device must have a judgment action when it is implemented, nor do they mean that there are other limitations.

[0064] The following is a brief description of the terms involved in this application.

[0065] 1. Static objects: objects that move only as the viewing angle changes.

[0066] 2. Dynamic objects: objects that not only move with changes in viewing angle, but also move in geometry themselves.

[0067] 3. Static pixels: pixels corresponding to static objects.

[0068] 4. Dynamic pixels: pixels corresponding to dynamic objects.

[0069] 5. RGBA: It is a color space representing the red (red, R) channel, the green (green, G) channel, the blue (blue, B) channel and the A channel. Among them, the three RGB channels can be used to determine the actual color displayed by the pixel, and the A channel can be used to determine the transparency of the pixel.

[0070] 6. Screen space: In the rendering field, it can refer to the pixel coordinate space of a two-dimensional (2D) image generated during three-dimensional (3D) rendering. It is the 3D projection result of the geometry in the camera space. It consists of a two-dimensional array of pixels, and each pixel has color channels such as RGBA.

[0071] 7. Main scene: In the rendering of game images, it can be divided into game main scene drawing and game user interface (UI) drawing. The game main scene can include game terrain, vegetation, buildings, characters, special effects, etc.

[0072] 8. Open Graphics Library (OpenGL): Also known as "Open Graphics Library", it is a cross-language, cross-platform application programming interface (API) for rendering 2D and 3D vector graphics.

[0073] 9. Vulkan: is a cross-platform 2D and 3D graphics API.

[0074] As screen resolution and refresh rates increase, the demand for shading calculations increases dramatically.

[0075] In the current image rendering method, the power consumption of shading calculation is relatively high. In the field of mobile rendering, it is limited by the computing power, bandwidth and heat dissipation of the mobile terminal. The mobile terminal will produce heating, lag and other phenomena, which will cause some applications (such as game applications) to limit the frequency and frame, thus affecting the user experience.

[0076] In order to solve the above problems, the present application provides an image rendering method and related devices, with N (N≥2) frames of images as a group, and the first frame of the N frames of images is obtained based on the currently known rendering process, that is, the first frame of the N frames of images can be rendered based on all pixel values ​​in the image data of the first frame of the image, and at least one frame of the N frames of images other than the first frame of the image can be rendered based on static pixels of a frame of the image before the image in the group of images, and at least part of the pixel values ​​in the image data of the image. In this way, the rendering generation of at least one frame of the image in each group of images reuses part of the pixel values ​​of other images in the group, which helps to reduce the power consumption of the shading calculation of the mobile terminal, reduce the occurrence of heat, freezes and other phenomena, and thus improve the user experience.

[0077] Before describing in detail the image rendering method provided by the embodiment of the present application, Figure 1 , Figure 2 and Figure 3 , an exemplary description is given of a terminal device applicable to an embodiment of the present application.

[0078] The image rendering method provided in the embodiment of the present application can be applied to mobile terminal devices, such as but not limited to smart phones, tablet computers, wearable devices, vehicle-mounted devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, virtual reality (VR) devices (such as VR glasses), augmented reality (AR) devices, projectors, etc. The embodiment of the present application does not impose any restrictions on the specific type of the device.

[0079] In addition, the method described in the embodiments of the present application can support operating systems such as Android operating system (Android OS), Harmony OS, OpenHarmony OS, iOS, Windows OS, and lightweight operating systems (such as LiteOS). The embodiments of the present application do not impose any limitation on this.

[0080] Figure 1 It is a structural schematic diagram of a terminal device suitable for the image rendering method provided in the embodiment of the present application.

[0081] For example, Figure 1 FIG. 1 shows a schematic diagram of the structure of the terminal device 100. Figure 1As shown, the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L and a bone conduction sensor 180M, etc.

[0082] The processor 110 may include one or more processing units, for example, the processor 110 may include one or more of an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0083] The application processor outputs sound signals through the audio module 170 (such as the speaker 170A, etc.), or displays images or videos through the display screen 194.

[0084] The controller may be the nerve center and command center of the terminal device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0085] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0086] The processor 110 can perform different operations to achieve different functions by executing instructions. The instructions can be, for example, instructions pre-stored in the memory before the device leaves the factory, or instructions read from the APP after the user installs a new APP during use, and the embodiments of the present application do not impose any limitation on this.

[0087] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0088] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transmit data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other devices, such as augmented reality (AR) devices, etc. It can be understood that the interface connection relationship between the modules illustrated in this application is only a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0089] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal device 100. While the charging management module 140 is charging the battery 142, it may also power the terminal device 100 through the power management module 141.

[0090] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0091] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0092] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0093] The mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the terminal device 100 .

[0094] The wireless communication module 160 can provide wireless communication solutions applied to the terminal device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC) technology, infrared (IR) technology, etc.

[0095] In some embodiments, antenna 1 of terminal device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that terminal device 100 can communicate with the network and other devices through wireless communication technology.

[0096] The terminal device 100 can realize the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0097] The display screen 194, which may also be referred to as a screen, may be used to display images, videos, etc. It should be understood that the display screen 194 may also include more components. For example, a backlight panel, a drive circuit, etc. Among them, the backlight panel may be used to provide a light source, and the display panel emits light based on the light source provided by the backlight panel. The drive circuit may be used to control whether the liquid crystal of the liquid crystal layer is light-transmissive or light-impermeable.

[0098] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0099] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0100] The internal memory 121 can be used to store device executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the terminal device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, etc.

[0101] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, such as the speaker 170A, the receiver 170B, the microphone 170C and the earphone interface 170D, and the application processor.

[0102] It is to be understood that the structure illustrated in the present application does not constitute a specific limitation on the terminal device 100. In other embodiments, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0103] The software system of the terminal device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture or a cloud architecture, wherein the layered architecture divides the software system of the terminal device 100 into several layers, each layer has a clear role and division of labor, and the layers communicate with each other through software interfaces. This application takes the layered architecture of the OpenHarmony OS and Android operating systems as examples to exemplify the software structure of the terminal device 100, but should not limit the type of operating system of the terminal device used in the image rendering method provided in the embodiments of this application.

[0104] Figure 2 It is a structural block diagram of software of a terminal device suitable for the image rendering method provided in an embodiment of the present application.

[0105] like Figure 2 As shown, in some embodiments, the operating system is divided into four layers, from top to bottom, namely, the application layer, the framework layer, the system service layer, and the kernel layer. In some embodiments, the terminal device 100 also includes hardware, such as a GPU, a central processing unit (CPU), a display screen, etc.

[0106] The application layer can include a series of application programs. Figure 2 As shown, the application layer can include system applications and third-party applications. System applications can be understood as some applications that come with the system, such as camera, gallery, calendar, call, map, Bluetooth, video and music applications. Third-party applications can be understood as non-system applications, which can be downloaded and installed from the application store or application market, such as some game applications.

[0107] Figure 2 Not shown in the figure, the framework layer may include a UI framework, a user program framework, and a component framework. The framework layer provides the operating system's application with a user program framework and a component framework in multiple languages, such as C, C++, and JavaScript (JS), as well as APIs open to the outside world for various software and hardware services; it also provides devices with a UI framework in multiple languages, such as C, C++, and JS. The APIs supported by different devices are related to the degree of componentization of the system.

[0108] In the embodiments of the present application, Figure 2 As shown, the framework layer may also include an interception module, a static-dynamic recognition module, a pixel-level static-dynamic separation module, a rendering rate update module, a rendering multiplexing module, a reprojection module, and an image output module.

[0109] Among them, the interception module can be used to intercept the rendering instructions and image data of the image, that is, in the framework layer of the operating system of the terminal device in the present application, the graphics API call can be intercepted, and the relevant rendering instructions and image data can be cached. The static and dynamic recognition module can be used to identify whether the rendering instruction is a static rendering instruction or a dynamic rendering instruction based on the characteristics of the rendering instruction of the image, so as to determine the static objects and dynamic objects in the image. The pixel-level static and dynamic separation module can determine and distinguish the static pixels and dynamic pixels of the image based on the static objects and dynamic objects in the image determined by the static and dynamic recognition module, that is, it can be understood as separating the static pixels and dynamic pixels of the image. The rendering rate update module can be used to update the shading rate during rendering, for example, using a variable rate shading (VRS) mechanism to adjust the rendering rate, which can include but is not limited to reducing the shading rate for static objects and maintaining the original shading rate for dynamic objects. The rendering multiplexing module can be used to reuse some pixels in a frame image before the mth frame image during the rendering of the mth frame image. The reprojection module can be used to match the screen space coordinates of the m-th frame image with the screen space coordinates of a frame image before the m-th frame image. The image output module can output the image generated by rendering.

[0110] The system service layer is a collection of core capabilities of the operating system, providing various system services to applications through the framework layer. Figure 2Not shown in the figure, the system service layer may include a system basic capability subsystem set, a basic software service subsystem set, an enhanced software service subsystem set and a hardware service subsystem set.

[0111] Among them, the system basic capability subsystem set can be composed of subsystems such as distributed soft bus and distributed data management. The system basic capability subsystem set can provide basic capabilities for the operation, scheduling, migration and other operations of distributed applications on multiple devices.

[0112] The basic software service subsystem set can be composed of subsystems such as event notification, telephone, multimedia, etc., and can provide public and general software services.

[0113] The enhanced software service subsystem set can be composed of subsystems such as smart screen proprietary services, wearable proprietary services, and Internet of Things (IoT) proprietary services, and can provide differentiated capability-enhanced software services for different devices.

[0114] The hardware service subsystem set can be composed of subsystems such as location services, biometric recognition, wearable proprietary hardware services, IoT proprietary hardware services, etc., which can provide hardware services.

[0115] It should be noted that, according to the deployment environment of different device forms, the basic software service subsystem set, enhanced software service subsystem set and hardware service subsystem set can be cut according to the subsystem granularity, and each subsystem can be cut according to the functional granularity. This embodiment of the application does not make any limitation to this.

[0116] Figure 2 Not shown in the figure, the kernel layer may include a kernel subsystem and a driver subsystem. The operating system may adopt a multi-kernel design. Based on the kernel subsystem, the operating system supports the selection of suitable OS kernels for different resource-constrained devices; the kernel abstract layer (KAL) shields the differences between multiple kernels and provides basic kernel capabilities to the upper layer, including process management / thread management, memory management, file system, network management, and peripheral management. The driver framework in the driver subsystem is the basis for the openness of the operating system hardware ecosystem, and can provide unified peripheral access capabilities and driver development and management frameworks.

[0117] Figure 3 It is another structural block diagram of the software of the terminal device suitable for the image rendering method provided in the embodiment of the present application.

[0118] like Figure 3 As shown, in some embodiments, the Android operating system is divided from top to bottom into an application layer, an application framework layer, an Android runtime (Android runtime), a system library, and a kernel layer.

[0119] In some embodiments, the terminal device 100 also includes hardware, such as a GPU, a CPU, a display screen, etc.

[0120] The application layer can include a series of applications. Figure 3 As shown, the application layer can include system applications and third-party applications. System applications can be understood as some applications that come with the system, such as camera, gallery, calendar, call, map, Bluetooth, video and music applications. Third-party applications can be understood as non-system applications that can be downloaded and installed from the application store.

[0121] The application framework layer provides API and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 3 As shown, the application framework layer may include an interception module, a static and dynamic recognition module, a pixel-level static and dynamic separation module, a rendering rate update module, a rendering multiplexing module, a reprojection module, and an image output module. For a detailed description of the interception module, the static and dynamic recognition module, the pixel-level static and dynamic separation module, the rendering rate update module, the rendering multiplexing module, the reprojection module, and the image output module, please refer to Figure 2 For the sake of brevity, the relevant contents in are not repeated here.

[0122] Figure 3 Not shown in the figure, the Android runtime may include a core library and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android operating system.

[0123] The core library can include two parts: one part is the function that needs to be called by the Java language, and the other part is the core library of the Android operating system.

[0124] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0125] Figure 3 Although not shown in the figure, the system library may include multiple functional modules, such as status monitoring service, surface manager, media library, 3D graphics processing library (e.g., OpenGLES), 2D graphics engine (e.g., SGL), etc.

[0126] The kernel layer is the layer between hardware and software. Figure 3Not shown in the figure, the kernel layer may include power management services, sensor services (also called sensor drivers), display services (also called display drivers), camera drivers, audio drivers, etc. The present application embodiment does not impose any limitation on this.

[0127] It should be understood that Figure 1 , Figure 2 and Figure 3 This is only an example and should not limit the embodiments of the present application.

[0128] Figure 4 It is a schematic flowchart of an image rendering method provided in an embodiment of the present application.

[0129] like Figure 4 As shown, method 400 may include step 410 and step 420. The steps of method 400 may be performed by a terminal device of a mobile terminal (hereinafter referred to as a mobile terminal), or the method 400 may be performed by a component configured in the terminal device (such as a chip, a chip system, etc.), or may be implemented by a logic module or software capable of implementing all or part of the functions of the terminal device, which is not limited in the present embodiment of the application. The terminal device may have the following features: Figure 1 , Figure 2 or Figure 3 The structure shown in the embodiment of the present application is not limited to this. Figure 4 Detailed description of each step in.

[0130] In step 410, the mth group of image data to be rendered sent by the application is received.

[0131] The mth group of image data includes a plurality of pixel values ​​used to generate an image, m≥1, and m is an integer.

[0132] The terminal device may receive the mth group of image data to be rendered sent by the application.

[0133] In a possible implementation, receiving the mth group of image data to be rendered sent by an application includes: receiving the mth group of image data to be rendered of a first application, where the first application is any application of one or more preset applications.

[0134] It is understandable that one or more applications may be preset on the terminal device, for example, including but not limited to a whitelist of applications pre-stored in the terminal device for use in the image rendering method provided in the present application, and the whitelist of applications is not limited to being stored in the form of a list. The first application may be any application in the whitelist of applications. In an embodiment of the present application, the first application may be a system application or a third-party application. The first application may include but is not limited to game applications, VR applications, AR applications, and the like.

[0135] The terminal device may obtain the mth group of image data to be rendered of the first application based on the installation path and / or resource path of the first application. In a possible implementation, as an example but not a limitation, when the first application is running on the terminal device, the terminal device may receive the mth group of image data from the server of the first application.

[0136] Figure 5 It is a schematic flowchart of a method for triggering the execution of an image rendering method provided by the present application, provided in an embodiment of the present application.

[0137] like Figure 5 As shown, when an application is started and running on the terminal device, the terminal device can determine whether the application is an application in the application whitelist. If the application is an application in the above-mentioned pre-stored application whitelist applied to the image rendering method provided by this application, the terminal device can execute the image rendering method provided by this application; if the application is not an application in the above-mentioned pre-stored application whitelist applied to the image rendering method provided by this application, the terminal device can execute a known image rendering method.

[0138] In step 420, the mth frame image is generated by rendering based on the mth group of image data.

[0139] Wherein, when m mod N=1, the m-th frame image is rendered based on all pixel values ​​in the m-th group of image data; when m mod N≠1, the m-th frame image is rendered based on static pixels of the ith frame image and at least part of pixel values ​​in the m-th group of image data, and the static pixels of the ith frame image are obtained based on the ith group of image data; wherein N is a preset integer, N≥2,

[0140] As you can understand, "mod" is the modulus operator, which is used to find the remainder when two numbers are divided; (m / N) rounded up.

[0141] When the mth group of image data of the first application is received, the terminal device may render and generate the mth frame image based on the mth group of image data.

[0142] When m mod N=1, the terminal device can execute a currently known image rendering method based on the mth group of image data, that is, the terminal device can render based on all pixel values ​​in the mth group of image data to obtain the mth frame image; when m mod N≠1, the terminal device can execute the image rendering method provided in the embodiment of the present application based on the mth group of image data, that is, the terminal device can render based on static pixels of the i-th frame image and at least part of the pixel values ​​in the mth group of image data to obtain the mth frame image, wherein N is a preset positive integer, That is to say, the rendering generation of the mth frame reuses part of the pixel values ​​of a previous frame image, which helps to reduce the power consumption of shading calculations on the mobile terminal, reduce the occurrence of heat, freezes and other phenomena, and thus improve the user experience.

[0143] For example, take N=2 (which can be understood as two frames of images as a group) as an example. When m=5, 5mod 2=1, and the terminal device can render the fifth frame of image based on all pixel values ​​in the fifth group of image data; when m=6, 6mod 2=0≠1, i=5 (according to The terminal device may render the sixth frame of image based on the static pixels of the fifth frame of image and at least part of the pixel values ​​in the sixth group of image data.

[0144] It can be understood that, in the case of N=2, it can be understood that the image of the 2x-1th frame can be rendered based on all pixel values ​​in its image data; the image of the 2xth frame can be rendered based on static pixels of the image of the 2x-1th frame and at least part of the pixel values ​​in the image data of the 2xth frame. Wherein, x is a positive integer.

[0145] Another example, taking N=3 (which can be understood as three frames of images as a group) as an example, when m=4, 4mod 3=1, the terminal device can render the fourth frame of image based on all pixel values ​​in the fourth group of image data; when m=5, 5mod 3=2≠1, i=4 (according to The terminal device may render the fifth frame of image based on the static pixels of the fourth frame of image and at least part of the pixel values ​​in the fifth group of image data; in the case of m=6, 6mod 3=0≠1, i=4 or 5 (according to The terminal device may render the sixth frame image based on the static pixels of the fourth frame image or the fifth frame image and at least part of the pixel values ​​in the sixth group of image data.

[0146] It can be understood that, in actual application, every N frames of images form a group, and among these N frames of images, at least one frame of image can be rendered based on the static pixels of any frame of image preceding it in the group, and at least part of the pixel values ​​in the image data of the at least one frame of image. As an example but not a limitation, for example, N=3, in the second group of images (including the fourth frame of image, the fifth frame of image, and the sixth frame of image), the fourth frame of image can be rendered based on all the pixel values ​​in the fourth group of image data; the fifth frame of image can be rendered based on all the pixel values ​​in the fifth group of image data; and the sixth frame of image can be rendered based on the static pixels of the fifth frame of image, and at least part of the pixel values ​​in the sixth group of image data. For another example, the fourth frame of image can be rendered based on all the pixel values ​​in the fourth group of image data; the fifth frame of image can be rendered based on the static pixels of the fourth frame of image, and at least part of the pixel values ​​in the fifth group of image data; and the sixth frame of image can be rendered based on all the pixel values ​​in the sixth group of image data.

[0147] In a possible implementation, the method 400 further includes: intercepting one or more rendering instructions corresponding to the m-th frame image.

[0148] It should be understood that the application needs to send the rendering instructions to the GPU or CPU, and the interception action can be to intercept the rendering instructions before the rendering instructions reach the GPU or CPU. Since the transmission of rendering instructions needs to go through the operating system, from a technical implementation point of view, the application can also send the rendering instructions to the operating system, and the operating system performs additional processing on the rendering instructions and then sends them to the GPU or CPU through the operating system.

[0149] It can be understood that each frame of image may include one or more objects (static objects or dynamic objects), and each rendering instruction is used to render one object. Therefore, each frame of image may correspond to one or more rendering instructions.

[0150] The terminal device pre-stores an application whitelist for the image rendering method provided in this application. After the first application in the application whitelist is started and run on the terminal device, the terminal device can use Figure 2 or Figure 3 An interception module is used to intercept one or more rendering instructions corresponding to the m-th frame image and the m-th group of image data corresponding to the m-th frame image.

[0151] In a possible implementation, before rendering to generate the m-th frame image, the method 400 further includes: determining static pixels and dynamic pixels of the m-th frame image based on features of one or more rendering instructions corresponding to the m-th frame image.

[0152] After intercepting one or more rendering instructions corresponding to the m-th frame image, the terminal device can use Figure 2 or Figure 3 The static and moving recognition module and the pixel-level static and moving separation module are used to determine the static pixels and dynamic pixels of the m-th frame image based on the features of one or more rendering instructions corresponding to the m-th frame image.

[0153] In a possible implementation, the feature of the rendering instruction includes one or more of the following: a program alias, a program variable name, or a program index buffer name.

[0154] It can be understood that the rendering instruction may include one or more attributes. For example, a program alias, a program variable name, or a program index buffer name may be the name of the attribute included in the rendering instruction. The name of each attribute may correspond to a corresponding attribute value. The characteristics of the rendering instruction may include the name of the attribute included in the rendering instruction and the attribute value corresponding to the attribute name.

[0155] The present application provides implementation A and implementation B to implement "determining static pixels and dynamic pixels of the mth frame image based on the features of one or more rendering instructions corresponding to the mth frame image". Implementation A and implementation B are described in detail below.

[0156] Implementation method A: Determine the static pixels and dynamic pixels of the m-th frame image based on the characteristics of one or more rendering instructions corresponding to the m-th frame image, including: determining the static objects and dynamic objects included in the m-th frame image based on the characteristics of each rendering instruction in the one or more rendering instructions; determining the static pixels based on the static objects; and determining the dynamic pixels based on the dynamic objects.

[0157] The attribute values ​​of some attributes can be preset (including but not limited to being set or limited in the form of a list). When a rendering instruction includes any one of the preset attribute values, the terminal device can determine that the rendering instruction is a dynamic rendering instruction, that is, in the m-th frame image, the object rendered by the rendering instruction is a dynamic object; when a rendering instruction does not include all of the preset attribute values, the terminal device can determine that the rendering instruction is a static rendering instruction, that is, in the m-th frame image, the object rendered by the rendering instruction is a static object.

[0158] As an example but not limitation, the one or more rendering instructions corresponding to the m-th frame image include rendering instruction 1, rendering instruction 2 and rendering instruction 3, etc., wherein, for example, the attribute value of the program index buffer name attribute of rendering instruction 1 is “character index buffer (used to drive character movement or character skeletal movement)”, and the terminal device can determine that rendering instruction 1 is a dynamic rendering instruction based on this feature of rendering instruction 1, that is, in the m-th frame image, the object rendered by rendering instruction 1 is a dynamic object; for another example, the attribute value of the program variable name attribute of rendering instruction 2 is “character colour (used to color the character)”, and the terminal device can determine that rendering instruction 2 is a dynamic rendering instruction based on this feature of rendering instruction 2, that is, in the m-th frame image, the object rendered by rendering instruction 2 is a dynamic object.

[0159] That is to say, when one or more rendering instructions corresponding to the mth frame image are intercepted, the terminal device can use Figure 2 or Figure 3 The dynamic and static identification module shown analyzes each rendering instruction based on the preset attribute values ​​of these attributes, and determines whether the attribute value of the rendering instruction includes any one of the preset attribute values ​​of these attributes. If included, it is determined that in the m-th frame image, the object rendered by the rendering instruction is a dynamic object; otherwise, it is determined that in the m-th frame image, the object rendered by the rendering instruction is a static object.

[0160] After determining the static object and the dynamic object included in the m-th frame image, the terminal device may determine the static pixel based on the static object, and determine the dynamic pixel based on the dynamic object. Figure 2 or Figure 3 The pixel-level static-dynamic separation module shown determines the pixel area corresponding to the static object included in the m-th frame image as the area of ​​static pixels, and identifies these static pixels with a first identification value; determines the pixel area corresponding to the dynamic object included in the m-th frame image as the area of ​​dynamic pixels, and identifies these dynamic pixels with a second identification value, so that static pixels can be distinguished from dynamic pixels.

[0161] In a possible implementation, after determining the static pixels and dynamic pixels of the mth frame image, the method 400 further includes: marking the static pixels and the dynamic pixels based on the A channel or the template buffer; wherein the static pixels and the dynamic pixels both include RGBA channels.

[0162] As an example but not limitation, after determining the static pixels and static pixels of the mth frame image, the first identification value for identifying the static pixels and the second identification value for identifying the dynamic pixels can be written into the template buffer or the A channel in RGBA to generate a static and dynamic separation mask (Mask), and the Mask is saved to facilitate subsequent calculations and analysis based on the Mask.

[0163] Figure 6 It is a schematic diagram of a static-dynamic separation Mask provided in an embodiment of the present application.

[0164] As an example but not a limitation, "0" is used as an example of the first identification value, and "1" is used as an example of the second identification value. Figure 6 As shown, the pixel point with a value of "0" is a static pixel, and the pixel point with a value of "1" is a dynamic pixel.

[0165] In order to understand the implementation method A, the following Figure 7 The generation of the static-dynamic separation mask is explained again.

[0166] Figure 7 It is a schematic flowchart of generating a dynamic and static separation Mask provided in an embodiment of the present application.

[0167] Step 701: intercept a rendering instruction of a first application.

[0168] The first application is any one of the preset one or more applications. The terminal device may intercept the rendering instruction of the first application. Each frame of image may include one or more objects (static objects or dynamic objects), and each rendering instruction is used to render an object. Therefore, each frame of image may correspond to one or more rendering instructions. For example, the terminal device may intercept one or more rendering instructions corresponding to the mth frame of image.

[0169] Step 702, recording the characteristics of the rendering instruction.

[0170] The terminal device may record the characteristics of each rendering instruction in the one or more rendering instructions corresponding to the m-th frame image. The characteristics of the rendering instruction include one or more of the following: a program alias, a program variable name, or a program index buffer name.

[0171] Step 703, classifying dynamic rendering instructions into static rendering instructions according to the characteristics of the rendering instructions.

[0172] The terminal device can classify the one or more rendering instructions corresponding to the m-th frame image into dynamic and static rendering instructions based on the characteristics of each of the one or more rendering instructions corresponding to the m-th frame image, that is, determine whether each of the one or more rendering instructions corresponding to the m-th frame image is a static rendering instruction or a dynamic rendering instruction.

[0173] Step 704, generating a dynamic and static separation Mask.

[0174] After determining whether each rendering instruction in one or more rendering instructions corresponding to the m-th frame image is a static rendering instruction or a dynamic rendering instruction, the terminal device will know which objects are static and which objects are dynamic in the m-th frame image, so that the static pixels can be further determined based on the static objects, and the dynamic pixels can be determined based on the dynamic objects. Furthermore, a static-dynamic separation Mask can be generated based on these static pixels and dynamic pixels.

[0175] Implementation method B: Based on the characteristics of one or more rendering instructions corresponding to the m-th frame image, determine the static pixels and dynamic pixels of the m-th frame image, including: based on the characteristics of each rendering instruction in the one or more rendering instructions, determine the static objects and dynamic objects included in the m-th frame image to obtain a motion vector set of the m-th frame image; based on the motion vector set, determine the static pixels and the dynamic pixels, the value of the motion vector corresponding to the static object is different from the value of the motion vector corresponding to the dynamic object.

[0176] In the implementation B, when one or more rendering instructions corresponding to the mth frame image are intercepted, the terminal device can also use Figure 2 or Figure 3 The static and dynamic identification module shown analyzes each rendering instruction based on the attribute value of the preset attribute, and determines whether the attribute value of the rendering instruction includes any one of the attribute values ​​of the preset attribute. In the case where the attribute value of the rendering instruction includes any one of the attribute values ​​of the preset attribute, it is determined that in the m-th frame image, the object rendered by the rendering instruction is a dynamic object; otherwise, it is determined that in the m-th frame image, the object rendered by the rendering instruction is a static object. For a detailed description of "determining the static objects and dynamic objects included in the m-th frame image based on the characteristics of one or more rendering instructions corresponding to the m-th frame image", please refer to the relevant description in the above-mentioned implementation method A, and for the sake of brevity, it will not be repeated here.

[0177] After determining the static objects and dynamic objects included in the m-th frame image, the terminal device can obtain a motion vector set of the m-th frame image based on the static objects and the dynamic objects. In the motion vector set of the m-th frame image, the value of the motion vector corresponding to the static object is different from the value of the motion vector corresponding to the dynamic object. As an example but not a limitation, for example, the value of the motion vector corresponding to the static object is zero, and the value of the motion vector corresponding to the dynamic object is non-zero. Therefore, the terminal device can determine the static pixels corresponding to the static object and the dynamic pixels corresponding to the dynamic object based on the motion vector set of the m-th frame image.

[0178] In one possible implementation, when m mod N≠1, and after determining the static objects included in the m-th frame image, method 400 further includes: reducing the shading rate of the static objects in the m-th frame image, and rendering the m-th frame image is performed based on the updated shading rate.

[0179] In the case where m mod N≠1, and after determining the static object included in the m-th frame image, the terminal device can use the method described in 2 or Figure 3 The rendering rate update module in the image processing module is used to update the shading rate when rendering the mth frame image. For example, the terminal device can reduce the shading rate of static objects in the mth frame image, while maintaining the original shading rate of dynamic objects. This is not limited in the present application. Reducing the shading rate of static objects can further reduce the power consumption of shading calculations on the mobile terminal, reduce the occurrence of heat, freezes, and other phenomena, and thus improve the user experience.

[0180] It is understood that in the case where m mod N = 1, the terminal device does not need to use 2 or Figure 3 The rendering rate update module in the image processing module is used to update the shading rate when rendering the m-th frame image. The terminal device can render the m-th frame image based on the currently known image rendering method.

[0181] For example, taking N=2, when m is 2x, after determining the static objects and dynamic objects included in the m-th frame image, the terminal device can reduce the shading rate of the static objects in the m-th frame image, while maintaining the shading rate of the dynamic objects in the m-th frame image; when m is 2x-1, the terminal device can render the m-th frame image based on currently known image rendering methods.

[0182] Figure 8 It is a schematic flowchart of whether to update the shading rate provided in an embodiment of the present application.

[0183] Step 801, dynamic and static rendering instruction identification.

[0184] In the case of m mod N≠1, the terminal device can perform dynamic and static rendering instruction identification on each of the one or more rendering instructions corresponding to the m-th frame image. For the convenience of description, in the embodiment of the present application, any one of the one or more rendering instructions corresponding to the m-th frame image is recorded as a "first rendering instruction". The terminal device can perform dynamic and static rendering instruction identification on the first rendering instruction based on the characteristics of the first rendering instruction, that is, determine whether the first rendering instruction is a static rendering instruction or a dynamic rendering instruction. The first rendering instruction is a static rendering instruction, that is, in the m-th frame image, the object rendered by the rendering instruction is a static object; the first rendering instruction is a dynamic rendering instruction, that is, in the m-th frame image, the object rendered by the rendering instruction is a dynamic object.

[0185] Step 802, determining whether it is a static rendering instruction.

[0186] After performing dynamic and static rendering instruction identification on the first instruction, the terminal device can determine whether the first rendering instruction is a static rendering instruction. If the first rendering instruction is a static rendering instruction, the terminal device can execute step 703; if the first rendering instruction is a dynamic rendering instruction, the terminal device can execute step 804.

[0187] Step 803, reduce the shading rate.

[0188] In the case where the first rendering instruction is a static rendering instruction, the terminal device may reduce the shading rate of the static object rendered by the first rendering instruction.

[0189] Step 804: the coloring rate remains unchanged.

[0190] In the case where the first rendering instruction is a dynamic rendering instruction, the terminal device may maintain a shading rate of the dynamic object rendered by the first rendering instruction unchanged.

[0191] Table 1 is a reference table of various coloring rates provided in the embodiments of the present application.

[0192] Table 1

[0193]

[0194]

[0195] Among the multiple shading rates shown in Table 1, the shading rate 1×1 corresponding to number 1 is the largest, and the shading rate 4×4 corresponding to number 7 is the smallest. For example, the shading rate of the static object rendered by the first rendering instruction is the shading rate 1×2 corresponding to number 2. When the terminal device determines that the first rendering instruction is a static rendering instruction, the terminal device can reduce the shading rate of the static object rendered by the first rendering instruction from the shading rate 1×2 corresponding to number 2 to the shading rate corresponding to any number from number 3 to number 7.

[0196] It is understandable that in actual application scenarios, multiple shading rates may be included but not limited to those shown in Table 1 above.

[0197] In a possible implementation, when m mod N≠1, rendering and generating an m-th frame image based on the m-th group of image data includes: making correspondence between the i-th frame image and the screen space coordinates of the m-th frame image; when the pixels of the first screen space corresponding to the i-th frame image and the m-th frame image are both static pixels, reusing the pixels of the first screen space of the i-th frame image to render the first screen space of the m-th frame image; or, when the pixels of the second screen space corresponding to the i-th frame image and the m-th frame image are both dynamic pixels, or when the pixels of the second screen space of the i-th frame image and the m-th frame image are different types of pixels, rendering the second screen space of the m-th frame image based on the pixels of the second screen space of the m-th frame image.

[0198] The terminal device may use a reprojection method to make a one-to-one correspondence between the screen space coordinates of the i-th frame image and the m-th frame image. As an example but not a limitation, in the case where m mod N≠1, the terminal device may use the following method: Figure 2 Or the reprojection module shown in 3 corresponds the screen space coordinates of the i-th frame image and the m-th frame image.

[0199] In more detail, the view projection transformation (VP) matrix of each frame of the main scene (i.e., image) can be obtained by using the operating system graphics interface hook layer of the terminal device. For example, the VP matrix includes but is not limited to a 4×4 matrix. The terminal device can also obtain the image data of each frame of the image, and the image data includes but is not limited to a pixel matrix, depth information (i.e., depth value) and pixel coordinates (used to indicate the position of the pixel point). The terminal device can correspond the i-th frame image to the screen space coordinates of the m-th frame image one by one based on the depth information, pixel coordinates and inverse matrix of the VP matrix of the i-th frame image, and the VP matrix of the m-th frame image. As an example and not a limitation, the depth information, pixel coordinates and W value of the i-th frame image are combined to obtain a matrix, for example, the depth information is a 1×1 matrix, the pixel coordinates are a 2×1 matrix, and the W value is a 1×1 matrix (for example, the W value is 1.0), then the depth information, pixel coordinates and W value of the i-th frame image are combined to obtain a 4×1 matrix, and for ease of description, the matrix is ​​recorded as a "combined matrix". For another example, the VP matrices of the i-th frame image and the m-th frame image are both 4×4 matrices. Then, the combination matrix × the inverse matrix of the VP matrix of the i-th frame image × the VP matrix of the m-th frame image can establish a one-to-one correspondence between the screen space coordinates of the i-th frame image and the m-th frame image, that is, a one-to-one correspondence can be established between the pixels of the i-th frame image and the pixels of the m-th frame image.

[0200] After establishing a one-to-one correspondence between the pixels of the i-th frame image and the pixels of the m-th frame image, the terminal device can determine which part of the static pixels in the i-th frame image can be reused by the m-th frame image based on the static and dynamic separation Mask of the i-th frame image and the static and dynamic separation Mask of the m-th frame image.

[0201] Table 2 is a schematic diagram of pixel reuse rules provided in an embodiment of the present application.

[0202] Table 2

[0203] Pixels of the i-th group of image data The pixels of the mth group of images The pixel source when rendering the mth frame image Static Static The i-th group of image data Static dynamic The mth group of image data dynamic Static The mth group of image data dynamic dynamic The mth group of image data

[0204] As shown in Table 2, based on the static-dynamic separation Mask of the i-th frame image and the static-dynamic separation Mask of the m-th frame image, when the pixels of the i-th group image data in the first screen space and the pixels of the m-th group image data in the first screen space are both static pixels, that is, when the pixels of the first screen space corresponding to the i-th frame image and the m-th frame image are both static pixels, the terminal device can use Figure 2 or Figure 3The rendering multiplexing module shown reuses the static pixels of the first screen space of the i-th frame image to render the first screen space of the m-th frame image. That is to say, when the pixels of the i-th group of image data in the first screen space and the pixels of the m-th group of image data in the first screen space are both static pixels, the pixels when rendering the first screen space of the m-th frame image are derived from the i-th group of image data. In this way, the pixels of the i-th group of image data in the first screen space are reused, which helps to reduce the power consumption of the shading calculation of the mobile terminal, reduce the occurrence of heat, freezes and other phenomena, and thus improve the user experience.

[0205] Based on the static-dynamic separation mask of the i-th frame image and the static-dynamic separation mask of the m-th frame image, when the pixels of the i-th group image data in the second screen space and the pixels of the m-th group image data in the second screen space are both dynamic pixels, that is, when the pixels of the second screen space corresponding to the i-th frame image and the m-th frame image are both dynamic pixels, the terminal device can render the second screen space of the m-th frame image based on the pixels of the m-th group image data in the second screen space. That is to say, when the pixels of the i-th group image data in the second screen space and the pixels of the m-th group image data in the second screen space are both dynamic pixels, the pixels when rendering the second screen space of the m-th frame image are derived from the m-th group image data, and the pixels of the i-th group image data in the second screen space are not reused.

[0206] Based on the static-dynamic separation mask of the i-th frame image and the static-dynamic separation mask of the m-th frame image, when the type of pixels of the i-th group image data in the second screen space is different from the type of pixels of the m-th group image data in the second screen space, that is, when the second screen space of the i-th frame image and the m-th frame image is different types of pixels, for example, the pixels of the i-th group image data in the second screen space are static pixels, and the pixels of the m-th group image data in the second screen space are dynamic pixels, or the pixels of the i-th group image data in the second screen space are dynamic pixels, and the pixels of the m-th group image data in the second screen space are static pixels, then the terminal device can render the second screen space of the m-th frame image based on the pixels of the m-th group image data in the second screen space. That is to say, when the type of pixels of the i-th group image data in the second screen space is different from the type of pixels of the m-th group image data in the second screen space, the pixels when rendering the second screen space of the m-th frame image are derived from the m-th group image data, and the pixels of the i-th group image data in the second screen space are not reused.

[0207] It can be understood that the first screen space is a screen space where the pixels of the i-th frame image and the m-th frame image are both static, and the second screen space is a screen space where the pixels of the i-th frame image and the m-th frame image are both dynamic, or the types of pixels of the i-th frame image and the m-th frame image are different.

[0208] Based on the above scheme, N (N≥2) frames of images are taken as a group, and the first frame of the N frames is obtained based on the currently known rendering process, that is, the first frame of the N frames can be rendered based on all pixel values ​​in the image data of the first frame, and at least one frame of the N frames except the first frame can be rendered based on static pixels of a frame before the image in the group of images, and at least part of the pixel values ​​in the image data of the image. In this way, the rendering generation of at least one frame of each group of images reuses part of the pixel values ​​of other images in the group, which helps to reduce the power consumption of the shading calculation on the mobile terminal, reduce the occurrence of heat, freezes and other phenomena, and thus improve the user experience.

[0209] In order to better understand the image rendering method provided in the embodiment of the present application, the following is combined with Fig. 9 , the software and hardware structure of the terminal device suitable for the image rendering method provided in the embodiment of the present application is described again.

[0210] Fig. 9 It is another structural block diagram of the software of the terminal device applicable to the image rendering method provided in the embodiment of the present application.

[0211] like Fig. 9 As shown, the application layer of the terminal device can be deployed with multiple applications suitable for the image rendering method provided in the embodiment of the present application, for example, game application one, game application two, and game application three, etc. The application layer can also be deployed with a game engine, for example, the game engine can include but is not limited to the Unity engine.

[0212] The framework layer of the terminal device can be deployed with interception module, static and dynamic recognition module, pixel-level static and dynamic separation module, rendering rate update module, rendering multiplexing module, reprojection module and image output module, etc. For detailed description of these modules, please refer to the above Figure 2 For the sake of brevity, the relevant descriptions in the figure are not repeated here. In one possible implementation, the terminal device may be deployed with a graphics interface Hook layer, and an interception module, a static and dynamic recognition module, a pixel-level static and dynamic separation module, a rendering rate update module, a rendering multiplexing module, a reprojection module, and an image output module may be deployed in the graphics interface Hook layer. The graphics interface Hook layer can intercept and process rendering instructions, and can also send the processed rendering instructions to the GPU or CPU.

[0213] The driver layer of the terminal device may be deployed with a graphics driver. For example, the graphics driver may include but is not limited to OpenGL (OpenGL ES) for embedded systems (ES).

[0214] The hardware layer of the terminal device may be deployed with a GPU and a CPU. The GPU may include but is not limited to an Arm Mali processor or a Qualcomm Adreno processor, etc. The CPU may include but is not limited to a Kirin 9000 processor or a Qualcomm processor, etc.

[0215] In the present application, the terminal device can intercept graphics API calls at the operating system framework layer, cache relevant instructions and associated data, and take N=2 as an example. The 2x-1 frame image can maintain the original rendering process and only separate dynamic objects from static objects. For the rendering process of the 2x frame image, the characteristics of the instruction stream can be combined to identify whether the DrawCall is dynamic or static. For the DrawCall of dynamic objects, the original shading rate is maintained, and for the DrawCall of static objects, the shading rate is reduced. In the post-processing part of the rendering, a static object rendering reuse solution is added to reuse the static pixels of the 2x-1 frame image into the 2x frame image, and finally the result image is output.

[0216] In some examples, Drawcall may be a drawing call interface, which may refer to a process in which a CPU sends a drawing instruction to a GPU.

[0217] Fig.10 It is a schematic flowchart of the image rendering method provided in the embodiment of the present application implemented on the operating system.

[0218] It is understandable that the software code (eg, program) for implementing the image rendering method provided in the embodiment of the present application may be deployed in the instruction reassembly layer of the operating system of the terminal device. Fig.10 As shown, in the embodiment of the present application, the terminal device can intercept the rendering instruction issued by the first application at the instruction reorganization layer, and the reorganization layer may include Figure 2 or Figure 3 The interception module, static-motion recognition module, pixel-level static-motion separation module, rendering rate update module, rendering multiplexing module and reprojection module are shown.

[0219] Step 1001: The rendering interface of the first application is called.

[0220] The first application is any one of one or more applications preset to be applicable to the image rendering method provided in this application. When the rendering interface of the first application is called, the terminal device can be triggered to execute the image rendering method provided in this application.

[0221] Step 1002, intercepting rendering instructions.

[0222] The terminal device may intercept one or more rendering instructions corresponding to the mth frame image of the first application.

[0223] Step 1003, identifying dynamic objects and static objects.

[0224] The terminal device may identify the dynamic objects and static objects included in the m-th frame image based on the features of each rendering instruction in the one or more rendering instructions corresponding to the m-th frame image.

[0225] It can be understood that, in the image rendering method provided in the present application, no matter m satisfies m mod N = 1 or m mod N ≠ 1, for the mth frame image of the first application, step 1002 and step 1003 need to be executed.

[0226] Step 1004, original drawing process.

[0227] In the case where m mod N=1, after intercepting one or more rendering instructions corresponding to the m-th frame image, the terminal device may execute an original drawing process in a known image rendering method based on a known image rendering method.

[0228] Step 1005, update the shading rate.

[0229] In the case where m mod N≠1, after executing step 1003, the terminal device may reduce the shading rate of static objects and maintain the shading rate of dynamic objects.

[0230] Step 1006, reprojection.

[0231] When the mth frame image reuses some pixels in the i-th frame image, the terminal device can reproject based on the m-th group of image data and the i-th group of image data to correspond the i-th frame image to the screen space coordinates of the m-th frame image.

[0232] Step 1007, separating static and dynamic pixels.

[0233] Regardless of whether m satisfies m mod N = 1 or m mod N ≠ 1, for the mth frame image of the first application, after executing step 1003, the terminal device can obtain static pixels and dynamic pixels based on the static objects and dynamic objects in the mth frame image to distinguish the static pixels from the dynamic pixels of the mth frame image.

[0234] Step 1008, static pixel rendering multiplexing.

[0235] In the case where m mod N≠1, the terminal device may reuse some static pixels in the i-th frame image when rendering and generating the m-th frame image based on the pixel reuse rule in Table 2 above.

[0236] Fig.11 It is another structural block diagram of the software and hardware of a terminal device suitable for the image rendering method provided in an embodiment of the present application.

[0237] like Fig.11 As shown, applications applicable to the image rendering method provided in the embodiment of the present application may include but are not limited to game applications, VR applications or AR applications, etc., and may also include other applications. A terminal device applicable to the image rendering method provided in the embodiment of the present application may include an instruction reset layer, which may be used to obtain calls to the rendering API, cache the rendering instruction stream and data related to the rendering instruction stream (for example, including but not limited to image data). A terminal device applicable to the image rendering method provided in the embodiment of the present application may include a module for executing the original drawing process, and may also include the modules provided in the present application (for example, including but not limited to Figure 2 or Figure 3 The motion-static recognition module, pixel-level motion-static separation module, rendering rate update module, rendering multiplexing module and reprojection module shown in FIG.

[0238] In order to better understand the image rendering method provided by this application, the following Figures 12 to 15 , further illustrating the image rendering method provided by this application.

[0239] Fig.12 is another schematic flowchart of the image rendering method provided in an embodiment of the present application.

[0240] like Fig.12 As shown, taking N=2 as an example, the terminal device processes the 2x-1th group of image data of the first application differently from the 2xth group of image data of the first application. For example, the terminal device can perform steps 1201 and 1202 in the process of rendering and generating the 2xth frame image; the terminal device can perform steps 1203 to 1208 in the process of rendering and generating the 2xth frame image. The rendering process of the adjacent 2x-1th frame image and the 2xth frame image is described below.

[0241] Step 1201: Identify dynamic and static rendering instructions for the 2x-1th frame image.

[0242] The terminal device can obtain the 2x-1th group of image data of the first application and the rendering instruction corresponding to the 2x-1th frame image, and then the terminal device can determine whether the rendering instruction is a dynamic rendering instruction or a static rendering instruction based on the rendering instruction corresponding to the 2x-1th frame image.

[0243] Step 1202: Normal rendering process.

[0244] The terminal device can perform a normal rendering process based on the 2x-1th group of image data to render and generate the 2x-1th frame image.

[0245] Step 1203: Generate a static and dynamic separation Mask for the 2x-1th frame image.

[0246] Before rendering and generating the 2xth frame image, the terminal device may generate a static and dynamic separation Mask for the 2x-1th frame image after executing step 1201. For a detailed description of generating the static and dynamic separation Mask for the 2x-1th frame image, please refer to the relevant description in the above implementation method A, which will not be repeated here for the sake of brevity.

[0247] Step 1204: Identify dynamic and static rendering instructions for the 2xth frame image.

[0248] The terminal device can obtain the 2xth group of image data of the first application and the rendering instruction corresponding to the 2xth frame image, and then the terminal device can determine whether the rendering instruction is a dynamic rendering instruction or a static rendering instruction based on the rendering instruction corresponding to the 2xth frame image.

[0249] Step 1205: Generate a static and dynamic separation Mask for the 2xth frame image.

[0250] Before rendering and generating the 2xth frame image, the terminal device may generate a static and dynamic separation Mask for the 2xth frame image after executing step 1204. For a detailed description of generating the static and dynamic separation Mask for the 2xth frame image, please refer to the relevant description in the above implementation method A, which will not be repeated here for the sake of brevity.

[0251] Step 1206: Reduce the shading rate of static objects.

[0252] After the terminal device executes step 1204, the terminal device determines which part of the 2xth frame image is a static object and which part is a dynamic object. Thus, the terminal device can reduce the shading rate of static objects while maintaining the shading rate of dynamic objects.

[0253] Step 1207: Reprojection.

[0254] The terminal device can use the reprojection method to make a one-to-one correspondence between the screen space coordinates of the odd frame image and the even frame image based on the reprojection method, so as to facilitate the subsequent determination of which part of the pixels of the odd frame image can be reused by the even frame image.

[0255] Step 1208: Static pixel shading reuse.

[0256] After executing step 1203, step 1205, step 1206 and step 1207, the terminal device can render the 2xth frame image. In the process of rendering and generating the 2xth frame image, some static pixels in the 2x-1th frame image are reused, and the rendering rate of the static objects in the 2xth frame image is based on the shading rate reduced in step 1206.

[0257] It can be understood that in the embodiment of the present application, there is no limitation on the order of step 1203, step 1205, step 1206 and step 1207.

[0258] Fig.13 It is a schematic diagram of data involved in an image rendering method provided in an embodiment of the present application.

[0259] like Fig.13 As shown, still taking N=2 as an example, the terminal device processes the 2x-1th group of image data of the first application differently from the 2xth group of image data of the first application. Fig.12 Taking the process of the image rendering method shown in as an example, in the process of rendering and generating the 2x frame image, reprojection and static pixel shading multiplexing are required. Among them, part of the image data of the 2x-1 frame image and part of the image data of the 2x frame image are required for reprojection. The image data may include but is not limited to the pixel matrix, depth information, pixel coordinates and VP matrix of the image. For example, the depth information, pixel coordinates and VP matrix of the 2x-1 frame image, as well as the VP matrix of the 2x frame image are required for reprojection. Static pixel shading multiplexing requires the static and dynamic separation mask of the 2x-1 frame pixel and the image data of the 2x-1 frame image, as well as the static and dynamic separation mask of the 2x frame pixel and the image data of the 2x frame image.

[0260] Fig.14 This is another schematic flowchart of the image rendering method provided in an embodiment of the present application.

[0261] and Fig.12 The difference between the image rendering method shown in is that Fig.14 The image rendering method shown in the process will use more native rendering data of the game application itself.

[0262] like Fig.14 As shown, still taking N=2 as an example, the terminal device processes the 2x-1th group of image data of the first application differently from the terminal device processes the 2xth group of image data of the first application.

[0263] exist Fig.14 In the image rendering method shown in the flow chart, the processing process of the 2x-1 group of image data can be referred to Fig.12 Steps 1201 and 1202 in the above are not described in detail here for the sake of brevity.

[0264] exist Fig.14 In the image rendering method flow shown, the process of processing the 2x-1th group of image data may include steps 1401 to 1403.

[0265] Step 1401, identifying dynamic and static rendering instructions for the 2xth frame image.

[0266] For a detailed description of step 1401, please refer to the above Fig.12 The relevant description of step 1204 in is not repeated here for the sake of brevity.

[0267] Step 1402, reduce the shading rate of static objects.

[0268] For a detailed description of step 1402, please refer to the above Fig.12 The relevant description of step 1206 in is not repeated here for the sake of brevity.

[0269] Step 1403, execution steps of the updated TAA module.

[0270] It is understandable that in this implementation, the updated TAA module can replace the currently known native TAA module. The updated TAA module adds the ability of static pixel shading multiplexing on the basis of the native TAA module. The terminal device can perform static pixel shading multiplexing based on the updated TAA module after executing the steps required by the native TAA module (including but not limited to softening the edges of objects) to render and generate the 2x frame image.

[0271] Fig.15 This is another schematic diagram of the data involved in the image rendering method provided in an embodiment of the present application.

[0272] like Fig.15 As shown, taking N=2 as an example, the terminal device's processing process for the 2x-1th group of image data of the first application is different from the processing process for the 2xth group of image data of the first application. In the process of rendering and generating the 2xth frame image, static pixel shading multiplexing is required.

[0273] like Fig.15As shown, since some game applications can distinguish between dynamic objects and static objects in the native rendering process, after the dynamic and static separation, the dynamic and static separation Mask is no longer generated, but the static pixels and dynamic pixels can be distinguished based on the motion vector set. After the execution steps required by the native TAA module are executed, the shading rate is controlled and the static pixel shading is reused. For example, the image data of the 2x-1 frame image and the motion vector set of the 2x-1 frame image, as well as the image data of the 2x frame image and the motion vector set of the 2x frame image, can be input into the native TAA module. After the native TAA module completes anti-aliasing, the terminal device can use the motion vector set of the 2x-1 frame image and the motion vector set of the 2x frame image to distinguish between static pixels and dynamic pixels. Based on the pixel reuse rules in Table 2 above, when rendering and generating the 2x frame image, some static pixels in the 2x-1 frame image of the previous frame are reused, and finally the 2x frame image is output.

[0274] In some examples, the screen displayed by an electronic device such as a mobile phone, such as a photo, a video, or a game screen, etc. Take a photo as an example. Each pixel of the photo may include four RBGA channels. Among them, the three RGB channels can be used to determine the color actually displayed by the pixel, and the A channel is used to determine the transparency of the pixel. For photos with the same A channel value, for example, the A channel value is 1, or the A channel value of the screen displayed by the mobile phone defaults to 1. In some scenarios where the A channel is not enabled, the A channel can be used to record other information, for example, it can be marked whether the pixel is dynamic or static.

[0275] Fig.16 It is a schematic flowchart of another image rendering method provided in an embodiment of the present application.

[0276] like Fig.16 As shown, method 1600 may include steps 1610 to 1640. The steps of method 1600 may be performed by a terminal device of a mobile terminal (hereinafter referred to as a mobile terminal), or the method 1600 may be performed by a component configured in the terminal device (such as a chip, a chip system, etc.), or may be implemented by a logic module or software capable of implementing all or part of the functions of the terminal device, which is not limited in the present embodiment of the application. The terminal device may have the following features: Figure 1 , Figure 2 or Figure 3 The structure shown in the embodiment of the present application is not limited to this. Fig.16 Detailed description of each step in.

[0277] In step 1610 , the mth group of image data to be rendered sent by the application is received.

[0278] The image data includes a plurality of pixel values ​​used to generate an image, m≥1, and m is an integer. In some examples, the image data includes but is not limited to a pixel matrix, depth information, and pixel coordinates.

[0279] The terminal device may receive the mth group of image data to be rendered sent by the application.

[0280] In a possible implementation, receiving the mth group of image data to be rendered sent by the application includes: receiving the mth group of image data to be rendered by a first application, where the first application is any one of the one or more preset applications. For a detailed description, please refer to the relevant description of step 410 in the above method 400, which will not be repeated here for brevity.

[0281] In step 1620, the mth frame image is generated by rendering based on all pixel values ​​in the mth group of image data.

[0282] The terminal device can execute a currently known image rendering method based on the mth group of image data, that is, the terminal device can perform rendering based on all pixel values ​​in the mth group of image data to obtain the mth frame of image.

[0283] In step 1630 , the (m+1)th group of image data to be rendered sent by the application is received.

[0284] After rendering and generating the mth frame image, the terminal device can receive image data of the next frame image of the mth frame image sent by the application, that is, the terminal device can receive the m+1th group of image data to be rendered sent by the application.

[0285] In step 1640, an m+1th frame of image is generated by rendering based on static pixels of the mth frame of image and at least part of pixel values ​​in the m+1th group of image data.

[0286] The static pixels of the m-th frame image are obtained based on the m-th group of image data.

[0287] The terminal device may determine whether the m+1th frame image can reuse some of the static pixels in the m-th frame image during the rendering and generation process based on the static pixels based on the m-th frame image, the m-th group of image data, and the m+1-th group of image data. In the case that the m+1-th frame image can reuse some of the static pixels in the m-th frame image during the rendering and generation process, the terminal device may render and generate the m+1-th frame image based on the static pixels based on the m-th frame image and at least some of the pixel values ​​in the m+1-th group of image data.

[0288] In one possible implementation, before rendering to generate the m+1th frame image, the method 1600 further includes: determining static pixels and dynamic pixels of the yth frame image based on features of one or more rendering instructions corresponding to the yth frame image, y∈[m, m+1].

[0289] That is to say, before rendering and generating the m+1th frame image, the terminal device can determine the static pixels and dynamic pixels of the m-th frame image, and can determine the static pixels and dynamic pixels of the m+1th frame image.

[0290] Terminal devices can use Figure 2 or Figure 3 The static and moving recognition module and the pixel-level static and moving separation module are used to determine the static pixels and dynamic pixels of the m-th frame image based on the features of one or more rendering instructions corresponding to the m-th frame image. Without loss of generality, the terminal device can also use Figure 2 or Figure 3 The static and moving identification module and the pixel-level static and moving separation module are used to determine the static pixels and dynamic pixels of the m+1th frame image based on the features of one or more rendering instructions corresponding to the m+1th frame image.

[0291] In a possible implementation, the feature of the rendering instruction includes one or more of the following: a program alias, a program variable name, or a program index buffer name. For a detailed description, please refer to the relevant description of the feature of the rendering instruction in the above method 400, which will not be repeated here for brevity.

[0292] The present application provides implementation method 1 and implementation method 2 to implement “determining static pixels and dynamic pixels of the mth frame image based on the features of one or more rendering instructions corresponding to the yth frame image”. Implementation method 1 and implementation method 2 are described in detail below.

[0293] Implementation method 1: Determine the static pixels and dynamic pixels of the yth frame image based on the characteristics of one or more rendering instructions corresponding to the yth frame image, including: determining the static objects and dynamic objects included in the yth frame image based on the characteristics of each rendering instruction in the one or more rendering instructions; determining the static pixels of the yth frame image based on the static objects; and determining the dynamic pixels of the yth frame image based on the dynamic objects.

[0294] For a detailed description of implementation method one, please refer to the relevant description in implementation method A in method 400 above. For the sake of brevity, it will not be repeated here.

[0295] In a possible implementation, after determining the static pixels and dynamic pixels of the yth frame image, the method 1600 further includes: marking the static pixels and the dynamic pixels based on the A channel or the template buffer; wherein the static pixels and the dynamic pixels both include RGBA channels.

[0296] For a detailed description of marking the static pixels and dynamic pixels of the yth frame image, please refer to the relevant description of marking the static pixels and dynamic pixels of the mth frame image in the above method 400, which will not be repeated here for the sake of brevity.

[0297] Implementation method two: Based on the characteristics of one or more rendering instructions corresponding to the yth frame image, determine the static pixels and dynamic pixels of the yth frame image, including: based on the characteristics of each rendering instruction in the one or more rendering instructions, determine the static objects and dynamic objects included in the yth frame image to obtain a motion vector set of the yth frame image; based on the motion vector set, determine the static pixels and dynamic pixels of the yth frame image, the value of the motion vector corresponding to the static object of the yth frame image is different from the value of the motion vector corresponding to the dynamic object of the yth frame image.

[0298] For a detailed description of implementation method 2, please refer to the relevant description of implementation method B in method 400 above. For the sake of brevity, it will not be repeated here.

[0299] In a possible implementation, the method 1600 further includes: intercepting one or more rendering instructions corresponding to the yth frame image.

[0300] For a detailed description of intercepting one or more rendering instructions corresponding to the yth frame image, please refer to the relevant description of intercepting one or more rendering instructions corresponding to the mth frame image in the above method 400, which will not be repeated here for brevity.

[0301] In one possible implementation, after determining the static objects included in the m+1th frame image, the method further includes: reducing the shading rate of the static objects in the m+1th frame image, and rendering the m+1th frame image is performed based on the updated shading rate.

[0302] For a detailed description of reducing the shading rate of the static object in the m+1th frame image, please refer to the above method 400, in which, when m mod N≠1, and after determining the static objects included in the mth frame image, the shading rate of the static objects in the mth frame image is reduced. For the sake of brevity, it will not be repeated here.

[0303] In a possible implementation, rendering and generating an m+1th frame image based on static pixels of the m-th frame image and at least part of the pixel values ​​in the m+1th group of image data includes: making correspondence between the m-th frame image and the screen space coordinates of the m+1th frame image; when the pixels of the first screen space corresponding to the m-th frame image and the m+1th frame image are both static pixels, reusing the pixels of the first screen space of the m-th frame image to render the first screen space of the m-th frame image; or, when the pixels of the second screen space corresponding to the m-th frame image and the m+1th frame image are both dynamic pixels, or when the pixels of the second screen space of the m-th frame image and the m+1th frame image are different types of pixels, rendering the second screen space of the m+1th frame image based on the pixels of the second screen space of the m+1th frame image.

[0304] For a detailed description of “rendering and generating the m+1th frame image based on the static pixels of the mth frame image and at least part of the pixel values ​​in the m+1th group of image data”, please refer to the relevant description of “rendering and generating the m+1th frame image based on the mth group of image data when m mod N≠1” in the above method 400. For the sake of brevity, it will not be repeated here.

[0305] In one possible implementation, the method 1600 also includes: receiving the m+nth group of image data to be rendered sent by an application, where n>1 and x is an integer; rendering to generate the m+nth frame image based on static pixels of the m+n-1th frame image and at least part of the pixel values ​​in the m+nth group of image data, wherein the static pixels of the m+n-1th frame image are obtained based on the m+n-1th group of image data.

[0306] In one example, for example, if n=2, then m+n is m+2. After rendering and generating the m+1th frame image, the terminal device can also receive the image data of the next frame image of the m+1th frame image sent by the application, that is, the terminal device can receive the m+2th group of image data to be rendered sent by the application. The terminal device can render and generate the m+2th frame image based on the static pixels based on the m+1th frame image and at least part of the pixel values ​​in the m+2th group of image data; or the terminal device can render and generate the m+2th frame image based on the static pixels based on the m+1th (based on m+2-1) frame image and at least part of the pixel values ​​in the m+2th group of image data.

[0307] In another example, for example, if n=5, then m+n is m+5. After rendering and generating the m+1th frame image, the terminal device can receive the m+5th group of image data to be rendered sent by the application. The terminal device can render and generate the m+5th frame image based on the static pixels based on the m-th frame image and at least part of the pixel values ​​in the m+5th group of image data; or, the terminal device can render and generate the m+5th frame image based on the static pixels based on the m+4th (according to m+5-1) frame image and at least part of the pixel values ​​in the m+5th group of image data.

[0308] Based on the above scheme, among the multiple frames of images to be rendered, the rendering generation of at least one frame of image reuses part of the pixel values ​​of the previous frame of image, which helps to reduce the power consumption of shading calculation of mobile terminal devices, reduce the occurrence of heat, freezes and other phenomena, and thus improve the user experience.

[0309] In some examples, by capturing a screen shot of an application (eg, a game application) while it is running and zooming in and comparing non-overlapping portions of two consecutive motion frames, it is possible to observe a difference in shading rates between the two consecutive motion frames.

[0310] In some examples, RenderDoc is used to open the corresponding application (such as a game application), capture the instruction stream and shader, analyze the graphics library (GL) instructions or Vulkan instructions, view the motion frames for dynamic and static separation, reduce the shading rate of static objects in the motion frames, and increase rendering instructions for rendering reuse.

[0311] The present application also provides a terminal device, which includes a terminal device for executing the above Figure 4 , Figure 5 , Figure 7 , Figure 8 , Fig.10 and Figures 12 to 16 The corresponding modules of the steps in any of the embodiments described above. The modules included in the terminal device can be implemented by software and / or hardware.

[0312] The embodiment of the present application also provides a terminal device, which includes a memory and a processor, wherein the memory is used to store a program, and the processor is used to call and execute the program, so that the terminal device executes the above Figure 4 , Figure 5 , Figure 7 , Figure 8 , Fig.10 and Figures 12 to 16 A step in any of the embodiments described.

[0313] The present application also provides a chip system, which includes at least one processor for implementing the above Figure 4 , Figure 5 , Figure 7 , Figure 8 , Fig.10 and Figures 12 to 16 The functions involved in the steps of any of the embodiments described herein.

[0314] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0315] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0316] The embodiment of the present application also provides a readable storage medium on which a program is stored. When the program is executed by a terminal device, the terminal device executes the above Figure 4 , Figure 5 , Figure 7 , Figure 8 , Fig.10 and Figures 12 to 16 A step in any of the embodiments described.

[0317] The embodiment of the present application also provides a program product, including a program, which, when executed by a terminal device, enables the terminal device to execute the above Figure 4 , Figure 5 , Figure 7 , Figure 8 , Fig.10 and Figures 12 to 16 A step in any of the embodiments described.

[0318] It should be understood that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined to perform. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0319] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory.

[0320] The terms "unit", "module" and the like used in this specification may be used to represent device-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution.

[0321] It will be appreciated by those skilled in the art that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in combination with software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. In the several embodiments provided in this application, it should be understood that the disclosed devices, equipment and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0322] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0323] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more units may be integrated into one module.

[0324] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the software product is stored in a storage medium, including several instructions for enabling a device to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0325] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image rendering method, characterized in that: Applied to a terminal device, the method comprises: receiving an mth group of image data to be rendered sent by an application, wherein the image data includes a plurality of pixel values ​​used to generate an image, where m≥1, and m is an integer; Rendering and generating an m-th frame of image based on all pixel values ​​in the m-th group of image data; Receiving the (m+1)th group of image data to be rendered sent by the application; The m+1th frame image is generated by rendering based on static pixels of the mth frame image and at least part of pixel values ​​in the m+1th group of image data, wherein the static pixels of the mth frame image are obtained based on the mth group of image data.

2. The method according to claim 1, characterized in that Before rendering to generate the (m+1)th frame image, the method further includes: Based on the features of one or more rendering instructions corresponding to the yth frame image, the static pixels and dynamic pixels of the yth frame image are determined, y∈[m, m+1].

3. The method according to claim 2, characterized in that The determining of static pixels and dynamic pixels of the yth frame image based on features of one or more rendering instructions corresponding to the yth frame image includes: Determine static objects and dynamic objects included in the yth frame image based on a feature of each rendering instruction in the one or more rendering instructions; Determine the static pixels of the yth frame image based on the static object; The dynamic pixels of the yth frame image are determined based on the dynamic object.

4. The method according to claim 3, characterized in that After determining the static pixels and dynamic pixels of the yth frame image, the method further includes: The static pixels and the dynamic pixels are marked based on an A channel or a template buffer; wherein the static pixels and the dynamic pixels both include RGBA channels.

5. The method according to claim 2, characterized in that The determining of static pixels and dynamic pixels of the yth frame image based on features of one or more rendering instructions corresponding to the yth frame image includes: Determine static objects and dynamic objects included in the yth frame image based on the characteristics of each rendering instruction in the one or more rendering instructions to obtain a motion vector set of the yth frame image; Based on the motion vector set, the static pixels and dynamic pixels of the yth frame image are determined, and the value of the motion vector corresponding to the static object of the yth frame image is different from the value of the motion vector corresponding to the dynamic object of the yth frame image.

6. The method according to any one of claims 3 to 5, characterized in that After determining the static object included in the m+1th frame image, the method further includes: The shading rate of the static object in the (m+1)th frame image is reduced, and the (m+1)th frame image is rendered based on the updated shading rate.

7. The method according to any one of claims 2 to 6, characterized in that The characteristics of the rendering instruction include one or more of the following: A program alias, program variable name, or program index buffer name.

8. The method according to any one of claims 2 to 7, characterized in that The method further comprises: One or more rendering instructions corresponding to the y-th frame image are intercepted.

9. The method according to any one of claims 1 to 8, characterized in that The rendering based on the static pixels of the mth frame image and at least part of the pixel values ​​in the m+1th group of image data to generate the m+1th frame image includes: Matching the screen space coordinates of the m-th frame image with the m+1-th frame image; In a case where the pixels of the first screen space corresponding to the m-th frame image and the m+1-th frame image are both static pixels, the pixels of the first screen space of the m-th frame image are reused to render the first screen space of the m-th frame image; or When the pixels of the second screen space corresponding to the m-th frame image and the m+1-th frame image are both dynamic pixels, or when the pixels of the second screen space of the m-th frame image and the m+1-th frame image are different types, the second screen space of the m+1-th frame image is rendered based on the pixels of the second screen space of the m+1-th frame image.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: Receive the m+nth group of image data to be rendered sent by the application, where n>1 and n is an integer; The m+n-th frame image is generated based on the static pixels of the m-th frame image or the m+n-1-th frame image and at least part of the pixel values ​​in the m+n-th group of image data, and the static pixels of the m+n-1-th frame image are obtained based on the m+n-1-th group of image data.

11. An image rendering method, characterized in that: Applied to a terminal device, the method comprises: receiving an mth group of image data to be rendered sent by an application, wherein the mth group of image data includes a plurality of pixel values ​​used to generate an image, where m≥1, and m is an integer; Based on the mth group of image data, the mth frame image is generated by rendering; wherein, In the case where m mod N=1, the m-th frame image is rendered based on all pixel values ​​in the m-th group of image data; In the case where m mod N≠1, the m-th frame image is rendered based on static pixels of the ith frame image and at least part of pixel values ​​in the m-th group of image data, and the static pixels of the ith frame image are obtained based on the ith group of image data; Wherein, N is a preset integer, N≥2, 12. The method according to claim 11, characterized in that Before rendering to generate the m-th frame image, the method further includes: Based on features of one or more rendering instructions corresponding to the m-th frame image, static pixels and dynamic pixels of the m-th frame image are determined.

13. The method according to claim 12, characterized in that The determining of static pixels and dynamic pixels of the m-th frame image based on features of one or more rendering instructions corresponding to the m-th frame image includes: Determine static objects and dynamic objects included in the m-th frame image based on a feature of each rendering instruction in the one or more rendering instructions; determining the static pixel based on the static object; The dynamic pixels are determined based on the dynamic object.

14. The method according to claim 13, characterized in that After determining the static pixels and dynamic pixels of the m-th frame image, the method further includes: The static pixels and the dynamic pixels are marked based on an A channel or a template buffer; wherein the static pixels and the dynamic pixels both include RGBA channels.

15. The method according to claim 12, characterized in that The determining of static pixels and dynamic pixels of the m-th frame image based on features of one or more rendering instructions corresponding to the m-th frame image includes: Determine static objects and dynamic objects included in the m-th frame image based on a feature of each rendering instruction in the one or more rendering instructions to obtain a motion vector set of the m-th frame image; The static pixel and the dynamic pixel are determined based on the motion vector set, and the value of the motion vector corresponding to the static object is different from the value of the motion vector corresponding to the dynamic object.

16. The method according to any one of claims 13 to 15, characterized in that In the case where m mod N≠1, and after determining the static object included in the m-th frame image, the method further includes: The shading rate of the static object in the m-th frame image is reduced, and the rendering of the m-th frame image is performed based on the updated shading rate.

17. The method according to any one of claims 12 to 16, characterized in that The characteristics of the rendering instruction include one or more of the following: A program alias, program variable name, or program index buffer name.

18. The method according to any one of claims 12 to 17, characterized in that The method further comprises: One or more rendering instructions corresponding to the m-th frame image are intercepted.

19. The method according to any one of claims 11 to 18, characterized in that In the case where m mod N≠1, rendering and generating the m-th frame image based on the m-th group of image data comprises: Matching the screen space coordinates of the i-th frame image with the m-th frame image; In a case where the pixels of the first screen space corresponding to the i-th frame image and the m-th frame image are both static pixels, the pixels of the first screen space of the i-th frame image are reused to render the first screen space of the m-th frame image; or When the pixels of the second screen space corresponding to the i-th frame image and the m-th frame image are both dynamic pixels, or when the pixels of the second screen space of the i-th frame image and the m-th frame image are different types, the second screen space of the m-th frame image is rendered based on the pixels of the second screen space of the m-th frame image.

20. A terminal device, characterized in that: comprising a processor and a memory, wherein: The memory is used to store programs; The processor is used to call the program so that the terminal device executes the method according to any one of claims 1 to 19.

21. A readable storage medium having a program stored thereon, characterized in that: When the program is executed, the terminal device executes the method according to any one of claims 1 to 19.

22. A program product, characterized in that The method comprises a program, which, when executed, causes a terminal device to execute the method according to any one of claims 1 to 19.

Images