Display method, readable storage medium and electronic equipment

Abstract

The invention relates to the technical field of image processing, and discloses a display method, a readable storage medium and electronic equipment. When a main image layer and a high-change-rate image layer exist in a user interface, the electronic device can render the main image layer and the high-change-rate image layer to obtain and store a main display image and a high-change-rate display image, and then a hardware synthesis module is called to synthesize the main display image and the high-change-rate display image into the user interface. When the first interface is rendered, if the display content of the main image layer in the first interface is the same as the display content of the main image layer in the previous frame of interface, the electronic equipment can only render the high-change-rate image layer without rendering the main image layer; and calling a hardware synthesis module to synthesize the high-change-rate display image corresponding to the rendered first interface and the stored main display image corresponding to the previous frame of interface into the first interface. In this way, the electronic device does not need to render the main layer again, and the power consumption of the electronic device in the process of rendering the display interface is reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a display method, a readable storage medium and an electronic device. Background Art

[0002] The graphical user interface (GUI) displayed in an electronic device is usually rendered by multiple layers corresponding to multiple applications running in the electronic device. When the electronic device renders multiple layers into a user interface, it usually calls the graphics processing unit (GPU) of the electronic device to render each layer into a display image. Then, the electronic device synthesizes the display images corresponding to the multiple applications according to the superposition order of the display images of each layer to obtain the user interface.

[0003] Generally speaking, when the content of any layer in the layers corresponding to the user interface of the electronic device changes, the electronic device needs to re-render all the layers, resulting in low efficiency of rendering the user interface of the electronic device. Summary of the invention

[0004] In view of this, the present application provides a display method, a readable storage medium and an electronic device.

[0005] In a first aspect, the present application provides a display method, which is applied to an electronic device, the method comprising: determining interface data of a first interface to be displayed, wherein the interface data comprises N layers corresponding to multiple applications, where N is an integer greater than 2; selecting P first layers from the N layers according to a frequency of change of display content in the N layers; rendering the P first layers to obtain at least one first display image; when the first display content corresponding to NP second layers other than the first layer in the N layers has not changed relative to the second interface, synthesizing at least one first display image and a second display image to obtain a first interface, wherein the second interface is a previous frame interface of the first interface, and the second display image is a display image corresponding to the first display content and used by the electronic device in a process of generating the second interface; and displaying the first interface.

[0006] Through this method, if the first display content corresponding to the NP second layers (such as the main layer hereinafter) in the first interface to be displayed by the electronic device has not changed compared to the previous frame interface (second interface) of the first interface, the electronic device does not need to re-render the second layer in the process of generating the first interface. The electronic device can render the P first layers (such as the high-change rate layers hereinafter) in the first interface to be displayed to obtain at least one first display image (such as the high-change rate display image hereinafter), and then synthesize the at least one display image and the second display image (such as the main display image hereinafter) corresponding to the first display content used by the electronic device in the process of generating the second interface to obtain the first interface. In this way, the number of layers rendered by the electronic device can be reduced, which is conducive to reducing the GPU resources occupied in the interface generation process, improving the efficiency of image rendering, and reducing the power consumption of the GPU.

[0007] In some implementations, the second display image may be rendered by the electronic device based on the first display content when generating the second interface.

[0008] In some implementations, if the display content in the layer corresponding to the first display content in the second interface has not changed from the previous frame interface of the second interface, the second display image may also be rendered by the electronic device based on the first display content in the process of generating the previous frame interface of the second interface.

[0009] In a possible implementation of the first aspect, the P first layers are selected from the N layers based on the change frequency of the displayed content in the N layers, including: determining a layer among the N layers that satisfies at least one of the following conditions as the first layer: the change frequency of the displayed content of the layer is greater than the first frequency; the layer is any one of the following layers: a layer corresponding to the game screen in a game application, a layer corresponding to the video, a layer corresponding to the web page content in a browser application, and a layer corresponding to the image captured by the camera of the electronic device.

[0010] In a possible implementation of the first aspect above, the method also includes: when the first display content corresponding to the NP second layers changes relative to the second interface, rendering the NP second layers to obtain a third display image; synthesizing at least one first display image and the third display image to obtain the first interface.

[0011] That is to say, when the first display content of the second image in the N layers of the first interface of the electronic device changes compared with the previous frame interface, the electronic device can render NP second layers to obtain a third display image, and synthesize at least one first display image and the third display image to obtain the first interface.

[0012] In a possible implementation of the first aspect, the electronic device includes a graphics processor; and rendering the P first layers to obtain at least one first display image includes: calling the image processor to render the P first layers to obtain at least one first display image.

[0013] That is to say, the electronic device renders the first layer through an image processor to obtain at least one first display image.

[0014] In a possible implementation of the first aspect above, the electronic device also includes a hardware synthesis module; and synthesizing at least one first display image and a second display image to obtain a first interface includes: calling the hardware synthesis module to obtain at least one first display image and a second display image to obtain the first interface.

[0015] That is, the electronic device synthesizes at least one first display image and a second display image into a first interface through a hardware synthesis module (such as a display subsystem, a hardware synthesizer, etc.). Since the power consumption of synthesizing images by a hardware synthesis module is usually lower than that of an image processor, synthesizing images through a hardware synthesis module can further reduce the power consumption of the electronic device in the process of generating a display interface.

[0016] In a possible implementation of the first aspect above, a first service is running in an operating system of an electronic device; and the electronic device determines, through a first process corresponding to the first service, interface data of a first interface to be displayed, and, when first display contents corresponding to NP second layers other than the first layer in N layers do not change relative to the second interface, synthesizes at least one first display image and a second display image to obtain a first interface.

[0017] That is, the electronic device displays the first image and the second image using a first service (eg, a rendering service described below).

[0018] In a possible implementation of the first aspect, the second display image is rendered by the first process based on the first display content.

[0019] That is to say, the rendering of the second display image is obtained by the first service (such as the rendering service mentioned below) based on the first display content in the process of generating the second interface or other interfaces before the second interface, rather than being rendered separately by the threads of the application corresponding to the first display content. This can reduce the data of the process or thread that calls GPU resources, reduce the complexity of GPU resource scheduling, and improve the efficiency of the GPU.

[0020] In a possible implementation of the first aspect above, the above-mentioned synthesis of at least one first display image and a second display image to obtain a first interface includes: synthesizing at least one first display image and a second display image based on the superposition order of the first display image and the second display image in a first direction to obtain the first interface, wherein the first direction is a direction perpendicular to the outward direction of the display screen of the electronic device.

[0021] In a possible implementation of the first aspect, the P first layers include Q first layers that are continuously stacked in a first direction, and at least one first display image includes a fourth display image corresponding to the Q first layers.

[0022] That is to say, multiple first layers in the P first layers that are consecutive in the stacking order in the first direction can be rendered as a first display image, and first layers in the P first layers that are not adjacent to other first layers in the stacking order in the first direction can be rendered as a first display image respectively.

[0023] In a second aspect, the present application provides a readable storage medium, which includes instructions. When the instructions are executed by a device, the device implements the display method provided by the first aspect and any possible implementation of the first aspect.

[0024] In a third aspect, the present application provides an electronic device, comprising: a memory for storing instructions; a processor for executing instructions to enable the electronic device to implement the display method provided by the first aspect and any possible implementation of the first aspect.

[0025] In a fourth aspect, the present application provides a program product, which, when running on an electronic device, enables the electronic device to implement the display method provided by the first aspect and any possible implementation of the first aspect.

[0026] It should be understood that the beneficial effects of the second to fourth aspects mentioned above can be referred to the description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 According to some embodiments of the present application, a user interface U1 and a schematic diagram of the superposition relationship of various layers in the user interface U1 are shown;

[0028] Figure 2 According to some embodiments of the present application, a schematic diagram of a process of synthesizing a user interface U1 is shown;

[0029] Figure 3 According to some embodiments of the present application, a schematic diagram of a process of an electronic device rendering a user interface U1 through a rendering service is shown;

[0030] Figure 4 According to some embodiments of the present application, a schematic diagram of a user interface U1 and a user interface U0 is shown;

[0031] Figure 5 According to some embodiments of the present application, a schematic diagram of synthesizing a main display image and a high-change rate display image into a user interface is shown;

[0032] Figure 6 According to some embodiments of the present application, a schematic diagram of a process in which an electronic device renders layers of a desktop 1 and a video 2 as a user interface is shown;

[0033] Figure 7 According to some embodiments of the present application, a schematic diagram of a process of generating a user interface U0 and a user interface U1 is shown;

[0034] Fig. 8A According to some embodiments of the present application, a schematic diagram of a process of obtaining a hole-punch image LM2 during a process of rendering a main display image LC by an electronic device is shown;

[0035] Figure 8B According to some embodiments of the present application, a schematic diagram of a process of an electronic device obtaining a main display image LC based on a hole-punch image LM2 is shown;

[0036] Fig. 9 According to some embodiments of the present application, a flow chart of a display method is shown;

[0037] Fig. 10A According to some embodiments of the present application, a schematic diagram of a user interface U2 is shown;

[0038] Fig. 10B According to some embodiments of the present application, a schematic diagram of a user interface U3 is shown;

[0039] Fig.11A According to some embodiments of the present application, a schematic diagram of rendering a main display image LC1 corresponding to a first interface is shown;

[0040] Fig. 11B According to some embodiments of the present application, a schematic diagram of synthesizing a main display image LC1 and a high-change rate display image LV1 into a user interface U2 is shown;

[0041] Fig.12 According to some embodiments of the present application, a flow chart of another display method is shown;

[0042] Fig.13 According to some embodiments of the present application, a schematic structural diagram of an electronic device 10 is shown;

[0043] Fig.14 According to some embodiments of the present application, a schematic diagram of a software architecture of an electronic device 10 is shown. DETAILED DESCRIPTION

[0044] Illustrative embodiments of the present application include, but are not limited to, a display method, a readable storage medium, and an electronic device.

[0045] The technical solution of the present application is introduced below in conjunction with the accompanying drawings.

[0046] As mentioned above, when the content of any layer in the layers corresponding to the user interface of the electronic device changes, the electronic device needs to re-render all the layers, which will result in low efficiency in rendering the user interface of the electronic device.

[0047] To improve the efficiency of rendering user interfaces by electronic devices, in some embodiments, the electronic device may configure a rendering thread for each application in the electronic device. When the electronic device renders multiple layers into a user interface, it may call the GPU of the electronic device through the rendering threads corresponding to each application to render the corresponding layers into display images. Then, the electronic device synthesizes the display images corresponding to the multiple applications according to the superposition order of the display images of each application to obtain a user interface. In this way, if the content of a layer of an application has not changed, the rendering thread of the application does not need to re-render the layer of the application, and the electronic device can use the display image corresponding to the application obtained by rendering the previous frame to synthesize the user interface.

[0048] For example, Figure 1 According to some embodiments of the present application, a user interface U1 and a schematic diagram of the superposition relationship of various layers in the user interface U1 are shown. Figure 2 According to some embodiments of the present application, a schematic diagram of a process of synthesizing an interface U1 is shown.

[0049] like Figure 1 As shown, the user interface U1 includes 6 layers corresponding to the desktop application (hereinafter referred to as the desktop) 1 and the video application (hereinafter referred to as the video) 2, namely the background layer U11 and the status bar layer U12 of the desktop 1, the background layer U21, the video layer U22, the title / control layer U23 and the function layer U24 of the video 2. Among them, the layer levels of the above 6 layers are from low to high: the background layer U11 of the desktop 1, the status bar layer U12, the background layer U21, the video layer U22, the title / control layer U23 and the function layer U24 of the video 2.

[0050] It should be understood that the layer level of a layer is used to indicate the stacking order of each layer (or image) along the positive direction of the Z axis (as the first direction) when synthesizing. The higher the layer level, the higher the arrangement order in the positive direction of the Z axis, where the Z axis is the direction perpendicular to the electronic device screen and outward. That is to say, in the process of synthesizing each layer, a layer with a higher layer level will be superimposed on a layer with a lower layer along the Z axis direction.

[0051] It should be understood that in some embodiments, a service or module (hereinafter referred to as a media service) for rendering multimedia files (such as videos, pictures, etc.) in an electronic device may render the video layer in the application, and store the rendering result in a video cache corresponding to the application. The rendering thread corresponding to the application or other applications may obtain the rendering result corresponding to the video layer from the video cache corresponding to the application.

[0052] refer to Figure 2 In some embodiments, when rendering the user interface U1, the electronic device can call the GPU to render the background layer U11 and the status bar layer U12 of the desktop 1 through the rendering thread of the desktop 1 according to the rendering tree of the desktop 1 (used to indicate the relationship between the layer levels of each layer in the desktop 1), obtain the display image P1, and store the display image P1 in the cache of the desktop 1. The electronic device can also call the GPU to render the background layer U21, the title / control layer U23, and the function layer U24 of the video 2 through the rendering thread of the video 2 according to the rendering tree of the video 2 (used to indicate the relationship between the layer levels of each layer in the video 2), obtain the display image P2, and store the display image P2 in the cache of the video 2. Then, the rendering service running in the electronic device performs hardware synthesis on the display image P1, the display image P2, and the display image V1 (rendered by the media service and stored in the video cache corresponding to the video 2) based on the synthetic rendering tree (used to indicate the relationship between the layer levels of the display images of each application), obtain the user interface U1, and store the user interface U1 in the display cache, so that the display screen of the electronic device can obtain and display the user interface U1 from the display cache.

[0053] It should be understood that the rendering tree of the application is used to indicate the stacking order of each layer (or image) of the application, and a node in the rendering tree is used to indicate a layer (it can also be one or more display elements, such as controls, images, text, etc.). The electronic device can obtain the arrangement order of the nodes in the rendering tree (that is, the arrangement order of the layer levels of the layers corresponding to each node) by traversing the nodes of the rendering tree in pre-order. The layer level of the layer traversed earlier in the rendering tree according to the pre-order traversal order is lower.

[0054] For example, refer to Figure 2, the rendering tree of desktop 1 includes root node 10, leaf node 11 and leaf node 12, wherein root node 10 corresponds to desktop 1, and leaf node 11 and leaf node 12 correspond to background layer U11 and status bar layer U12 of desktop 1 respectively. The result of the pre-order traversal of the rendering tree of desktop 1 by the rendering process of desktop 1 is root node 10→leaf node 11→leaf node 12, indicating that the relationship between the layer levels of background layer U11 and status bar layer U12 of desktop 1 is: background layer U11 of desktop 1<status bar layer U12. For another example, refer to Figure 2 , the rendering tree of video 2 includes root node 20, leaf node 21, leaf node 22, leaf node 23 and leaf node 24, wherein root node 20 corresponds to video 2, and leaf nodes 21, leaf node 22, leaf node 23 and leaf node 24 correspond to the background layer U21, video layer U22, title / control layer U23 and function layer U24 of video 2 respectively. The result of the pre-order traversal of the rendering tree of video 2 by the rendering process of video 2 is root node 20→leaf node 21→leaf node 22→leaf node 23→leaf node 24, indicating that the relationship between the layer levels of the background layer U21, video layer U22, title / control layer U23 and function layer U24 of video 2 is: background layer U21 of video 2<video layer U22<title / control layer U23<function layer U24.

[0055] It should be understood that the composite rendering tree is used to indicate the stacking order of display images corresponding to each application, a node in the composite rendering tree is used to indicate a display image corresponding to an application or a layer in an application, and the pre-order traversal result of the nodes of the composite rendering tree indicates the arrangement order of the display images or layers of the application corresponding to each node. For example, Figure 2 The pre-order traversal result of the composite rendering tree shown is R (root node of the user interface) → leaf node 10 (root node of desktop 1) → leaf node 22 (video layer U22) → 20 (root node of video 2), that is, the display image corresponding to desktop 1 is at the bottom layer, the video layer U22 is superimposed on the display image corresponding to desktop 1 along the Z-axis direction, and the display image corresponding to video 2 is superimposed on the video layer U22 along the Z-axis direction.

[0056] It should be understood that in other embodiments, the traversal order of the rendering tree may also be other orders, such as in-order traversal, post-order traversal, etc., which is not limited here.

[0057] However, in Figure 2In the rendering process shown, the layers of desktop 1 and video 2 are obtained by the rendering thread of desktop 1 and the rendering thread of video 2 respectively. In this way, there will be two threads to call the computing resources of the GPU, which may make the resource scheduling of the GPU complicated and affect the efficiency of the GPU. In addition, the more applications corresponding to the user interface to be displayed by the electronic device, the more threads that call the GPU resources, the more complicated the computing resource scheduling process of the GPU of the electronic device, and the lower the efficiency of the GPU.

[0058] In order to improve the rendering efficiency of the electronic device when rendering the user interface, in some embodiments, the electronic device can merge the rendering trees of various applications in the user interface to be displayed into a unified rendering tree, and a rendering service can render the user interface based on the unified rendering tree. In this way, only one rendering service is needed to perform the rendering of the user interface, and there is no need to schedule the computing resources of the GPU through a complex resource scheduling strategy, which is conducive to improving the rendering efficiency of the electronic device when rendering the user interface.

[0059] For example, Figure 3 According to some embodiments of the present application, a schematic diagram of a process in which an electronic device renders a user interface U1 through a rendering service is shown.

[0060] refer to Figure 3 , electronic devices can Figure 2 The rendering tree of desktop 1 and the rendering tree of video 2 are shown to be merged into a unified rendering tree, which includes leaf nodes and root nodes in the rendering tree of desktop 1 and the rendering tree of video 2, and the arrangement order of each node in the pre-order traversal result of the unified rendering tree corresponds to the relationship between the layer levels of each layer in the user interface U1. For example, for the aforementioned user interface U1, the pre-order traversal result of the unified rendering tree by the electronic device based on the rendering service is root node R (root node of user interface U1) → leaf node 10 → leaf node 11 → leaf node 12 → leaf node 20 → leaf node 21 → leaf node 22 → leaf node 23 → leaf node 24, indicating that the relationship between the layer levels of the background layer U11 of desktop 1, the status bar layer U12, the background layer U21 of video 2, the video layer U22, the title / control layer U23 and the function layer U24 is: background layer U11 of desktop 1 < status bar layer U12 < background layer U21 of video 2 < video layer U22 < title / control layer U23 < function layer U24.

[0061] Then, the electronic device can call the GPU through the rendering service to sequentially superimpose each layer on the layer of the previous layer with a lower layer level to obtain a unified rendering image (that is, the image corresponding to the user interface U1), and store the unified rendering image in the display cache, so that the display screen of the electronic device can obtain and display the user interface U1 from the display cache. For example, refer to Figure 1 The electronic device can use a rendering service to sequentially superimpose the status bar layer U12 on the background layer U11 of the desktop 1 along the Z-axis direction, superimpose the background layer U21 of the video 2 on the status bar layer U12 along the Z-axis direction, superimpose the video layer U22 on the background layer U21 of the video 2 along the Z-axis direction, superimpose the title / control layer U23 on the video layer U22 along the Z-axis direction, and superimpose the function layer U24 on the title / control layer U23 along the Z-axis direction to obtain the user interface U1.

[0062] Based on the above method, the electronic device can call the GPU through a rendering service to sequentially superimpose the layers corresponding to the user interface onto the layers with lower layer levels to obtain the user interface, without the need for complex scheduling of the GPU's computing resources, which is beneficial to improving the efficiency of the GPU. However, in the process of rendering the user interface in the above manner, the user interface is obtained by superimposing each layer layer by layer according to the relationship between the layer levels of each layer. Even in a scene where only a part of the content in the layers of the user interface to be displayed has changed compared to the previous frame of the user interface, the rendering service needs to re-render all the layers in the user interface to be displayed, which affects the efficiency of the electronic device in rendering the user interface and increases the power consumption of the GPU.

[0063] For example, refer to Figure 4 , the previous frame of the user interface U1 is the user interface U0, and the only thing that changes in the user interface U1 is the content in the video layer U22 (the position of the character in the video layer U22 changes). When the electronic device renders the user interface U1 through the rendering service, it still needs to sequentially stack the background layer U11 of the desktop 1, the status bar layer U12, the background layer U21 of the video 2, the video layer U22, the title / control layer U23, and the function layer U24 to render the user interface U1, which increases the power consumption of the GPU.

[0064] In view of this, an embodiment of the present application provides a display method, in which, when there is at least one layer with a low content change frequency in the user interface (hereinafter referred to as the main layer, such as the background layer U11 of the desktop 1, the status bar layer U12, the background layer U21 of the video 2, the title / control layer U23, and the function layer U24 in the aforementioned user interface U0 / U1) (as the first layer) and at least one layer with a high content change rate (hereinafter referred to as the high change rate layer, such as the video layer U22 in the aforementioned user interface U0 or / user interface U1), the electronic device can render the main layer as the main display image (as the first display image) and render the high change rate layer as at least one high change rate display image (for example, all layers in the high change rate layer are rendered as a high change rate display image, or the high change rate layer is split into multiple layer groups and the layers in each layer group are rendered as a high change rate display image) (as the second display image) through the GPU. Then, the rendering service (as the first service) synthesizes each main display image and each high change rate image through the hardware synthesis module to obtain a user interface.

[0065] Specifically, assuming that the previous frame interface of the first interface (for example, the aforementioned user interface U1) is the second interface (for example, the aforementioned user interface U0), the display contents of the main layers corresponding to the first interface and the second interface (for example, the background layer U11 of desktop 1, the status bar layer U12, the background layer U21 of video 2, the title / control layer U23, and the function layer U24 in the aforementioned user interface U0 / U1) are all first display contents, the high change rate layer in the first interface has second display content, and the high change rate layer in the second interface has third display content.

[0066] When rendering the second interface, the electronic device can first call the GPU through the rendering service to render the main layer in the second interface as the main display image based on the first display content, and render the high-change rate layer into at least one first high-change rate display image based on the third display content through the service or application corresponding to the high-change rate layer (such as the media service corresponding to the video layer or the image collected by the camera, the game application corresponding to the layer corresponding to the game screen, the browser application corresponding to the layer corresponding to the web page content, etc., hereinafter referred to as the second service). Then, the electronic device can call the hardware synthesis module through the rendering service to synthesize the main display image with each high-change rate display image to obtain the second interface.

[0067] It should be understood that in other embodiments, the high-change rate layer may also be rendered into a high-change rate display image by a rendering service, without limitation thereto.

[0068] When the electronic device renders the first interface, the second service first calls the GPU to synthesize the high-change rate layer corresponding to the second interface based on the second display content to obtain at least one second high-change rate display image. Then, the rendering service in the electronic device can call the hardware synthesis module to synthesize the second high-change rate display image and the main display image to obtain the second interface.

[0069] In this way, when the content in the main layer corresponding to the first interface has not changed compared to the second interface, the electronic device does not need to render the main layer. Instead, it uses the hardware synthesis module to synthesize the main display image corresponding to the main layer in the second interface with the second high-change rate display image corresponding to the high-change rate layer in the first interface to obtain the first interface, which is beneficial to reducing the power consumption of the electronic device in rendering the first interface.

[0070] It should be understood that the hardware synthesis module (as the first image processing module) is a device or circuit in the electronic device that is independent of the GPU and is used to synthesize multiple images into one image. In some embodiments, the hardware synthesis module can be integrated with the GPU, central processing unit (CPU) and the like in the electronic device in the same device, or it can be a separate device different from the GPU and the CPU, which is not limited here.

[0071] It should be understood that after the electronic device calls the GPU to synthesize the main layer into the main display image, the main display image can be stored in a cache. If the display content in the main layer does not change, the main display image stored in the cache will not change. When the electronic device uses the hardware synthesis module to synthesize the user interface, it can directly obtain the main display image from the cache.

[0072] In some embodiments, the hardware composition module may also be referred to as a display subsystem (DSS) or a hardware composer.

[0073] In some embodiments, a high-change rate layer may be a layer whose content changes at a frequency exceeding a preset frequency, or may be a layer of a specific type. For example, a high-change rate layer may include, but is not limited to, a layer corresponding to a game screen in a game application, a layer corresponding to a video (e.g., a layer for playing a video, a layer for displaying a video captured by a camera of an electronic device), a layer corresponding to web page content in a browser application, a clock layer whose time refresh frequency is higher than a preset frequency (e.g., a layer of a stopwatch application, etc.), etc.

[0074] In some embodiments, a layer in the main layer whose layer level is higher than a certain high change rate layer can be called an upper layer of the high change rate layer, and a layer in the main layer whose layer level is lower than the high change rate layer can be called a lower layer of the high change rate layer. For example, for the aforementioned user interface U1, the video layer U22 can be a high change rate layer, and the other layers are main layers. The layer levels of the background layer U11 of the desktop 1, the status bar layer U12, and the background layer U21 of the video 2 are lower than the video layer U22, and can be called lower layers of the video layer U22; the layer levels of the title / control layer U23 and the function layer U24 are higher than the video layer U22, and can be called upper layers of the video layer U22.

[0075] In some embodiments, the content of the lower layer of the high-change rate layer in the main display image in the area where the high-change rate layer is located is transparent, and the content of the upper layer of the high-change rate layer in the main display image in the area where the high-change rate layer is located is retained. In this way, when the electronic device synthesizes the main display image and the high-change rate display image, the main display image can be superimposed on the high-change rate display image along the Z-axis direction. For example, referring to Figure 5 In the area D1 corresponding to the high-change rate display image in the main display image, the content of the lower layer of the high-change rate layer in the area where the high-change rate layer is located is transparent, and the content of the upper layer of the high-change rate layer in the area where the high-change rate layer is located (for example Figure 5 The electronic device can superimpose the main display image on the upper layer of the high-change rate display image to obtain a user interface.

[0076] In some embodiments, there may be one or more high-change rate layers, and correspondingly there may be one or more high-change rate display images. For example, in a case where there are multiple high-change rate layers whose layer levels are not completely continuous in the user interface (i.e., there are main layers between the high-change rate layers), the electronic device may group the continuous high-change rate layers among the multiple high-change rate layers whose layer levels are not completely continuous, and render each group of high-change rate layers into a high-change rate display image. For another example, in a case where the layer levels of the high-change rate layers in the user interface are continuous (i.e., there are no main layers between the high-change rate layers), the electronic device may render the high-change rate layers into a high-change rate display image.

[0077] For the purpose of description, the technical solution of the present application is introduced below by taking the high-change rate layer in the user interface as a continuous high-change rate layer, and the high-change rate layer is rendered as one high-change rate layer as an example.

[0078] The following takes the example where the first interface is the aforementioned user interface U1 and the second interface is the aforementioned user interface U0 to introduce the technical solution of the present application.

[0079] For example, Figure 6 According to some embodiments of the present application, a schematic diagram of a process in which an electronic device renders layers of a desktop 1 and a video 2 as a user interface is shown; Figure 7 According to some embodiments of the present application, a schematic diagram of a process of generating a user interface U0 and a user interface U1 is shown.

[0080] for Figure 4 In the case shown, in the user interface U0 and the user interface U1, the high-change rate layer is the video layer U22, and the main layers include the background layer U11 of the desktop 1, the status bar layer U12, the background layer U21 of the video 2, the title / control layer U23 and the function layer U24. Figure 6 After obtaining the unified rendering tree of user interface U0 and user interface U1, the rendering service of the electronic device can call the GPU to render the main layer (layers corresponding to nodes 11, 12, 21, 13, and 14) according to the layer hierarchy relationship of each layer to obtain the main display image. The media service in the electronic device can render the high-change rate layer (layer corresponding to node 22) to obtain a high-change rate display image. Then, the rendering service can call the hardware synthesis module to synthesize the main display image and the high-change rate layer to obtain the user interface, and store the user interface in the display cache.

[0081] Specifically, refer to Figure 7 In the process of generating the user interface U0, the electronic device can call the GPU through the rendering service to render the background layer U11 of the desktop 1, the status bar layer U12, the background layer U21 of the video 2, the title / control layer U23 and the function layer U24 to obtain the main display image LC. In the main display image LC, the content of the background layer U11 of the desktop 1, the status bar layer U12, and the background layer U21 of the video 2 is transparent in the area corresponding to the video layer U22. Then, the electronic device can superimpose the main display image LC on the high-change rate display image LV0 corresponding to the video layer U22 along the Z-axis direction to obtain the user interface U0.

[0082] During the process of generating the user interface U1, since the contents in the main layers of the user interface U1 (background layer U11 of the desktop 1, the status bar layer U12, the background layer U21 of the video 2, the title / control layer U23 and the function layer U24) have not changed relative to the user interface U0, while the contents in the high-change-rate layers (video layer U22) of the user interface U1 have changed relative to the user interface U0, the electronic device can superimpose the main display image LC obtained in the process of generating the user interface U0 along the Z-axis direction on top of the high-change-rate display image LV1 corresponding to the changed video layer U22 to obtain the user interface U1.

[0083] Based on the above process, when the electronic device generates the user interface U1, it does not need to re-render the background layer U11 of the desktop 1, the status bar layer U12, the background layer U21 of the video 2, the title / control layer U23 and the function layer U24, but directly superimposes the main display image LC generated in the process of rendering the user interface U0 on the high-change rate display image LV1 along the Z-axis direction through the hardware synthesis module to obtain the user interface U1. In this way, the GPU resources occupied by the electronic device in the process of generating the user interface U1 can be reduced.

[0084] In some embodiments, when rendering the main layer into the aforementioned main display image, the electronic device may first render the lower layer corresponding to the high-change rate layer to obtain an intermediate image. Then, the electronic device may set the area corresponding to the high-change rate layer in the intermediate image to be transparent to obtain a hole-punching image. Secondly, the electronic device then superimposes the upper layer corresponding to the high-change rate layer layer by layer along the Z-axis direction on top of the hole-punching image to obtain the main display image.

[0085] For example, Fig. 8A According to some embodiments of the present application, a schematic diagram of a process of obtaining a hole-punch image LM2 during a process of rendering a main display image LC by an electronic device is shown; Figure 8B According to some embodiments of the present application, a schematic diagram of a process in which an electronic device obtains a main display image LC based on a hole-punch image LM2 is shown.

[0086] For the aforementioned user interface U0, the layer with a high rate of change is the video layer U22. The lower layers corresponding to the video layer U22 include the background layer U11 of the desktop 1, the status bar layer U12, and the background layer U21 of the video 2. The upper layers corresponding to the video layer U22 include the title / control layer U23 and the function layer U24. Fig. 8A , the rendering service of the electronic device can call the GPU to sequentially superimpose the background layer U11 of the desktop 1, the status bar layer U12, and the background layer U21 of the video 2 to obtain the intermediate image LM1. Then, the rendering service of the electronic device can call the GPU to set the area D3 corresponding to the video layer U22 in the intermediate image LM1 to be transparent to obtain the hole-punch image LM2.

[0087] Then, refer to Figure 8B The rendering service of the electronic device can call the GPU to sequentially superimpose the title / control layer U23 and the function layer U24 on the pore image LM2 along the Z-axis direction to obtain the main display image LC.

[0088] Next, combined with the above figure Figure 6 The process of rendering the user interface is shown in Figure 7 , Fig. 8A and Figure 8B The situation shown introduces the technical solution of the present application.

[0089] Fig. 9 According to some embodiments of the present application, a flow chart of a display method is shown. The execution subject of the flow chart may be an electronic device. Fig. 9 As shown, the process includes the following steps:

[0090] S901: Determine whether a layer with a high rate of change exists in the first interface to be rendered.

[0091] The electronic device can determine whether there is a high-change rate layer in the first interface to be rendered, and trigger the display method provided in the embodiment of the present application when it is determined that there is a high-change rate layer in the first interface to be rendered.

[0092] For example, for the aforementioned Figures 4 to 8B In the case shown, the first interface may be the user interface U1. When there is a video layer (e.g., video layer U22) in the user interface U1, the electronic device may determine that there is a high-change rate layer in the first interface to be rendered, triggering the display method provided in the embodiment of the present application.

[0093] Exemplarily, in some embodiments, the electronic device can obtain the types of each layer included in the first interface from the rendering tree of each application corresponding to the first interface (or other information recording the types of each layer, which may also be referred to as interface data), and determine that there is a high-change rate layer in the first interface to be rendered if there is a layer type that matches the type corresponding to a preset high-change rate layer.

[0094] In some embodiments, the types of layers may include, but are not limited to, videos, web pages, game screens, canvases, clocks, text, pictures, buttons, data panels, date selectors, card decks, frame animations, menus, etc. The types corresponding to high-change-rate layers may include videos, web pages, game screens, canvases, etc.

[0095] Exemplarily, in some embodiments, the electronic device may also obtain the content change frequency of each layer from the rendering tree of each application corresponding to the first interface (or other information that records the content change frequency of each layer (for example, the frame rate, refresh rate, etc. of the layer)), and determine whether there is a high-change rate layer in the first interface to be rendered if there is a layer with a content change frequency greater than a preset frequency.

[0096] Exemplarily, in some embodiments, the electronic device may also determine that there is a high-change rate layer in the first interface to be rendered based on the applications corresponding to each layer in the first interface. For example, the electronic device may pre-store a list of applications with high-change rate layers, and when there is an application in the list among the applications corresponding to the first interface, it is determined that there is a high-change rate layer in the first interface to be rendered.

[0097] It should be understood that in other embodiments, the electronic device may also determine that there is a high-change rate layer in the first interface to be rendered by other means.

[0098] S902: When the content in the main layer corresponding to the first interface does not change compared with the previous frame interface of the first interface, call the GPU to render the high-change rate layer to obtain a high-change rate display image corresponding to the first interface.

[0099] After determining that there is a high-change rate layer in the first interface to be rendered, the electronic device can first determine the main layer and the high-change rate layer in the layer corresponding to the first interface. Then, the electronic device can determine whether the content of the main layer in the first interface has changed compared to the content of the main layer in the previous frame interface of the first interface (hereinafter referred to as the second interface). If the content in the main layer corresponding to the first interface has not changed compared to the second interface, the electronic device can call the GPU to render the high-change rate layer to obtain a high-change rate display image corresponding to the first interface.

[0100] For example, for the aforementioned Figures 4 to 8B In the situation shown, the first interface may be the user interface U1, and the second interface may be the user interface U0. The electronic device may determine that the video layer U22 in the user interface U1 is a high-change rate layer, and the other layers in the user interface U1 (the background layer U11 of the desktop 1, the status bar layer U12, the background layer U21 of the video 2, the title / control layer U23, and the function layer U24) are the main layers. Since the display content in the video layer U22 in the user interface U1 has changed relative to the video layer U22 in the user interface U0, and the content in the main layer in the user interface U1 is the same as that in the user interface U0, the electronic device may call the GPU to re-render the changed video layer U22 to obtain the aforementioned high-change rate display image LV1.

[0101] In some embodiments, the electronic device may determine the high-change rate layer according to the types of the layers corresponding to the first interface, and determine the layers other than the high-change rate layer as the main layer. For example, if the type of a layer is the type corresponding to the aforementioned high-change rate layer, the electronic device may determine the layer as the high-change rate layer.

[0102] In some embodiments, the electronic device may also store identifiers of high change rate layers in different applications (e.g., layer names, unique identification codes, etc.), thereby determining that a layer is a high change rate layer when the identifier of a layer of an application is one of the identifiers of the high change rate layers of the application.

[0103] In some embodiments, the electronic device may also obtain the frequency of change of the content of each layer (eg, frame rate, refresh rate, etc.), and when the frequency of change of the content of a certain layer is greater than a preset frequency, the layer is determined as a high-change-rate layer.

[0104] It should be understood that in other embodiments, the electronic device may also determine the high-change rate layer and the main layer in other ways, which are not limited here.

[0105] In some embodiments, the electronic device can determine whether the content of the main layer in the first interface has changed compared to the content of the main layer in the second interface according to whether there is a dirty area in the main layer of the first interface. For example, if there is no dirty area in the main layer of the first interface, it is determined that the content of the main layer in the first interface has not changed compared to the content of the main layer in the second interface.

[0106] It should be understood that a dirty area refers to an area in a user interface that has changed relative to the content of the previous frame of the user interface. The operating system of the electronic device can determine the dirty area in the user interface through relevant services or modules, and provide a calling interface for dirty area information (such as the size of the dirty area, the layer corresponding to the dirty area, etc.) for other services, applications or modules in the electronic device. The rendering service can determine whether there is a dirty area in the interface to be displayed through the calling interface.

[0107] In some embodiments, when there are multiple high-change rate layers, the high-change rate display images corresponding to the high-change rate layers may be one or more.

[0108] For example, Fig. 10A According to some embodiments of the present application, a schematic diagram of a user interface U2 is shown.

[0109] like Fig. 10A As shown, compared with the aforementioned user interface U1, the user interface U2 also superimposes a clock layer U31 corresponding to the stopwatch application on the upper layer of the function layer U24. Since the refresh frequency of the time in the clock layer U31 is higher than the preset frequency, the electronic device can determine both the clock layer U31 and the video layer U22 as high-change rate layers. Since the clock layer U31 and the video layer U22 are discontinuous, the electronic device can call the GPU to render the clock layer U31 and the video layer U22 respectively to obtain two high-resolution display images.

[0110] For example, Fig. 10B According to some embodiments of the present application, a schematic diagram of a user interface U3 is shown.

[0111] like Fig. 10B As shown, compared with the aforementioned user interface U1, the user interface U3 inserts a clock layer U25 between the video layer U22 and the title / control layer U23, and the user displays the video playback progress. Since the refresh frequency of the time in the clock layer U25 is higher than the preset frequency, the electronic device can determine both the clock layer U25 and the video layer U22 as high-change rate layers. Since the clock layer U25 and the video layer U22 are continuous, the electronic device can call the GPU to render the clock layer U25 and the video layer U22 to obtain a high-resolution display image.

[0112] In some embodiments, the electronic device can render the high-change rate layer through the service or application corresponding to the high-change rate layer (for example, the media service corresponding to the video layer or the image captured by the camera, the game application corresponding to the layer corresponding to the game screen, the browser application corresponding to the layer corresponding to the web page content, etc.) to obtain a high-change rate display image corresponding to the first interface.

[0113] S903: Calling a hardware synthesis module to synthesize a main display image corresponding to a previous frame interface of the first interface and a high-change rate display image corresponding to the first interface to obtain a first interface.

[0114] After obtaining the high-change rate display image corresponding to the first interface, the electronic device can call the hardware synthesis module to synthesize the main display image corresponding to the previous frame interface (second interface) of the first interface and the high-change rate display image corresponding to the first interface to obtain the first interface.

[0115] For example, for the aforementioned Figures 4 to 8B In the situation shown, after obtaining the high-change rate display image LV1 corresponding to the user interface U1, the electronic device can call the hardware synthesis module to synthesize the high-change rate display image LV1 and the main display image LC corresponding to the user interface U0 to obtain the user interface U1.

[0116] In some embodiments, the content of the lower layer of the high-change rate layer in the main display image corresponding to the second interface in the area where the high-change rate layer is located is transparent, and the content of the upper layer of the high-change rate layer in the main display image corresponding to the second interface in the area where the high-change rate layer is located is retained. The electronic device can superimpose the main display image corresponding to the second interface on the high-change rate display image corresponding to the first interface along the Z-axis direction to obtain the first interface. For example, for the aforementioned Figures 4 to 8B In the situation shown, the electronic device can call the hardware synthesis module to superimpose the main display image LC on the high-change rate display image LV1 along the Z-axis direction to obtain the user interface U1.

[0117] For example, for Fig. 10A In the case shown, the main display image may be the aforementioned main display image LC. The rendering service in the electronic device may call the hardware synthesis module to superimpose the main display image LC on the high-change rate display image corresponding to the video layer U22 along the Z-axis direction, and to superimpose the high-change rate display image corresponding to the clock layer U31 on the main display image LC along the Z-axis direction, to obtain the user interface U2.

[0118] For example, for Fig. 10B In the situation shown, the main display image may be the aforementioned main display image LC, and the high-change rate display image may be a high-change rate display image obtained by rendering the clock layer U25 and the video layer U22. The rendering service in the electronic device may call the hardware synthesis module to superimpose the main display image LC on a high-change rate display image corresponding to the rendered clock layer U25 and the video layer U22 along the Z-axis direction to obtain the user interface U3.

[0119] S904: When the content in the main layer corresponding to the first interface changes compared with the previous frame interface of the first interface, the GPU is called to render the main layer to obtain the main display image corresponding to the first interface, and to render the high-change rate layer to obtain the high-change rate display image corresponding to the first interface.

[0120] After determining that there is a high-change rate layer in the first interface to be rendered, the electronic device can first determine the main layer and the high-change rate layer in the layer corresponding to the first interface. Then, the electronic device can determine whether the content of the main layer in the first interface has changed compared to the content of the main layer in the previous frame interface of the first interface (hereinafter referred to as the second interface). If the content in the main layer corresponding to the first interface has changed compared to the second interface, the electronic device can call the GPU to render the main layer to obtain the main display image corresponding to the first interface, and render the high-change rate layer to obtain the high-change rate display image corresponding to the first interface.

[0121] For example, refer to Fig.11A , assuming that the first interface is the user interface U4, and the time in the status bar layer U12 in the user interface U4 changes from "10:08" in the user interface U1 to "10:09", the electronic device can call the GPU to render the background layer U11, status bar layer U12, background layer U21, title / control layer U23 and function layer U24 of the desktop 1 corresponding to the user interface U4 to obtain the main display image LC1. The electronic device can also call the GPU to render the video layer U22 to obtain the aforementioned high-change rate display image LV1.

[0122] S905: Calling a hardware synthesis module to synthesize a main display image corresponding to the first interface and a high-change rate display image corresponding to the first interface to obtain a first interface.

[0123] After obtaining the main display image corresponding to the first interface and the high-change rate display image corresponding to the first interface, the electronic device can call the hardware synthesis module to synthesize the main display image corresponding to the first interface and the high-change rate display image corresponding to the first interface to obtain the first interface.

[0124] For example, refer to Fig. 11B ,for Fig.11A In the situation shown, the electronic device can superimpose the main display image LC1 on the high-change rate display image LV1 along the Z-axis direction to obtain the user interface U4 (first interface).

[0125] S906: Display the first interface.

[0126] After obtaining the first interface, the electronic device may display the first interface when a vertical synchronization signal corresponding to the first interface is detected.

[0127] Based on the above method, when the electronic device is rendering the first interface, if the display content in the main layer in the first interface does not change corresponding to the second interface, there is no need to render the main layer, which is beneficial to reducing the power consumption of the electronic device in rendering the user interface.

[0128] Based on the aforementioned embodiments, the embodiment of the present application further provides a display method.

[0129] For example, Fig.12 According to some embodiments of the present application, a flow chart of another display method is shown. The execution subject of the flow can be an electronic device, such as Fig.12 As shown, the method comprises the following steps:

[0130] S1201: Determine interface data of a first interface to be displayed.

[0131] Before generating the first interface to be displayed, the electronic device may obtain interface data of the first interface. In some embodiments, the interface data may include N layers corresponding to multiple application programs, where N is an integer greater than 2.

[0132] In some embodiments, the electronic device may obtain the interface data of the first interface from the rendering tree of each application corresponding to the first interface.

[0133] In some embodiments, the interface data may further include the type of each layer in the N layers corresponding to the first interface, the frequency of content change of each layer, etc.

[0134] S1202: Select P first layers from the N layers according to the change frequency of the displayed content in the N layers corresponding to the first interface.

[0135] After obtaining the interface data of the first interface, the electronic device can select P first layers (i.e., the aforementioned high-change rate layers) from the N layers according to the change frequency of the displayed content in the N layers corresponding to the first interface, where P is an integer greater than 1 and less than N.

[0136] In some embodiments, the electronic device may determine a layer among the N layers that satisfies at least one of the following conditions as the first layer:

[0137] The change frequency of the displayed content is greater than the first frequency;

[0138] The layer is a layer corresponding to the game screen in a game application, or a layer corresponding to a video, or a layer corresponding to web page content in a browser application, or a layer corresponding to an image captured by a camera of an electronic device.

[0139] Exemplarily, for the aforementioned user interface U1, the first layer may include a video layer U22; for the aforementioned user interface U2, the first layer may include a video layer U22 and a clock layer U31.

[0140] In other embodiments, the electronic device may also determine the first layer in other ways, which are not limited here.

[0141] S1203: Render the P first layers to obtain at least one first display image.

[0142] After determining the P first layers, the electronic device may render the P first layers to obtain at least one first display image (eg, the aforementioned high-change rate display image).

[0143] In some embodiments, the electronic device may render the P first layers through a GPU to obtain at least one first display image.

[0144] In some embodiments, the electronic device can render the P first layers through applications (such as camera applications, video applications, browser applications, etc.) or services (such as media services, etc.) corresponding to the P first layers to obtain at least one first display image.

[0145] In some embodiments, the electronic device may render multiple first layers in a continuous stacking order in the positive direction of the Z axis among the P first layers as a first display image; and render each first layer in the P first layers that is not adjacent to other first layers in the stacking order in the positive direction of the Z axis as a first display image.

[0146] For example, for the user interface U1, the electronic device may combine the video layer 22 into a change rate display image LV1 (as a first display image). For the user interface U3, the electronic device may combine the video layer 22 and the clock layer U25 into a first display image.

[0147] S1204: When the first display contents corresponding to the NP second layers except the first layer in the N layers are unchanged relative to the second interface, synthesize at least one first display image and a second display image to obtain a first interface.

[0148] After obtaining the first display image, the electronic device can determine whether the first display content corresponding to the NP second layers in the N layers except the first layer has changed relative to the previous frame interface (second interface) of the first interface, and synthesize at least one first display image and a second display image (such as the aforementioned main display image) to obtain the first interface when the content of the second layer has not changed relative to the second interface. The specific synthesis process can refer to the aforementioned step S903, which will not be described in detail here.

[0149] It should be understood that the second display image is a display image corresponding to the first display content used by the electronic device in the process of generating the second interface. For example, the second display image can be rendered by the electronic device based on the first display content in the process of generating the second interface. For another example, if the display content in the layer corresponding to the first display content in the second interface has not changed from the previous frame interface of the second interface, the second display image can also be rendered by the electronic device based on the first display content in the process of generating the previous frame interface of the second interface.

[0150] In some embodiments, the electronic device may synthesize at least one first display image and a second display image into a first interface through a hardware synthesis module.

[0151] In some embodiments, the electronic device may synthesize at least one first display image and a second display image into a first interface by using a process or thread corresponding to a rendering service.

[0152] It should be understood that the specific manner in which the electronic device synthesizes at least one first display image and a second display image can refer to the description of S903 above, and will not be elaborated here.

[0153] S1205: Display the first interface.

[0154] After synthesizing the first interface, the electronic device can display the first interface.

[0155] Through this method, if the first display content corresponding to the NP second layers (such as the main layer hereinafter) in the first interface to be displayed by the electronic device has not changed compared to the previous frame interface (second interface) of the first interface, the electronic device does not need to re-render the second layer in the process of generating the first interface. The electronic device can render the P first layers (such as the high-change rate layers hereinafter) in the first interface to be displayed to obtain at least one first display image (such as the high-change rate display image hereinafter), and then synthesize the at least one display image and the second display image (such as the main display image hereinafter) corresponding to the first display content used by the electronic device in the process of generating the second interface to obtain the first interface. In this way, the number of layers rendered by the electronic device can be reduced, which is conducive to reducing the GPU resources occupied in the interface generation process, improving the efficiency of image rendering, and reducing the power consumption of the GPU.

[0156] Furthermore, the embodiment of the present application also provides an electronic device 10 for implementing the display methods provided in the aforementioned embodiments.

[0157] For example, Fig.13 According to some embodiments of the present application, a schematic structural diagram of an electronic device 10 is shown.

[0158] like Fig.13 As shown, the electronic device 10 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

[0159] The processor 110 may include one or more processing units, for example, the processor 110 may include a CPU, an application processor, a modem processor, a GPU, an image signal processor (ISP), a microcontroller unit (MCU), a video codec, a DSP, a baseband processor, an NPU, a field programmable gate array (FPGA), etc. In some embodiments, different processing units may be independent devices or integrated into one or more processors.

[0160] In some embodiments, the processor 110 may further include a hardware synthesis module for synthesizing the multiple display images synthesized by the GPU to obtain a user interface of the electronic device 10 .

[0161] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. When the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0162] In some embodiments, the processor 110 may be used to execute instructions corresponding to the display methods provided in the aforementioned embodiments. For example, the processor 110 may be used to execute instructions for determining a high-change rate layer and a main layer in a first interface to be displayed, an instruction for determining whether the display content of the main layer in the first interface is the same as the display content of the main layer in the previous frame user interface (second interface) of the first interface, an instruction for calling a GPU to render the main layer as a main display image, an instruction for calling a GPU to render a high-change rate layer as a high-change rate display image, an instruction for calling a hardware synthesis module to synthesize the main display image and the high-change rate display image into a user interface, etc.

[0163] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, etc.

[0164] The USB interface 130 is an interface that complies with USB standard specifications, and may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 may be used to connect a charger to charge the electronic device 10, and may also be used to transmit data between the electronic device 10 and a peripheral device.

[0165] The charging management module 140 is used to receive charging input from a charger. While the charging management module 140 is charging the battery 142 , it can also power the electronic device through the power management module 141 .

[0166] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160.

[0167] The wireless communication function of the electronic device 10 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0168] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0169] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 10. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0170] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 10. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0171] The electronic device 10 implements the display function through a GPU, a display screen 194, and a CPU.

[0172] The display screen 194 is used to display images, videos, etc. In some embodiments, the electronic device 10 may include 1 or N display screens 194 , where N is a positive integer greater than 1.

[0173] In some embodiments, display screen 194 may be used to display the aforementioned user interface.

[0174] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 10 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0175] In some embodiments, the image captured by the camera 193 can be used as the content of the layer in the application. When the content in the layer of the application is the image captured by the camera 193, the layer can be used as a high-change rate layer.

[0176] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 10. The external memory card communicates with the processor 110 via the external memory interface 120 to implement a data storage function.

[0177] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. In some embodiments, the program storage area may store an operating system, an application required for at least one function, etc., such as instructions for the display method provided in the aforementioned embodiments. The data storage area may store data created during the use of the electronic device 10, etc. In addition, the internal memory 121 may include a volatile memory and a non-volatile memory. The processor 110 executes various functional applications of the electronic device 10 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor 110.

[0178] The electronic device 10 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0179] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals.

[0180] The microphone 170C, also called a “microphone” or “microphone”, is used to convert sound signals into electrical signals. For example, in some embodiments, the microphone 170C can be used to collect audio data and send electrical signals corresponding to the collected audio data to the audio detection module 61.

[0181] The earphone jack 170D is used to connect a wired earphone.

[0182] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194 .

[0183] The air pressure sensor 180C may be used to detect the air pressure at the location where the electronic device 10 is located.

[0184] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 10 in various directions (generally three axes).

[0185] The ambient light sensor 180L is used to sense the brightness of ambient light.

[0186] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 10 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0187] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it.

[0188] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 10 may receive key input and generate key signal input related to user settings and function control of the electronic device 10.

[0189] Motor 191 can generate vibration prompts.

[0190] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.

[0191] The SIM card interface 195 is used to connect a SIM card.

[0192] It is understood that the structure of the electronic device 10 shown in the embodiment of the present application does not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0193] It should be understood that the electronic device or the aforementioned electronic device 10 mentioned in the embodiments of the present application can be any electronic device, including but not limited to mobile phones, tablet computers, desktops, laptops, handheld computers, netbooks, and wearable devices such as augmented reality (AR) / virtual reality (VR) devices, smart TVs, smart watches, car equipment, portable game consoles, reader devices, televisions embedded or coupled with one or more processors, or other electronic devices capable of accessing the Internet. The embodiments of the present application do not limit the specific form of the electronic device.

[0194] further, Fig.14 According to some embodiments of the present application, a schematic diagram of a software architecture of an electronic device 10 is shown.

[0195] like Fig.14 As shown, the software architecture of the electronic device 10 may include an application layer 01 , a framework layer 02 , a system service layer 03 and a kernel layer 04 .

[0196] The application layer 01 may include applications in the electronic device 10, such as the aforementioned desktop 1, video 2, etc. The applications in the electronic device are used to render respective high-change rate layers to obtain high-change rate display images.

[0197] When the application in the application layer 01 is running, the interface data (such as the rendering tree) of the application can be passed to the rendering service 031 so that the rendering service can generate a unified rendering tree according to the rendering tree of the application and render the user interface of the electronic device 10.

[0198] It should be understood that in some embodiments, the application layer 01 may include more or fewer application programs, which is not limited here.

[0199] The framework layer 02 provides a multi-language framework for the application layer, including the interface Ark development framework (ArkUI) 021, the user program framework 022 and the capability framework 023, etc.

[0200] Ark Development Framework (ArkUI) 021 is used to provide a support framework for different user interface development paradigms, such as declarative development paradigms, Web-like development paradigms, etc.

[0201] The user program framework 022 and the capability framework 023 can provide the application with the capabilities of various capability components required for the application, such as computing power (including CPU computing power, GPU computing power, ISP computing power, image synthesis computing power, etc.), sound pickup capability (including microphone sound pickup capability, voice recognition capability, etc.), device security protection security capabilities (including trusted operating environment security level, etc.), display capability, playback capability, etc., without limitation here.

[0202] The system service layer 03 is used to provide services to the application programs in the application layer 01 through the framework layer 02, including but not limited to basic services corresponding to the operation, scheduling, migration and other operations of the application, distributed services corresponding to distributed soft buses, distributed data management, distributed task scheduling, etc., software services such as event notifications, telephones, multimedia, and hardware services such as location services.

[0203] In some embodiments, the system service layer 03 may include a rendering service 031 and a media service 032 .

[0204] The rendering service 031 is used to obtain a unified rendering tree according to the rendering trees of each application program in the application layer 01 , and render the user interface of the electronic device 10 based on the unified rendering tree.

[0205] In some embodiments, the rendering service 031 can be used to implement the display methods provided in the aforementioned embodiments. For example, the rendering service 031 can be used to determine the main layer and the high-change rate layer in the user interface to be displayed, call the GPU to render the main layer to obtain the main display image, call the hardware synthesis module to synthesize the main display image and the high-change rate display image into a user interface, etc.

[0206] In some embodiments, the rendering service 031 can also be used to determine whether the main layer in the user interface to be displayed (e.g., the aforementioned first interface) has changed relative to the previous frame user interface of the user interface (e.g., the aforementioned second interface), and if there is no change, call the hardware synthesis module to synthesize the main display image corresponding to the main layer in the previous frame user interface and the high-change rate display image corresponding to the high-change rate layer in the user interface to be displayed to obtain the user interface to be displayed.

[0207] In some embodiments, the rendering service 031 may also be used to render at least part of the high-change rate layers to obtain a high-change rate display image.

[0208] The media service 032 is used for multimedia playback and collection in the electronic device 10. In some embodiments, the media service 032 can be used to render the video data corresponding to the video layer in the application or the video data collected by the camera of the electronic device into an image and store it in the video cache.

[0209] The kernel layer 04 includes a kernel subsystem 041 and a driver subsystem 042. The kernel subsystem 041 provides basic kernel capabilities to the upper layer by shielding multi-kernel differences, including process / thread management, memory management, file system, network management, and peripheral management. The driver subsystem 042 includes a hardware driver framework, which can provide the electronic device 10 with a unified peripheral access capability and management framework.

[0210] In some embodiments, the hardware driver framework can provide hardware access interfaces to the application layer 01, the framework layer 02, and the kernel layer 03, such as the GPU call interface, the hardware synthesis module call interface, etc. In this way, the application layer 01, the framework layer 02, and the kernel layer 03, the application layer / service / module can call the GPU rendering layer through the GPU call interface to obtain the display interface, and call the hardware synthesis module through the hardware synthesis module call interface to synthesize the display image rendered by the GPU into a user interface.

[0211] Understandably, Fig.14 The software architecture shown is only an example. In other embodiments, the software architecture of the electronic device 10 may include more or fewer modules, or may merge or split some modules, or may adopt a different approach from the above. Fig.14 As for other software architectures shown, the embodiment of the present application does not limit the software architecture of the electronic device 10.

[0212] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0213] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "includes one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A display method, applied to an electronic device, characterized in that: include: Determine interface data of a first interface to be displayed, wherein the interface data includes N layers corresponding to a plurality of application programs, where N is an integer greater than 2; Selecting P first layers from the N layers according to a change frequency of displayed content in the N layers; Rendering the P first layers to obtain at least one first display image; When the first display contents corresponding to the NP second layers among the N layers except the first layer are unchanged relative to the second interface, the at least one first display image and the second display image are synthesized to obtain the first interface, The second interface is a previous frame interface of the first interface, and the second display image is a display image corresponding to the first display content and used by the electronic device in the process of generating the second interface; The first interface is displayed.

2. The method according to claim 1, characterized in that: The selecting P first layers from the N layers according to the change frequency of the displayed contents in the N layers comprises: A layer among the N layers that satisfies at least one of the following conditions is determined as the first layer: The change frequency of the display content of the layer is greater than the first frequency; The layer is any one of the following layers: a layer corresponding to a game screen in a game application, a layer corresponding to a video, a layer corresponding to web page content in a browser application, and a layer corresponding to an image captured by a camera of an electronic device.

3. The method according to claim 1, characterized in that: The method further comprises: When the first display contents corresponding to the NP second layers are changed relative to the second interface, the NP second layers are rendered to obtain a third display image; The at least one first display image and the third display image are synthesized to obtain the first interface.

4. The method according to claim 1, characterized in that The electronic device includes a graphics processor; and the rendering of the P first layers to obtain at least one first display image includes: The image processor is called to render the P first layers to obtain at least one first display image.

5. The method according to any one of claims 1 to 4, characterized in that The electronic device further includes a hardware synthesis module; and the synthesizing the at least one first display image and the second display image to obtain the first interface includes: The at least one first display image and the second display image of the hardware synthesis module are called to obtain the first interface.

6. The method according to any one of claims 1 to 4, characterized in that A first service is running in the operating system of the electronic device; and the electronic device determines interface data of a first interface to be displayed through a first process corresponding to the first service, and synthesizes the at least one first display image and the second display image when the first display contents corresponding to NP second layers among the N layers except the first layer do not change relative to the second interface, so as to obtain the first interface.

7. The method according to claim 6, characterized in that The second display image is rendered by the first process based on the first display content.

8. The method according to claim 1, characterized in that The synthesizing the at least one first display image and the second display image to obtain the first interface includes: The first interface is obtained by synthesizing the at least one first display image and the second display image based on the superposition sequence of the first display image and the second display image in a first direction, wherein the first direction is a direction perpendicular to the display screen of the electronic device and facing outward.

9. The method according to claim 8, characterized in that The P first layers include Q first layers that are continuously stacked in the first direction, and the at least one first display image includes a fourth display image corresponding to the Q first layers.

10. A readable storage medium, characterized in that: The readable storage medium includes instructions, and when the instructions are executed by the electronic device, the electronic device implements the display method according to any one of claims 1 to 9.

11. An electronic device, characterized in that: include: A memory for storing instructions; A processor, configured to execute the instructions so that the electronic device implements the display method according to any one of claims 1 to 9.

Images