Interface generating method and electronic device

By merging render trees and optimizing rendering processes, the method reduces power consumption and improves user experience in electronic devices by minimizing redundant computations and resource usage.

JP2024544612AActive Publication Date: 2024-12-03HUAWEI TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2024531374
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-25
Filing Date
2022-11-23
Publication Date
2024-12-03
Estimated Expiration
2042-11-23

AI Technical Summary

Technical Problem

Electronic devices consume significant computing and storage resources to generate application interfaces, particularly when multiple applications or windows are displayed, leading to increased power consumption and reduced user experience.

Method used

The method involves merging render trees of multiple applications into a single target render tree, optimizing parameters to reduce rendering times and power consumption by eliminating fully occluded views and operations, and performing off-screen rendering directly on the bitmap without layer compositing.

Benefits of technology

This approach reduces rendering times, power consumption, and frame freezing, thereby improving user experience by minimizing redundant computations and resource usage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024544612000001_ABST
    Figure 2024544612000001_ABST
Patent Text Reader

Abstract

The present application discloses an interface generating method and an electronic device, and relates to the field of electronic technology. According to the interface generating method provided in the present application, the render trees of one or more applications in one display area may be merged, and an interface including one or more application interfaces is generated through one rendering based on the merged target render tree, thereby reducing the rendering times, reducing the power consumption of the electronic device, and improving the user experience.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of International Application No. PCT / CN2022 / 133838, filed on November 23, 2022, which claims priority to Chinese Patent Application No. 202111410513.6, filed on November 25, 2021. The disclosures of the aforementioned applications are incorporated herein by reference in their entireties.

[0002] The present application relates to the field of electronic technology, and in particular to an interface generating method and electronic device. [Background technology]

[0003] With the development of electronic technology, more and more electronic devices are involved in users' daily lives. In addition, as parameters such as the resolution and size of the display of electronic devices increase, more and more content can be displayed on the electronic devices.

[0004] However, before displaying the interface of the application, the electronic device needs to consume computing and storage resources to generate the interface of the application, which increases the power consumption of the electronic device. In addition, when there are multiple applications or multiple windows on the screen of the electronic device, the electronic device needs to consume more computing and storage resources to perform rendering to generate the interfaces of the multiple applications or multiple windows. Summary of the Invention [Means for solving the problem]

[0005] The embodiments of the present application provide an interface generation method and an electronic device. According to the interface generation method provided in the present application, the render trees of one or more applications in one display area may be merged, and an interface including one or more application interfaces is generated through one rendering based on the merged target render tree, thereby reducing the rendering times, reducing the power consumption of the electronic device, and improving the user experience.

[0006] According to a first aspect, an interface generating method provided in an embodiment of the present application is applied to a first electronic device, the first electronic device determines that a content displayed in a first display area includes an interface of a first process and an interface of a second process, the method includes: the first process generates a first render tree, the first render tree is used to render the interface of the first process; the second process generates a second render tree, the second render tree is used to render the interface of the second process; and the third process generates a first target interface based on the first render tree and the second render tree, the first target interface includes the interface of the first process and the interface of the second process, and the first target interface is displayed in the first display area.

[0007] In the aforementioned embodiment, an interface including one or more application interfaces is generated based on a render tree of one or more applications in a display area, thereby reducing rendering times, reducing power consumption of the electronic device, and improving user experience.

[0008] With reference to some embodiments of the first aspect, in some embodiments, the third process generating the first target interface based on the first render tree and the second render tree specifically includes: the third process generating the first target render tree based on the first render tree and the second render tree; and the third process generating the first target interface based on the first target render tree.

[0009] In the above embodiment, the third process may merge the render trees of one or more applications into one target render tree, so that an interface may be generated based on the target render tree. The interface includes the interfaces of one or more applications. It is clear that the interface including one or more applications can be generated through one rendering, i.e., one rendering process from the render tree to a bitmap, so that the rendering times are reduced and the power consumption of the electronic device is reduced.

[0010] With reference to some embodiments of the first aspect, in some embodiments, the third process generating the first target render tree based on the first render tree and the second render tree specifically includes the third process creating a root render node as a root node of the first target render tree and using the first render tree and the second render tree as child nodes of the root render node.

[0011] In the above embodiment, a new root node may be created and the render trees of multiple applications may be mounted on child nodes of the root node to facilitate the creation of a target render tree.

[0012] Referring to some embodiments of the first aspect, in some embodiments, the method further includes a third process removing a render node in the first target render tree based on the Z-order of the first render tree and the Z-order of the second render tree, where the removed render node corresponds to a fully occluded view.

[0013] In the aforementioned embodiment, a third process may optimize parameters in the target render tree to reduce overdraw, for example by removing render nodes that correspond to fully occluded views in the interface.

[0014] Referring to some embodiments of the first aspect, in some embodiments, the method further includes a third process removing drawing operations in the first target render tree based on the Z-order of the first render tree and the Z-order of the second render tree, where the removed drawing operations correspond to fully occluded graphics.

[0015] In such an embodiment, a third process may optimize parameters in the target render tree to reduce overdraw, for example by removing render nodes that correspond to fully occluded graphics in the interface.

[0016] Referring to some embodiments of the first aspect, in some embodiments, the method further includes, in a process in which a third process generates a first target interface based on the first render tree and the second render tree, the third process performs merging or batching on the first drawing operation and the second drawing operation, where the first drawing operation belongs to the first render tree and the second drawing operation belongs to the second render tree.

[0017] In the aforementioned embodiment, the third process may merge or batch drawing operations in the target render tree to increase the interface generation rate, reduce frame freezes, reduce power consumption of the electronic device, and improve the user experience.

[0018] Referring to some embodiments of the first aspect, in some embodiments, the method further includes: a third process determining an off-screen rendering logic of an interface of the first process, the off-screen rendering logic including at least one of window rounding, color conversion, rotation, and scaling; and the third process adding off-screen rendering properties to the rendering properties of the first render tree based on the off-screen rendering logic of the interface of the first process, the off-screen rendering properties including at least one of a rounding property, a color property, a rotation property, and a scaling property. The off-screen rendering properties correspond one-to-one to the off-screen rendering logic, and the off-screen rendering properties are used to modify drawing operations in the process in which the third process generates the first target interface based on the first render tree and the second render tree to realize the off-screen rendering logic.

[0019] In the above embodiment, the off-screen rendering property is added to the render tree, so that in the process of performing drawing operations on a surface, the off-screen rendered graphics can be directly drawn based on the off-screen rendering property, and there is no need to perform off-screen rendering on the entire bitmap and then perform layer compositing after the drawing is completed. This can avoid off-screen rendering, thereby reducing the number of rendering times and off-screen rendering overhead, reducing the power consumption of electronic devices, and improving the user experience.

[0020] Referring to some embodiments of the first aspect, in some embodiments, a reference coordinate system of the first render tree is a first coordinate system, and a reference coordinate system corresponding to the first display area is a second coordinate system, and when the first coordinate system and the second coordinate system are different, the method further includes: a third process determining a first parameter based on the first coordinate system and the second coordinate system and adding the first parameter to a rendering property of the first render tree; and in a process in which the third process generates a first target interface based on the first render tree and the second render tree, the third process modifies a reference coordinate system of a first drawing operation based on the first parameter, and the first drawing operation belongs to the first render tree.

[0021] In the above embodiment, when the reference coordinate system of the render tree generated by the application is different from the reference coordinate system in which the third process performs drawing operations in the render tree, in order to achieve a correct drawing operation, a first parameter is added to the rendering properties, and then the reference coordinate system of the drawing operation is transformed based on the first parameter.

[0022] According to a second aspect, an interface generation method provided in an embodiment of the present application includes: a third process operating on a second electronic device receiving a first render tree and a second render tree, where the first render tree is generated by the first process operating on the first electronic device, and the first render tree is used to render an interface of the first process, and the second render tree is generated by the second process operating on the second electronic device, and the second render tree is used to render an interface of the second process; and the third process generating a target interface based on the first render tree and the second render tree, where the target interface includes an interface of the first process and an interface of the second process, and the target interface is displayed in the first display area.

[0023] In the aforementioned embodiment, an interface including one or more application interfaces is generated based on a render tree of one or more applications in a display area, thereby reducing rendering times, reducing power consumption of the electronic device, and improving user experience.

[0024] With reference to some embodiments of the second aspect, in some embodiments, the third process generating the first target interface based on the first render tree and the second render tree specifically includes: the third process generating the first target render tree based on the first render tree and the second render tree; and the third process generating the first target interface based on the first target render tree.

[0025] In the above embodiment, the third process may merge the render trees of one or more applications into one target render tree, so that an interface may be generated based on the target render tree. The interface includes the interfaces of one or more applications. It is clear that the interface including one or more applications can be generated through one rendering, i.e., one rendering process from the render tree to a bitmap, so that the rendering times are reduced and the power consumption of the electronic device is reduced.

[0026] With reference to some embodiments of the second aspect, in some embodiments, the third process generating the first target render tree based on the first render tree and the second render tree specifically includes the third process creating a root render node as a root node of the first target render tree and using the first render tree and the second render tree as child nodes of the root render node.

[0027] In the above embodiment, a new root node may be created and the render trees of multiple applications may be mounted on child nodes of the root node to facilitate the creation of a target render tree.

[0028] Referring to some embodiments of the second aspect, in some embodiments, the method further includes a third process removing a render node in the first target render tree based on the Z-order of the first render tree and the Z-order of the second render tree, where the removed render node corresponds to a fully occluded view.

[0029] In the aforementioned embodiment, a third process may optimize parameters in the target render tree to reduce overdraw, for example by removing render nodes that correspond to fully occluded views in the interface.

[0030] Referring to some embodiments of the second aspect, in some embodiments, the method further includes a third process removing drawing operations in the first target render tree based on the Z-order of the first render tree and the Z-order of the second render tree, where the removed drawing operations correspond to fully occluded graphics.

[0031] In such an embodiment, a third process may optimize parameters in the target render tree to reduce overdraw, for example by removing render nodes that correspond to fully occluded graphics in the interface.

[0032] Referring to some embodiments of the second aspect, in some embodiments, the method further includes, in a process in which a third process generates a first target interface based on the first render tree and the second render tree, the third process performs merging or batching on the first drawing operation and the second drawing operation, where the first drawing operation belongs to the first render tree and the second drawing operation belongs to the second render tree.

[0033] In the aforementioned embodiment, the third process may merge or batch drawing operations in the target render tree to increase the interface generation rate, reduce frame freezes, reduce power consumption of the electronic device, and improve the user experience.

[0034] Referring to some embodiments of the second aspect, in some embodiments, the method further includes: a third process determining an off-screen rendering logic of an interface of the first process, the off-screen rendering logic including at least one of window rounding, color conversion, rotation, and scaling; and the third process adding off-screen rendering properties to the rendering properties of the first render tree based on the off-screen rendering logic of the interface of the first process, the off-screen rendering properties including at least one of a rounding property, a color property, a rotation property, and a scaling property. The off-screen rendering properties correspond one-to-one to the off-screen rendering logic, and the off-screen rendering properties are used to modify drawing operations in the process in which the third process generates the first target interface based on the first render tree and the second render tree to realize the off-screen rendering logic.

[0035] In the above embodiment, the off-screen rendering property is added to the render tree, so that in the process of performing drawing operations on a surface, the off-screen rendered graphics can be directly drawn based on the off-screen rendering property, and there is no need to perform off-screen rendering on the entire bitmap and then perform layer compositing after the drawing is completed. This can avoid off-screen rendering, thereby reducing the number of rendering times and off-screen rendering overhead, reducing the power consumption of electronic devices, and improving the user experience.

[0036] Referring to some embodiments of the second aspect, in some embodiments, a reference coordinate system of the first render tree is a first coordinate system, and a reference coordinate system corresponding to the first display area is a second coordinate system, and when the first coordinate system and the second coordinate system are different, the method further includes: a third process determining a first parameter based on the first coordinate system and the second coordinate system and adding the first parameter to a rendering property of the first render tree; and in a process in which the third process generates a first target interface based on the first render tree and the second render tree, the third process modifies a reference coordinate system of a first drawing operation based on the first parameter, and the first drawing operation belongs to the first render tree.

[0037] In the above embodiment, when the reference coordinate system of the render tree generated by the application is different from the reference coordinate system in which the third process performs drawing operations in the render tree, in order to achieve a correct drawing operation, a first parameter is added to the rendering properties, and then the reference coordinate system of the drawing operation is transformed based on the first parameter.

[0038] According to a third aspect, an embodiment of the present application provides an electronic device, the electronic device including one or more processors and a memory, the memory coupled to the one or more processors, the memory configured to store a computer program code, the computer program code including computer instructions, the one or more processors calling the computer instructions, such that the electronic device performs the following operations: a first process generates a first render tree, the first render tree is used to render an interface of the first process; a second process generates a second render tree, the second render tree is used to render an interface of the second process; and a third process generates a first target interface based on the first render tree and the second render tree, the first target interface includes an interface of the first process and an interface of the second process, and the first target interface is displayed in a first display area.

[0039] In the aforementioned embodiment, an interface including one or more application interfaces is generated based on a render tree of one or more applications in a display area, thereby reducing rendering times, reducing power consumption of the electronic device, and improving user experience.

[0040] With reference to some embodiments of the third aspect, in some embodiments, the one or more processors are specifically configured to invoke computer instructions such that the electronic device performs the following operations: a third process generates a first target render tree based on the first render tree and the second render tree; and the third process generates a first target interface based on the first target render tree.

[0041] With reference to some embodiments of the third aspect, in some embodiments, the one or more processors are specifically configured to invoke computer instructions such that the electronic device performs the following operation: the third process creates a root render node as a root node of a first target render tree, and uses the first render tree and the second render tree as child nodes of the root render node.

[0042] With reference to some embodiments of the third aspect, in some embodiments, the one or more processors are further configured to invoke computer instructions such that the electronic device performs the following operation: a third process removes a render node in the first target render tree based on an order of the first render tree and a Z-order of the second render tree, where the removed render node corresponds to a fully occluded view.

[0043] With reference to some embodiments of the third aspect, in some embodiments, the one or more processors are further configured to invoke computer instructions such that the electronic device performs the following operation: a third process removes drawing operations in the first target render tree based on the order of the first render tree and the Z-order of the second render tree, where the removed drawing operations correspond to fully occluded graphics.

[0044] Referring to some embodiments of the third aspect, in some embodiments, the one or more processors are further configured to invoke computer instructions such that the electronic device performs the following operation: in a process in which the third process generates a first target interface based on the first render tree and the second render tree, the third process performs merging or batching on the first drawing operation and the second drawing operation, where the first drawing operation belongs to the first render tree and the second drawing operation belongs to the second render tree.

[0045] Referring to some embodiments of the third aspect, in some embodiments, the one or more processors are further configured to invoke computer instructions, such that the electronic device performs the following operations: the third process determines an off-screen rendering logic of the interface of the first process, the off-screen rendering logic including at least one of window rounding, color conversion, rotation, and scaling; and the third process adds off-screen rendering properties to the rendering properties of the first render tree based on the off-screen rendering logic of the interface of the first process, the off-screen rendering properties including at least one of a rounding property, a color property, a rotation property, and a scaling property. The off-screen rendering properties correspond one-to-one to the off-screen rendering logic, and the off-screen rendering properties are used to modify drawing operations in the process of the third process generating the first target interface based on the first render tree and the second render tree to realize the off-screen rendering logic.

[0046] Referring to some embodiments of the third aspect, in some embodiments, the one or more processors are further configured to invoke computer instructions such that the electronic device performs the following operations: a third process determines a first parameter based on the first coordinate system and the second coordinate system and adds the first parameter to a rendering property of the first render tree; and in the process of the third process generating a first target interface based on the first render tree and the second render tree, the third process modifies a reference coordinate system of a first drawing operation based on the first parameter, where the first drawing operation belongs to the first render tree.

[0047] According to a fourth aspect, an embodiment of the present application provides an electronic device, the electronic device including one or more processors and a memory, the memory coupled to the one or more processors, the memory configured to store a computer program code, the computer program code including computer instructions, the one or more processors calling the computer instructions, such that the electronic device performs the following operations: a third process running on a second electronic device receives a first render tree and a second render tree, the first render tree being generated by the first process running on the first electronic device, the first render tree being used to render an interface of the first process, the second render tree being generated by the second process running on the second electronic device, the second render tree being used to render an interface of the second process, and the third process generating a target interface based on the first render tree and the second render tree, the target interface including an interface of the first process and an interface of the second process, the target interface being displayed in a first display area.

[0048] In the aforementioned embodiment, an interface including one or more application interfaces is generated based on a render tree of one or more applications in a display area, thereby reducing rendering times, reducing power consumption of the electronic device, and improving user experience.

[0049] With reference to some embodiments of the fourth aspect, in some embodiments, the one or more processors are specifically configured to invoke computer instructions such that the electronic device performs the following operations: a third process generates a first target render tree based on the first render tree and the second render tree; and the third process generates a first target interface based on the first target render tree.

[0050] With reference to some embodiments of the fourth aspect, in some embodiments, the one or more processors are specifically configured to invoke computer instructions such that the electronic device performs the following operation: the third process creates a root render node as a root node of the first target render tree, and uses the first render tree and the second render tree as child nodes of the root render node.

[0051] With reference to some embodiments of the fourth aspect, in some embodiments, the one or more processors are further configured to invoke computer instructions such that the electronic device performs the following operation: a third process removes a render node in the first target render tree based on the order of the first render tree and the Z-order of the second render tree, where the removed render node corresponds to a fully occluded view.

[0052] With reference to some embodiments of the fourth aspect, in some embodiments, the one or more processors are further configured to invoke computer instructions such that the electronic device performs the following operation: a third process removes drawing operations in the first target render tree based on the order of the first render tree and the Z-order of the second render tree, where the removed drawing operations correspond to fully occluded graphics.

[0053] With reference to some embodiments of the fourth aspect, in some embodiments, the one or more processors are further configured to invoke computer instructions such that the electronic device performs the following operation: in a process in which the third process generates a first target interface based on the first render tree and the second render tree, the third process performs merging or batching on the first drawing operation and the second drawing operation, where the first drawing operation belongs to the first render tree and the second drawing operation belongs to the second render tree.

[0054] Referring to some embodiments of the fourth aspect, in some embodiments, the one or more processors are further configured to invoke computer instructions, such that the electronic device performs the following operations: the third process determines an off-screen rendering logic of the interface of the first process, the off-screen rendering logic including at least one of window rounding, color conversion, rotation, and scaling; and the third process adds off-screen rendering properties to the rendering properties of the first render tree based on the off-screen rendering logic of the interface of the first process, the off-screen rendering properties including at least one of a rounding property, a color property, a rotation property, and a scaling property. The off-screen rendering properties correspond one-to-one to the off-screen rendering logic, and the off-screen rendering properties are used to modify drawing operations in the process of the third process generating the first target interface based on the first render tree and the second render tree to realize the off-screen rendering logic.

[0055] Referring to some embodiments of the fourth aspect, in some embodiments, the one or more processors are further configured to invoke computer instructions such that the electronic device performs the following operations: a third process determines a first parameter based on the first coordinate system and the second coordinate system and adds the first parameter to a rendering property of the first render tree; and in the process of the third process generating a first target interface based on the first render tree and the second render tree, the third process modifies a reference coordinate system of a first drawing operation based on the first parameter, where the first drawing operation belongs to the first render tree.

[0056] According to a fifth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, the chip system including one or more processors, the processors being configured to invoke computer instructions, such that the electronic device performs the method according to any one of the first and second aspects and possible implementations of the first and second aspects.

[0057] According to a sixth aspect, an embodiment of the present application provides a computer program product comprising instructions, which when run on an electronic device, enables the electronic device to perform a method according to any one of the first and second aspects and possible implementations of the first and second aspects.

[0058] According to a seventh aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions, which when run on an electronic device, enable the electronic device to perform a method according to any one of the first and second aspects and possible implementations of the first and second aspects.

[0059] It should be understood that the electronic device provided in the third and fourth aspects, the chip system provided in the fifth aspect, the computer program product provided in the sixth aspect, and the computer storage medium provided in the seventh aspect are all used to perform the method provided in the embodiments of the present application. Therefore, for the beneficial effects that can be achieved, please refer to the beneficial effects of the corresponding methods above. Details will not be described again in this specification. [Brief description of the drawings]

[0060] [Figure 1A] FIG. 2 is a schematic diagram of an example of an interface of an electronic device according to an embodiment of the present application. [Figure 1B] FIG. 2 is a schematic diagram of an example of a display interface of an electronic device according to an embodiment of the present application. [Diagram 2] FIG. 2 is a schematic diagram of an example of generating a bitmap by an application according to an embodiment of the present application; [Diagram 3] FIG. 2 is a schematic diagram of an example of a render tree according to an embodiment of the present application; [Figure 4] FIG. 2 is a schematic diagram of an example in which a bitmap is used as a layer for participating in layer compositing, according to an embodiment of the present application; [Diagram 5] FIG. 2 is a schematic diagram of an example of layer compositing according to an embodiment of the present application; [Figure 6] FIG. 1 is a schematic diagram of another example of how SurfaceFlinger performs layer compositing, according to an embodiment of the present application. [Figure 7A] FIG. 13 is a schematic diagram of another example of layer compositing according to an embodiment of the present application. [Figure 7B-1] FIG. 13 is a schematic diagram of another example of layer compositing according to an embodiment of the present application. [Figure 7B-2] FIG. 13 is a schematic diagram of another example of layer compositing according to an embodiment of the present application. [Figure 8] FIG. 2 is a schematic diagram of an example of a system architecture of an interface generation method according to an embodiment of the present application. [Figure 9]FIG. 2 is a schematic diagram of an example of an interface generation method according to an embodiment of the present application. [Figure 10] FIG. 2 is a schematic diagram of an example of the architecture of UniRender according to an embodiment of the present application. [Figure 11] FIG. 2 is a schematic diagram of an example of a shared memory transferring a render tree according to an embodiment of the present application; [Figure 12A] FIG. 2 is a schematic diagram of an example of a storage structure of a render tree in a shared memory according to an embodiment of the present application; [Figure 12B] FIG. 2 is a schematic diagram of an example of a storage structure of a render tree in a shared memory according to an embodiment of the present application; [Figure 13] FIG. 2 is a schematic diagram of an example of the architecture of UniRender according to an embodiment of the present application. [Figure 14] FIG. 2 is a schematic diagram of an example of an application writing a render tree to a shared memory according to an embodiment of the present application; [Figure 15] FIG. 13 is a schematic diagram of another example in which an application writes a render tree to a shared memory according to an embodiment of the present application. [Figure 16A-1] FIG. 2 is a schematic diagram of an example of a scenario of drawing logic triggering off-screen rendering, according to an embodiment of the present application; [Figure 16A-2] FIG. 2 is a schematic diagram of an example of a scenario of drawing logic triggering off-screen rendering, according to an embodiment of the present application; [Figure 16B] FIG. 2 is a schematic diagram of an example of a scenario of drawing logic triggering off-screen rendering, according to an embodiment of the present application; [Figure 16C] FIG. 2 is a schematic diagram of an example of a scenario of drawing logic triggering off-screen rendering, according to an embodiment of the present application; [Figure 17A] FIG. 2 is a schematic diagram of an example of how UniRender transitions off-screen rendering trigger instructions according to an embodiment of the present application. [Figure 17B]FIG. 2 is a schematic diagram of an example of how UniRender transitions off-screen rendering trigger instructions according to an embodiment of the present application. [Figure 18] FIG. 2 is a schematic diagram of an example of how UniRender modifies render node properties in a render tree according to an embodiment of the present application. [Figure 19] FIG. 1 is a schematic diagram of an example of merging multiple render trees obtained after off-screen rendering instructions are migrated to one target render tree according to an embodiment of the present application. [Figure 20] FIG. 1 is a schematic diagram of an example of merging multiple render trees obtained after off-screen rendering instructions are migrated to one target render tree according to an embodiment of the present application. [Figure 21] FIG. 13 is a schematic diagram of an example of setStaticMatrix() according to an embodiment of the present application. [Figure 22A] FIG. 2 is a schematic diagram of an example of a multi-display region scenario according to an embodiment of the present application. [Figure 22B] FIG. 2 is a schematic diagram of an example of a multi-display region scenario according to an embodiment of the present application. [Figure 22C] FIG. 2 is a schematic diagram of an example of a multi-display region scenario according to an embodiment of the present application. [Figure 23A-1] FIG. 2 is a schematic diagram of another example of a multi-display region scenario according to an embodiment of the present application. [Figure 23A-2] FIG. 2 is a schematic diagram of another example of a multi-display region scenario according to an embodiment of the present application. [Figure 23B] FIG. 2 is a schematic diagram of another example of a multi-display region scenario according to an embodiment of the present application. [Figure 23C] FIG. 2 is a schematic diagram of another example of a multi-display region scenario according to an embodiment of the present application. [Figure 24] FIG. 13 is a schematic diagram of another example of an interface generation method according to an embodiment of the present application. [Diagram 25] A schematic diagram of an example in which the UniRender process performs frequency division or frequency multiplication on a vertical synchronization signal, according to an embodiment of the present application. [Figure 26A] 2 is a schematic diagram of an example of data flow when an electronic device implements an interface generation method according to an embodiment of the present application; [Figure 26B] 2 is a schematic diagram of an example of data flow when an electronic device implements an interface generation method according to an embodiment of the present application; [Figure 27] FIG. 13 is a schematic diagram of another example of an interface generation method according to an embodiment of the present application. [Figure 28A] FIG. 2 is a schematic diagram of an example of merging a render tree into a target render tree according to an embodiment of the present application. [Figure 28B] FIG. 2 is a schematic diagram of an example of merging a render tree into a target render tree according to an embodiment of the present application; [Figure 29] FIG. 2 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application. [Diagram 30] FIG. 2 is a schematic diagram of an example of a software structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0061] The terms used in the following embodiments are intended to describe specific embodiments only and are not intended to limit the present application. As used in the specification of this application, the singular terms "a," "one," "the," "said," "this," and "one of" are intended to include the plural terms unless the context clearly dictates otherwise. It is further understood that the term "and / or" as used in this application denotes and includes any and all possible combinations of one or more listed items.

[0062] The terms "first" and "second" below are used for descriptive purposes only and should not be understood as indicating or implying a relative importance or quantity of technical features indicated. Thus, a feature qualified by "first" or "second" may explicitly or implicitly include one or more such features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0063] The term "user interface (UI)" in the following embodiments of this application refers to a medium interface for interaction and information exchange between an application or an operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code compiled in a specific computer language, such as Java or extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and is finally presented as a content that can be recognized by the user. A common presentation form of the user interface is a graphic user interface (GUI), which refers to a user interface that is displayed in a graphical manner and related to computer operation. The GUI may be a visible interface element, such as text, an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, or a widget, displayed on the display of the electronic device.

[0064] For ease of understanding, the following will first describe the relevant terms and concepts in the embodiments of the present application. The terms used in the embodiments of the present invention are only used to describe specific embodiments of the present invention, and are not intended to limit the present invention.

[0065] The interface is used as a medium interface for interaction and information exchange between the application and the user. Every time a vertical synchronization signal arrives, the electronic device needs to generate the interface of the application for the foreground application. The frequency of the vertical synchronization signal is related to the refresh rate of the screen of the electronic device. For example, the frequency of the vertical synchronization signal is the same as the refresh rate of the screen of the electronic device.

[0066] Specifically, before the electronic device refreshes the content displayed on the screen, the electronic device must generate an application interface for the foreground application each time, and then when the screen is refreshed, the newly generated application interface is displayed to the user.

[0067] The interface displayed on the electronic device by the electronic device may include interfaces of one or more applications, i.e. the electronic device needs to generate interfaces of one or more applications and composite the interfaces to obtain a composite interface displayed on the screen.

[0068] When an electronic device generates an interface for an application, the application needs to render and generate bitmaps and transfer the application's bitmaps to a surface composer (SurfaceFlinger). That is, the application acts as a producer, drawing and generating bitmaps and storing them in a buffer queue (BufferQueue) provided by SurfaceFlinger. SurfaceFlinger acts as a consumer, continuously retrieving the bitmaps generated by the application from the BufferQueue. The bitmaps are placed on the surfaces generated by the application. The surfaces are put into the BufferQueue.

[0069] After SurfaceFlinger obtains the bitmaps of visible applications, SurfaceFlinger and the Hardware Composer (HWC) determine the layer compositing mode in which the bitmaps are used as layers. SurfaceFlinger may determine the visible applications by using the Window Manager Service (WMS).

[0070] After obtaining off-screen rendering logic, such as rounding, rotation, color transformation, and scaling, that acts on the application's window from the window manager service, SurfaceFlinger may copy the application's bitmaps to off-screen buffers for rendering and obtain bitmaps for layer compositing via off-screen rendering.

[0071] After the application bitmap is composited, SurfaceFlinger / HWC puts the composited bitmap (the bitmap may be referred to as a layer on SurfaceFlinger) into a Frame Buffer for transfer to the Display Subsystem (DSS). After getting the composited bitmap, the DSS may display the composited bitmap on the screen. The frame buffer may be an on-screen buffer.

[0072] (1) First, we separately describe (1.1) the process of generating a bitmap by an application, (1.2) the process of compositing a bitmap by SurfaceFlinger / HWC, and (1.3) an example of off-screen rendering.

[0073] (1.1) The process of generating a bitmap by an application 1A and 1B are schematic diagrams of an example of an interface of an electronic device, respectively, according to an embodiment of the present application.

[0074] As shown in Fig. 1A, the screen of an electronic device displays interfaces of Application 1 and Application 2. The interface of Application 1 may be referred to as a status bar. In Fig. 1A, Application 1 may be an operating system, and Application 2 may be a music application.

[0075] Before displaying the interface shown in FIG. 1A, the electronic device needs to generate bitmaps of the applications separately and independently. The status bar interface is maintained by the operating system, i.e., the application that generates the status bar bitmap may be considered to be the operating system. Specifically, the operating system generates bitmap 1 and then transfers bitmap 1 to SurfaceFlinger, and the music application generates bitmap 2 and then transfers bitmap 2 to SurfaceFlinger. Bitmap 1 carries image information for the status bar, and bitmap 2 carries image information for the music application.

[0076] After receiving Bitmap 1 and Bitmap 2, SurfaceFlinger / HWC uses Bitmap 1 and Bitmap 2 as layers to perform layer compositing. For the contents of layer compositing, please refer to the description of (1.2) The process of compositing bitmaps by SurfaceFlinger / HWC, and the details will not be described again in this specification.

[0077] As shown in Fig. 1B, the screen of the electronic device displays interfaces of Application 1, Application 2, and Application 3. The interface of Application 1 may be referred to as a status bar. In Fig. 1B, Application 1 may be an operating system, Application 2 may be a browser application, and Application 3 may be an SMS application.

[0078] Before displaying the interface shown in FIG. 1B, the electronic device also needs to generate bitmaps of the applications separately and independently.

[0079] Specifically, after generating bitmap 3, the operating system forwards bitmap 3 to SurfaceFlinger, after generating bitmap 4, the SMS application forwards bitmap 4 to SurfaceFlinger, and after generating bitmap 5, the news application forwards bitmap 5 to SurfaceFlinger. Bitmap 3 carries image information for the status bar, bitmap 4 carries image information for the SMS application, and bitmap 5 carries image information for the news application.

[0080] After receiving Bitmap 3, Bitmap 4, and Bitmap 5, SurfaceFlinger / HWC uses Bitmap 3, Bitmap 4, and Bitmap 5 as layers to perform layer compositing. For the contents of layer compositing, please refer to the description of (1.2) The process of compositing bitmaps by SurfaceFlinger / HWC, and the details will not be described again in this specification.

[0081] The process of generating a bitmap by an application is shown in FIG.

[0082] FIG. 2 is a schematic diagram of an example of generating a bitmap by an application according to an embodiment of the present application.

[0083] As shown in Figure 2, after receiving a vertical synchronization signal (Vsync-APP), the application starts to generate a bitmap. The specific steps include the following three steps:

[0084] [1] The view hierarchy is invalidated. The main thread (UI Thread) traverses the views of the application, determines and stores the drawing operations for each view, and records the views and their associated drawing operations (Draw Operation Struct, DrawOP) in the displaylist of the RenderNode in the render tree.

[0085] Views are the basic elements that form an application interface, and one control on the interface may correspond to one or more views.

[0086] A drawing operation is a data structure used to draw graphics, e.g., draw line, draw width, draw rectangle, draw text, etc. A drawing operation is translated into an API call of an image processing library on a render node, e.g., an API call of OpenGL. For example, DrawLineOp is a data structure that contains information about drawing data such as the length and width of a line.

[0087] A drawing instruction list may be a buffer that records all drawing operations or identifiers of all drawing operations in one frame of an application's interface, such as addresses and sequence numbers. When an application has multiple windows or is displayed in different displays, multiple render trees need to be generated independently. Multiple drawing instruction lists corresponding to different windows or displays are generated independently.

[0088] In an embodiment of the present application, the display area may be a screen, a virtual screen (VirtualDisplay), etc. The virtual screen may be an area used by an electronic device to carry the content displayed on the screen during screen recording.

[0089] The render tree is generated by the UI thread and is used to generate the data structures of the application interface. The render tree may contain multiple render nodes, each of which contains rendering properties and a list of drawing instructions. The render tree records all the information to generate one frame interface of the application.

[0090] [2] The UI thread transfers / synchronizes the render tree to the render thread, and the render tree is located in the stack of the process corresponding to the application and does not have to be contiguously distributed in terms of physical addresses.

[0091] [3] The render thread first obtains a hardware canvas and performs the drawing operations in the render tree on the hardware canvas to generate a bitmap. The hardware canvas is placed on a surface held by the application that carries a bitmap or data in another format used to store image information.

[0092] [1] may be thought of as a construction phase primarily responsible for determining properties such as size, position, and transparency of each view in an application. For example, a drawLine in a view may be packaged into a DrawLineOp during construction, which contains drawing data such as the line's length and width, and may further contain an interface call corresponding to a drawLineOP in an underlying graphics processing library, which is used to invoke the underlying graphics library to generate a bitmap in the rendering phase.

[0093] Similarly, [3] may be considered as the rendering phase, which is primarily responsible for traversing the render nodes of the render tree, performing drawing operations for each render node, and generating a bitmap on the hardware canvas. In this process, the render thread calls the underlying graphics processing library, e.g., OpenGL, to invoke the GPU to perform the rendering and generate the bitmap.

[0094] FIG. 3 is a schematic diagram of an example of a render tree according to an embodiment of the present application.

[0095] The interface that needs to be displayed by an application includes multiple nested views, and different views have parent-child relationships. Therefore, the parent-child relationships between render nodes in the render tree generated by traversing the views are the same as the parent-child relationships between views. In other words, the parent-child relationships between views determine the nesting relationships between different render nodes. Then, the render thread can correctly render the interface of the application when it generates a bitmap according to the render tree.

[0096] A view may correspond to one or more render nodes. The root view (DecorView) corresponds to the root render node (RootRenderNode). In other words, the nesting relationship between render nodes corresponds to the parent-child relationship between views. A render node further includes rendering properties that are used to determine the position, size, transparency, etc. of the view corresponding to the render node on the surface when a bitmap is generated through rendering.

[0097] For example, the structure of an application's interface is as follows: the application's PhoneWindow carries a root view, the child views of the root view are view 1 and view 2, and the child view of view 2 is view 3. In this case, the structure of the render tree generated by the application's UI thread is as follows: the root render node corresponding to the PhoneWindow is the root node of the render tree, the child node of the root render node is render node 0 which corresponds to the root view, the child nodes of render node 0 are render node 1 which corresponds to view 1 and render node 2 which corresponds to view 2, and the child node of render node 2 is render node 3 which corresponds to view 3.

[0098] The correspondence between a view and a render node means that the render node contains all the drawing operations on the corresponding view.

[0099] After receiving the render tree synchronized by the UI thread, the render thread calls OpenGL interfaces to render bitmaps on the application's surfaces and sends the surfaces to SurfaceFlinger for compositing and display.

[0100] Please note that during the construction phase, the computing resources of the CPU need to be occupied, and during the rendering phase, the resources of the GPU need to be occupied.

[0101] Note that when hardware acceleration is not enabled, an application performs all operations in the construction and rendering phases by using the UI thread and does not need to perform packaging to a render node. After the application's views and view drawing operations are traversed, the application requests anonymous shared memory from SurfaceFlinger and directly calls the underlying graphics library in memory to generate bitmaps.

[0102] Please note that the content shown in Figure 2 is a process in which an application generates a bitmap when hardware acceleration is enabled. When hardware acceleration is not enabled on an electronic device, an application generates a bitmap through software drawing. Software drawing includes:

[0103] [1] The view hierarchy is invalidated. The UI thread traverses the views of the application and records the drawing operations for each view. [2] The UI thread obtains a software canvas for drawing through an interface, for example, Surface.lockCanvas(), and performs drawing on the canvas based on the stored list of drawing operations to generate a bitmap. The software canvas is located on a surface created by the application.

[0104] Surfaces held by an application are assigned to the application by SurfaceFlinger via binder communication. The number of surfaces held by an application may be the same as the number of current windows (PhoneWindows) of the application.

[0105] After the process of generating a bitmap by an application has been described, the process of compositing the bitmap will be described using an example.

[0106] (1.2) The process of compositing bitmaps using SurfaceFlinger / HWC SurfaceFlinger is a system service on an electronic device that is used to assign surfaces to applications and use bitmaps on one or more surfaces as layers for layer compositing. HWC is a functional module of the Hardware Abstraction Layer (HAL) responsible for compositing and display in electronic devices, providing an interface for the upper layer SurfaceFlinger and calling the functions of the underlying hardware (e.g., display drivers other than GPU) to perform layer compositing.

[0107] FIG. 4 is a schematic diagram of an example in which bitmaps are used as layers to participate in layer compositing, according to an embodiment of the present application.

[0108] As shown in Figure 4, an application's render thread generates a bitmap on a surface assigned to the application by SurfaceFlinger. SurfaceFlinger maintains one or more application's surfaces by using the BufferQueue mechanism. In other words, SurfaceFlinger can retrieve bitmaps for different applications.

[0109] After obtaining bitmaps for one or more applications, SurfaceFlinger calls the GPU to composite multiple bitmaps into one bitmap (bitmap compositing is called layer compositing). Compositing may also be called client compositing or GLES compositing.

[0110] After obtaining the application's bitmap, SurfaceFlinger may call the underlying hardware (excluding the GPU) through the HWC for compositing. This compositing mode is also called device compositing.

[0111] Client compositing requires invoking the GPU, and may composite multiple layers and perform layer compositing in a pixel-by-pixel processing manner such as linear deepening.

[0112] Device compositing may composite a limited number of layers and does not support compositing in a multi-pixel manner. If device compositing composites multiple layers that do not have a position intersection on the screen, layer compositing may not be performed. Instead, data on different surfaces is read and displayed when different positions on the screen are displayed.

[0113] 4, after obtaining bitmap 1 of application 1, bitmap 2 of application 2, ..., and bitmap N of application N, SurfaceFlinger may perform client compositing on bitmap 1 and bitmap 2 to generate bitmap 11. SurfaceFlinger then performs device compositing on bitmap 11 and bitmap N by using the HWC to invoke the underlying hardware.

[0114] In Device compositing, bitmap 11 and bitmap N are temporarily stored. When they need to be displayed on the screen of the electronic device, corresponding pixels are retrieved from bitmap 11 / bitmap N for display on the screen. For example, in the interface shown in Figure 1B, bitmap 11 is the bitmap composited by the SMS application and the news application, and bitmap N is the bitmap of the status bar.

[0115] For the underlying hardware that supports SurfaceFlinger or HWC, each bitmap corresponds to one layer.

[0116] The layer compositing mode may be determined by the underlying hardware corresponding to the HWC or may be determined by SurfaceFlinger.

[0117] For example, after getting a bitmap, SurfaceFlinger forwards the set layers to the underlying hardware by using the HWC, and the underlying hardware determines which specific layers should be Client compositing performed and which specific layers should be Device compositing performed. The underlying hardware marks the compositing modes of the layers in the layer list and returns the compositing modes of the different layers to SurfaceFlinger. SurfaceFlinger composites the layers marked with GPU compositing and stores the composited results in a buffer. SurfaceFlinger forwards the buffer and other layers marked with Overlay compositing mode to the underlying hardware via the HWC. The underlying hardware then performs the layer compositing.

[0118] In another example, SurfaceFlinger directly marks layers associated with off-screen rendering trigger logic, such as window animation, in the layer compositing process with GPU compositing. The off-screen rendering trigger logic further includes logic that cannot be processed by the underlying hardware corresponding to the HWC, such as rounding, scaling, rotation, and color conversion.

[0119] In addition to the bitmap generated by the application's render thread, a surface assigned to an application by the application's SurfaceFlinger further includes window control information, such as the layer's Z-order, obtained from the window manager service. Thus, SurfaceFlinger may obtain the layer's window control information from the surface to determine whether the layer requires GPU compositing. The layer's Z-order determines the ordering of the layers on the Z-axis, which is the direction perpendicular to the screen, and is used to calculate the hierarchical relationship between different layers.

[0120] FIG. 5 is a schematic diagram of an example of layer compositing according to an embodiment of the present application.

[0121] As shown in Figure 5, after getting the application's bitmaps, SurfaceFlinger writes each bitmap as a layer in a layer list. SurfaceFlinger forwards the layer list to the underlying hardware through the HWC. The underlying hardware determines the compositing mode for different layers based on the capabilities of the underlying hardware and returns the result to SurfaceFlinger through the HWC.

[0122] After getting the results returned by the HWC, SurfaceFlinger may get the compositing mode for each layer in the layer list. For layers marked with GPU compositing, SurfaceFlinger composites the layer and forwards the composited layers and layers marked with overlay compositing to the underlying hardware through the HWC. The underlying hardware corresponding to the HWC is then used to composite the layers.

[0123] FIG. 6 is a schematic diagram of another example of how SurfaceFlinger performs layer compositing, according to one embodiment of the present application.

[0124] After getting the multiple surfaces, SurfaceFlinger may determine a compositing mode in which a bitmap is used as a layer on each surface. After getting the multiple layers and determining that the layer compositing mode is GPU, SurfaceFlinger may perform client compositing. The layer compositing modes may include Mode.CLEAR (display the topmost layer in the Z-order), Mode.SRC_OVER (display layers in order according to the Z-order), Mode.DST_IN (display the non-intersection of the topmost layer with the layer below it in the Z-order), etc.

[0125] For example, as shown in Figure 6, when the layer compositing mode is Mode.SRC_OVER, SurfaceFlinger composites the contents of three layers, determines the hierarchical relationship between different layers according to the Z-order of the layers, and stacks the layers on top of one another sequentially.

[0126] The Z-order of layer 1 is a, the Z-order of layer 2 is a+1, and the Z-order of layer 3 is a+2. After SurfaceFlinger composites the content of the three layers, the content of layers 1 and 2 is completely occluded and only the content of layer 3 is visible.

[0127] Total occlusion means that a view in a lower Z-order layer is hidden from view due to occlusion by a layer higher in the Z-order.

[0128] Layers on which GPU compositing is performed trigger off-screen rendering. Layer compositing is performed via off-screen rendering. The following describes concepts related to off-screen rendering.

[0129] (1.3) Offscreen Rendering If SurfaceFlinger determines that any layers need to be composited by using the GPU, off-screen rendering needs to be enabled to perform the layer compositing.

[0130] Off-screen rendering requests an off-screen buffer for SurfaceFlinger and calls the GPU to process the images in the off-screen buffer, which is memory outside the current screen buffer, and multiple layers are composited into the off-screen buffer.

[0131] Off-screen rendering may include the following steps.

[0132] [1] Bitmaps in layers that are determined to be GPU composited by SurfaceFlinger must be converted to textures, then uploaded to GPU memory (i.e., off-screen buffer) or mapped to OpenGL textures via shared memory, and OpenGL binds the textures (binding the texture involves binding the context).

[0133] [2] Textures are rendered according to the instructions associated with the window animations associated with the layers. In addition, textures from multiple applications are merged. Layer compositing performs pixel-by-pixel rendering according to the layer compositing mode.

[0134] [3] SurfaceFlinger retrieves rendered textures either from GPU memory or directly from shared memory.

[0135] It is clear that off-screen rendering causes a context switch, and a context switch adds additional performance overhead.

[0136] The compositing process in which a bitmap is used as a layer is described in detail below with reference to (1.1) the process of generating a bitmap by an application, (1.2) the process of compositing a bitmap by SurfaceFlinger / HWC, and (1.3) the content of off-screen rendering.

[0137] FIG. 7A and FIG. 7B-1 and FIG. 7B-2 are schematic diagrams of another example of layer compositing according to an embodiment of the present application.

[0138] As shown in FIG. 7A, UI threads of different applications generate their respective render trees independently and transfer the render trees to respective render threads for rendering to generate bitmaps.

[0139] A render thread first generates a bitmap through rendering based on the properties of the render nodes in the render tree and the drawing instruction list, and performs compositing by using the bitmap as a layer. When the layer compositing mode is GPU compositing, SurfaceFlinger composites multiple layers into one layer through off-screen rendering.

[0140] For example, application 1 generates render tree 1, and then generates bitmap 1 based on render tree 1, and application 2 generates render tree 2, and then generates bitmap 2 based on render tree 2. After receiving bitmap 1 and bitmap 2, SurfaceFlinger performs off-screen rendering on bitmap 1 and bitmap 2 to generate bitmap 5. In the off-screen rendering process, bitmap 1 is first copied to an off-screen buffer, bitmap 1 is converted to bitmap 3 in the off-screen buffer based on window animation information, similarly, bitmap 2 is converted to bitmap 4, and finally, bitmap 3 and bitmap 4 are superimposed based on a layer compositing mode to generate bitmap 5.

[0141] As shown in FIG. 7B-1 and FIG. 7B-2, to realize the interface shown in FIG. 1B, after receiving an application's bitmap, SurfaceFlinger first performs off-screen rendering on the application's bitmap related to the window animation bitmap to accomplish transformation, and then copies the transformed layer to an on-screen buffer for transmission and display.

[0142] For example, if application 3's bitmap needs to be scaled down, it cannot be copied directly to an on-screen buffer for transformation, which could affect other applications' bitmaps. Instead, application 3's bitmap needs to be copied to an off-screen buffer for a separate transformation, and the transformation result is copied back to the on-screen buffer.

[0143] It is clear that the correct interface can only be generated when the bitmaps generated by each application are first modified and then superimposed based on the layer compositing mode.

[0144] Because the processes that invoke the GPU change from when an application's main thread generates the render tree to when SurfaceFlinger performs layer compositing (from application 1, to application 2, ..., application N, and SurfaceFlinger), the GPU needs to be started at least N+1 separate times, where N is the number of applications.

[0145] It is understood that different applications build and render their bitmaps independently, but a layer with a lower Z-order may be completely occluded by a layer with a higher Z-order during layer compositing. This inevitably results in overdraw of bitmaps of different applications. For example, the overlapping parts of the three graphics in FIG. 6 are overdrawn, i.e., layer 1 and layer 2 are overdrawn. The composited interface is not affected by layer 1 and layer 2. In addition, off-screen rendering requires new off-screen buffer allocation and context switching, which increases the memory overhead and computing overhead of the electronic device.

[0146] (2) Next, the interface generation method provided in the embodiment of the present application will be described below.

[0147] According to the interface generation method provided in the embodiment of the present application, firstly, render trees of one or more applications are obtained by using a UniRender process, and the one or more render trees are regrouped to generate a target render tree. Then, the UniRender process performs rendering based on the target render tree to directly obtain a bitmap carrying image information of the interface of the one or more applications without performing layer compositing.

[0148] First, in an interface generation method provided in an embodiment of the present application, the target render trees of one or more applications are merged into a target render tree. In the process of generating the target render tree, the UniRender process determines the off-screen rendering logic of each layer, and adds or modifies the properties of the corresponding render nodes in the target render tree according to the off-screen rendering logic, so that the UniRender process directly generates a bitmap without performing off-screen rendering.

[0149] Then, in the interface generation method provided in the embodiment of the present application, the target render trees of one or more applications are merged into a target render tree, and there is no need to first generate multiple bitmaps as layers for compositing. In the process in which the UniRender process performs rendering based on the target render tree to generate bitmaps, the Z-order of the layers is used as the Z-order of the render tree, and render nodes corresponding to views that are not displayed or do not affect the display may be deleted, thereby avoiding overdraw.

[0150] Finally, according to the interface generation method provided in the embodiment of the present application, the application does not need to generate a render thread, but the UniRender process contributes to improving the rendering speed of the interface by performing integrated rendering.

[0151] (2.1) System architecture of the interface generation method provided in the embodiment of the present application

[0152] FIG. 8 is a schematic diagram of an example of a system architecture of an interface generation method according to an embodiment of the present application.

[0153] When an application's interface is updated, the application may request a vertical synchronization signal (Vsync-APP) from the UniRender process (SurfaceFlinger not shown in FIG. 8). When SurfaceFlinger is present, the application may request a vertical synchronization signal (Vsync-APP) from SurfaceFlinger.

[0154] The vertical synchronization signal (Vsync-APP) for the UniRender process may be generated from SurfaceFlinger, or may be generated directly from the underlying hardware (such as a screen) that corresponds to the HWC, or the vertical synchronization signal (Vsync-APP) may be generated by the UniRender process by starting a thread that is periodically woken up. When SurfaceFlinger is present, the Vsync-APP may come from SurfaceFlinger.

[0155] After the application gets the vertical synchronization signal (Vsync-APP), the application creates a render tree and transfers the render tree to the UniRender process.

[0156] After receiving a vertical synchronization signal (Vsync-UR), the UniRender process merges one or more render trees to generate a target render tree. The UniRender process then uses a render engine to traverse and execute drawing operations in the drawing instruction list of each render node in the render tree, and generates a bitmap that carries image information for the one or more render trees. The bitmap may be located in an on-screen buffer.

[0157] The difference between Vsync-UR and Vsync-APP is Vsync-Offset, which may be determined by the UniRender process. If SurfaceFlinger is present, Vsync-Offset may be determined by SurfaceFlinger.

[0158] After generating the bitmap, the UniRender process transfers the bitmap via the HWC to the display subsystem for display.

[0159] After the system architecture of the interface generating method in the present application is described, the method steps of the interface generating method provided in the embodiments of the present application will be described using an example.

[0160] (2.2) Method steps of the interface generation method provided in the embodiment of the present application With reference to FIG. 9, an example of an interface generation method provided in an embodiment of the present application will now be described.

[0161] FIG. 9 is a schematic diagram of an example of an interface generation method according to an embodiment of the present application.

[0162] As shown in FIG. 9, the interface generation method provided in this embodiment of the present application includes the following steps:

[0163] S901: Construct and generate a render tree after receiving a vertical synchronization signal.

[0164] When the interface needs to be updated, the application may request a vertical sync signal (Vsync-APP) from the UniRender process. After receiving the vertical sync signal, the application executes the measure(), layout(), and draw() methods in the UI thread. When executing the draw() method, the UI thread traverses the application's views, determines the drawing instructions required to render each view, and sequentially records the drawing instructions in the drawing instruction list of the render node corresponding to the view.

[0165] The interface that needs to be displayed by the application includes multiple nested views, and the drawing instruction list corresponding to the root view (DecorView) includes drawing instruction list entries of the child views of the root view, i.e., the nesting relationship between the drawing instruction lists is the same as the nesting relationship between the views. Therefore, the nesting relationship between the render nodes is the same as the nesting relationship between the views. For the definition of the related concepts of render tree and render node, please refer to the above description corresponding to FIG. 3.

[0166] FIG. 10 is a schematic diagram of an example of constructing a render tree according to an embodiment of the present application.

[0167] After performing measurement, layout, and drawing, the application's UI thread may obtain the parent-child structure of the interface's multiple views to be updated, and in the process of traversing the views may determine the content to be displayed in each view and the interface calls, e.g., drawCircle or drawLine, needed to generate the content.

[0168] An application packages drawing interface calls, such as drawCircle and drawLine, into corresponding DrawOps, such as DrawCircleOp and DrawLineOp, and stores the DrawOps in a drawing instruction list. DrawLineOp is a graphics drawing interface call of the underlying graphics library (e.g., OpenGL), which is further translated into a graphics drawing instruction for invoking the GPU.

[0169] 10, the view structure of the application interface includes a root view, the child views of the root view are view 1 and view 2, and the child view of view 2 is view 3. View 2 includes a rectangle and a circle, and the drawing operations in the drawing command list of render node 2 corresponding to view 2 include an operation to draw a rectangle, an operation to draw a circle, etc.

[0170] When executing the draw() method, the application's UI thread may start from the root view, traverse all views based on the parent-child relationships between the views, determine the drawing operations in each view, and package the drawing operations into a DrawOp. After generating the drawing instruction list, the application's UI thread further packages the drawing instruction list into a render tree.

[0171] A render node in the render tree contains a drawing instruction list and rendering properties; the rendering properties are used to determine properties such as the position, size, and transparency of the view to be rendered by the render node on the surface, and the drawing instruction list is used to determine the content of the view, such as a line, rectangle, or circle, to be rendered by the render node.

[0172] Surfaces are claimed by applications. The application determines the size of the surface. If SurfaceFlinger is present, the surface may be claimed by the application from SurfaceFlinger. If SurfaceFlinger is not present, the surface may be claimed from the UniRender process. SurfaceFlinger may not allocate the surface to the application.

[0173] If necessary, in some embodiments of the present application, after determining the screen refresh rate of each display area, the UniRender process may adjust the frequency of the vertical synchronization signal (Vsync-APP) in each display area, so that the application displayed in display area 1 generates a render tree at the frequency of the screen refresh rate of display area 1.

[0174] Note that an application's UI thread may generate multiple render trees in a multi-display scenario, such as a multi-screen scenario, a virtual screen scenario, or a multi-window scenario. For information about multi-display scenarios, see the description of (2.2) Interface Generation Methods in Multi-Display-Area Scenarios.

[0175] S902: The render tree is transferred in a cross-process manner.

[0176] After generating the render tree, the UI thread of the application transfers the render tree to the UniRender process via IPC communication. The render tree resides on the stack of the process corresponding to the application. The corresponding UniRender process receives the render trees transferred by different applications and needs to determine the correspondence between the render trees and the applications.

[0177] Applications in the foreground transfer their render trees to UniRender. An application is a foreground application if it meets any one of the following three conditions: it has a visible Activity, it has a foreground service, and other foreground applications are associated with it.

[0178] Since the memory of different processes is not shared, data exchange between processes must be accomplished through Inter-Process Communication (IPC). To achieve IPC communication, an application may transfer a render tree to the UniRender process using methods such as Binder, AIDL, shared memory, or Socket. This is not limited in this specification.

[0179] In the following, a shared memory is used as an example of IPC communication to explain an example of a method for transferring a render tree in a cross-process manner.

[0180] (a) The application writes the render tree to shared memory.

[0181] FIG. 11 is a schematic diagram of an example of shared memory transferring a render tree according to an embodiment of the present application.

[0182] As shown in Fig. 11, first, each time an application is launched, the application applies for shared memory to the UniRender process. After receiving a request for shared memory initiated by the application, the UniRender process applies for the shared memory and a handle corresponding to the shared memory through the Anonymous Shared Memory (Ashmem) subsystem.

[0183] After successfully requesting shared memory from the Ashmem subsystem, the UniRender process receives a handle returned by the Ashmem subsystem for reading and writing physical memory. The UniRender process returns the handle to the application so that the application can use it to write the render tree to physical memory. The UniRender process can directly read the physical memory from the UniRender process's process space and then directly read the application's render tree.

[0184] Shared memory may be a virtual file created in memory (RAM) by using a temporary file system (tmpfs) and mapped separately into the user space of different processes.

[0185] For example, cross-process communication may be realized via Binder communication, where an application requests shared memory from the UniRender process, the UniRender process requests shared memory from the Ashmem subsystem, and the UniRender process returns a handle to the application corresponding to the shared memory obtained via the application.

[0186] It will be appreciated that the render tree stored in the stack of the application process may be transferred to the UniRender process in other IPC manners, such as memory sharing, which is not limited herein.

[0187] Optionally, in some embodiments of the present application, a trust list is configured in a local configuration file of the electronic device or cloud server, and the trust list stores other identifiers that can uniquely determine the application process, such as the package name of the application. If the application is in the trust list, the render tree is transferred to the UniRender process. If the application is not in the trust list, after the bitmap is obtained via software drawing in the UI thread or rendering in the render thread, the bitmap is transferred to UniRender for process compositing, or the bitmap is transferred to SurfaceFlinger. SurfaceFlinger composites the bitmap of the application not in the trust list with the bitmap generated by the UniRender process.

[0188] Optionally, in some embodiments of the present application, if an application has multiple layers, i.e., if the application generates multiple render trees, the application may request two shared memories from the UniRender process, each of which is used to store a different layer, i.e., to store data for a different render tree.

[0189] If necessary, in some embodiments of the present application, the application may apply two shared memories to the UniRender process to write the render trees alternately. For example, when the render tree corresponding to the interface of a frame is written to the first shared memory, the render tree corresponding to the interface of the next frame is written to the second shared memory, and then the render tree corresponding to the interface of the next frame is written to the first shared memory. This helps to avoid read / write conflicts in which the data of the render tree cannot be written to the shared memory in a timely manner when there is only one shared memory.

[0190] (b) Storage data structure of the render tree in shared memory In order to further improve IPC communication efficiency, in this embodiment of the present application, the render tree is stored in the shared memory in the form of a memory tree. An example of the data structure form of the render tree stored in the shared memory is described below.

[0191] The memory tree may include multiple data segments, and different data segments store layer information, rendering data, etc., respectively. In the following, the contents shown in Figures 12A and 12B are used as an example to describe the structure of the memory tree. The offsets of different data segments in the shared memory may be fixed or may be adaptive instead of fixed.

[0192] 12A and 12B are schematic diagrams of an example of a storage structure of a render tree in a shared memory according to an embodiment of the present application.

[0193] As shown in FIG. 12A, the storage structure of the render tree includes three data segments: a HEAD field, a MAPPING field, and a NODES field.

[0194] The HEAD field contains the layerkey and the rootid, the MAPPING field contains the nodeid and the address corresponding to the nodeid, and the NODES field contains the currentproperties, stagingproperties, stagingdisplaylist, and currentdisplaylist.

[0195] The layerkey is the ID of the entire render tree as a layer, the rootid is the ID of the root node of the render tree, the nodeid is the ID of a render node other than the root node of the render tree, one nodeid corresponds to one address, the address is the starting address of the rendering properties (renderproperties / properties) and the drawing instruction list (displaylist) in the render tree node, the stagingproperties are the rendering properties written by the application, the stagingdisplaylist is the drawing instruction list written by the application, the currentproperties are the rendering properties read by the UniRender process, and the currentdisplaylist is the drawing instruction list read by the UniRender process.

[0196] Note that "stagingproperties and stagingdisplaylist" are considered to be the first data group, and "currentproperties and currentdisplaylist" are considered to be the second data group. In this case, the data written to the application is the first data group, and the data written to the application next is the second data group, thereby implementing a dual buffer mechanism. Similarly, the data read by the UniRender process is the first data group, and the data read next is the second data group.

[0197] Optionally, in some embodiments of the present application, the storage structure of the render tree in the shared memory may be as shown in Figure 12B. The NODES field may contain only one data group, i.e., only the rendering properties and the drawing instruction list.

[0198] The layerkey is used to get the ID of the layer in the application that needs to be displayed and composited from the WMS before the UniRender process reads the layer data in shared memory by using the handle. The UniRender process matches the layer ID with the layer ID contained in the layerkey in shared memory.

[0199] The rootid is used for: As an entry in the render tree, the rootid stores the entries of other render nodes. After obtaining the rootid, the UniRender process may read the data of the render tree and restore the nested structure of the render tree.

[0200] currentproperties, stagingproperties, stagingdisplaylist, and currentdisplaylist are used as follows: after an application completes writing of the drawing instruction list and rendering properties, the values ​​of currentproperties and stagingproperties are swapped, and the values ​​of currentdisplaylist and stagingdisplaylist are swapped. The UniRender process reads the rendering properties and drawing instruction list from currentproperties and currentdisplaylist.

[0201] For example, application 1 is an application displayed in the foreground, and application 1 may have multiple layers. Specifically, after receiving a vertical synchronization signal (Vsync-APP), application 1 generates multiple render trees, for example, render tree 1 and render tree 2. The UniRender process determines from the WMS that the render tree corresponding to the layers involved in layer compositing is render tree 1.

[0202] Since the offsets of layerkey and rootid are fixed, the UniRender process can determine the address of the root node. The UniRender process looks up the location of the root node on the NODES edge in the MAPPING field and reads the rendering instructions of the node. If a child node exists, there is a corresponding DrawRenderNode instruction. The instruction stores the ID of the child node. The location of the child node in the MAPPING field is looked up based on the hash value. For example, the parent-child relationship between render nodes in the render tree is stored in a DrawOP operation. For example, the drawing instruction list of render node 2 includes several DrawOP operations and an operation of "Draw RenderNode3" (draw render node 3), so the UniRender process may determine that render node 3 is a child node of render node 2.

[0203] It will be appreciated that the render tree in shared memory still stores the same nesting relationships between the views of the application. Thus, the UniRender process may read data from the root node and then all of the data in the render tree.

[0204] The currentproperties, stagingproperties, stagingdisplaylist, and currentdisplaylist are classified to ensure the security of the display data read by the UniRender process and the data written by the application, and to prevent data that has not been completely written by the application from being read by the UniRender process as the latest data for rendering and generating the application's interface. For security guarantees of concurrent reading and writing of the render tree, please refer to the following description in (2.3) Reading and Writing the Render Tree in Shared Memory. Details will not be described again in this specification.

[0205] If necessary, in some embodiments of the present application, the size of the three data segments may be fixed. Specifically, after the UniRender process applies for the shared memory from the Ashmem subsystem, the size of the acquired shared memory is a+b+c. The starting address (physical address) to the starting address+a is the location where the HEAD field is filled, the starting address+a+1 to the starting address+a+b is the location where the MAPPING field is filled, and the starting address+a+b+1 to the starting address+a+b+c is the location where the NODES field is filled.

[0206] It can be understood that if the sizes of the three data segments are fixed, the UniRender process may determine the beginning of each data segment according to a fixed offset to search the Mapping field, which stores the offset of the render node of the render tree in the NODES field to search the data of each render node.

[0207] Optionally, in some embodiments of the present application, the size of the three data segments is fixed, and when the size of the render node written by the application exceeds b, the application applies for a second shared memory to the UniRender process. The format of the second shared memory may be the same as the format of the first shared memory, and the rendering instruction list and the render node properties of the render node continue to be stored in the NODES field of the second shared memory. The HEAD field and / or the MAPPING field of the second shared memory may be null or may not exist. In other words, in some embodiments of the present application, the second shared memory includes only the NODES field.

[0208] (c) UniRender reads the render tree in shared memory.

[0209] (c.1) UniRender Architecture FIG. 13 is a schematic diagram of an example of the architecture of UniRender according to an embodiment of the present application.

[0210] As shown in FIG. 13, the UniRender provided in this application may include four parts: NodeManager, LayerManager, DisplayManager, and UniRenderCore.

[0211] The NodeManager is a node management module in the UniRender process, and is responsible for receiving the render tree sent by the application. For the composition of the target render tree, see the description of step S903.

[0212] The LayerManager is the layer management module in the UniRender process, and is responsible for synchronizing layer information such as layer creation, layer destruction, and property changes from the Window Manager Service (WMS). One bitmap corresponds to one layer.

[0213] DisplayerManager is the display device management module in the UniRender process, and is responsible for synchronizing display device information such as screen size from the Display Manager Service (DMS).

[0214] UniRenderCore is a rendering management module in the UniRender process, which is responsible for establishing a render node corresponding to each layer, receiving render trees corresponding to different applications maintained in the NodeManager, making layer information of the application into commands in the LayerManager and inserting the commands into the render nodes, merging render trees corresponding to all visible layers of the activated display devices maintained in the DisplayManager, and traversing the merged render tree for each display area. UniRender generates bitmaps for the buffers allocated by UniRender.

[0215] (c.2) UniRender reads the render tree.

[0216] The UniRender process may first determine from the DMS and WMS the applications displayed in each display area, which are applications that are involved in layer compositing. The UniRender process may further determine the applications that perform layer compositing in the UniRender process by referring to the trust list. The UniRender process may determine the layer IDs of each application by using the WMS.

[0217] The DisplayerManager in the UniRender process is responsible for communicating with the DMS, and the LayerManager in the UniRender process is responsible for communicating with the WMS.

[0218] The UniRender process stores a handle to the shared memory corresponding to the application, so that after determining the application involved in layer compositing, the UniRender process may determine the shared memory corresponding to the application by using the handle. UniRender reads the render tree from the shared memory via the handle.

[0219] The NodeManager in the UniRender process is responsible for managing the shared memory handle and reading the render tree from the shared memory.

[0220] The process by which the UniRender process reads the render tree from shared memory includes the following:

[0221] The UniRender process first reads the layerkey from the start address in shared memory and matches the layer ID. The UniRender process compares the layer ID determined by the WMS with the layer ID determined from the layerkey. After the layer IDs match, the UniRender process reads the render tree from the root node rootid.

[0222] Then, after retrieving the address of the root node, the UniRender process determines the start address of the root node in the NODES field in the address field in the MAPPING field, and starts reading the drawing instruction list and rendering properties of the render node. If the root node has a child node, the drawing list command of the root node stores the entry of the child node, for example, the DrawRenderNode command. Since the DrawRenderNode command contains the ID of the child node, the UniRender process retrieves the corresponding nodeid in the MAPPING through a hash operation, determines the location of the drawing instruction list and rendering properties of the child node in the NODES field, and reads the drawing instruction list and rendering properties of the child node.

[0223] (d) Reading and writing the render tree in shared memory The render tree located in the shared memory may be read and written by two or more processes. To reduce and avoid errors in the render tree data caused by read and write conflicts, synchronization locks between the processes may be configured to ensure security of reads and writes to the render tree.

[0224] An example of a process for reading and writing a render tree is described below with reference to the contents shown in Figures 14 and 15.

[0225] FIG. 14 is a schematic diagram of an example in which an application writes a render tree to a shared memory according to an embodiment of the present application.

[0226] Each application stores at least one lock variable A to prevent the application and UniRender from reading / writing the shared memory at the same time. UniRender gets the lock variable status (held or released) on different applications via IPC communication.

[0227] As shown in FIG. 14, [1] first, when an application needs to update the interface, a vertical synchronization signal (Vsync-APP) is requested and received, then the lock variable A in the shared memory corresponding to the application is obtained, and the properties of the render node and the drawing instruction list are calculated and updated from the root node.

[0228] [2] The application then writes the updated properties and drawing instruction list of the render node to the stagingproperties and stagingdisplaylist data segments of the NODES field in the shared memory, and adds the IDs of the modified render nodes to the properties_dirty and displaylist_dirty queues, which are stored in the application-side shared memory management class singleton.

[0229] It will be appreciated that modified render nodes are marked in the properties_dirty and displaylist_dirty queues to effect differential updates of the render tree.

[0230] Optionally, in some embodiments of the present application, the properties_dirty and displaylist_dirty queues may not be populated to achieve a complete render tree update.

[0231] [3] The application then copies the stagingproperties segment of the corresponding render node in the properties_dirty queue to the currentproperties segment. The application exchanges the draw_pointer and record_pointer of the corresponding render node in the displaylist_dirty queue, i.e., copies the stagingdisplaylist segment of the corresponding render node in the displaylist_dirty queue to the currentdisplaylist, or copies the stagingdisplaylist segment to the currentdisplaylist.

[0232] It will be understood that, in response to the current vertical synchronization signal (Vsync-APP), compared with the previous vertical synchronization signal (Vsync-APP), the application only changes the data of the render node corresponding to the displaylist_dirty, and the application exchanges the draw_pointer and record_pointer of the corresponding render node in the displaylist_dirty queue, so that a differential update of the currentdisplaylist can be realized.

[0233] To achieve a complete update, the application copies the stagingdisplaylist segment to the currentdisplaylist, i.e., all data of the render tree generated by the application in response to the vertical synchronization signal (Vsync-APP) is written directly to the shared memory, which can be understood to be relatively easy to achieve.

[0234] Optionally, in some embodiments of the present application, when the properties_dirty queue and the displaylist_dirty queue are not stored, the stagingproperties segment of all render nodes of the application is copied to the currentproperties segment, and the stagingdisplaylist segment is copied to the currentdisplaylist segment. The copying may be accomplished by changing the position of the pointer.

[0235] [4] The application then transfers information that lock variable A has been released to the UniRender process via IPC communication.

[0236] [5] The UniRender process then holds the lock variable A.

[0237] [6] Finally, corresponding to [3], the UniRender process reads the current displaylist and current properties from shared memory, or reads the stagingdisplaylist segment of the corresponding render node in the displaylist_dirty queue, and copies the stagingdisplaylist segment to the currentdisplaylist.

[0238] After reading the data, the UniRender process may release lock variable A and notify the application that lock variable A has been released. When the next vertical synchronization signal (Vsync-APP) arrives, lock variable A is held and the render tree is written to the shared memory. In this case, the functions of the staging data segment and the current data segment are swapped. The UniRender process reads the stagingdisplaylist and stagingproperties segments last to realize the "dual buffer" mechanism and ensure the robustness of the interface generation.

[0239] Optionally, in some embodiments of the present application, when there is one lock variable between the application and the UniRender process, the NODES field may contain only stagingdisplaylist and stagingproperties, or only currentdisplaylist and currentproperties. The application and the UniRender process achieve read and write security by using the lock variable, so that the UniRender process reads the correct render tree.

[0240] If necessary, in some embodiments of the present application, each application may hold more lock variables. For example, each application holds lock variable A and lock variable B. In this way, after an application releases lock A, the application does not need to wait for UniRender to release lock A. After holding lock variable B, the application writes the render tree data directly to the shared memory after receiving the next vertical synchronization signal (Vsync-APP).

[0241] The hold, release, and inter-process synchronization of the lock variable B are the same as the hold, release, and inter-process synchronization of the lock variable A. For details, please refer to the description of FIG. 14. The details will not be described again in this specification.

[0242] FIG. 15 is a schematic diagram of another example in which an application writes a render tree to a shared memory according to an embodiment of the present application.

[0243] 15, when an application stores one lock variable A, the application holds the lock variable A after vertical synchronization signal 1 (Vsync-APP) is received, and writes the generated render tree to the shared memory in the holding phase. After the application writes the render tree to the shared memory, the lock variable A is released. After determining that the lock variable A has been released by the application, UniRender holds the lock variable A and starts reading the render tree generated by the application from the shared memory in response to the vertical synchronization signal 1.

[0244] During the period that the UniRender process holds lock variable A, the application receives vertical sync signal 2 (Vsync-APP) after vertical sync signal 1. Because lock variable A is held by the UniRender process, the application cannot write the render tree to shared memory in a timely manner and must wait a period of time until the application determines that the UniRender process has released lock A.

[0245] However, in the content of Figure 13, it is clear that the UniRender process reads the currentdisplaylist and currentproperties fields, and the application writes the stagingdisplaylist and stagingproperties fields. Simultaneous reading and writing of different fields in the shared memory may be achieved by increasing the number of lock variables.

[0246] 15, when the application stores two lock variables B, after vertical synchronization signal 1 (Vsync-APP) is received, the application holds lock variable A and writes the generated render tree to the shared memory in the holding phase. After the application writes the render tree to the shared memory, lock variable A is released. After determining that lock variable A has been released by the application, the UniRender process holds the lock variable and starts reading the render tree generated by the application from the shared memory in response to vertical synchronization signal 1.

[0247] During the period when the UniRender process holds lock variable A, the application receives vertical sync signal 2 (Vsync-APP) after vertical sync signal 1. In this case, because lock variable B is held, the application can write the render tree to shared memory in a timely manner without having to wait a period of time until the application determines that the UniRender process has released lock A.

[0248] Correspondingly, after determining that lock variable B has been released by the application, the UniRender process holds lock variable B and begins reading the render tree generated by the application from the shared memory in response to vertical sync signal 2.

[0249] Note that the number of lock variables held by an application may be related to the contents contained in the NODES field in the shared memory, or may be related to the value of Vsync-offset.

[0250] For example, if currentdisplaylist and currentproperties are a first group and stagingdisplaylist and stagingproperties are a second group, then two synchronized variables may be configured in the application and UniRender processes, which correspond to the two data groups, respectively. Similarly, in another example, if the NODES field contains three data groups, then three synchronized lock variables may be configured in the application and UniRender.

[0251] One lock variable corresponds to one data group. For example, if lock variable A corresponds to currentDisplayList and currentProperties, a change of lock variable A from held to released indicates that the application has successfully updated the data of currentDisplayList and currentProperties in the shared memory, and the data of currentDisplayList and currentProperties can be read by the UniRender process. Alternatively, a change of lock variable A from held to released indicates that the UniRender process has completed reading the data of currentDisplayList and currentProperties from the shared memory, and the data of currentDisplayList and currentProperties can be updated by the application.

[0252] The number of lock variables may be related to the value of Vsync-offset.

[0253] In other words, the number of lock variables may be related to the difference Vsync-offset between the vertical synchronization signal (Vsync-APP) and the vertical synchronization signal (Vsync-UR). If the Vsync-offset is large, the lock variable may not be configured. If the lock variable is not configured, the UniRender process reads the render tree from the shared memory after receiving the vertical synchronization signal (Vsync-UR). Because the Vsync-offset is large, the application has completely written the render tree to the shared memory when the UniRender process reads the render tree.

[0254] S903: Window control information and display area information of the application are transferred.

[0255] The LayerManager in the UniRender process obtains window control information of one or more applications from the window manager service, and further determines whether drawing logic that triggers off-screen rendering exists in any application's layer by referring to the layer of the one or more applications obtained in step S802.

[0256] The UniRender process may further obtain the Z-order of the layers of different applications, which is the order of the Z axis between the different layers.

[0257] The DisplayerManager in the UniRender process obtains display area information from the display manager service, and the display area information includes the size of the display device. The UniRender process determines the size of the allocated surface based on the display area information. The surface is used to carry the bitmap generated by the UniRender process based on the target render tree. For information about generating a bitmap based on the target render tree by the UniRender process, please refer to the description of step S905. Details will not be described again in this specification.

[0258] The instructions related to off-screen rendering include instructions such as rounding, scaling, rotation, or color conversion. For the definition of off-screen rendering, please refer to the description of (1.3) Off-screen Rendering above. The details will not be described again in this specification.

[0259] The drawing logic associated with off-screen rendering triggers such as rounding, scaling, rotation, and color transformation is described below with the aid of an example.

[0260] 16A-1 and 16A-2 through 16C are schematic diagrams of an example of a scenario of drawing logic triggering off-screen rendering, according to an embodiment of the present application.

[0261] In response to a user interaction, such as a bezel-less gesture (e.g., sliding upward from the bottom to the center of the screen) or tapping a multitasking control on the bottom navigation bar, the electronic device displays a multitasking interface, as shown in FIGS. 16A-1 and 16A-2.

[0262] The bitmap generated by the gallery application has the same size as the screen, but the interface of the gallery application in the multitasking interface shown in Figures 16A-1 and 16A-2 is the interface obtained after scaling and rounding processes.

[0263] As shown in Figure 16B, in small window mode, views of multiple applications are displayed on the interface of the electronic device, some of the applications' interfaces being in floating windows.

[0264] For example, a news application interface and an SMS application interface are displayed on the interface of the electronic device, the SMS application interface being within a rendering floating window.

[0265] The bitmap generated by the SMS application is the same size as the screen, but the interface of the SMS application in small window mode shown in FIG. 16B is the interface obtained after scaling and rounding processes.

[0266] As shown in Figure 16C, when a user taps an icon of the reader application on the desktop application, the electronic device launches the reader application. In the launching process of the reader application, the main interface (interface corresponding to the Main Activity) or the starting window of the reader application is continuously enlarged.

[0267] The bitmap corresponding to the main interface or launching window of the reader application has the same size as the screen, and a continuous scale-up effect is realized by adjusting the scaling ratio, and the bitmap corresponding to the main interface or launching window of the reader application is the bitmap obtained after the rounding process and is displayed on the interface of the electronic device.

[0268] It will be appreciated that in the scenarios illustrated in Figures 16A-1 and 16A-2 through 16C, and many other scenarios, in order to improve the user experience, an interface needs to be generated that better matches the user's visual habits through rounding, scaling, rotation, and color transformation.

[0269] Note that in the scenarios illustrated in Figures 16A-1 and 16A-2 through 16C, and many other scenarios, the UniRender process receives window control information for each application's layer from the window manager service to determine whether rounding, scaling, rotation, and color conversion needs to be performed for each application's layer.

[0270] Optionally, in some embodiments of the present application, the UniRender process may first obtain window control information for one or more applications from a window manager service and then obtain the render tree for one or more applications, i.e., the time order of steps S902 and S903 may be reversed.

[0271] S904: A target render tree is generated based on the obtained render tree, window control information, and display area information.

[0272] First, after receiving the render trees generated by one or more applications, the UniRender process determines from the window control information whether each application's layer has logic to trigger off-screen rendering in response to receiving a vertical synchronization signal and window control information.

[0273] When the application is displayed in a display area of ​​the local electronic device, the window control information may come from a local window manager service. When the application is displayed in a display area of ​​another electronic device, the window control information may come from a window manager service of the peer electronic device.

[0274] If the UniRender process determines that logic for triggering off-screen rendering exists in any layer of the application, the UniRender process converts the logic for triggering off-screen rendering into off-screen rendering instructions and converts the off-screen rendering instructions into properties of render nodes in the corresponding render tree. For ease of explanation, the process by which the UniRender process converts logic for triggering off-screen rendering into off-screen rendering instructions and converts the off-screen rendering instructions into properties of render nodes in the corresponding render tree may be briefly referred to as forwarding the off-screen rendering instructions.

[0275] Then, after the UniRender process transitions the off-screen rendering trigger instruction for one or more render trees, the UniRender process merges, for each display area, the render trees corresponding to the visible layers of each display area to generate a target render tree. In other words, the number of target render trees may be related to the number of display areas.

[0276] After receiving a vertical synchronization signal (Vsync-UR), the UniRender process may start reading the render tree from the shared memory, and after obtaining multiple render trees, start transferring off-screen rendering instructions and merging the render trees. Alternatively, the UniRender process may start reading the render tree from the shared memory while holding the lock variable, and start transferring off-screen rendering instructions and merging the render trees when a vertical synchronization signal (Vsync-UR) is received.

[0277] The process of triggering and transitioning off-screen rendering instructions and the process of generating the target render tree are described separately below.

[0278] (a) The process of advancing instructions to trigger off-screen rendering First, the UniRender process obtains the window control information of each application layer and determines whether the window control information of the layer includes drawing logic for triggering off-screen rendering. If the UniRender process determines that none of the window control information of the layers of all applications in the display area includes drawing logic for triggering off-screen rendering, one or more render trees may be directly merged into the target render tree. If the UniRender process determines that the window control information of the layers of any application in the display area includes drawing logic for triggering off-screen rendering, the UniRender process first transfers instructions for triggering off-screen rendering, and then merges the multiple render trees into the target render tree.

[0279] The process of transferring instructions to trigger offscreen rendering: The UniRender process first determines the drawing logic that triggers off-screen rendering from the window control information, and converts the drawing logic that triggers off-screen rendering into instructions (also called drawing logic instructions) that can be configured into the rendering properties of a render node. After determining the binding relationship between layers and render trees, the UniRender process updates the off-screen rendering instructions into the rendering properties of the corresponding render node.

[0280] When the rendering properties of the render node include a corresponding scaling property (scale), rounding property (roundrect), color transform property (colortransform), or rotation property (transform), the parameters in the scaling instruction, rounding property, color transform property, or rotation instruction in the instruction to trigger off-screen rendering are assigned to the scaling property, rounding property, color transform property, or rotation property in the properties of the render node.When the rendering properties of the render node do not include a corresponding scaling property, rounding property, color transform property, or rotation property, a scaling property, rounding property, color transform property, or rotation property is added to the render node, and the parameters in the scaling instruction, rounding property, color transform property, or rotation instruction in the instruction to trigger off-screen rendering are assigned to the scaling property, rounding property, color transform property, or rotation property in the properties of the render node.

[0281] 17A and 17B are schematic diagrams of an example of UniRender transitioning off-screen rendering trigger instructions according to an embodiment of the present application.

[0282] As shown in Figure 17A, the UI threads of Application 1 and Application 2 generate render trees independently and transfer the render trees independently to the UniRender process via IPC communication. The UniRender process then determines the instructions from the window manager service to trigger offscreen rendering on each application's layer. The render tree generated by Application 1 is Render Tree 1, and the render tree generated by Application 2 is Render Tree 2.

[0283] UniRender updates properties of render nodes in different render trees independently, i.e. UniRender process updates properties of render nodes in render tree 1, UniRender process updates properties of render nodes in render tree 2, etc.

[0284] If an instruction to trigger off-screen rendering applies to an entire layer, the UniRender process assigns the parameters in the off-screen rendering instruction to the root node of the render tree corresponding to the layer, or directly to all render nodes in the render tree.

[0285] When the UniRender process assigns parameters in an off-screen rendering instruction to the root node of a render tree corresponding to a layer, the UniRender process automatically configures these properties of the render node to the rendering properties of the child nodes of the root node when it generates a bitmap based on the render tree.

[0286] For example, if the child node of render node 1 is render node 2, after a rotation property (transform) is configured in the rendering properties of render node 1, the UniRender process configures the same rotation property (transform) for the rendering properties of RenderNode2 when generating the render tree.

[0287] As shown in FIG. 17B, after receiving bitmaps corresponding to the render trees generated by the applications before the off-screen rendering instructions are migrated, SurfaceFlinger needs to perform different processing on the bitmaps of different applications based on the window control information to perform layer compositing.

[0288] After the off-screen rendering instructions for an application's render tree are migrated, the UniRender process assigns the parameters in the off-screen rendering instructions to the rendering properties of the render tree, so that the transformed bitmap is drawn directly onto the canvas in the process of generating the bitmap based on the render tree. Since the bitmaps of different applications do not need to be processed differently, the bitmaps of different applications can be drawn sequentially into one on-screen buffer.

[0289] The UniRender process assigns parameters in the off-screen rendering instructions to rendering properties in the render tree, as shown in FIG.

[0290] FIG. 18 is a schematic diagram of an example of UniRender modifying render node properties of a render tree according to an embodiment of the present application.

[0291] As shown in Figure 18, the UniRender process modifies the properties of the root node of the render tree and adds instructions to realize the addition of properties of the render nodes. The instruction corresponding to the scaling property is setScale(), the instruction corresponding to the rounding property is outline.setRoundRect(), the instruction corresponding to the color transformation property is setColorFilter(), the instruction corresponding to the rotation property is setLeftTopRightBottom(), etc.

[0292] In the following step S905, when generating a bitmap based on the render tree, the UniRender process modifies the drawing operations in the drawing instruction list according to the rendering properties, and then generates a bitmap obtained after scaling, rounding, color conversion and rotation. For details, please refer to the description of step S905. The details will not be described again in this specification.

[0293] Optionally, in some embodiments of the present application, the instruction in the property of the root node of the render tree may further include an instruction setStaticMatrix(), etc. Since the UI thread of the application generates a render tree based on the surface requested from SurfaceFlinger, in the process of transferring an instruction for triggering off-screen rendering to change the reference system when UniRender generates a bitmap based on the render tree in step S905, the instruction setStaticMatrix() is configured in the root node of the render tree. For the specific content of the instruction setStaticMatrix(), please refer to the following description of step S905. Details will not be described again in this specification.

[0294] (b) The process of generating the target render tree The UniRender process obtains one or more processed render trees after migrating off-screen rendering instructions for all render trees.

[0295] When there is one processed render tree, the processed render tree is the target render tree, or when there are multiple processed render trees, the multiple processed render trees are merged to form one target render tree.

[0296] FIG. 19 is a schematic diagram of an example of compositing multiple render trees obtained after off-screen rendering instructions are migrated to one target render tree according to an embodiment of the present application.

[0297] As shown in Figure 19, the UniRender process may create a new root render node, e.g., the new root render node of Figure 19, and use the processed render trees of multiple applications as child nodes of the root render node. Specifically, the root render node may store a parent-child relationship between the processed render trees of different applications, such that the UniRender process may traverse the target render trees from the root render node to retrieve the processed render trees of different applications.

[0298] FIG. 20 is a schematic diagram of an example of compositing multiple render trees obtained after off-screen rendering instructions are migrated to one target render tree according to an embodiment of the present application.

[0299] The UniRender process may first determine the Z-order of layers corresponding to different applications according to the window manager service, i.e., determine the top-bottom occlusion relationship between layers of the applications. Furthermore, in the process of generating the target render tree, render nodes corresponding to completely occluded views are deleted, thereby reducing the amount of calculation in the process of generating a bitmap in step S905 and improving the speed of generating the bitmap.

[0300] For example, as shown in FIG. 20, the layer corresponding to the render tree generated by application 1 is layer 1, and the layer corresponding to the render tree generated by application 2 is layer 2. The Z-order of layer 1 is higher than the Z-order of layer 2. The render tree corresponding to layer 1 is root render node 1, render node 1, render node 2, and render node 3, where render node 1 and render node 2 are child render nodes of root render node 1, and render node 3 is a child render node of render node 2. The render tree corresponding to layer 2 is root render node 2, render node 4, render node 5, and render node 6, where render node 4 and render node 5 are child render nodes of the new root render node, and render node 6 is a child render node of render node 5.

[0301] The UniRender process may traverse the child render nodes in the render tree corresponding to layer 1 and the child render nodes in the render tree corresponding to layer 2, determine the position of the views corresponding to each render node on the surface (the surface assigned by the UniRender process), and refer to the Z-order of layer 1 and the Z-order of layer 2 to determine the render nodes of the fully occluded views and the fully occluded views.

[0302] For example, UniRender may determine that the view corresponding to render node 6 in the render tree corresponding to layer 2 is fully occluded, and then remove render node 6.

[0303] After removing the render nodes corresponding to the fully occluded views, the UniRender process may merge all render trees obtained after transitioning the off-screen rendering trigger instruction into the target render tree according to the contents shown in FIG. 18.

[0304] Optionally, in some embodiments of the present application, UniRender may further traverse the render nodes and optimize the parameters of the target render tree based on the granularity of the DrawOP drawing operations. For example, the DrawOP drawing operations that do not affect the interface are deleted, and not affecting means that the graphics drawn by the DrawOP drawing operations are not displayed in the interface. As another example, in different applications, the position of the DrawOP operations in the drawing nodes is modified, so that the same type of DrawOP drawing operations can be executed simultaneously. For example, the DrawOP drawing operation of the render node 2 of application 1 is modified to the drawing instruction list of the render node 1 of application 2. This is not limited in this specification.

[0305] If necessary, in some embodiments of the present application, the multiple render trees obtained after the off-screen rendering trigger instruction is migrated may be merged into one render tree according to the contents shown in FIG. 18, and then the fully occluded views are determined to remove the render nodes corresponding to the fully occluded views to generate a target render tree.

[0306] Optionally, in some embodiments of the present application, for a partially occluded view, the render node corresponding to that view may also be clipped by using the clip instruction. Clipping may be performed before the target render trees are merged or after the target render trees are merged.

[0307] Optionally, in some embodiments of the present application, after step S904 and before step S905, after generating the target render tree, the UniRender process depackages the drawing instruction list of the RenderNode to obtain a series of DrawOPs, and then performs DrawOP operations on the entire target render tree to perform batching and merging. In step S805, a target render tree is obtained from which a bitmap can be generated with less computational effort.

[0308] It will be appreciated that in the scenarios shown in Figures 16A-1 and 16A-2 and Figure 16B as well as other scenarios, multiple render trees are merged and the merged render tree is optimized to avoid or reduce overdraw, thereby improving the speed of generating the bitmap in step S905, reducing power consumption of the electronic device, and improving the user experience.

[0309] It should be noted that the UniRender process may further optimize other parameters of the target render tree in the process of generating the target render tree, which is not limited herein.

[0310] S905: A bitmap is generated based on the target render tree.

[0311] After the target render tree is obtained, the UniRender process assigns a surface to the target render tree, and the UniRender process generates a bitmap on the surface based on the target render tree, where the bitmap corresponds to a composited interface of one or more applications.

[0312] When the electronic device has multiple display areas, the surface may be associated with one of the display areas, and the size of the surface may be the same as the size of the display area associated with the surface.

[0313] The UniRender process of generating a bitmap on a surface based on a target render tree includes the following:

[0314] First, the UniRender process begins traversing the target render tree from the root node, and may traverse the child nodes of the root node in multiple ways.

[0315] The UniRender process can traverse different layers under the root node based on the Z-order of the layers. The UniRender process may traverse different layers in descending Z-order or may traverse different layers in ascending Z-order.

[0316] 20, if the Z-order of layer 1 is higher than the Z-order of layer 2, the UniRender process may traverse the render tree corresponding to layer 1 first, i.e., traverse the render tree that includes root render node 1, render node 1, render node 2, and render node 3, then traverse the render tree that includes root render node 2, render node 4, and render node 5. Similarly, the UniRender process may traverse the render tree corresponding to layer 2 first, then traverse the render tree corresponding to layer 1.

[0317] Note that as the UniRender process traverses layers in descending Z-order, when it performs the drawing operations in the drawing instruction list, if layers are composited with a top-bottom occlusion relationship, drawing may be performed only where they have not been drawn, to reduce overdraw.

[0318] Note that as the UniRender process traverses the layers in ascending Z-order, it sequentially executes the drawing operations in the drawing instruction lists in the render tree that correspond to the different layers to generate a bitmap.

[0319] The UniRender process then modifies the drawing operations in the drawing instruction list according to the rendering properties and executes the drawing operations to generate a bitmap.

[0320] As it traverses each render node, the UniRender process reads the rendering properties of the render node. When the rendering properties include scaling properties, rounding properties, color transformation properties, rotation properties, etc., the UniRender process modifies the parameters of the drawing operations in the drawing instruction list according to these properties, and then executes the drawing operations in the modified drawing instruction list. When the rendering properties do not include scaling properties, rounding properties, color transformation properties, or rotation properties, the UniRender process directly executes the drawing operations in the drawing instruction list.

[0321] The manner in which the UniRender process modifies the parameters of drawing operations in a drawing instruction list according to rendering properties is described below with the aid of an example.

[0322] For example, the rendering properties of a render node include setscale(0.5), which indicates scaling down to 0.5 times the original size, and the drawing instruction list includes drawCircle(x0, y0, 5). In this case, when executing the drawing operation drawCircle(x0, y0, 5), the UniRender process changes the drawing operation to drawCircle(x0, y0, 2.5). In drawCircle(), the first parameter is the coordinate of the center of the circle on the X axis, the second parameter is the coordinate of the center of the circle on the Y axis, and the third parameter is the radius of the circle.

[0323] Considering that the drawing instruction list generated by an application's UI thread may be related to the size of the surface requested by the application, in this embodiment of the application, when traversing a render tree node, the UniRender process configures a setStaticMatrix() in the render node's properties to transform the reference coordinate system for drawing and rendering.

[0324] The UniRender process may use the WMS to determine the size of the reference coordinate system or reference surface when each application generates a drawing command list in order to determine the parameters of setStaticMatrix().

[0325] FIG. 21 is a schematic diagram of an example of setStaticMatrix() according to an embodiment of the present application.

[0326] After determining the surface coordinate system referenced by the application to generate the drawing instruction list, the UniRender process determines a coordinate transformation matrix Transformation based on the coordinate system of the surface held by the UniRender process.

number

[0327] In this specification, scale x is the scaling in the x-axis direction, skew x is the twist / tilt along the x-axis, translate x is the translation along the x-axis, scale y is the scaling in the y-axis direction, skew y is the twist / tilt along the y axis, translate y is the translation along the y axis.

[0328] After executing the setStaticMatrix() command, the UniRender process obtains the coordinate transformation matrix Transformation through calculation and applies the transformation matrix to each drawing operation in the drawing command list. As shown in Fig. 21, when the coordinate system referenced by the UI thread of the application to generate the render tree is not the coordinate system on the surface held by the UniRender process, the UniRender process determines the coordinate transformation matrix Transformation by using the setStaticMatrix() command and transforms the reference coordinate system when drawing the interface on the surface based on the drawing command list in order to draw the interface correctly.

[0329] The above mainly describes the specific procedure for an electronic device to realize the interface generation method provided in the embodiment of the present application in a single display area scenario. The following mainly describes the specific procedure for an electronic device to realize the interface generation method provided in the embodiment of the present application in a multi-display area scenario.

[0330] (2.3) How to generate interfaces in multi-area scenarios First, the multi-display area scenario is explained with an example.

[0331] (a) Multi-area scenario In an embodiment of the present application, the display area may be a screen, a virtual screen, etc. The virtual screen may be an area used by an electronic device to carry the content displayed on the screen during screen recording.

[0332] 22A to 22C are schematic diagrams of an example of a multiple display region scenario according to an embodiment of the present application.

[0333] When multiple screens are configured on an electronic device, each screen is a display area. As shown in Figure 22A, the primary screen of the electronic device is used as display area 1 for displaying a desktop application, and the secondary screen of the electronic device is used as display area 2 for displaying a music application.

[0334] The screen of the electronic device may have multiple states, and the screen in each state may be one or more display areas. As shown in FIG. 22B, when the screen of the electronic device is in a folded state, each part of the screen may be a display area, or when the screen of the electronic device is in an unfolded state, the left half of the screen and the right half of the screen may each be a display area.

[0335] As shown in Figure 22C, when the screen recording function is enabled, the screen of the electronic device is the display area and the virtual screen carrying the screen recording content is the display area. The content displayed in the two display areas may be the same.

[0336] 23A-1 and 23A-2 through 23C are schematic diagrams of another example of a multiple display region scenario according to an embodiment of the present application.

[0337] 23A-1 and 23A-2, electronic device 1 shares the content displayed on its screen with the screen of electronic device 2. In this case, for electronic device 1, the screen of electronic device 1 is display area 1, and the screen of electronic device 2 is display area 2. The content displayed on the screen of electronic device 2 includes the content on the screen of electronic device 1.

[0338] 23B, electronic device 1 projects the content displayed on its screen onto the screen of electronic device 2. In this case, for electronic device 1, the screen of electronic device 1 is display area 1, and the screen of electronic device 2 is display area 2. Unlike the content shown in FIGS. 23A-1 and 23A-2, the content displayed on the screen of electronic device 2 is only the content on the screen of electronic device 1.

[0339] 23C, electronic device 1 projects the content displayed on its screen onto the screen of electronic device 2. In this case, for electronic device 1, the screen of electronic device 1 is display area 1, and the screen of electronic device 2 is display area 2. Unlike the content shown in FIGS. 23A-1, 23A-2, and 23B, the content displayed on the screen of electronic device 2 is a part of the content on the screen of electronic device 1.

[0340] It should be noted that in a multi-display area scenario, in the case of multiple electronic devices, for example where there is display area 1 on electronic device 1 and display area 2 on electronic device 2, only the content displayed on the screen of electronic device 1 may be displayed in display area 2, or some or all of the content displayed on the screen of electronic device 1 may be superimposed on the content originally displayed on the screen of electronic device 2 and displayed in display area 2.

[0341] (b) How to generate an interface in a single device, multiple display areas scenario FIG. 24 is a schematic diagram of another example of an interface generation method according to an embodiment of the present application.

[0342] As shown in FIG. 24, the interface generation method in a single device multi-display area scenario includes the following steps.

[0343] S2401: Construct and generate a render tree after receiving a vertical signal.

[0344] For specific content, please refer to the corresponding description of step S901 in Figure 9. Details will not be described again in this specification.

[0345] S2402: The render tree is transferred in a cross-process manner.

[0346] For specific content, please refer to the corresponding description of step S902 in Figure 9. Details will not be described again in this specification.

[0347] If necessary, in some embodiments of the present application, the DisplayRender sub-process may directly obtain the render tree through IPC communication with the corresponding application. The UniRender process determines the correspondence between the DisplayRender sub-process and the application.

[0348] Optionally, in some embodiments of the present application, a shared memory set containing multiple shared memories may exist between all DisplayRender sub-processes and all applications, and each DisplayRender sub-process holds a handle to the multiple shared memories.

[0349] For the concept of the DisplayRender sub-process, please refer to the corresponding description of step S2404, and the details will not be described again in this specification.

[0350] S2403: Window control information and display area information of the application are transferred.

[0351] For specific content, please refer to the corresponding description of step S903 in Figure 9. Details will not be described again in this specification.

[0352] S2404: Allocate a render tree.

[0353] The UniRender process may create sub-processes DisplayRender (DisplayRender1 to DisplayRenderN in FIG. 24) whose number is equal to the number of display areas, and transfer the render tree of the application displayed in the display area to the DisplayRender.

[0354] For example, application 1 and application 2 are displayed in display area 1, which corresponds to sub-process DisplayRender1, and application 3 and application 4 are displayed in display area 2, which corresponds to sub-process DisplayRender2. In this case, the UniRender process transfers the render tree of application 1 and the render tree of application 2 to sub-process DisplayRender1, and transfers the render tree of application 3 and the render tree of application 4 to sub-process DisplayRender2.

[0355] DisplayRender1 and DisplayRender2 may obtain a handle to the shared memory of the corresponding application from the UniRender process to obtain the render tree of the corresponding application. Alternatively, the render tree of the application is obtained from the UniRender process by other IPC communication methods such as Binder, which are not limited in this specification.

[0356] If necessary, in some embodiments of the present application, the UniRender process may first merge the render trees of multiple applications into a target render tree, and then transfer the target render tree to the corresponding DisplayRender sub-process. The transfer of the target render tree may be realized by multiple IPC communication methods, such as shared memory and Binder, which are not limited in this specification.

[0357] Optionally, in some embodiments of the present application, the UniRender process may transition off-screen rendering instructions for a render tree and then transfer the render tree to a corresponding DisplayRender sub-process, or may transfer the window control information and render tree of the application in the corresponding display area to the corresponding DisplayRender sub-process, which transitions off-screen rendering instructions for the render tree and merges multiple render trees into a target render tree.

[0358] Optionally, in some embodiments of the present application, two threads, for example an I / O thread and a render thread, may be configured for the DisplayRender sub-process. The I / O thread is responsible for receiving a render tree, and the render thread is responsible for generating a target render tree and generating a bitmap based on the target render tree.

[0359] It will be appreciated that by creating multiple DisplayRender sub-processes, the UniRender process may perform frequency division and frequency multiplication on the vertical synchronization signal (Vsync-UR) and send the frequency-divided and frequency-multiplied vertical synchronization signal (Vsync-UR) to the different DisplayRender sub-processes, such that the different DisplayRender sub-processes generate bitmaps at different frequencies to match the refresh rates of the different display areas.

[0360] FIG. 25 is a schematic diagram of an example in which the UniRender process performs frequency division or frequency multiplication on a vertical synchronization signal, according to an embodiment of the present application.

[0361] As shown in FIG. 25, if the refresh rates of the display areas corresponding to different DisplayRender sub-processes are different, UniRender may perform frequency division or frequency multiplication on the vertical synchronization signal (Vsync-UR) in order to forward the vertical synchronization signal to the different DisplayRender sub-processes.

[0362] For example, the refresh rate of display area 1 is 60 Hz, the refresh rate of display area 2 is 30 Hz, the DisplayRender sub-process corresponding to display area 1 is DisplayRender1, and the DisplayRender sub-process corresponding to display area 2 is DisplayRender2. In this case, after receiving a 60 Hz vertical synchronization signal (Vsync) or generating a 60 Hz vertical synchronization signal (Vsync), the UniRender process performs frequency division to generate a 30 Hz vertical synchronization signal (Vsync-UR) and a 60 Hz vertical synchronization signal (Vsync-UR), transfers the 60 Hz vertical synchronization signal (Vsync-UR) to the DisplayRender1 sub-process, and transfers the 30 Hz vertical synchronization signal (Vsync-UR) to the DisplayRender2 sub-process.

[0363] It will be appreciated that frequency division or frequency multiplication is performed on the vertical synchronization signal (Vsync-UR) so that the frequency at which the DisplayUniRender process receives the render tree or target render tree and the frequency at which the bitmaps are generated matches the refresh frequency of the display area.

[0364] After receiving a vertical synchronization signal (Vsync-UR), the DisplayRender sub-process may begin reading the render tree, generating the target render tree, and generating the bitmap.

[0365] It should be noted that the UniRender process may receive the display area information sent by the display manager service, determine the connection status of each display area, and create or destroy a DisplayRender process corresponding to each display area according to the connection status of the display area.

[0366] For the content of concepts such as UniRender and shared memory architecture, please refer to the description of step S904. Details will not be described again in this specification.

[0367] S2405: A target render tree is generated based on the acquired render tree, window control information, and display area information.

[0368] The DisplayRender sub-process merges one or more target render trees into a target render tree and transitions off-screen rendering instructions for an application's render trees when the application's layers are associated with off-screen rendering logic.

[0369] It should be noted that the DisplayRender sub-process may obtain window control information within the display area and display area information from the UniRender process.

[0370] Optionally, in some embodiments of the present application, the UniRender process may merge one or more render trees into a target render tree and then forward the target render tree to the DisplayRender sub-process.

[0371] For the content of concepts such as the process of generating a target render tree and migrating off-screen rendering instructions, please refer to the description of step S904, and the details will not be described again in this specification.

[0372] S2406: Generate a bitmap based on the target render tree.

[0373] After generating the target render tree, the DisplayRender sub-process generates a bitmap based on the target render tree. After generating the bitmap, the DisplayRender sub-process transfers the surface carrying the bitmap to the UniRender process. The UniRender process sends the contents of the surface to each display area via the DSS.

[0374] For the process of generating a bitmap based on the target render tree, please refer to the description of step S905, and the details will not be described again in this specification.

[0375] Because the calls to the underlying graphics library are context dependent, different DisplayRender sub-processes correspond to different contexts, so that bitmaps can be generated in parallel for different display areas.

[0376] It will be appreciated that the UniRender process may generate bitmaps for each display area in parallel by creating DisplayRender sub-processes that correspond one-to-one with the display areas. In a multi-display area scenario, interface generation speed can be improved, frame freezes can be reduced, and the user experience can be improved.

[0377] With reference to the contents shown in Figures 26A and 26B, an example of a data flow process when an electronic device implements the interface generation method is described below.

[0378] 26A and 26B are schematic diagrams of an example of data flow when an electronic device implements an interface generation method according to an embodiment of the present application.

[0379] As shown in Figure 26A, the electronic device has two display areas: display area 1 and display area 2. Display area 1 displays the contents of application 1 and application 3, and display area 2 displays the contents of application 2, application 3, and application 4.

[0380] The UniRender process creates a DisplayRender1 sub-process corresponding to display area 1 and a DisplayRender2 sub-process corresponding to display area 2. After receiving Render Tree 1 of Application 1, Render Tree 2 of Application 2, Render Tree 3 of Application 3, and Render Tree 4 of Application 4, the UniRender process transfers Render Tree 1 and Render Tree 3 to the DisplayRender1 sub-process, and transfers Render Tree 2, Render Tree 3, and Render Tree 4 to the DisplayRender2 sub-process. The UniRender process transfers relevant window control information and display area information to the DisplayRender sub-process.

[0381] The DisplayRender1 sub-process and the DisplayRender2 sub-process generate bitmaps separately based on the target render tree. The DisplayRender1 sub-process generates surface1 based on the display area information of display area 1, and the DisplayRender2 sub-process generates surface2 based on the display area information of display area 2. Surface1 is used to carry bitmap 1 generated by the DisplayRender1 sub-process, and surface2 is used to carry bitmap 2 generated by the DisplayRender1 sub-process. The size of surface1 may be related to the size of display area 1, and the size of surface2 may be related to the size of display area 2.

[0382] In this embodiment of the present application, the contents displayed by the application on different display areas may be the same or different. If the contents displayed by the application on different display areas are the same, one render tree generated by the application is assigned to multiple DisplayRender sub-processes. If the contents displayed by the application on different display areas are different, the application generates multiple different render trees and assigns the render trees to multiple DisplayRender sub-processes accordingly.

[0383] For example, in the content shown in Fig. 26B, since the sizes (e.g., resolutions) of display area 1 and display area 2 are different, the interfaces displayed by application 3 in display area 1 and display area 2 are different (e.g., the layout structures are different but the contents are the same). Therefore, the application generates render trees corresponding to the different display areas, which are render tree 31 and render tree 32, respectively. Render tree 31 is transferred to a DisplayRender1 sub-process, and render tree 32 is transferred to a DisplayRender2 sub-process.

[0384] Two threads may be configured for any DisplayRender sub-process, for example an I / O thread and a render thread. The I / O thread is responsible for receiving a render tree, and the render thread is responsible for generating a target render tree and generating a bitmap based on the target render tree.

[0385] (c) How to generate interfaces in multi-device, multi-display area scenarios As shown in the scenarios of Figures 23A-1 and 23A-2 to 23C, an electronic device may display some or all of its on-screen content on a screen of a peer electronic device in a manner such as projection or multi-screen collaboration.

[0386] In a multi-device multi-display area scenario, after establishing a connection to a peer electronic device, the local electronic device may use the screen of the peer electronic device as display area 2 and transfer the render trees of applications that need to be displayed in display area 2 to the peer electronic device. After receiving the render trees of the local electronic device, the peer electronic device merges the render trees of all applications displayed in display area 2 into the target render tree to generate a bitmap and send the bitmap for display.

[0387] Alternatively, the local electronic device may send the interface of the application displayed in the display area 2 to the peer electronic device.

[0388] FIG. 27 is a schematic diagram of another example of an interface generation method according to an embodiment of the present application.

[0389] As shown in FIG. 27, the interface generation method provided in this embodiment of the present application includes the following steps:

[0390] S2701: Establish a connection.

[0391] The electronic device 1 and the electronic device 2 establish communication connections through multiple methods such as Bluetooth, Wi-Fi, HiLink, etc. The electronic device 1 uses the screen of the electronic device 2 as a display area 2, and the screen of the electronic device 1 is used as a display area 1.

[0392] After determining that a connection has been established between electronic device 1 and electronic device 2, or after receiving a display manager service and window manager service projection request, multi-screen collaboration request, etc., the UniRender process on electronic device 2 may apply for heap memory to store data sent by electronic device 1 and used to render and generate bitmaps, e.g., render trees.

[0393] S2702: The application displayed in the display area 2 is determined.

[0394] The window manager service and the display manager service on the electronic device 1 determine the application to be displayed in the display area 2 and transfer the result to the UniRender1 process on the electronic device 1. The UniRender1 process on the electronic device 1 then obtains the render tree of the application displayed in the display area 2 via IPC communication.

[0395] For obtaining the render tree of the application by the UniRender1 process through IPC communication, please refer to the above description of step S902, and the details will not be described again in this specification.

[0396] S2703: The render tree of the application displayed in display area 2 is sent.

[0397] The UniRender1 process on electronic device 1 sends the render tree of the application being displayed in display area 2 to electronic device 2.

[0398] UniRender1 may determine the starting address of the shared memory and the size of the shared memory, and transfer the render tree to the created heap memory of the UniRender2 process on the electronic device 2 by using the communication connection in step S2701.

[0399] The storage structure of data in the heap memory may be the same as the storage structure of data in the shared memory in UniRender1. For the storage structure of data in the heap memory and the read and write security of the heap memory, please refer to the description of step S902. Details will not be described again in this specification.

[0400] It will be appreciated that by electronic device 1 sending the render tree to electronic device 2, the amount of data transmission between electronic device 1 and electronic device 2 can be reduced, thereby reducing delays and improving the speed of interface generation.

[0401] S2704: The render tree of the application displayed in display area 2 is merged into the target render tree.

[0402] After receiving a vertical synchronization signal (Vsync-UR2), the UniRender2 process on the electronic device 2 may merge the render tree of the application being displayed in the display area 2 into the target render tree, and perform the transition of off-screen rendering commands in this process.

[0403] Electronic device 2 may merge the render tree sent by electronic device 1 and the local render tree of electronic device 2 into a target render tree.

[0404] 28A and 28B are schematic diagrams of an example of merging a render tree into a target render tree, respectively, according to an embodiment of the present application.

[0405] 28A, application 1 and application 2 run on electronic device 1, the render tree generated by application 1 is render tree 1, and the render tree generated by application 2 is render tree 2. Application 3 runs on electronic device 2, and the render tree generated by application 3 is render tree 3.

[0406] The UniRender1 process on electronic device 1 determines that the interface of application 2 needs to be displayed in display area 2 and sends render tree 2 to the UniRender2 process on electronic device 2.

[0407] The UniRender1 process transfers off-screen rendering instructions to Render Tree 1 and Render Tree 2 separately, and merges Render Tree 1 and Render Tree 2 into target Render Tree 1 to generate Bitmap 1. The UniRender2 process transfers off-screen rendering instructions to Render Tree 2 and Render Tree 3 separately, and merges Render Tree 2 and Render Tree 3 into target Render Tree 2 to generate Bitmap 2.

[0408] Note that when the UniRender1 process sends the render tree 2 data to the UniRender2 process, it may send all of the render tree 2 data or the render tree 2 difference data to the UniRender2 process. The render tree 2 all data may be the current properties, current displaylist, staging displaylist, or staging properties in shared memory, and the render tree 2 difference data may be the drawing instruction lists and rendering properties of the render nodes of render tree 2 in the properties_dirty and displaylist_dirty queues.

[0409] 28B, application 1, application 2, and application 3 are running on electronic device 1, the render tree generated by application 1 is render tree 1, the render tree generated by application 2 is render tree 2, and the render tree generated by application 2 is render tree 3. Application 4 is running on electronic device 2, and the render tree generated by application 4 is render tree 4.

[0410] If the UniRender1 process on electronic device 1 determines that the interfaces of application 2 and application 3 need to be displayed in display area 2, the UniRender1 process sends render tree 2 and render tree 3 to the UniRender2 process on electronic device 2, or after render tree 2 and render tree 3 are merged into target render tree 2, sends target render tree 2 to the UniRender2 process on electronic device 2.

[0411] The UniRender1 process transfers off-screen rendering instructions for Render Tree 1, Render Tree 2, and Render Tree 3 separately, and merges Render Tree 1, Render Tree 2, and Render Tree 3 into target Render Tree 1 to generate Bitmap 1. The UniRender1 process transfers off-screen rendering instructions for Render Tree 2, and merges Render Tree 2 into target Render Tree 2.

[0412] If necessary, in some embodiments of the present application, the UniRender1 process may create multiple sub-processes DisplayRender to separately receive render trees from different applications. Specifically, in the content shown in FIG. 28B, the UniRender1 process may create two sub-processes DisplayRender: DisplayRender1 and DisplayRender2. The DisplayRender1 sub-process receives Render Tree 1, Render Tree 2, and Render Tree 3, and merges Render Tree 1, Render Tree 2, and Render Tree 3 into a target render tree 1. The DisplayRender2 sub-process receives Render Tree 2 and Render Tree 3, merges Render Tree 2 and Render Tree 3 into a target render tree 2, and sends the target render tree 2 to the UniRender2 process.

[0413] The UniRender2 process transfers off-screen rendering instructions to Render Tree 2 and Render Tree 3 separately, and merges Render Tree 2 and Render Tree 3 into Target Render Tree 2 to generate Bitmap 2.

[0414] It should be noted that in response to a vertical synchronization signal (Vsync-UR1) on electronic device 1, the UniRender1 process retrieves the render trees of one or more applications from the shared memory and starts generating a target render tree. In response to a vertical synchronization signal (Vsync-UR2) on electronic device 2, the UniRender2 retrieves the render trees of one or more applications from the heap memory and / or the shared memory and starts generating a target render tree.

[0415] Optionally, in some embodiments of the present application, when the frequencies of the vertical synchronization signal (Vsync-UR1) on the electronic device 1 and the vertical synchronization signal (Vsync-UR2) on the electronic device 2 do not match, the UniRender on the electronic device 1 may adjust the frequency of the vertical synchronization signal (Vsync-UR1) to be the same as the frequency of Vsync-UR2 through frequency division or frequency multiplication. Alternatively, the UniRender2 on the electronic device 2 may adjust the frequency of the vertical synchronization signal (Vsync-UR2) to be the same as the frequency of the vertical synchronization signal (Vsync-UR1) through frequency division or frequency multiplication.

[0416] If necessary, in some embodiments of the present application, multiple frequencies of the vertical synchronization signal (Vsync-UR1) may be configured for the electronic device 1. For example, in the scenario shown in FIG. 28A and FIG. 28B, the original frequency of the vertical synchronization signal (Vsync-UR) on the electronic device 1 is 60 Hz, and the original frequency of the vertical synchronization signal (Vsync-UR2) on the electronic device 2 is 90 Hz. After the electronic device 1 determines to display the interface of the application 2 on the electronic device 2, the UniRender process of the electronic device 1 may generate vertical synchronization signal (Vsync-UR1) frequencies that are 60 Hz and 90 Hz, respectively, so that the application 2 generates a render tree at a frequency of 90 Hz. In this way, the UniRender1 process may transfer the render tree at a frequency of 90 Hz to the UniRender2 process. The application 1 may continue to generate a render tree at a frequency of 60 Hz.

[0417] On any electronic device, the vertical synchronization signal received or generated by UniRender (Vsync-UR) may be the same as the vertical synchronization signal received by an application (Vsync-APP).

[0418] S2705: Generate a bitmap based on the target render tree.

[0419] For specific details, please refer to the description of step S905. Details will not be described again in this specification.

[0420] If necessary, in some embodiments of the present application, the UniRender1 process of electronic device 1 may first merge one or more render trees in display area 2 into a target render tree, generate a bitmap based on the target render tree, and transfer the bitmap to the UniRender2 process of electronic device 2.

[0421] It should be noted that when the electronic device 2 displays the interface of the application 1 on the electronic device 1, the user may tap the interface of the application 1 on the electronic device 2 with other interaction methods such as tapping, and the electronic device 2 may send the position of the tap operation performed by the user in the interface of the application 1 to the application 1 on the electronic device 1, so that the application 1 can respond correctly to the user interaction. Alternatively, since the interface of the application 1 on the electronic device 2 is generated by the UniRender2 process on the electronic device 2 based on the render tree, and the render tree includes the position information of the view, the electronic device 2 may determine the view tapped by the user and directly send a view tap event to the application 1 on the electronic device 1, so that the application 1 can respond correctly to the user interaction.

[0422] (3) Finally, the hardware and software architectures of the electronic device provided in the embodiments of the present application are described.

[0423] FIG. 29 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application.

[0424] In the following, the embodiment will be specifically described using an electronic device as an example. It should be understood that the electronic device may have more or less components than those shown in the figure, may combine two or more components, or may have a different component configuration. The components shown in the figure may be realized in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0425] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display 194, a subscriber identification module (SIM) card interface 195, and the like. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, an optical proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, and a bone conduction sensor 180M, among others.

[0426] It will be understood that the structure shown in this embodiment of the present invention is not a specific limitation on the electronic device. In some other embodiments of the present application, the electronic device may include more or fewer components than those shown in the figures, may combine some components, may split some components, or may have a different arrangement of components. The components shown in the figures may be realized by hardware, software, or a combination of software and hardware.

[0427] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be separate components or may be integrated into one or more processors.

[0428] The controller may be the brain and command center of the electronic device, and may generate operation control signals based on the instruction operation codes and time series signals to perform control of instruction fetching and instruction execution.

[0429] A memory may further be disposed within the processor 110 to store instructions and data. In some embodiments, the memory within the processor 110 is a cache. The memory may store instructions or data that have just been used or that are used periodically by the processor 110. If the processor 110 needs to use the instructions or data again, the instructions or data may be retrieved directly from the memory. This avoids repeated accesses and reduces the latency of the processor 110, thereby improving system efficiency.

[0430] In some embodiments, the processor 110 may include one or more interfaces, which may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0431] The I2C interface is a bidirectional synchronous serial bus and includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be separately coupled to the touch sensor 180K, the charger, the flash, the camera 193, and the like through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K by the processor 110 using the I2C bus interface, so that the processor 110 communicates with the touch sensor 180K by using the I2C interface to realize the touch function of the electronic device.

[0432] The I2S interface may be used for audio communication. The PCM interface may be configured to perform audio communication and sample, quantize, and encode analog signals. The UART interface is a universal serial data bus and is used for asynchronous communication.

[0433] The MIPI interface may be configured to connect the processor 110 to peripheral components such as the display 194 or the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate with each other by using the CSI interface to realize the photographing function of the electronic device. The processor 110 communicates with the display 194 by using the DSI interface to realize the display function of the electronic device. The GPIO interface may be configured by using software. The GPIO interface may be configured as a control signal or as a data signal. In some embodiments, the GPIO interface may be configured to connect the processor 110 to the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface may alternatively be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc. The SIM interface may be configured to communicate with a SIM card interface 195 to realize a function of sending data to the SIM card or a function of reading data from the SIM card. The USB interface 130 is an interface conforming to the USB standard specification, and specifically may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc.

[0434] It will be understood that the interface connection relationships between modules shown in this embodiment of the present invention are merely illustrative examples and are not limiting on the structure of the electronic device. In some other embodiments of the present application, the electronic device may alternatively use a different interface connection scheme or a combination of multiple interface connection schemes than the aforementioned embodiment.

[0435] The charging management module 140 is configured to receive a charging input from a charger. The power management module 141 is configured to connect the battery 142, the charging management module 140, and the processor 110.

[0436] The wireless communication function of the electronic device may be realized by using an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, a modem processor, a baseband processor, etc. The antenna 1 and the antenna 2 are configured to transmit and receive electromagnetic wave signals. Each antenna in the electronic device may be configured to cover one or more communication bands. Different antennas may be multiplexed to improve the utilization rate of the antennas. The mobile communication module 150 may provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc., applied to the electronic device. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 may receive electromagnetic waves by using the antenna 1, perform processing such as filtering and amplification on the received electromagnetic waves, and send the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 may further amplify the signal modulated by the modem processor and convert the signal into electromagnetic waves for radiation via the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be disposed in the same component as at least some of the modules of the processor 110.

[0437] The modem processor may include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into a medium / high-frequency signal. The demodulator is configured to demodulate a received electromagnetic signal into a low-frequency baseband signal. The demodulator then transmits the low-frequency baseband signal obtained by demodulation to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs an acoustic signal by using an audio device (such as but not limited to the speaker 170A and the receiver 170B) or displays an image or video by using the display 194. In some embodiments, the modem processor may be an independent device. In some other embodiments, the modem processor may be independent from the processor 110 and may be located in the same device as the mobile communication module 150 or other functional modules.

[0438] The wireless communication module 160 may provide solutions for wireless communication applied to electronic devices, including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. The wireless communication module 160 may be one or more components integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may further receive signals to be transmitted from the processor 110, perform frequency modulation and amplification on the signals to be transmitted, and convert the signals to be transmitted into electromagnetic waves for emission via the antenna 2.

[0439] In some embodiments, in the electronic device, antenna 1 is coupled to a mobile communication module 150 and antenna 2 is coupled to a wireless communication module 160, so that the electronic device can communicate with networks and other devices according to a wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, IR technology, etc. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0440] The electronic device realizes display functions by using a GPU, a display 194, an application processor, etc. The GPU is a microprocessor for image processing and is connected to the display 194 and the application processor. The GPU is configured to perform mathematical and geometric calculations and render graphics. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0441] The display 194 is configured to display images, videos, and the like. The display 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (QLED), and the like. In some embodiments, the electronic device may include one or N displays 194, where N is a positive integer greater than 1.

[0442] The electronic device may achieve the photographing function by using an ISP, a camera 193, a video codec, a GPU, a display 194, an application processor, etc., and obtain real-time video data.

[0443] The ISP is configured to process data fed back by the camera 193. For example, during shooting, the shutter is pressed, light is sent through the lens to the camera's light receiving element, the light signal is converted into an electrical signal, and the camera's light receiving element sends the electrical signal to the ISP for processing, which converts the electrical signal into a visible image. The ISP may further perform algorithmic optimization on image noise and brightness. The ISP may further optimize parameters such as exposure and color temperature for the shooting scenario. In some embodiments, the ISP may be located in the camera 193.

[0444] The camera 193 is configured to capture still images or videos. An optical image of an object is generated by using a lens and projected onto a light receiving element. The light receiving element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The light receiving element converts the optical signal into an electrical signal, and then sends the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into a standard image signal in a format such as RGB or YUV. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1.

[0445] The digital signal processor is configured to process the digital signal. In addition to processing the digital image signal, the digital signal processor may further process other digital signals. For example, when the electronic device selects a frequency, the digital signal processor is configured to perform a Fourier transform or the like on the frequency energy.

[0446] A video codec is configured to compress or decompress digital video. An electronic device may support one or more video codecs. In this manner, an electronic device may play or record video in multiple encoding formats, for example, moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, and MPEG 4.

[0447] The NPU is a neural network (NN) computing processor that can rapidly process input information and further perform continuous self-learning by referring to the structure of biological neural networks, for example, by referring to the transmission mode between human brain neurons. The NPU may be used to realize applications such as intelligent cognition of electronic devices, for example, image recognition, face recognition, speech recognition, and text understanding.

[0448] The internal memory 121 may include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs).

[0449] The random access memory may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, where, for example, a fifth generation DDR SDRAM is commonly referred to as DDR5 SDRAM), and the like.

[0450] Non-volatile memory may include disk storage and flash memory.

[0451] In an embodiment of the present application, the non-real-time video may be located in non-volatile memory.

[0452] According to the operating principle, the flash memory may include NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. Depending on the number of potential levels of storage units, the flash memory may include single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. According to the storage specification, the flash memory may include universal flash storage (UFS), embedded multi media card (eMMC), etc.

[0453] The random access memory may be directly read and written by the processor 110 and may be configured to store executable programs (e.g., machine instructions) for the operating system or other executing programs, and may be further configured to store user and application data.

[0454] The non-volatile memory may also store executable programs, user and application data, and the like, which may be previously loaded into the random access memory for direct reading and writing by the processor 110.

[0455] The external memory interface 120 may be configured to connect to an external non-volatile memory to expand the storage capacity of the electronic device. The external non-volatile memory communicates with the processor 110 by using the external memory interface 120 to realize data storage functions. The external non-volatile memory stores files such as music and videos.

[0456] The electronic device may achieve audio functions such as music playback and recording by using an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, an application processor, and the like.

[0457] The audio module 170 is configured to convert digital audio information into analog audio signals for output, and is also configured to convert analog audio input into digital audio signals. The speaker 170A, also referred to as a "loudspeaker", is configured to convert audio electrical signals into acoustic signals. The receiver 170B, also referred to as an "earphone", is configured to convert audio electrical signals into acoustic signals. The microphone 170C, also referred to as a "mike" or "mic", is configured to convert acoustic signals into electrical signals.

[0458] The headset jack 170D is configured to connect to a wired headset. The headset jack 170D may be a USB interface 130, or a 3.5 mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0459] The pressure sensor 180A may be configured to sense a pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A may be disposed on the display 194. The gyro sensor 180B may be configured to determine a motion gesture of the electronic device. The air pressure sensor 180C is configured to measure air pressure. The magnetic sensor 180D includes a Hall sensor. The acceleration sensor 180E may detect the acceleration value of the electronic device in all directions (usually three axes). The distance sensor 180F is configured to measure distance. The optical proximity sensor 180G may include a light detector, such as a light-emitting diode (LED) and a photodiode. The ambient light sensor 180L is configured to sense the brightness of the ambient light. The fingerprint sensor 180H is configured to capture a fingerprint. The electronic device may realize fingerprint-based unlocking, application lock access, fingerprint-based photography, fingerprint-based phone answering, and the like based on the captured fingerprint features. The temperature sensor 180J is configured to detect temperature.

[0460] The touch sensor 180K is also referred to as a touch panel. The touch sensor 180K may be disposed on the display 194. The touch sensor 180K and the display 194 form a touch screen, also referred to as a "touch screen." The touch sensor 180K is configured to detect touch operations performed on or near the touch sensor. The touch sensor may forward the detected touch operations to an application processor to determine a type of touch event. A visual output associated with the touch operation may be provided by using the display 194. In some other embodiments, the touch sensor 180K may alternatively be disposed at a location on the surface of the electronic device different from the location of the display 194.

[0461] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The electronic device may receive the button input and generate a button signal input related to user settings and function control of the electronic device. The motor 191 may generate a vibration prompt. The indicator 192 may be an indicator light and may be used to indicate a charging status or a power change, or may be used to indicate a message, a missed call, a notification, and the like. The SIM card interface 195 is configured to connect to a SIM card. The electronic device interacts with a network by using a SIM card to realize functions such as conversation and data communication.

[0462] FIG. 30 is a schematic diagram of an example of a software structure of an electronic device according to an embodiment of the present application.

[0463] In a layered architecture, software is divided into layers, each of which has a distinct role and task. The layers communicate with each other through software interfaces. In some embodiments, the system is divided into four layers, from top to bottom: application layer, application framework layer, system library layer, and kernel layer.

[0464] The application layer may include a series of application packages, which may include application programs (also referred to as applications) such as camera, gallery, calendar, phone, map, navigation, WLAN, Bluetooth, music, video, and messaging, as shown in FIG.

[0465] The application framework layer provides an application programming interface (API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0466] As shown in FIG. 30, the application framework layer may include a window manager service, a display manager service, a content provider, a view system, a phone manager, a resource manager, a notification manager, a Local Profile Assistant (LPA), etc.

[0467] The window manager service is responsible for starting, adding, and removing windows, determining which applications are displayed in windows, creating, destroying, and changing the properties of application layers, determining the presence or absence of a status bar, locking the screen, capturing the screen, etc.

[0468] The display manager service can get the number of display areas and the size of the display areas and is responsible for starting, adding, and removing display areas.

[0469] Content providers are configured to store and retrieve data and make the data accessible to applications. Data may include video, images, audio, calls made and received, browsing history and bookmarks, address books, etc.

[0470] The telephone manager is configured to provide communication functions of the electronic device, such as managing call status (including answering or rejecting).

[0471] The resource manager provides various resources for an application, such as localized strings, icons, pictures, layout files, video files, etc.

[0472] The notification manager may be configured to allow applications to display notification information in the status bar and to forward notification type messages. The information may disappear automatically after a short time without user interaction. For example, the notification manager is configured to notify of download completion, message reminders, etc. The notification manager may alternatively be notifications that appear in the form of graphs or scrollbar text in a status bar at the top of the system, notifications that appear on the screen, for example, notifications of applications running in the background, or in the form of a dialog interface. For example, the status bar displays text information, a prompt tone is emitted, the electronic device vibrates, and an indicator flashes.

[0473] The view system includes a visual control, such as a text display control or a picture display control. The view system may be configured to build an application. A display interface may include one or more views. For example, a display interface that includes an SMS notification icon may include a text display view and a picture display view.

[0474] The view system may further include UniRender, where UniRender may receive the render trees of one or more applications. UniRender may synchronize layer information, such as layer creation, layer destruction, and property changes, by using a window manager service. UniRender may synchronize information about the display area, such as screen size, from a display manager service.

[0475] Optionally, in some embodiments of the present application, the view system further includes a SurfaceFlinger. On an electronic device configured with a trusted list, when an application does not belong to the trusted list, after a UI thread of the application generates a render tree, the render thread of the application generates a bitmap, and then the bitmap is sent to SurfaceFlinger for layer compositing.

[0476] Optionally, in some embodiments of the present application, when an application in the trusted list and an application not in the trusted list are displayed in a display area, UniRender is responsible for generating bitmaps for the applications in the trusted list. After generating the bitmaps, UniRender forwards the bitmaps to SurfaceFlinger. SurfaceFlinger then performs layer compositing on the bitmaps and other applications not in the trusted list to generate the bitmaps that are sent for display.

[0477] The runtime includes a kernel library and a virtual machine. The runtime is responsible for scheduling and managing the operating system.

[0478] The kernel library includes two parts: the functions that need to be called using the java language and the kernel library.

[0479] The application layer and the application framework layer run on a virtual machine. The virtual machine executes the java files of the application layer and the application framework layer as binary files. The virtual machine is configured to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0480] The system library may include multiple functional modules, such as a surface manager, media libraries, and graphics processing libraries, including a 3D graphics processing library (e.g., OpenGL ES), a 2D graphics engine (e.g., SGL), and the like.

[0481] The surface manager is configured to manage the display subsystem and provide a blend of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0482] The media library supports multiple commonly used audio and video formats, as well as the playback and recording of still image files. The media library can support multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0483] The 3D graphics processing library is configured to realize 3D graphics drawing, image rendering, layer compositing, layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0484] The kernel layer is a layer between the hardware and the software, and includes at least a display driver, a camera driver, an audio driver, a sensor driver, and a virtual card driver.

[0485] In an embodiment of the present application, one or more applications in the application layer transfer the render tree generated by the UI thread of the application to the UniRender process of the view system. The UniRender process obtains window control information and display area information from the window manager service and the display manager service, and then merges the render tree of the application in the display area into a target render tree. After generating the target render tree, the UniRender process calls the layer processing library to execute the DrawOP operations in the drawing instruction list of the target render tree to generate a bitmap. The UniRender transfers the generated bitmap to the display driver for display.

[0486] Depending on the context, the term "when" used in the foregoing embodiments may be interpreted to mean "when," "after," "upon determining," or "upon detecting." Similarly, depending on the context, the phrases "upon determining" or "upon detecting (a stated condition or event)" may be interpreted as "upon determining," "upon determining," "upon detecting (a stated condition or event)," or "upon detecting (a stated condition or event)."

[0487] All or part of the above-mentioned embodiments may be realized by software, hardware, firmware, or any combination thereof. When software is used to realize the embodiments, all or part of the embodiments may be realized in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded into a computer and executed, the procedures or functions according to the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired (e.g., coaxial cable, optical fiber, digital subscriber line) manner or in a wireless (e.g., infrared, radio wave, microwave) manner. The computer-readable storage medium may be any available medium accessible by a computer, or may be a data storage device, such as a server or a data center, that integrates one or more available media. The usable medium may be a magnetic medium (eg, a floppy disk, hard disk, or magnetic tape), an optical medium (eg, a DVD), a semiconductor medium (eg, a solid state drive), or the like.

[0488] Those skilled in the art will understand that all or part of the steps of the method in the above embodiment may be performed by a computer program that instructs relevant hardware. The program may be stored in a computer-readable storage medium. When the program is executed, the steps of the above method embodiment may be included. The above storage medium includes any medium that can store program code, such as a ROM, a random access memory RAM, a magnetic disk, an optical disk, etc. [Explanation of symbols]

[0489] 110 Processor 120 External Memory Interface 121 Internal Memory 130 Universal Serial Bus Interface 140 Charging Management Module 141 Power Management Module 142 Battery 150 Mobile communication module 160 Wireless communication module 170 Audio Module 170A Speaker 170B Receiver 170C Microphone 170D Headset Jack 180 Sensor Module 180A PRESSURE SENSOR 180B Gyro sensor 180C Barometric Pressure Sensor 180D Magnetic Sensor 180E Acceleration Sensor 180F Distance Sensor 180G Optical Proximity Sensor 180H Fingerprint Sensor 180J Temperature Sensor 180K touch sensor 180L Ambient Light Sensor 180M Bone Conduction Sensor 190 Button 191 Motor 192 Indicator 193 Camera 194 Display 195 Subscriber Identity Module Card Interface

Claims

1. 1. An interface generation method for application to a first electronic device, the first electronic device determining that content displayed in a first display area includes an interface of a first process and an interface of a second process, the method comprising: generating, by the first process, a first render tree, the first render tree being used to render the interface of the first process; generating, by the second process, a second render tree, the second render tree being used to render the interface of the second process; generating, by a third process, a first target interface based on the first render tree and the second render tree, the first target interface including the interface of the first process and the interface of the second process, the first target interface being displayed in the first display area; The method of generating an interface includes:

2. The step of generating a first target interface based on the first render tree and the second render tree by a third process includes: generating, by the third process, a first target render tree based on the first render tree and the second render tree; generating, by the third process, the first target interface based on the first target render tree; The method of claim 1 , specifically comprising:

3. The step of generating a first target render tree based on the first render tree and the second render tree by the third process includes: creating a root render node by the third process as a root node of the first target render tree, and using the first render tree and the second render tree as child nodes of the root render node; The method of claim 2, specifically comprising:

4. removing, by the third process, a render node in the first target render tree based on a Z-order of the first render tree and a Z-order of the second render tree, the removed render node corresponding to a fully occluded view; 4. The method of claim 3, further comprising:

5. removing, by the third process, drawing operations in the first target render tree based on a Z-order of the first render tree and a Z-order of the second render tree, the removed drawing operations corresponding to fully occluded graphics.

4. The method of claim 3, further comprising:

6. a step of performing a merge or batch on a first drawing operation and a second drawing operation by the third process in a process of the third process generating the first target interface based on the first render tree and the second render tree, the first drawing operation belonging to the first render tree and the second drawing operation belonging to the second render tree; 6. The method of claim 1, further comprising:

7. determining, by the third process, off-screen rendering logic for the interface of the first process, the off-screen rendering logic including at least one of window rounding, color conversion, rotation, and scaling; adding, by the third process, off-screen rendering properties to rendering properties of the first render tree based on the off-screen rendering logic of the interface of the first process, the off-screen rendering properties including at least one of a rounding property, a color property, a rotation property, and a scaling property; Further comprising: the off-screen rendering properties have a one-to-one correspondence with the off-screen rendering logic, and the off-screen rendering properties are used to modify drawing operations in the process of the third process generating the first target interface based on the first render tree and the second render tree to realize the off-screen rendering logic; 7. The method according to any one of claims 1 to 6.

8. a reference coordinate system of the first render tree is a first coordinate system, a reference coordinate system corresponding to the first display area is a second coordinate system, and when the first coordinate system is different from the second coordinate system, determining, by the third process, a first parameter based on the first coordinate system and the second coordinate system, and adding the first parameter to the rendering properties of the first render tree; In the process of generating the first target interface based on the first render tree and the second render tree, modifying a reference coordinate system of the first drawing operation by the third process based on the first parameter, the first drawing operation belonging to the first render tree; 8. The method of claim 1, further comprising:

9. 13. An interface generation method for application to a second electronic device, the second electronic device determining that content displayed in a first display area includes an interface of a first process and an interface of a second process, the method comprising: receiving, by a third process operating on the second electronic device, a first render tree and a second render tree, the first render tree being generated by the first process operating on the first electronic device and the first render tree being used to render the interface of the first process, and the second render tree being generated by the second process operating on the second electronic device and the second render tree being used to render the interface of the second process; generating, by the third process, a target interface based on the first render tree and the second render tree, the target interface including the interface of the first process and the interface of the second process, the target interface being displayed in the first display area; The method of generating an interface includes:

10. An electronic device comprising one or more processors and a memory, 10. An electronic device, wherein the memory is coupled to the one or more processors, the memory being configured to store computer program code, the computer program code including computer instructions, and the one or more processors invoke the computer instructions, such that the electronic device performs the method of any one of claims 1 to 9.

11. A chip system for application to an electronic device, the chip system comprising one or more processors, the processors configured to invoke computer instructions such that the electronic device executes the method according to any one of claims 1 to 8.

12. A computer readable storage medium comprising instructions which, when executed on an electronic device, enable the electronic device to perform the method of any one of claims 1 or 9.

13. 10. A computer program product comprising computer readable instructions which, when executed by one or more processors, effectuate the method of any one of claims 1 to 9.

Citation Information

Patent Citations

  • Rendering method and device

    CN112346811A

  • Image processing device, image processing method and image processing program

    JP2006235926A

  • Magnification engine and interface for computers

    US20070013723A1

  • Method of rendering a user interface

    US20130097534A1