Interface generation method and electronic device

Abstract

The present application discloses an interface generating method and an electronic device, and relates to the field of electronic technology. According to the interface generating method provided in the present application, the render trees of one or more applications in one display area may be merged, and an interface including one or more application interfaces is generated through one rendering based on the merged target render tree, thereby reducing the rendering times, reducing the power consumption of the electronic device, and improving the user experience.

Description

Technical Field

[0001] Cross - reference to Related Applications This application is a continuation of International Application No. PCT / CN2022 / 133838, filed on November 23, 2022, claiming the priority of Chinese Patent Application No. 202111410513.6, filed on November 25, 2021. The disclosure of the aforementioned application is hereby incorporated by reference in its entirety into this specification.

[0002] This application relates to the field of electronic technology, and in particular, to an interface generation method and an electronic device.

Background Art

[0003] With the development of electronic technology, more and more electronic devices have been involved in users' daily lives. In addition, as parameters such as the resolution and size of the display of electronic devices increase, more content can be displayed on electronic devices.

[0004] However, before displaying the interface of an application, an electronic device needs to consume computing resources and storage resources to generate the interface of the application. This increases the power consumption of the electronic device. In addition, when there are multiple applications or multiple windows on the screen of an electronic device, the electronic device needs to consume more computing resources and storage resources to perform rendering for generating the interfaces of multiple applications or multiple windows.

Summary of the Invention

Means for Solving the Problems

[0005] Embodiments of this application provide an interface generation method and an electronic device. According to the interface generation method provided in this application, the render trees of one or more applications within a single display area may be merged, and an interface including one or more application interfaces is generated through a single rendering based on the merged target render tree, thereby reducing the number of renderings, reducing the power consumption of the electronic device, and improving the user experience.

[0006] According to a first aspect, an interface generation method provided in one embodiment of the present application is applied to a first electronic device. The first electronic device determines that the content displayed in a first display area includes an interface for a first process and an interface for a second process, the method comprising: a first process generating a first render tree, the first render tree being used to draw the interface for the first process; a second process generating a second render tree, the second render tree being used to draw the interface for the second process; and a third process generating a first target interface based on the first and second render trees, the first target interface including the interface for the first process and the interface for the second process, the first target interface being displayed in a first display area.

[0007] In the embodiments described above, an interface including one or more application interfaces is generated based on the render trees of one or more applications within the display area, thereby reducing the number of rendering cycles, lowering the power consumption of electronic devices, and improving the user experience.

[0008] Referring to some embodiments of the first aspect, in some embodiments, the third process generating a first target interface based on a first render tree and a second render tree specifically includes the third process generating a first target render tree based on a first render tree and a second render tree, and the third process generating a first target interface based on the first target render tree.

[0009] In the embodiments described above, the third process may merge the render trees of one or more applications into a single target render tree, and as a result, the interface may be generated based on the target render tree. The interface includes the interfaces of one or more applications. The interface including one or more applications can be generated through a single rendering, i.e., a single rendering process from the render tree to a bitmap, and as a result, the number of renderings is reduced, and the power consumption of the electronic device is reduced.

[0010] Referring to some embodiments of the first aspect, in some embodiments, the third process generating a first target render tree based on a first render tree and a second render tree specifically includes the third process creating a root render node as the root node of the first target render tree and using the first render tree and the second render tree as child nodes of the root render node.

[0011] In the embodiments described above, a new root node may be created, and the render trees of multiple applications may be mounted as child nodes of the root node to enable the creation of a target render tree.

[0012] Referring to some embodiments of the first aspect, in some embodiments, the method further includes a third process deleting render nodes in a first target render tree based on the Z-order of a first render tree and the Z-order of a second render tree, wherein the deleted render nodes correspond to a fully occluded view.

[0013] In the embodiments described above, the third process may optimize parameters in the target render tree to reduce overdraw, for example, by removing render nodes corresponding to completely occluded views within the interface.

[0014] Referring to some embodiments of the first aspect, in some embodiments, the method further includes a third process deleting drawing operations in a first target render tree based on the Z-order of a first render tree and the Z-order of a second render tree, wherein the deleted drawing operations correspond to graphics that are completely obscured.

[0015] In the embodiments described above, the third process may optimize parameters in the target render tree to reduce overdraw, for example, by removing render nodes corresponding to completely occluded graphics in the interface.

[0016] Referring to some embodiments of the first aspect, in some embodiments, the method further includes a process in which a third process generates a first target interface based on a first render tree and a second render tree, wherein the third process performs a merge or batch on a first drawing operation and a second drawing operation, the first drawing operation belonging to the first render tree and the second drawing operation belonging to the second render tree.

[0017] In the embodiments described above, the third process may merge or batch drawing operations in the target render tree in order to increase the interface generation rate, reduce frame freezes, reduce power consumption of electronic devices, and improve the user experience.

[0018] Referring to some embodiments of the first aspect, in some embodiments, the method further includes a third process determining off-screen rendering logic for the interface of the first process, the off-screen rendering logic comprising at least one of window rounding, color conversion, rotation, and scaling, and the third process adding off-screen rendering properties to the rendering properties of a first render tree based on the off-screen rendering logic for the interface of the first process, the off-screen rendering properties comprising at least one of rounding properties, color properties, rotation properties, and scaling properties. The off-screen rendering properties correspond one-to-one with the off-screen rendering logic, and the off-screen rendering properties are used to modify drawing operations in a process in which the third process generates a first target interface based on a first render tree and a second render tree in order to implement the off-screen rendering logic.

[0019] In the embodiment described above, off-screen rendering properties are added to the render tree, and as a result, in the process of performing drawing operations on a surface, off-screen rendered graphics can be drawn directly based on the off-screen rendering properties, eliminating the need to perform off-screen rendering on the entire bitmap and then perform layer compositing after drawing is complete. This avoids off-screen rendering, thereby reducing the number of rendering cycles and off-screen rendering overhead, lowering the power consumption of electronic devices, and improving the user experience.

[0020] Referring to some embodiments of the first aspect, in some embodiments, the reference coordinate system of a first render tree is a first coordinate system, the reference coordinate system corresponding to a first display area is a second coordinate system, and when the first coordinate system and the second coordinate system are different, the method further includes a third process determining first parameters based on the first and second coordinate systems and adding the first parameters to the rendering properties of the first render tree, and in a process in which the third process generates a first target interface based on the first and second render trees, the third process modifies the reference coordinate system of a first drawing operation based on the first parameters, wherein the first drawing operation belongs to the first render tree.

[0021] In the embodiment described above, when the reference coordinate system of the render tree generated by the application differs from the reference coordinate system in which the third process performs drawing operations within the render tree, a first parameter is added to the rendering properties, and then the reference coordinate system of the drawing operation is transformed based on the first parameter, in order to achieve the correct drawing operation.

[0022] According to a second aspect, an interface generation method provided in one embodiment of the present application includes a third process operating on a second electronic device receiving a first render tree and a second render tree, wherein the first render tree is generated by a first process operating on the first electronic device and is used to draw the interface of the first process, and the second render tree is generated by a second process operating on the second electronic device and is used to draw the interface of the second process, and the third process generates a target interface based on the first render tree and the second render tree, wherein the target interface includes the interface of the first process and the interface of the second process, and the target interface is displayed in a first display area.

[0023] In the embodiments described above, an interface including one or more application interfaces is generated based on the render trees of one or more applications within the display area, thereby reducing the number of rendering cycles, lowering the power consumption of electronic devices, and improving the user experience.

[0024] Referring to some embodiments of the second aspect, in some embodiments, the third process generating a first target interface based on a first render tree and a second render tree specifically includes the third process generating a first target render tree based on a first render tree and a second render tree, and the third process generating a first target interface based on the first target render tree.

[0025] In the foregoing embodiments, the third process may merge the render trees of one or more applications into one target render tree, and as a result, the interface may be generated based on the target render tree. The interface includes the interfaces of one or more applications. The interface including one or more applications can be generated through one rendering, that is, one rendering process from the render tree to the bitmap. As a result, it is clear that the number of renderings is reduced and the power consumption of the electronic device is reduced.

[0026] Referring to some embodiments of the second aspect, in some embodiments, the third process generating the first target render tree based on the first render tree and the second render tree specifically includes that the third process creates a root render node as the root node of the first target render tree and uses the first render tree and the second render tree as the child nodes of the root render node.

[0027] In the foregoing embodiments, a new root node may be created, and the render trees of multiple applications may be mounted as child nodes of the root node to realize the creation of the target render tree.

[0028] Referring to some embodiments of the second aspect, in some embodiments, the method further includes that the third process deletes the render nodes in the first target render tree based on the Z-order of the first render tree and the Z-order of the second render tree, and the deleted render nodes correspond to the fully occluded views.

[0029] In the foregoing embodiments, the third process may optimize the parameters in the target render tree to reduce overdraw, for example, delete the render nodes corresponding to the fully occluded views in the interface.

[0030] Referring to some embodiments of the second aspect, in some embodiments, the method further includes a third process deleting drawing operations in a first target render tree based on the Z-order of a first render tree and the Z-order of a second render tree, wherein the deleted drawing operations correspond to graphics that are completely obscured.

[0031] In the embodiments described above, the third process may optimize parameters in the target render tree to reduce overdraw, for example, by removing render nodes corresponding to completely occluded graphics in the interface.

[0032] Referring to some embodiments of the second aspect, in some embodiments, the method further includes a process in which a third process generates a first target interface based on a first render tree and a second render tree, wherein the third process performs a merge or batch on a first drawing operation and a second drawing operation, the first drawing operation belonging to the first render tree and the second drawing operation belonging to the second render tree.

[0033] In the embodiments described above, the third process may merge or batch drawing operations in the target render tree in order to increase the interface generation rate, reduce frame freezes, reduce power consumption of electronic devices, and improve the user experience.

[0034] Referring to some embodiments of the second aspect, in some embodiments, the method further includes a third process determining off-screen rendering logic for the interface of the first process, the off-screen rendering logic comprising at least one of window rounding, color conversion, rotation, and scaling, and the third process adding off-screen rendering properties to the rendering properties of a first render tree based on the off-screen rendering logic for the interface of the first process, the off-screen rendering properties comprising at least one of rounding properties, color properties, rotation properties, and scaling properties. The off-screen rendering properties correspond one-to-one with the off-screen rendering logic, and the off-screen rendering properties are used to modify drawing operations in a process in which the third process generates a first target interface based on a first render tree and a second render tree in order to implement the off-screen rendering logic.

[0035] In the embodiment described above, off-screen rendering properties are added to the render tree, and as a result, in the process of performing drawing operations on a surface, off-screen rendered graphics can be drawn directly based on the off-screen rendering properties, eliminating the need to perform off-screen rendering on the entire bitmap and then perform layer compositing after drawing is complete. This avoids off-screen rendering, thereby reducing the number of rendering cycles and off-screen rendering overhead, lowering the power consumption of electronic devices, and improving the user experience.

[0036] Referring to some embodiments of the second aspect, in some embodiments, the reference coordinate system of the first render tree is a first coordinate system, the reference coordinate system corresponding to the first display area is a second coordinate system, and when the first coordinate system and the second coordinate system are different, the method further includes a third process determining first parameters based on the first and second coordinate systems and adding the first parameters to the rendering properties of the first render tree, and in a process in which the third process generates a first target interface based on the first and second render trees, the third process modifies the reference coordinate system of a first drawing operation based on the first parameters, the first drawing operation belonging to the first render tree.

[0037] In the embodiment described above, when the reference coordinate system of the render tree generated by the application differs from the reference coordinate system in which the third process performs drawing operations within the render tree, a first parameter is added to the rendering properties, and then the reference coordinate system of the drawing operation is transformed based on the first parameter, in order to achieve the correct drawing operation.

[0038] According to a third aspect, one embodiment of the present application provides an electronic device. The electronic device includes one or more processors and memory, the memory being coupled to one or more processors, the memory being configured to store computer program code, the computer program code including computer instructions, the one or more processors calling computer instructions, and as a result the electronic device performs the following operations: a first process generating a first render tree, the first render tree being used to draw the interface of the first process; a second process generating a second render tree, the second render tree being used to draw the interface of the second process; and a third process generating a first target interface based on the first and second render trees, the first target interface including the interface of the first process and the interface of the second process, the first target interface being displayed in a first display area.

[0039] In the embodiments described above, an interface including one or more application interfaces is generated based on the render trees of one or more applications within the display area, thereby reducing the number of rendering cycles, lowering the power consumption of electronic devices, and improving the user experience.

[0040] Referring to some embodiments of the third aspect, in some embodiments, one or more processors are specifically configured to call computer instructions, so that the electronic device performs the following operations: a third process generates a first target render tree based on a first render tree and a second render tree, and a third process generates a first target interface based on the first target render tree.

[0041] Referring to some embodiments of the third aspect, in some embodiments, one or more processors are specifically configured to call computer instructions, and as a result, the electronic device performs the following operation: namely, a third process creates a root render node as the root node of a first target render tree, and uses the first render tree and the second render tree as child nodes of the root render node.

[0042] Referring to some embodiments of the third aspect, in some embodiments, one or more processors are further configured to call computer instructions, so that the electronic device performs the following operation: a third process deletes render nodes in a first target render tree based on the order of a first render tree and the Z-order of a second render tree, the deleted render nodes corresponding to a completely occluded view.

[0043] Referring to some embodiments of the third aspect, in some embodiments, one or more processors are further configured to call computer instructions, so that the electronic device performs the following operation: a third process deletes drawing operations in a first target render tree based on the order of a first render tree and the Z-order of a second render tree, the deleted drawing operations corresponding to a completely occluded graphic.

[0044] Referring to some embodiments of the third aspect, in some embodiments, one or more processors are further configured to call computer instructions, so that the electronic device performs the following operations: namely, in the process in which the third process generates a first target interface based on a first render tree and a second render tree, the third process performs a merge or batch on a first drawing operation and a second drawing operation, where the first drawing operation belongs to the first render tree and the second drawing operation belongs to the second render tree.

[0045] Referring to some embodiments of the third aspect, in some embodiments, one or more processors are further configured to call computer instructions, so that the electronic device performs the following operations: namely, the third process determines off-screen rendering logic for the interface of the first process, the off-screen rendering logic includes at least one of window rounding, color conversion, rotation, and scaling; and the third process adds off-screen rendering properties to the rendering properties of the first render tree based on the off-screen rendering logic for the interface of the first process, the off-screen rendering properties include at least one of rounding properties, color properties, rotation properties, and scaling properties. The off-screen rendering properties correspond one-to-one with the off-screen rendering logic, and the off-screen rendering properties are used to modify drawing operations in the process in which the third process generates a first target interface based on the first and second render trees in order to implement the off-screen rendering logic.

[0046] Referring to some embodiments of the third aspect, in some embodiments, one or more processors are further configured to call computer instructions, so that the electronic device performs the following operations: a third process determines first parameters based on a first coordinate system and a second coordinate system and adds the first parameters to the rendering properties of a first render tree; and in a process in which the third process generates a first target interface based on a first render tree and a second render tree, the third process modifies the reference coordinate system of a first drawing operation based on the first parameters, the first drawing operation belonging to a first render tree.

[0047] According to a fourth aspect, one embodiment of the present application provides an electronic device. The electronic device includes one or more processors and memory, the memory being coupled to one or more processors, the memory being configured to store computer program code, the computer program code including computer instructions, the one or more processors calling computer instructions, and as a result the electronic device performs the following operations: a third process operating on a second electronic device receives a first render tree and a second render tree, the first render tree being generated by a first process operating on the first electronic device and used to draw the interface of the first process, the second render tree being generated by a second process operating on the second electronic device and used to draw the interface of the second process, and the third process generating a target interface based on the first render tree and the second render tree, the target interface including the interface of the first process and the interface of the second process, the target interface being displayed in a first display area.

[0048] In the embodiments described above, an interface including one or more application interfaces is generated based on the render trees of one or more applications within the display area, thereby reducing the number of rendering cycles, lowering the power consumption of electronic devices, and improving the user experience.

[0049] Referring to some embodiments of the fourth aspect, in some embodiments, one or more processors are specifically configured to call computer instructions, so that the electronic device performs the following operations: a third process generates a first target render tree based on a first render tree and a second render tree, and a third process generates a first target interface based on the first target render tree.

[0050] Referring to some embodiments of the fourth aspect, in some embodiments, one or more processors are specifically configured to call computer instructions, and as a result, the electronic device performs the following operation: a third process creates a root render node as the root node of a first target render tree, and uses the first render tree and the second render tree as child nodes of the root render node.

[0051] Referring to some embodiments of the fourth aspect, in some embodiments, one or more processors are further configured to call computer instructions, so that the electronic device performs the following operation: a third process deletes render nodes in a first target render tree based on the order of the first render tree and the Z-order of the second render tree, the deleted render nodes corresponding to a completely occluded view.

[0052] Referring to some embodiments of the fourth aspect, in some embodiments, one or more processors are further configured to call computer instructions, so that the electronic device performs the following operation: a third process deletes drawing operations in a first target render tree based on the order of the first render tree and the Z-order of the second render tree, the deleted drawing operations corresponding to completely occluded graphics.

[0053] Referring to some embodiments of the fourth aspect, in some embodiments, one or more processors are further configured to call computer instructions, so that the electronic device performs the following operations: a process in which a third process generates a first target interface based on a first render tree and a second render tree, wherein the third process performs a merge or batch on a first drawing operation and a second drawing operation, the first drawing operation belonging to the first render tree and the second drawing operation belonging to the second render tree.

[0054] Referring to some embodiments of the fourth aspect, in some embodiments, one or more processors are further configured to call computer instructions, so that the electronic device performs the following operations: a third process determines off-screen rendering logic for the interface of a first process, the off-screen rendering logic includes at least one of window rounding, color conversion, rotation, and scaling; and the third process adds off-screen rendering properties to the rendering properties of a first render tree based on the off-screen rendering logic for the interface of the first process, the off-screen rendering properties include at least one of rounding properties, color properties, rotation properties, and scaling properties. The off-screen rendering properties correspond one-to-one with the off-screen rendering logic and are used to modify drawing operations in a process in which the third process generates a first target interface based on a first render tree and a second render tree in order to implement the off-screen rendering logic.

[0055] Referring to some embodiments of the fourth aspect, in some embodiments, one or more processors are further configured to call computer instructions, so that the electronic device performs the following operations: a third process determines first parameters based on a first coordinate system and a second coordinate system and adds the first parameters to the rendering properties of a first render tree; and in a process in which the third process generates a first target interface based on a first render tree and a second render tree, the third process modifies the reference coordinate system of a first drawing operation based on the first parameters, the first drawing operation belonging to a first render tree.

[0056] According to a fifth aspect, one embodiment of the present application provides a chip system. The chip system is applied to an electronic device, the chip system includes one or more processors, the processors are configured to call computer instructions, and as a result the electronic device performs a method according to any one of the first and second aspects and possible implementations of the first and second aspects.

[0057] According to a sixth aspect, one embodiment of the present application provides a computer program product including instructions. When the computer program product operates on an electronic device, the electronic device is enabled to perform any one of the methods of the first and second aspects and any possible implementations of the first and second aspects.

[0058] According to a seventh aspect, one embodiment of the present application provides a computer-readable storage medium containing instructions. When computer instructions are executed on an electronic device, the electronic device is enabled to perform any one of the methods of the first and second aspects and any possible implementations of the first and second aspects.

[0059] It will be understood that the electronic devices provided in the third and fourth embodiments, the chip systems provided in the fifth embodiment, the computer program products provided in the sixth embodiment, and the computer storage media provided in the seventh embodiment are all used to carry out the methods provided in the embodiments of this application. Therefore, for the beneficial effects that can be achieved, please refer to the beneficial effects of the corresponding methods described above. Further details are not described herein. [Brief explanation of the drawing]

[0060] [Figure 1A] This is a schematic diagram of an example of an interface for an electronic device according to one embodiment of this application. [Figure 1B] This is a schematic diagram of an example of a display interface for an electronic device according to one embodiment of this application. [Figure 2] This is a schematic diagram illustrating an example of an application generating a bitmap according to one embodiment of the present application. [Figure 3] This is a schematic diagram of an example of a render tree according to one embodiment of this application. [Figure 4] This is a schematic diagram of an example in which a bitmap is used as a layer for layer compositing, according to one embodiment of the present application. [Figure 5] This is a schematic diagram of an example of layer synthesis according to one embodiment of this application. [Figure 6] This is a schematic diagram of another example in which SurfaceFlinger performs layer synthesis according to one embodiment of the present application. [Figure 7A] This is a schematic diagram of another example of layer synthesis according to one embodiment of this application. [Figure 7B-1] This is a schematic diagram of another example of layer synthesis according to one embodiment of this application. [Figure 7B-2] This is a schematic diagram of another example of layer synthesis according to one embodiment of this application. [Figure 8] This is a schematic diagram of an example of a system architecture for an interface generation method according to one embodiment of this application. [Figure 9]This is a schematic diagram of an example of an interface generation method according to one embodiment of this application. [Figure 10] This is a schematic diagram of an example of a UniRender architecture according to one embodiment of this application. [Figure 11] This is a schematic diagram of an example in which a shared memory transfers a render tree according to one embodiment of the present application. [Figure 12A] This is a schematic diagram of an example of a render tree storage structure in shared memory according to one embodiment of this application. [Figure 12B] This is a schematic diagram of an example of a render tree storage structure in shared memory according to one embodiment of this application. [Figure 13] This is a schematic diagram of an example of a UniRender architecture according to one embodiment of this application. [Figure 14] This is a schematic diagram illustrating an example of how an application writes a render tree to shared memory, according to one embodiment of this application. [Figure 15] This is a schematic diagram of another example in which an application writes a render tree to shared memory, according to one embodiment of the present application. [Figure 16A-1] This is a schematic diagram of an example scenario of drawing logic that triggers off-screen rendering according to one embodiment of the present application. [Figure 16A-2] This is a schematic diagram of an example scenario of drawing logic that triggers off-screen rendering according to one embodiment of the present application. [Figure 16B] This is a schematic diagram of an example scenario of drawing logic that triggers off-screen rendering according to one embodiment of the present application. [Figure 16C] This is a schematic diagram of an example scenario of drawing logic that triggers off-screen rendering according to one embodiment of the present application. [Figure 17A] This is a schematic diagram illustrating an example of how UniRender transitions an off-screen rendering trigger command according to one embodiment of this application. [Figure 17B]This is a schematic diagram illustrating an example of how UniRender transitions an off-screen rendering trigger command according to one embodiment of this application. [Figure 18] This is a schematic diagram illustrating an example of how UniRender modifies render node properties in a render tree according to one embodiment of this application. [Figure 19] This is a schematic diagram illustrating an example of merging multiple render trees acquired after an off-screen rendering command has been transferred to a single target render tree, according to one embodiment of the present application. [Figure 20] This is a schematic diagram illustrating an example of merging multiple render trees acquired after an off-screen rendering command has been transferred to a single target render tree, according to one embodiment of the present application. [Figure 21] This is a schematic diagram of an example of setStaticMatrix() according to one embodiment of this application. [Figure 22A] This is a schematic diagram of an example of a multi-display area scenario according to one embodiment of this application. [Figure 22B] This is a schematic diagram of an example of a multi-display area scenario according to one embodiment of this application. [Figure 22C] This is a schematic diagram of an example of a multi-display area scenario according to one embodiment of this application. [Figure 23A-1] This is a schematic diagram of another example of a multi-display area scenario according to one embodiment of the present application. [Figure 23A-2] This is a schematic diagram of another example of a multi-display area scenario according to one embodiment of the present application. [Figure 23B] This is a schematic diagram of another example of a multi-display area scenario according to one embodiment of the present application. [Figure 23C] This is a schematic diagram of another example of a multi-display area scenario according to one embodiment of the present application. [Figure 24] This is a schematic diagram of another example of an interface generation method according to one embodiment of this application. [Figure 25] This is a schematic diagram illustrating an example in which a UniRender process performs frequency division or frequency multiplication on a vertical synchronization signal, according to one embodiment of the present application. [Figure 26A] This is a schematic diagram of an example of a data flow when an electronic device implements an interface generation method according to one embodiment of this application. [Figure 26B] This is a schematic diagram of an example of a data flow when an electronic device implements an interface generation method according to one embodiment of this application. [Figure 27] This is a schematic diagram of another example of an interface generation method according to one embodiment of this application. [Figure 28A] This is a schematic diagram illustrating an example of merging a render tree into a target render tree according to one embodiment of this application. [Figure 28B] This is a schematic diagram illustrating an example of merging a render tree into a target render tree according to one embodiment of this application. [Figure 29] This is a schematic diagram of the hardware structure of an electronic device according to one embodiment of this application. [Figure 30] This is a schematic diagram of an example of the software structure of an electronic device according to one embodiment of this application. [Modes for carrying out the invention]

[0061] The terms used in the following embodiments are intended solely to describe specific embodiments and are not intended to limit this application. The singular expressions “a,” “one,” “that,” “the aforementioned,” “this,” and “that one” used in this specification are intended to include plural expressions unless explicitly specified otherwise in the context. It should be further understood that the terms “and / or” used in this application indicate and include any possible combination of one or more described items.

[0062] The terms “first” and “second” used below are for illustrative purposes only and should not be understood as indicating or implying relative importance, or implicitly referring to any given technical feature. Therefore, a feature limited by “first” or “second” may explicitly or implicitly include one or more such features. In the description of embodiments of this application, unless otherwise specified, “multiple” means two or more.

[0063] In the following embodiments of this application, the term "user interface (UI)" refers to a medium interface for interaction and information exchange between an application or operating system and a user, enabling the conversion of information between its internal format and a format acceptable to the user. A user interface is source code compiled in a specific computer language, such as Java or an extensible markup language (XML). The interface source code is parsed and rendered on an electronic device and ultimately presented as user-recognizable content. A common form of presentation of a user interface is a graphic user interface (GUI), which refers to a user interface displayed graphically and related to computer operation. A GUI may be a visible interface element displayed on the display of an electronic device, such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, or widgets.

[0064] For the sake of clarity, the following will first explain the relevant terms and concepts in the embodiments of this application. The terms used in the embodiments of the present invention are used solely to describe specific embodiments of the invention and are not intended to limit the invention.

[0065] The interface is used as a medium interface for interaction and information exchange between the application and the user. Each time a vertical synchronization signal arrives, the electronic device must generate the application's interface for the foreground application. The frequency of the vertical synchronization signal is related to the refresh rate of the electronic device's screen. For example, the frequency of the vertical synchronization signal is the same as the refresh rate of the electronic device's screen.

[0066] Specifically, before an electronic device can refresh the content displayed on its screen, it must generate an application interface for the foreground application each time. In this case, when the screen is refreshed, the newly generated application interface is displayed to the user.

[0067] An interface displayed on an electronic device by an electronic device may include the interfaces of one or more applications; that is, the electronic device needs to generate and combine the interfaces of one or more applications in order to obtain a composite interface displayed on the screen.

[0068] When an electronic device generates an application interface, the application needs to render and generate a bitmap and transfer the application's bitmap to a SurfaceFlinger. That is, the application acts as a producer, drawing and generating the bitmap and storing it in a buffer queue provided by the SurfaceFlinger. The SurfaceFlinger acts as a consumer, continuously retrieving the bitmap generated by the application from the BufferQueue. The bitmap resides on a surface generated by the application. The surface is placed in the BufferQueue.

[0069] After SurfaceFlinger obtains the bitmap of the visible application, SurfaceFlinger and the Hardware Composer (HWC) determine the layer compositing mode in which the bitmap will be used as a layer. SurfaceFlinger may also determine the visible application by using the Window Manager Service (WMS).

[0070] After obtaining off-screen rendering logic such as rounding, rotation, color conversion, and scaling that acts on the application's windows from the window manager service, SurfaceFlinger may copy the application's bitmaps to an off-screen buffer for rendering and obtain the bitmaps for layer compositing via off-screen rendering.

[0071] After the application bitmap is composited, SurfaceFlinger / HWC places the composited bitmap (which may also be referred to as a layer on SurfaceFlinger) into a frame buffer for transfer to the Display Subsystem (DSS). After retrieving the composited bitmap, the DSS may display it on the screen. The frame buffer may also be an on-screen buffer.

[0072] (1) First, we will explain separately the process of generating bitmaps by an application, the process of compositing bitmaps by SurfaceFlinger / HWC, and an example of off-screen rendering.

[0073] (1.1) The process by which an application generates a bitmap Figures 1A and 1B are schematic diagrams of an example of an interface for an electronic device according to one embodiment of this application.

[0074] As shown in Figure 1A, the screen of the electronic device displays the interfaces of Application 1 and Application 2. The interface of Application 1 may also be called the status bar. In Figure 1A, Application 1 may be an operating system, and Application 2 may be a music application.

[0075] Before displaying the interface shown in Figure 1A, the electronic device needs to generate the application's bitmap separately and independently. The status bar interface is maintained by the operating system; that is, the application that generates the status bar bitmap can be considered the operating system itself. Specifically, the operating system generates bitmap 1 and then transfers bitmap 1 to SurfaceFlinger, and the music application generates bitmap 2 and then transfers bitmap 2 to SurfaceFlinger. Bitmap 1 carries the status bar image information, and bitmap 2 carries the music application's image information.

[0076] After receiving bitmap 1 and bitmap 2, SurfaceFlinger / HWC uses bitmap 1 and bitmap 2 as layers to perform layer compositing. For details on layer compositing, please refer to (1.2) the description of the process of compositing bitmaps by SurfaceFlinger / HWC, as further details are not provided herein.

[0077] As shown in Figure 1B, the screen of the electronic device displays the interfaces of Application 1, Application 2, and Application 3. The interface of Application 1 may also be called the status bar. In Figure 1B, Application 1 may be an operating system, Application 2 may be a browser application, and Application 3 may be an SMS application.

[0078] Before displaying the interface shown in Figure 1B, the electronic device also needs to generate the application's bitmap separately and independently.

[0079] Specifically, after generating bitmap 3, the operating system transfers bitmap 3 to SurfaceFlinger; after generating bitmap 4, the SMS application transfers bitmap 4 to SurfaceFlinger; and after generating bitmap 5, the news application transfers bitmap 5 to SurfaceFlinger. Bitmap 3 carries the image information for the status bar, bitmap 4 carries the image information for the SMS application, and bitmap 5 carries the image information for the news application.

[0080] After receiving bitmaps 3, 4, and 5, SurfaceFlinger / HWC uses bitmaps 3, 4, and 5 as layers to perform layer compositing. For details on layer compositing, please refer to section (1.2) describing the process of compositing bitmaps by SurfaceFlinger / HWC, as further details are not provided herein.

[0081] Figure 2 shows the process by which an application generates a bitmap.

[0082] Figure 2 is a schematic diagram of an example of an application generating a bitmap according to one embodiment of the present application.

[0083] As shown in Figure 2, after receiving the vertical synchronization signal (Vsync-APP), the application begins generating the bitmap. The specific steps include the following three steps:

[0084] [1] The view hierarchy is disabled. The main thread (UI thread) traverses the application's views, determines and stores the drawing operations for each view, and records the views and their associated drawing operations (Draw Operation Struct, DrawOP) in the displaylist of the render node in the render tree.

[0085] A view is a fundamental element that forms an application interface, and one control on the interface may correspond to one or more views.

[0086] A drawing operation is a data structure used for drawing graphics, such as drawing lines, widths, rectangles, and text. Drawing operations are translated into API calls to image processing libraries on the render node, such as OpenGL API calls. For example, DrawLineOp is a data structure that contains information about drawing data, such as the length and width of a line.

[0087] The drawing command list may be a buffer that records all drawing operations or identifiers of all drawing operations contained in one frame of the application's interface, such as addresses and sequence numbers. When an application has multiple windows or is displayed in different display areas, multiple render trees must be generated independently. Multiple drawing command lists corresponding to different windows or display areas are generated independently.

[0088] In the embodiments of this application, the display area may be a screen, a virtual display, or the like. The virtual display may be an area used by an electronic device to carry content displayed on the screen during screen recording.

[0089] The render tree is generated by the UI thread and used to generate the data structures for the application interface. The render tree may contain multiple render nodes, each containing rendering properties and a list of drawing instructions. The render tree records all the information necessary to generate a single frame interface for the application.

[0090] [2] The UI thread transfers / synchronizes the render tree to the render thread, which is located on the stack of the process corresponding to the application and does not have to be contiguous in terms of physical addresses.

[0091] [3] The render thread first acquires a HardwareCanvas and performs the drawing operations in the render tree on the HardwareCanvas to generate a bitmap. The HardwareCanvas resides on a surface held by the application, which carries bitmaps or data in other formats used to store image information.

[0092] [1] can be thought of as a construction phase, primarily responsible for determining properties such as the size, position, and transparency of each view in the application. For example, a drawLine in a view may be packaged into a DrawLineOp during construction, which contains drawing data such as the length and width of the line, and may further contain interface calls to the drawLineOP of the underlying graphics processing library, which is used to call the underlying graphics library to generate a bitmap in the rendering phase.

[0093] Similarly, [3] may be considered a rendering phase primarily responsible for traversing the render nodes of the render tree, performing drawing operations for each render node, and generating bitmaps on the hardware canvas. In this process, the render thread generates bitmaps by calling the underlying graphics processing library, such as OpenGL, in order to invoke the GPU to perform rendering.

[0094] Figure 3 is a schematic diagram of an example of a render tree according to one embodiment of this application.

[0095] An interface that needs to be displayed by an application contains multiple nested views, and different views have parent-child relationships. Therefore, the parent-child relationships between render nodes in the render tree generated by traversing views are the same as the parent-child relationships between views. In other words, the parent-child relationships between views determine the nesting relationships between different render nodes. Then, the render thread can correctly render the application's interface when generating bitmaps according to the render tree.

[0096] A single view may correspond to one or more render nodes. The root view (DecorView) corresponds to the root render node (RootRenderNode). In other words, nesting relationships between render nodes correspond to parent-child relationships between views. A render node further includes rendering properties, which are used to determine the position, size, transparency, etc., of the view corresponding to the render node on the surface when a bitmap is generated through rendering.

[0097] For example, the structure of the application interface is as follows: The application's PhoneWindow carries the root view, the root view's child views are View 1 and View 2, and View 2's child view is View 3. In this case, the structure of the render tree generated by the application's UI thread is as follows: The root render node corresponding to PhoneWindow is the root node of the render tree, the root render node's child node is render node 0 corresponding to the root view, the child nodes of render node 0 are render node 1 corresponding to View 1 and render node 2 corresponding to View 2, and the child node of render node 2 is render node 3 corresponding to View 3.

[0098] The correspondence between a view and a render node means that the render node includes all drawing operations in the corresponding view.

[0099] After receiving the render tree synchronized by the UI thread, the render thread calls the OpenGL interface to render the bitmap onto the application's surface and sends the surface to SurfaceFlinger for compositing and display.

[0100] Please note that the CPU computing resources need to be occupied during the build phase, and the GPU resources need to be occupied during the rendering phase.

[0101] Note that if hardware acceleration is not enabled, the application will perform all operations in the build and rendering phases by using the UI thread and will not need to perform packaging to the render node. After the application's view and view drawing operations have been traversed, the application requests anonymous shared memory from SurfaceFlinger and directly calls the underlying graphics library in memory to generate the bitmap.

[0102] Note that the content shown in Figure 2 represents the process by which an application generates a bitmap when hardware acceleration is enabled. When hardware acceleration is not enabled on the electronic device, the application generates the bitmap via software drawing. Software drawing includes the following:

[0103] [1] The view hierarchy is disabled. The UI thread traverses the application's views and records the drawing operations for each view. [2] The UI thread obtains a software canvas for drawing via an interface, such as Surface.lockCanvas(), and performs drawing on the canvas based on the stored list of drawing operations, generating a bitmap. The software canvas resides on a surface created by the application.

[0104] Surfaces held by an application are assigned to the application by SurfaceFlinger via binder communication. The number of surfaces held by an application may be the same as the number of the application's current windows (PhoneWindow).

[0105] After the process of generating bitmaps by an application is explained, the process of compositing bitmaps is explained using an example.

[0106] (1.2) Process of compositing bitmaps using SurfaceFlinger / HWC SurfaceFlinger is a system service on an electronic device used to assign surfaces to applications and to use bitmaps on one or more surfaces as layers for layer compositing. HWC is a functional module of the Hardware Abstraction Layer (HAL), which is responsible for compositing and display in electronic devices. It provides an interface for the higher-layer SurfaceFlinger and calls the functions of the underlying hardware (e.g., display drivers, excluding the GPU) to perform layer compositing.

[0107] Figure 4 is a schematic diagram of an example in which a bitmap is used as a layer for layer compositing, according to one embodiment of the present application.

[0108] As shown in Figure 4, the application's render thread generates bitmaps on the surface assigned to the application by SurfaceFlinger. SurfaceFlinger maintains the surfaces of one or more applications by using the BufferQueue mechanism. In other words, SurfaceFlinger can retrieve bitmaps from different applications.

[0109] After obtaining bitmaps from one or more applications, SurfaceFlinger invokes the GPU to combine multiple bitmaps into a single bitmap (this bitmap compositing is also called layer compositing). The compositing may also be referred to as client compositing or GLES compositing.

[0110] After obtaining the application's bitmap, SurfaceFlinger may invoke the underlying hardware (excluding the GPU) via HWC for synthesis. This synthesis mode is also known as device synthesis.

[0111] Client compositing requires calling the GPU, and may perform compositing of multiple layers using a pixel-by-pixel processing method such as linear deepening.

[0112] Device compositing may combine a limited number of layers, but it does not support compositing using a multi-pixel processing method. If Device compositing combines multiple layers that do not intersect on the screen, layer compositing does not need to be performed. Instead, data on different surfaces is read and displayed when different locations on the screen are shown.

[0113] As shown in Figure 4, after obtaining bitmap 1 from application 1, bitmap 2 from application 2, ..., and bitmap N from application N, SurfaceFlinger may perform client synthesis on bitmap 1 and bitmap 2 to generate bitmap 11. SurfaceFlinger then performs device synthesis on bitmap 11 and bitmap N by using HWC to invoke the underlying hardware.

[0114] During device synthesis, bitmap 11 and bitmap N are temporarily stored. When display on the electronic device screen is required, the corresponding pixels are retrieved from bitmap 11 / bitmap N for display on the screen. For example, in the interface shown in Figure 1B, bitmap 11 is a bitmap synthesized by the SMS application and the news application, and bitmap N is the bitmap for the status bar.

[0115] For the underlying hardware compatible with SurfaceFlinger or HWC, each bitmap corresponds to one layer.

[0116] The layer blending mode may be determined by the underlying hardware compatible with HWC, or it may be determined by SurfaceFlinger.

[0117] For example, after acquiring a bitmap, SurfaceFlinger transfers the configured layers to the underlying hardware using HWC, and the underlying hardware determines which layers should undergo Client synthesis and which layers should undergo Device synthesis. The underlying hardware marks the synthesis mode of the layers in the layer list and returns the synthesis modes of the different layers to SurfaceFlinger. SurfaceFlinger synthesizes the layers marked for GPU synthesis and stores the synthesis results in a buffer. SurfaceFlinger transfers the buffers marked for Overlay synthesis mode and other layers to the underlying hardware via HWC, where the underlying hardware then performs layer synthesis.

[0118] In another example, SurfaceFlinger, in its layer compositing process, directly marks layers related to off-screen rendering trigger logic, such as window animations, in GPU compositing. Off-screen rendering trigger logic further includes logic that cannot be handled by the underlying hardware corresponding to HWC, such as rounding, scaling, rotation, and color conversion.

[0119] In addition to the bitmap generated by the application render thread, the surface assigned to the application by the application's SurfaceFlinger further includes window control information such as the Z-order of layers obtained from the window manager service. Therefore, SurfaceFlinger may obtain the layer's window control information from the surface to determine whether the layer requires GPU synthesis. The Z-order of layers determines the order of the layers on the Z-axis, which is perpendicular to the screen and is used to calculate the hierarchy between different layers.

[0120] Figure 5 is a schematic diagram of an example of layer synthesis according to one embodiment of this application.

[0121] As shown in Figure 5, after obtaining the application's bitmap, SurfaceFlinger writes each bitmap as a layer to a layer list. SurfaceFlinger then transfers the layer list to the underlying hardware via the HWC. The underlying hardware determines the blending mode for the different layers based on the capabilities of the underlying hardware and returns the result to SurfaceFlinger via the HWC.

[0122] After obtaining the results returned by the HWC, SurfaceFlinger may obtain the blending mode for each layer in the layer list. For layers marked as GPU blending, SurfaceFlinger blends the layers and transfers the blended layers and layers marked as overlay blending to the underlying hardware via the HWC. The underlying hardware corresponding to the HWC is then used to blend the layers.

[0123] Figure 6 is a schematic diagram of another example in which SurfaceFlinger performs layer synthesis according to one embodiment of the present application.

[0124] After acquiring multiple surfaces, SurfaceFlinger may determine the compositing mode in which bitmaps are used as layers on each surface. After acquiring multiple layers and determining that the layer compositing mode is GPU, SurfaceFlinger may perform Client compositing. Layer compositing modes may include Mode.CLEAR (displays the topmost layer in Z order), Mode.SRC_OVER (displays layers sequentially according to Z order), Mode.DST_IN (displays the non-intersecting portion between the topmost layer in Z order and the layers below it), etc.

[0125] For example, as shown in Figure 6, when the layer blending mode is Mode.SRC_OVER, SurfaceFlinger blends the contents of the three layers, determines the hierarchical relationship between different layers according to the Z-order of the layers, and sequentially stacks the layers on top of a single layer.

[0126] The Z-order of Layer 1 is a, the Z-order of Layer 2 is a+1, and the Z-order of Layer 3 is a+2. After SurfaceFlinger combines the contents of the three layers, the contents of Layers 1 and 2 are completely occluded, and only the contents of Layer 3 are visible.

[0127] Complete occlusion means that layers with a higher Z-order are occluded, preventing views of layers with a lower Z-order from being displayed.

[0128] Layers on which GPU compositing is performed trigger off-screen rendering. Layer compositing is performed via off-screen rendering. The following sections explain concepts related to off-screen rendering.

[0129] (1.3) Off-screen rendering If SurfaceFlinger determines that any multiple layers need to be composited using the GPU, off-screen rendering must be enabled to perform the layer compositing.

[0130] Off-screen rendering requests an off-screen buffer for SurfaceFlinger and invokes the GPU for image processing within the off-screen buffer. The off-screen buffer is memory outside the current screen buffer, and multiple layers are composited within it.

[0131] Off-screen rendering may include the following steps:

[0132] [1] Bitmaps in layers that SurfaceFlinger determines to be GPU synthesized must be converted to textures, which then need to be uploaded to GPU memory (i.e., off-screen buffer) or mapped to OpenGL textures via shared memory. OpenGL then combines the textures (combining textures includes combining contexts).

[0133] [2] Textures are rendered according to instructions related to the window animation corresponding to the layer. In addition, textures from multiple applications are merged. Layer blending is performed pixel by pixel according to the layer blending mode.

[0134] [3] SurfaceFlinger retrieves textures rendered from GPU memory or directly from shared memory.

[0135] It is clear that off-screen rendering triggers a context switch, and this context switch increases further performance overhead.

[0136] The following sections describe in detail the compositing process in which bitmaps are used as layers, referring to (1.1) the process of generating bitmaps by the application, (1.2) the process of compositing bitmaps by SurfaceFlinger / HWC, and (1.3) the content of off-screen rendering.

[0137] Figures 7A, 7B-1, and 7B-2 are schematic diagrams of other examples of layer synthesis according to one embodiment of this application.

[0138] As shown in Figure 7A, UI threads for different applications independently generate their own render trees and transfer these render trees to their respective rendering threads to generate bitmaps.

[0139] The render thread first generates bitmaps through rendering based on the properties of the render nodes in the render tree and the list of drawing instructions, and then performs compositing by using the bitmaps as layers. When the layer compositing mode is GPU compositing, SurfaceFlinger composites multiple layers into a single layer via off-screen rendering.

[0140] For example, Application 1 generates Render Tree 1 and then generates Bitmap 1 based on Render Tree 1, while Application 2 generates Render Tree 2 and then generates Bitmap 2 based on Render Tree 2. After receiving Bitmap 1 and Bitmap 2, SurfaceFlinger performs off-screen rendering on Bitmap 1 and Bitmap 2 to generate Bitmap 5. In the off-screen rendering process, Bitmap 1 is first copied to an off-screen buffer, Bitmap 1 is converted to Bitmap 3 in the off-screen buffer based on window animation information, similarly Bitmap 2 is converted to Bitmap 4, and finally Bitmap 3 and Bitmap 4 are superimposed based on the layer blending mode to generate Bitmap 5.

[0141] As shown in Figures 7B-1 and 7B-2, to implement the interface shown in Figure 1B, after receiving the application's bitmap, SurfaceFlinger first performs off-screen rendering on the application's bitmap related to the window animation bitmap in order to carry out the conversion, and then copies the converted layer to an on-screen buffer for transmission and display.

[0142] For example, if a bitmap in Application 3 needs to be scaled down, the bitmap in Application 3 cannot be directly copied to the on-screen buffer for conversion. This could affect bitmaps in other applications. Instead, the bitmap in Application 3 needs to be copied to an off-screen buffer for a separate conversion, and the conversion result is copied back to the on-screen buffer.

[0143] It is clear that the correct interface can only be generated when the bitmaps generated by each application are first modified and then superimposed based on the layer blending mode.

[0144] Since the process calling the GPU changes continuously from when the application's main thread generates the render tree until SurfaceFlinger performs layer compositing (from Application 1 to Application 2, ..., Application N, and SurfaceFlinger), the GPU needs to be started at least N+1 times separately, where N is the number of applications.

[0145] While different applications build and render their bitmaps independently, it will be understood that layers with lower Z-orders can be completely obscured by layers with higher Z-orders during layer compositing. This inevitably leads to overdraw of the bitmaps of different applications. For example, the overlapping portion of the three graphics in Figure 6 is overdrawn, i.e., layers 1 and 2 are overdrawn. The composite interface is unaffected by layers 1 and 2. In addition, off-screen rendering requires new off-screen buffer allocation and context switching, increasing the memory and computing overhead of electronic devices.

[0146] (2) Next, the interface generation method provided in the embodiment of this application will be described below.

[0147] According to the interface generation method provided in the embodiments of this application, first, render trees of one or more applications are obtained by using a UniRender process, and the one or more render trees are regrouped to generate a target render tree. Next, the UniRender process performs rendering based on the target render tree to directly obtain bitmaps carrying image information of the interface of one or more applications without performing layer compositing.

[0148] First, in the interface generation method provided in the embodiments of this application, the target render trees of one or more applications are merged into a single target render tree. In the process of generating the target render tree, the UniRender process determines the off-screen rendering logic for each layer and adds or modifies the properties of the corresponding render nodes in the target render tree according to the off-screen rendering logic, so that the UniRender process directly generates bitmaps without performing off-screen rendering.

[0149] Next, in the interface generation method provided in the embodiments of this application, the target render trees of one or more applications are merged into a single target render tree, eliminating the need to initially generate multiple bitmaps as layers for compositing. In the process in which the UniRender process performs rendering based on the target render tree to generate bitmaps, the Z-order of the layers is used as the Z-order of the render tree, and render nodes corresponding to views that are not displayed or do not affect the display may be removed, thereby avoiding overdraw.

[0150] Finally, according to the interface generation method provided in the embodiments of this application, the application does not need to generate render threads, but the UniRender process contributes to improving the rendering speed of the interface by performing integrated rendering.

[0151] (2.1) System architecture of the interface generation method provided in the embodiments of this application

[0152] Figure 8 is a schematic diagram of an example of a system architecture of an interface generation method according to one embodiment of this application.

[0153] When the application interface is updated, the application may request a vertical synchronization signal (Vsync-APP) from the UniRender process (SurfaceFlinger is not shown in Figure 8). If SurfaceFlinger is present, the application may request a vertical synchronization signal (Vsync-APP) from SurfaceFlinger.

[0154] The vertical synchronization signal (Vsync-APP) for the UniRender process may be generated by SurfaceFlinger, or directly by the underlying hardware (such as a screen) that supports HWC, or it may be generated by the UniRender process by starting a thread that is periodically woken up. If SurfaceFlinger is present, Vsync-APP may originate from SurfaceFlinger.

[0155] After the application receives the vertical synchronization signal (Vsync-APP), it generates a render tree and transfers the render tree to the UniRender process.

[0156] After receiving the vertical synchronization signal (Vsync-UR), the UniRender process merges one or more render trees to generate a target render tree. The UniRender process then uses the render engine to traverse and execute the draw operations within the draw instruction list of each render node in the render tree, generating bitmaps that carry the image information of one or more render trees. These bitmaps may reside in an on-screen buffer.

[0157] The difference between Vsync-UR and Vsync-APP is Vsync-Offset, which may be determined by the UniRender process. If SurfaceFlinger is present, Vsync-Offset may also be determined by SurfaceFlinger.

[0158] After generating the bitmap, the UniRender process transfers the bitmap to the display subsystem via HWC for display.

[0159] After describing the system architecture of the interface generation method in this application, the method procedure of the interface generation method provided in the embodiments of this application will be described using examples.

[0160] (2.2) Method and procedure of interface generation method provided in embodiments of this application Referring to Figure 9, an example of an interface generation method provided in one embodiment of this application will be described below.

[0161] Figure 9 is a schematic diagram of an example of an interface generation method according to one embodiment of this application.

[0162] As shown in Figure 9, the interface generation method provided in this embodiment of the present application includes the following steps:

[0163] S901: Constructs and generates a render tree after receiving a vertical synchronization signal.

[0164] When an interface needs to be updated, the application may request a vertical synchronization signal (Vsync-APP) from the UniRender process. After receiving the vertical synchronization signal, the application executes the measure(), layout(), and draw() methods on the UI thread. When executing the draw() method, the UI thread traverses the application's views, determines the drawing commands required to render each view, and sequentially records the drawing commands in the draw command list of the render node corresponding to the view.

[0165] An interface that needs to be displayed by an application includes multiple nested views, and the drawing command list corresponding to the root view (DecorView) includes the drawing command list entries of the root view's child views; that is, the nesting relationships between drawing command lists are the same as the nesting relationships between views. Therefore, the nesting relationships between render nodes are the same as the nesting relationships between views. For definitions of the related concepts of render trees and render nodes, please refer to the above explanation corresponding to Figure 3.

[0166] Figure 10 is a schematic diagram of an example of constructing a render tree according to one embodiment of this application.

[0167] After performing measurement, layout, and drawing, the application's UI thread may obtain a parent-child structure of multiple views of the interface to be updated, and in the process of traversing the views, may determine what should be displayed in each view and the interface calls necessary to generate that content, such as drawCircle or drawLine.

[0168] The application packages drawing interface calls, such as drawCircle and drawLine, into corresponding DrawOps, such as DrawCircleOp and DrawLineOp, and stores these DrawOps in a list of drawing instructions. DrawLineOps are graphics drawing interface calls to the underlying graphics library (such as OpenGL), which are further translated into graphics drawing instructions to invoke the GPU.

[0169] As shown in Figure 10, the view structure of the application interface includes a root view, the root view's child views are View 1 and View 2, and View 3 is the child view of View 2. View 2 includes rectangles and circles, and the drawing operations in the drawing command list of render node 2 corresponding to View 2 include operations to draw rectangles, operations to draw circles, and so on.

[0170] When executing the draw() method, the application's UI thread may start with the root view, traverse all views based on their parent-child relationships, determine the drawing operations for each view, and package those drawing operations into DrawOps. After generating the list of drawing instructions, the application's UI thread further packages that list of drawing instructions into a render tree.

[0171] A render node in the render tree contains a list of drawing instructions and rendering properties. Rendering properties are used to determine properties such as the position, size, and transparency of the view that should be rendered by the render node on the surface, while the list of drawing instructions is used to determine the content of the view, such as lines, rectangles, or circles, that should be rendered by the render node.

[0172] A surface is claimed by the application. The application determines the size of the surface. If a SurfaceFlinger exists, the surface may be claimed by the application from the SurfaceFlinger. If a SurfaceFlinger does not exist, the surface may be claimed from the UniRender process. The SurfaceFlinger is not required to assign the surface to the application.

[0173] If necessary, in some embodiments of this application, after determining the screen refresh rate of each display area, the UniRender process may adjust the frequency of the vertical synchronization signal (Vsync-APP) in each display area, so that the application displayed in display area 1 generates a render tree at the frequency of the screen refresh rate of display area 1.

[0174] Note that the application's UI thread may generate multiple render trees in multi-display scenarios, such as multi-screen scenarios, virtual screen scenarios, or multi-window scenarios. For more information on multi-display scenarios, see (2.2) Description of Interface Generation Methods in Multi-Display Scenarios.

[0175] S902: Transfer the render tree using the cross-process method.

[0176] After generating the render tree, the application's UI thread transfers the render tree to the UniRender process via IPC communication. The render tree resides on the stack of the process corresponding to the application. The corresponding UniRender process must receive the render tree transferred by a different application and determine the correspondence between the render tree and the application.

[0177] Multiple applications in the foreground forward their render tree to UniRender. An application is a foreground application if it meets any one of the following three conditions: it has a visible activity, it has a foreground service, and other foreground applications are associated with it.

[0178] Since memory is not shared between different processes, data exchange between processes must be performed via inter-process communication (IPC). Applications may transfer render trees to the UniRender process using methods such as Binder, AIDL, shared memory, or Socket to achieve IPC communication. This is not limited to the foregoing.

[0179] In the following section, to illustrate an example of a method for transferring render trees using a cross-process approach, shared memory is used as an example of IPC communication.

[0180] (a) The application writes the render tree to shared memory.

[0181] Figure 11 is a schematic diagram of an example in which a shared memory transfers a render tree according to one embodiment of the present application.

[0182] As shown in Figure 11, initially, each time an application is launched, the application requests shared memory from the UniRender process. After receiving the request for shared memory initiated by the application, the UniRender process requests the shared memory and the handle corresponding to it via the Anonymous Shared Memory (Ashmem) subsystem.

[0183] After successfully requesting shared memory from the Ashmem subsystem, the UniRender process receives a handle returned by the Ashmem subsystem for reading and writing to physical memory. The UniRender process returns the handle to the application, which can then use to write the render tree to physical memory. The UniRender process can then directly read physical memory from its own process space and subsequently read the application's render tree directly.

[0184] Shared memory may be a virtual file created in RAM using a temporary file system (tmpfs), and is mapped separately to the user space of different processes.

[0185] For example, cross-process communication in which an application requests shared memory from the UniRender process, the UniRender process requests shared memory from the Ashmem subsystem, and the UniRender process returns a handle to the application corresponding to the shared memory obtained via the application may be implemented via Binder communication.

[0186] It will be understood that the render tree stored in the application process's stack may be transferred to the UniRender process using other IPC methods, such as memory sharing. This is not limited to the foregoing.

[0187] In some embodiments of this application, a trust list is configured in a local configuration file of an electronic device or cloud server, where the trust list stores other identifiers that can uniquely determine an application process, such as the application's package name. If the application is in the trust list, the render tree is forwarded to the UniRender process. If the application is not in the trust list, the bitmap is obtained via software drawing on the UI thread or rendering on the render thread, and then the bitmap is forwarded to UniRender for process compositing, or the bitmap is forwarded to SurfaceFlinger. SurfaceFlinger combines the bitmap of the application that is not in the trust list with the bitmap generated by the UniRender process.

[0188] If necessary, in some embodiments of this application, if the application has multiple layers, i.e., if the application generates multiple render trees, the application may request two shared memories from the UniRender process, each to be used to store different layers, i.e., to store data for different render trees.

[0189] If necessary, in some embodiments of this application, the application may request two shared memories from the UniRender process to write render trees alternately. For example, if the render tree corresponding to a frame interface is written to the first shared memory, the render tree corresponding to the interface of the next frame is written to the second shared memory, and then the render tree corresponding to the interface of the next frame is written to the first shared memory. This helps to avoid read / write conflicts, which would occur if there were only one shared memory, where the render tree data could not be written to the shared memory in a timely manner.

[0190] (b) Storage data structure of the render tree in shared memory To further improve IPC communication efficiency, in this embodiment of the present application, the render tree is stored in shared memory in the form of a memory tree. An example of the data structure of the render tree stored in shared memory is described below.

[0191] A memory tree may contain multiple data segments, each storing different data segments such as layer information and rendering data. To illustrate the structure of the memory tree, Figures 12A and 12B are used as an example. The offsets of different data segments within shared memory may be fixed or they may be adaptive.

[0192] Figures 12A and 12B are schematic diagrams of an example of a render tree storage structure in shared memory according to one embodiment of the present application.

[0193] As shown in Figure 12A, the memory structure of the render tree includes three data segments: the HEAD field, the MAPPING field, and the NODES field.

[0194] The HEAD field contains the layer key and root ID, the MAPPING field contains the node ID and the address corresponding to the node ID, and the NODES field contains current properties, staging properties, staging displaylist, and current displaylist.

[0195] `layerkey` is the ID of the entire render tree as a layer, `rootid` is the ID of the root node of the render tree, `nodeid` is the ID of a render node other than the root node of the render tree, one `nodeid` corresponds to one `address`, where `address` is the starting address of the rendering properties (renderproperties / properties) and the displaylist (displaylist) in the render tree node, `stagingproperties` are the rendering properties written by the application, `stagingdisplaylist` is the displaylist written by the application, `currentproperties` are the rendering properties read by the UniRender process, and `currentdisplaylist` is the displaylist read by the UniRender process.

[0196] It should be noted that "stagingproperties and stagingdisplaylist" are considered the first data group, and "currentproperties and currentdisplaylist" are considered the second data group. In this case, the data written to the application is from the first data group, and the data written to the application next is from the second data group, thereby realizing a dual buffer mechanism. Similarly, the data read by the UniRender process is from the first data group, and the data read next is from the second data group.

[0197] If necessary, in some embodiments of this application, the storage structure of the render tree in shared memory may be shown in Figure 12B. The NODES field may contain only one data group, i.e., only rendering properties and a list of drawing instructions.

[0198] The layerkey is used by the UniRender process to retrieve the layer data in shared memory using a handle, and to obtain the IDs of the application's layers that need to be rendered and composited from the WMS. The UniRender process then matches these layer IDs with the layer IDs contained in the layerkey in shared memory.

[0199] The rootid is used as follows: As an entry in the render tree, the rootid stores entries for other render nodes. After obtaining the rootid, the UniRender process may read the render tree data and reconstruct the nested structure of the render tree.

[0200] currentproperties, stagingproperties, stagingdisplaylist, and currentdisplaylist are used, namely, after the application has written the draw command list and rendering properties, the values ​​of currentproperties and stagingproperties are swapped, and the values ​​of currentdisplaylist and stagingdisplaylist are swapped. The UniRender process reads the rendering properties and draw command list from currentproperties and current displaylist.

[0201] For example, Application 1 is an application displayed in the foreground, and Application 1 may have multiple layers. Specifically, after receiving a vertical synchronization signal (Vsync-APP), Application 1 generates multiple render trees, for example, Render Tree 1 and Render Tree 2. The UniRender process determines from the WMS that Render Tree 1 is the render tree corresponding to the layers involved in layer compositing.

[0202] Since the offsets for layerkey and rootid are fixed, the UniRender process can determine the address of the root node. The UniRender process searches for the location of the root node on the NODES end in the MAPPING field and reads the node's rendering command. If child nodes exist, there is a corresponding DrawRenderNode command. The command stores the ID of the child node. The location of the child node in the MAPPING field is searched based on its hash value. For example, the parent-child relationships between render nodes in the render tree are stored in the DrawOP operations. For example, the draw command list for render node 2 may include several DrawOP operations and the "Draw RenderNode3" operation (draw render node 3), and as a result, the UniRender process may determine that render node 3 is a child node of render node 2.

[0203] It will be understood that the render tree in shared memory still stores the same nesting relationships as the nesting relationships between the application's views. Therefore, the UniRender process may read data from the root node and then read all the data in the render tree.

[0204] The currentproperties, stagingproperties, stagingdisplaylist, and currentdisplaylist are categorized to ensure the security of display data read by the UniRender process and data written by the application, preventing data that has not been fully written by the application from being read by the UniRender process as the latest data for rendering and generating the application's interface. For security assurances regarding simultaneous reading and writing of the render tree, see the following description in (2.3) Reading and Writing the Render Tree in Shared Memory. Further details are not provided herein.

[0205] In some embodiments of this application, the sizes of the three data segments may be fixed as needed. Specifically, after the UniRender process requests shared memory from the Ashmem subsystem, the size of the acquired shared memory is a + b + c. The area from the start address (physical address) to the start address + a is where the HEAD field is filled, the area from the start address + a + 1 to the start address + a + b is where the MAPPING field is filled, and the area from the start address + a + b + 1 to the start address + a + b + c is where the NODES field is filled.

[0206] If the sizes of the three data segments are fixed, it will be understood that the UniRender process may determine the beginning of each data segment according to a fixed offset in order to look up the Mapping field. The Mapping field stores the offset of the render node in the render tree within the NODES field in order to look up the data for each render node.

[0207] If necessary, in some embodiments of this application, the sizes of the three data segments are fixed, and if the size of the render node written by the application exceeds b, the application requests a second shared memory from the UniRender process. The format of the second shared memory may be the same as the format of the first shared memory, and the draw instruction list and render node properties of the render node continue to be stored in the NODES field of the second shared memory. The HEAD field and / or MAPPING field of the second shared memory may be null or not exist. In other words, in some embodiments of this application, the second shared memory contains only the NODES field.

[0208] (c) UniRender reads the render tree from shared memory.

[0209] (c.1) UniRender Architecture Figure 13 is a schematic diagram of an example of a UniRender architecture according to one embodiment of this application.

[0210] As shown in Figure 13, the UniRender provided in this application may include four parts: NodeManager, LayerManager, DisplayManager, and UniRenderCore.

[0211] The NodeManager is the node management module in the UniRender process and is responsible for receiving render trees sent by the application. For information on compositing the target render tree, please refer to the description in step S903.

[0212] The LayerManager is the layer management module in the UniRender process, responsible for synchronizing layer information such as layer creation, layer destruction, and property changes from the Window Manager Service (WMS). Each bitmap corresponds to one layer.

[0213] DisplayerManager is a display device management module in the UniRender process, and its role is to synchronize display device information, such as screen size, from the Display Manager Service (DMS).

[0214] UniRenderCore is the rendering management module in the UniRender process. It establishes a corresponding render node for each layer, receives render trees corresponding to different applications maintained by NodeManager, converts the application's layer information into commands in LayerManager, inserts those commands into the render nodes, merges the render trees corresponding to all visible layers of the activated display device maintained in DisplayManager, and traverses the merged render trees for each display area. UniRender generates the bitmap for the buffer assigned by UniRender.

[0215] (c.2) UniRender reads the render tree.

[0216] The UniRender process first determines the applications displayed in each display area from the DMS and WMS, and these applications are those involved in layer compositing. The UniRender process may further determine which applications perform layer compositing in the UniRender process by referring to a trust list. The UniRender process may also determine the layer ID of each application by using the WMS.

[0217] In the UniRender process, the DisplayerManager is responsible for communicating with the DMS, and the LayerManager in the UniRender process is responsible for communicating with the WMS.

[0218] The UniRender process stores a handle to the shared memory corresponding to the application. After determining which applications are involved in layer compositing, the UniRender process may use the handle to determine which shared memory corresponds to the application. UniRender reads the render tree from the shared memory via the handle.

[0219] In the UniRender process, the NodeManager is responsible for managing the handle to shared memory and reading the render tree from shared memory.

[0220] The UniRender process reads the render tree from shared memory, including the following:

[0221] The UniRender process first reads the layerkey from the start address of shared memory and matches it with the layer ID. The UniRender process compares the layer ID determined by WMS with the layer ID determined from the layerkey. After the layer IDs match, the UniRender process reads the render tree from the rootid of the root node.

[0222] Next, after finding the address of the root node, the UniRender process determines the starting address of the root node in the NODES field using the address field in the MAPPING field, and begins reading the draw command list and rendering properties of the render node. If the root node has child nodes, the draw list commands of the root node store the entries of the child nodes, for example, the DrawRenderNode command. Since the DrawRenderNode command contains the ID of the child node, the UniRender process searches for the corresponding nodeid in MAPPING via a hash operation, determines the location of the draw command list and rendering properties of the child node in the NODES field, and reads the draw command list and rendering properties of the child node.

[0223] (d) Reading and writing render trees in shared memory A render tree located in shared memory may be read from and written to by two or more processes. To reduce and avoid errors in the render tree data caused by read and write contention, inter-process synchronization locks may be configured to ensure the security of read and write operations to the render tree.

[0224] Referring to the contents shown in Figures 14 and 15, an example of the render tree read and write process is described below.

[0225] Figure 14 is a schematic diagram illustrating an example of an application writing a render tree to shared memory according to one embodiment of the present application.

[0226] Each application stores at least one lock variable A to prevent the application and UniRender from simultaneously reading / writing to shared memory. UniRender obtains the lock variable status (held or released) on different applications via IPC communication.

[0227] As shown in Figure 14, [1] first, when the application needs to update the interface, a vertical synchronization signal (Vsync-APP) is requested and received, then the lock variable A in the shared memory corresponding to the application is acquired, and the properties of the render node and the draw instruction list are calculated and updated from the root node.

[0228] [2] The application then writes the updated render node properties and draw instruction list to the stagingproperties and stagingdisplaylist data segments of the NODES field in shared memory, and adds the IDs of the changed render nodes to the properties_dirty and displaylist_dirty queues. The queues are stored in a shared memory management class singleton on the application side.

[0229] It should be understood that modified render nodes are marked in the properties_dirty queue and displaylist_dirty queue to enable differential updates of the render tree.

[0230] In some embodiments of this application, the properties_dirty queue and displaylist_dirty queue may not be stored in order to achieve a complete render tree update, as needed.

[0231] [3] The application then copies the stagingproperties segment of the corresponding render node in the properties_dirty queue to the currentproperties segment. The application then swaps the draw_pointer and record_pointer of the corresponding render node in the displaylist_dirty queue, i.e., copies the stagingdisplaylist segment of the corresponding render node in the displaylist_dirty queue to the currentdisplaylist, or the application copies the stagingdisplaylist segment to the currentdisplaylist.

[0232] It will be understood that, in accordance with the current vertical synchronization signal (Vsync-APP) compared to the previous vertical synchronization signal (Vsync-APP), the application can change only the data of the render node corresponding to displaylist_dirty, and the application can swap the draw_pointer and record_pointer of the corresponding render node in the displaylist_dirty queue, thereby achieving a differential update of currentdisplaylist.

[0233] To achieve a complete update, the application copies the stagingdisplaylist segment to currentdisplaylist, meaning all data in the render tree generated by the application in response to the vertical synchronization signal (Vsync-APP) is written directly to shared memory, which is relatively easy to implement.

[0234] If necessary, in some embodiments of this application, when the properties_dirty queue and displaylist_dirty queue are not stored, the stagingproperties segment of all render nodes in the application is copied to the currentproperties segment, and the stagingdisplaylist segment is copied to the currentdisplaylist segment. The copy may be achieved by changing the position of the pointer.

[0235] [4] The application then transmits information to the UniRender process via IPC communication that lock variable A has been released.

[0236] [5] Next, the UniRender process holds the lock variable A.

[0237] [6] Finally, corresponding to [3], the UniRender process reads the current displaylist and current properties from shared memory, or reads the stagingdisplaylist segment of the corresponding render node in the displaylist_dirty queue and copies the stagingdisplaylist segment to the currentdisplaylist.

[0238] After reading the data, the UniRender process may release lock variable A and notify the application that lock variable A has been released. When the next vertical synchronization signal (Vsync-APP) arrives, lock variable A is retained and the render tree is written to shared memory. In this case, the functions of the staging data segment and the current data segment are swapped. The UniRender process implements a "dual buffer" mechanism and reads the stagingdisplaylist and stagingproperties segments last to ensure robustness of interface generation.

[0239] If necessary, in some embodiments of this application, when there is one lock variable between the application and the UniRender process, the NODES field may contain only stagingdisplaylist and stagingproperties, or only currentdisplaylist and currentproperties. The application and the UniRender process use the lock variable to achieve read and write security, and as a result the UniRender process reads the correct render tree.

[0240] If necessary, in some embodiments of this application, each application may hold more lock variables. For example, each application holds lock variable A and lock variable B. In this way, after the application releases lock A, the application does not need to wait for UniRender to release lock A. After holding lock variable B, the application writes the render tree data directly to shared memory after receiving the next vertical synchronization signal (Vsync-APP).

[0241] The holding, release, and inter-process synchronization of lock variable B are the same as those of lock variable A. See the caption for Figure 14 for details. Further details are not provided in this specification.

[0242] Figure 15 is a schematic diagram of another example in which an application writes a render tree to shared memory, according to one embodiment of the present application.

[0243] As shown in Figure 15, when an application stores a lock variable A, the application holds the lock variable A after receiving the vertical synchronization signal 1 (Vsync-APP), and during the holding phase, writes the generated render tree to shared memory. After the application has written the render tree to shared memory, the lock variable A is released. After determining that the lock variable A has been released by the application, UniRender holds the lock variable A and begins reading the render tree generated by the application from shared memory in response to the vertical synchronization signal 1.

[0244] While the UniRender process holds lock variable A, the application receives vertical synchronization signal 2 (Vsync-APP) after vertical synchronization signal 1. Because lock variable A is held by the UniRender process, the application cannot write the render tree to shared memory in a timely manner and must wait for a certain period of time until the application determines that the UniRender process has released lock A.

[0245] However, as can be seen in Figure 13, the UniRender process reads the currentdisplaylist and currentproperties fields, while the application writes to the stagingdisplaylist and stagingproperties fields. Simultaneous reading and writing of different fields in shared memory may be achieved by increasing the number of lock variables.

[0246] As shown in Figure 15, when the application stores two lock variables B, after receiving vertical synchronization signal 1 (Vsync-APP), the application holds lock variable A and, during the holding phase, writes the generated render tree to shared memory. After the application has written the render tree to shared memory, lock variable A is released. After determining that lock variable A has been released by the application, the UniRender process holds the lock variable and, in response to vertical synchronization signal 1, begins reading the render tree generated by the application from shared memory.

[0247] During the period when the UniRender process holds lock variable A, the application receives vertical synchronization signal 2 (Vsync-APP) after vertical synchronization signal 1. In this case, since lock variable B is held, the application can write the render tree to shared memory in a timely manner and does not need to wait for a certain period of time until the application determines that the UniRender process has released lock A.

[0248] Accordingly, after determining that lock variable B has been released by the application, the UniRender process holds lock variable B and begins reading the render tree generated by the application from shared memory in response to vertical synchronization signal 2.

[0249] Note that the number of lock variables held by the application may be related to the contents of the NODES field in shared memory, or it may be related to the value of Vsync-offset.

[0250] For example, if currentdisplaylist and currentproperties are the first group and stagingdisplaylist and stagingproperties are the second group, then two synchronized variables may be configured in the application and the UniRender process, corresponding to the two data groups, respectively. Similarly, in another example, if the NODES field contains three data groups, then three synchronized lock variables may be configured in the application and UniRender.

[0251] A single lock variable corresponds to a single data group. For example, if lock variable A corresponds to currentdisplaylist and currentproperties, a change in the lock variable A state from held to released indicates that the application has successfully updated the data in currentdisplaylist and currentproperties in shared memory, and that the data in currentdisplaylist and currentproperties can now be read by the UniRender process. Alternatively, a change in the lock variable A state from held to released indicates that the UniRender process has successfully read the data in currentdisplaylist and currentproperties from shared memory, and that the data in currentdisplaylist and currentproperties can now be updated by the application.

[0252] The number of lock variables may be related to the value of Vsync-offset.

[0253] In other words, the number of lock variables may be related to the difference Vsync-offset between the vertical synchronization signal (Vsync-APP) and the vertical synchronization signal (Vsync-UR). If Vsync-offset is large, lock variables may not be configured. If lock variables are not configured, the UniRender process reads the render tree from shared memory after receiving the vertical synchronization signal (Vsync-UR). Because Vsync-offset is large, when the UniRender process reads the render tree, the application has written the render tree completely to shared memory.

[0254] S903: Transfers application window control information and display area information.

[0255] In the UniRender process, the LayerManager obtains window control information for one or more applications from the window manager service and, referring to the layers of one or more applications obtained in step S802, further determines whether any application layer contains drawing logic that triggers off-screen rendering.

[0256] The UniRender process may further obtain the Z-order of layers from different applications. The Z-order is the order of the Z-axis between different layers.

[0257] In the UniRender process, the DisplayerManager obtains display area information from the display manager service, which includes the size of the display device. Based on the display area information, the UniRender process determines the size of the allocated surface. The surface is used to carry the bitmap generated by UniRender based on the target render tree. For details on how the UniRender process generates the bitmap based on the target render tree, see the description in step S905. Further details are not described herein.

[0258] Instructions related to off-screen rendering include instructions such as rounding, scaling, rotation, or color conversion. For the definition of off-screen rendering, please refer to the description of (1.3) off-screen rendering mentioned above. Details will not be described again in this specification.

[0259] The rendering logic related to the triggers of off-screen rendering such as rounding, scaling, rotation, and color conversion will be described below using examples.

[0260] Figures 16A-1, 16A-2 to 16C are schematic diagrams of an example of a scenario of rendering logic that triggers off-screen rendering according to an embodiment of the present application.

[0261] In response to user interaction, such as a bezel-less gesture (e.g., sliding upward from the bottom to the center of the screen), or tapping the multitasking control on the lower navigation bar, the electronic device displays the multitasking interface as shown in Figures 16A-1 and 16A-2.

[0262] The bitmap generated by the gallery application has the same size as the screen, but the interface of the gallery application in the multitasking interface shown in Figures 16A-1 and 16A-2 is the interface obtained after scaling and rounding processing.

[0263] As shown in Figure 16B, in the small window mode, views of multiple applications are displayed on the interface of the electronic device. The interfaces of some applications are within floating windows.

[0264] For example, the interfaces for news applications and SMS applications are displayed on the electronic device's interface, with the SMS application interface located within a rendering floating window.

[0265] The bitmap generated by the SMS application is the same size as the screen, but the interface of the SMS application in small window mode shown in Figure 16B is the interface obtained after scaling and rounding.

[0266] As shown in Figure 16C, the electronic device launches the reader application in response to the user tapping the reader application icon on the desktop application. During the reader application launch process, the reader application's main interface (the interface corresponding to the Main Activity) or launch window (starting window) expands sequentially.

[0267] The bitmap corresponding to the main interface or startup window of the reader application has the same size as the screen, and a continuous scaling effect is achieved by adjusting the scaling ratio. Furthermore, the bitmap corresponding to the main interface or startup window of the reader application is a bitmap obtained after rounding processing and is displayed on the interface of the electronic device.

[0268] In the scenarios shown in Figures 16A-1 and 16A-2 through 16C, and many other scenarios, it will be understood that in order to improve the user experience, interfaces that better suit the user's visual habits need to be generated through rounding, scaling, rotation, and color transformation.

[0269] Note that in the scenarios shown in Figures 16A-1 and 16A-2 through 16C, and many other scenarios, the UniRender process receives window control information for each application layer from the window manager service to determine whether rounding, scaling, rotation, and color transformation should be performed for each application layer.

[0270] If necessary, in some embodiments of this application, the UniRender process may first obtain window control information for one or more applications from the window manager service, and then obtain the render trees for one or more applications. That is, the time order of steps S902 and S903 may be reversed.

[0271] S904: Generate a target render tree based on the acquired render tree, window control information, and display area information.

[0272] First, after receiving render trees generated by one or more applications, the UniRender process receives vertical synchronization signals and window control information, and then, based on the window control information, determines whether each application's layer has logic to trigger off-screen rendering.

[0273] When an application is displayed in the display area of ​​a local electronic device, window control information may come from the local window manager service. When an application is displayed in the display area of ​​another electronic device, window control information may come from the window manager service of the peer electronic device.

[0274] If the UniRender process determines that logic for triggering off-screen rendering exists in any application layer, the UniRender process translates the logic for triggering off-screen rendering into an off-screen rendering instruction, and then translates the off-screen rendering instruction into a property of the render node in the corresponding render tree. For simplicity of explanation, the process by which the UniRender process translates the logic for triggering off-screen rendering into an off-screen rendering instruction, and then translates the off-screen rendering instruction into a property of the render node in the corresponding render tree, may be concisely referred to as advancing the off-screen rendering instruction.

[0275] Next, after the UniRender process has triggered off-screen rendering commands for one or more render trees, the UniRender process merges the render trees corresponding to the visible layers of each display area to generate a target render tree. In other words, the number of target render trees may be related to the number of display areas.

[0276] After receiving the vertical synchronization signal (Vsync-UR), the UniRender process may begin reading the render tree from shared memory, and after obtaining multiple render trees, it may initiate an off-screen rendering instruction and begin merging the render trees. Alternatively, the UniRender process may begin reading the render tree from shared memory while holding a lock variable, and when it receives the vertical synchronization signal (Vsync-UR), it may initiate an off-screen rendering instruction and begin merging the render trees.

[0277] The process of triggering and transitioning to off-screen rendering commands, and the process of generating the target render tree, are described separately below.

[0278] (a) The process of advancing the instructions to trigger off-screen rendering. First, the UniRender process retrieves window control information for each application layer and determines whether the layer's window control information contains drawing logic to trigger off-screen rendering. If the UniRender process determines that none of the window control information for any application layers in the display area contains drawing logic to trigger off-screen rendering, one or more render trees may be merged directly into the target render tree. If the UniRender process determines that the window control information for any application layer in the display area contains drawing logic to trigger off-screen rendering, the UniRender process first transfers the instructions to trigger off-screen rendering, and then merges the multiple render trees into the target render tree.

[0279] The process of transferring instructions to trigger off-screen rendering: The UniRender process first determines the drawing logic that triggers off-screen rendering from the window control information, and then translates the drawing logic that triggers off-screen rendering into instructions (or referred to as drawing logic instructions) that can be configured in the rendering properties of the render nodes. After determining the coupling relationships between layers and the render tree, the UniRender process updates the off-screen rendering instructions in the rendering properties of the corresponding render nodes.

[0280] When a render node's rendering properties include corresponding scaling, rounding, colortransform, or transformation properties, the parameters within the scaling, rounding, colortransform, or transformation instructions in the command to trigger off-screen rendering are assigned to the scaling, rounding, colortransform, or transformation properties within the render node's properties. When a render node's rendering properties do not include corresponding scaling, rounding, colortransform, or transformation properties, scaling, rounding, colortransform, or transformation properties are added to the render node, and the parameters within the scaling, rounding, colortransform, or transformation instructions in the command to trigger off-screen rendering are assigned to the scaling, rounding, colortransform, or transformation properties within the render node's properties.

[0281] Figures 17A and 17B are schematic diagrams illustrating an example of how UniRender transitions an off-screen rendering trigger command according to one embodiment of the present application.

[0282] As shown in Figure 17A, the UI threads of Application 1 and Application 2 independently generate render trees and independently transfer them to the UniRender process via IPC communication. The UniRender process then determines the commands from the window manager service to trigger off-screen rendering at each application's layer. The render tree generated by Application 1 is render tree 1, and the render tree generated by Application 2 is render tree 2.

[0283] UniRender updates the properties of render nodes in different render trees independently. That is, the UniRender process updates the properties of the render nodes in render tree 1, and the UniRender process updates the properties of the render nodes in render tree 2.

[0284] When commands for triggering off-screen rendering are applied to the entire layer, the UniRender process assigns the parameters in the off-screen rendering command directly to the root node of the render tree corresponding to the layer or to all render nodes of the render tree.

[0285] When the UniRender process assigns the parameters in the off-screen rendering command to the root node of the render tree corresponding to the layer, the UniRender process automatically configures these properties of the render nodes as the rendering properties of the child nodes of the root node when generating a bitmap based on the render tree.

[0286] For example, if the child node of render node 1 is render node 2, after the rotation property (transform) is configured in the rendering properties of render node 1, the UniRender process configures the same rotation property (transform) for the rendering properties of RenderNode2 when generating the render tree.

[0287] As shown in Figure 17B, after receiving the bitmap corresponding to the render tree generated by the application before the off-screen rendering command is migrated, SurfaceFlinger needs to perform different processes on the bitmaps of different applications based on the window control information in order to perform layer composition.

[0288] After the off-screen rendering instructions are transferred to the application's render tree, the UniRender process assigns the parameters within the off-screen rendering instructions to the rendering properties of the render tree. As a result, the converted bitmaps are drawn directly onto the canvas in a process that generates bitmaps based on the render tree. Bitmaps from different applications do not need to be processed differently, so bitmaps from different applications can be drawn sequentially into a single on-screen buffer.

[0289] The UniRender process assigns parameters within the off-screen rendering instructions to rendering properties in the render tree, as shown in Figure 18.

[0290] Figure 18 is a schematic diagram illustrating an example of how UniRender modifies the render node properties of a render tree according to one embodiment of the present application.

[0291] As shown in Figure 18, the UniRender process modifies the properties of the root node in the render tree and adds instructions to enable the addition of properties to render nodes. For example, the instruction for scaling properties is setScale(), the instruction for rounding properties is outline.setRoundRect(), the instruction for color conversion properties is setColorFilter(), and the instruction for rotation properties is setLeftTopRightBottom().

[0292] In the subsequent step S905, when generating a bitmap based on the render tree, the UniRender process modifies the drawing operations in the drawing instruction list according to the rendering properties, and then generates the bitmap obtained after scaling, rounding, color conversion, and rotation. For details, please refer to the description of step S905. Details are again not described herein.

[0293] If necessary, in some embodiments of this application, the instructions in the properties of the root node of the render tree may further include the instruction setStaticMatrix(), etc. Since the application's UI thread generates the render tree based on the surface requested by SurfaceFlinger, the instruction setStaticMatrix() is configured in the root node of the render tree in the process of shifting the instruction to trigger off-screen rendering in order to change the reference system when UniRender generates a bitmap based on the render tree in step S905. For the specific content of the instruction setStaticMatrix(), see the following description of step S905. Further details are not described herein.

[0294] (b) Process for generating the target render tree The UniRender process retrieves one or more processed render trees after transferring off-screen rendering instructions for all render trees.

[0295] When there is one processed render tree, that processed render tree is the target render tree; or, when there are multiple processed render trees, the multiple processed render trees are merged to form a single target render tree.

[0296] Figure 19 is a schematic diagram of an example of compositing multiple render trees obtained after an off-screen rendering command has been transferred to a single target render tree, according to one embodiment of the present application.

[0297] As shown in Figure 19, the UniRender process may create a new root render node, for example, the new root render node in Figure 19, and use the processed render trees of multiple applications as child nodes of the root render node. Specifically, the root render node may store parent-child relationships between processed render trees of different applications, and as a result, the UniRender process may traverse the target render trees from the root render node to find the processed render trees of different applications.

[0298] Figure 20 is a schematic diagram of an example of compositing multiple render trees acquired after an off-screen rendering command has been transferred to a single target render tree, according to one embodiment of the present application.

[0299] The UniRender process may, according to the window manager service, first determine the Z-order of layers corresponding to different applications, that is, it may determine the top-to-bottom occlusion relationships between application layers. Furthermore, in the process of generating the target render tree, the rendering nodes corresponding to completely occluded views are removed, thereby reducing the computational complexity in the process of generating the bitmap in step S905 and improving the bitmap generation speed.

[0300] For example, as shown in Figure 20, the layer corresponding to the render tree generated by Application 1 is Layer 1, and the layer corresponding to the render tree generated by Application 2 is Layer 2. The Z-order of Layer 1 is higher than that of Layer 2. The render tree corresponding to Layer 1 consists of Root Render Node 1, Render Node 1, Render Node 2, and Render Node 3, where Render Node 1 and Render Node 2 are child render nodes of Root Render Node 1, and Render Node 3 is a child render node of Render Node 2. The render tree corresponding to Layer 2 consists of Root Render Node 2, Render Node 4, Render Node 5, and Render Node 6, where Render Node 4 and Render Node 5 are child render nodes of the new Root Render Node, and Render Node 6 is a child render node of Render Node 5.

[0301] The UniRender process may traverse the child render nodes of the render tree corresponding to Layer 1 and the child nodes of the render tree corresponding to Layer 2 to determine the position of the view corresponding to each render node on the surface (the surface assigned by the UniRender process), and refer to the Z-order of Layer 1 and the Z-order of Layer 2 to determine the render nodes of the completely occluded view.

[0302] For example, UniRender may determine that the view corresponding to render node 6 in the render tree corresponding to layer 2 is completely occluded, and then delete render node 6.

[0303] After removing the render node corresponding to the completely occluded view, the UniRender process may merge all render trees obtained after the off-screen rendering trigger instruction has been migrated into the target render tree, as shown in Figure 18.

[0304] If necessary, in some embodiments of this application, UniRender may further traverse render nodes and optimize the parameters of the target render tree based on the granularity of DrawOP drawing operations. For example, DrawOP drawing operations that do not affect the interface may be removed, meaning that graphics drawn by the DrawOP drawing operation are not displayed in the interface. As another example, the location of DrawOP operations within the drawing nodes may be modified in different applications so that the same type of DrawOP drawing operation can be executed simultaneously. For example, DrawOP drawing operations on render node 2 of application 1 may be modified in the drawing instruction list of render node 1 of application 2. This is not limited herein.

[0305] If necessary, in some embodiments of this application, multiple render trees acquired after an off-screen rendering trigger instruction has been transferred may be merged into a single render tree as shown in Figure 18, then a fully occluded view is determined in order to remove the render node corresponding to the fully occluded view, and a target render tree is generated.

[0306] If necessary, in some embodiments of this application, for partially obscured views, the render node corresponding to that view may also be clipped by using the clip instruction. Clipping may be performed before the target render trees are merged or after the target render trees are merged.

[0307] If necessary, in some embodiments of this application, after step S904 and before step S905, after generating the target render tree, the UniRender process depackages the list of draw instructions of the RenderNode to obtain a set of DrawOPs, and then performs DrawOP operations on the entire target render tree to perform batch and merge operations. In step S805, a target render tree is obtained from which bitmaps can be generated with less computation.

[0308] In the scenarios shown in Figures 16A-1, 16A-2, and 16B, as well as in other scenarios, it will be understood that multiple render trees are merged to avoid or reduce overdraw, and the merged render tree is optimized, thereby improving the speed at which bitmaps are generated in step S905, reducing power consumption of electronic devices, and improving the user experience.

[0309] It should be noted that the UniRender process may further optimize other parameters of the target render tree during the process of generating the target render tree. This is not limited to the foregoing.

[0310] S905: Generates a bitmap based on the target render tree.

[0311] After the target render tree is obtained, the UniRender process assigns a surface to the target render tree, and the UniRender process generates a bitmap on the surface based on the target render tree, and the bitmap corresponds to the composite interface of one or more applications.

[0312] When an electronic device has multiple display areas, the surface may be joined to one of the display areas, and the size of the surface may be the same as the size of the display area joined to the surface.

[0313] The UniRender process generating bitmaps on a surface based on the target render tree includes the following:

[0314] First, the UniRender process begins its traverse from the root node to the target render tree, and may traverse the root node's child nodes in multiple ways.

[0315] The UniRender process can traverse different layers under the root node based on the Z-order of the layers. The UniRender process may traverse different layers in descending Z-order or ascending Z-order.

[0316] For example, in the target render tree shown in Figure 20, if the Z-order of Layer 1 is higher than the Z-order of Layer 2, the UniRender process may first traverse the render tree corresponding to Layer 1, that is, it may first traverse the render tree containing Root Render Node 1, Render Node 1, Render Node 2, and Render Node 3, and then traverse the render tree containing Root Render Node 2, Render Node 4, and Render Node 5. Similarly, the UniRender process may first traverse the render tree corresponding to Layer 2, and then traverse the render tree corresponding to Layer 1.

[0317] Note that when the UniRender process traverses layers in descending Z-order, and when the UniRender process executes drawing operations in the drawing command list, if the layers are composited in an occluding relationship, drawing may only be performed where it has not been drawn in order to reduce overdrawing.

[0318] Note that when the UniRender process traverses layers in ascending Z-order, it sequentially executes the drawing operations in the list of drawing instructions within the render tree corresponding to each different layer in order to generate a bitmap.

[0319] Next, the UniRender process modifies the drawing operations in the list of drawing instructions according to the rendering properties and executes the drawing operations to generate the bitmap.

[0320] When traversing each render node, the UniRender process reads the render properties of the render node. If the rendering properties include scaling properties, rounding properties, color conversion properties, rotation properties, etc., the UniRender process modifies the parameters of the drawing operations in the draw command list according to these properties, and then executes the drawing operations in the modified draw command list. If the rendering properties do not include scaling properties, rounding properties, color conversion properties, or rotation properties, UniRender executes the drawing operations in the draw command list directly.

[0321] The following example illustrates how the UniRender process modifies the parameters of drawing operations in the drawing command list according to rendering properties.

[0322] For example, the rendering properties of a render node include setscale(0.5), which indicates scaling down to 0.5 times the original size, and the draw command list includes drawCircle(x0, y0, 5). In this case, when executing the draw operation drawCircle(x0, y0, 5), the UniRender process changes the draw operation to drawCircle(x0, y0, 2.5). In drawCircle(), the first parameter is the coordinates of the circle's center on the X axis, the second parameter is the coordinates of the circle's center on the Y axis, and the third parameter is the radius of the circle.

[0323] Considering that the list of drawing instructions generated by the application's UI thread may be related to the size of the surface requested by the application, in this embodiment of the application, when traversing a render tree node, the UniRender process configures setStaticMatrix() within the properties of the render node to transform the reference coordinate system for drawing and rendering.

[0324] The UniRender process may determine the size of the reference coordinate system or reference surface when each application generates its drawing instruction list, in order to determine the parameters of setStaticMatrix(), by using the WMS.

[0325] Figure 21 is a schematic diagram of an example of setStaticMatrix() according to one embodiment of this application.

[0326] After determining the surface coordinate system referenced by the application to generate the drawing command list, the UniRender process determines the coordinate transformation matrix Transformation based on the surface coordinate system maintained by the UniRender process.

number

[0327] In this specification, scale x This involves scaling in the x-axis direction, skew x This is the twist / tilt in the x-axis direction, translate x This involves translation along the x-axis and scaling. y This involves scaling in the y-axis direction, skew y This is the twist / tilt in the y-axis direction, translate y This represents a translation along the y-axis.

[0328] After executing the setStaticMatrix() command, the UniRender process obtains the Transformation matrix through computation and applies that transformation matrix to each drawing operation in the draw command list. As shown in Figure 21, when the coordinate system referenced by the application's UI thread to generate the render tree is not the coordinate system on the surface held by the UniRender process, the UniRender process determines the Transformation matrix by using the setStaticMatrix() command and transforms the reference coordinate system when drawing the interface on the surface based on the draw command list in order to draw the interface correctly.

[0329] The above primarily describes the specific steps by which an electronic device implements the interface generation method provided in the embodiments of this application in a single display area scenario. Below, the specific steps by which an electronic device implements the interface generation method provided in the embodiments of this application in a multi-display area scenario will be primarily described.

[0330] (2.3) Interface generation method in a multi-display area scenario First, the multi-display area scenario will be explained with an example.

[0331] (a) Multi-display area scenario In the embodiments of this application, the display area may be a screen, a virtual display, or the like. The virtual display may be an area used by an electronic device to carry content displayed on the screen during screen recording.

[0332] Figures 22A to 22C are schematic diagrams of an example of a multi-display area scenario according to one embodiment of this application.

[0333] When an electronic device is configured with multiple screens, each screen is a display area. As shown in Figure 22A, the primary screen of the electronic device is used as display area 1 for displaying desktop applications, and the secondary screen of the electronic device is used as display area 2 for displaying music applications.

[0334] The screen of an electronic device may have multiple states, and in each state, the screen may have one or more display areas. As shown in Figure 22B, when the screen of an electronic device is folded, each part of the screen may be a display area, or when the screen of an electronic device is unfolded, the left half and the right half of the screen may each be a display area.

[0335] As shown in Figure 22C, when the screen recording function is enabled, the screen of the electronic device is the display area, and the virtual screen that carries the screen recording content is also a display area. The content displayed in the two display areas may be the same.

[0336] Figures 23A-1 and 23A-2 through 23C are schematic diagrams of other examples of a multi-display area scenario according to one embodiment of the present application.

[0337] As shown in Figures 23A-1 and 23A-2, electronic device 1 shares the content displayed on its screen with the screen of electronic device 2. In this case, for electronic device 1, its screen is display area 1, and the screen of electronic device 2 is display area 2. The content displayed on the screen of electronic device 2 includes the content on the screen of electronic device 1.

[0338] As shown in Figure 23B, electronic device 1 projects the content displayed on its screen onto the screen of electronic device 2. In this case, for electronic device 1, its screen is display area 1, and the screen of electronic device 2 is display area 2. Unlike the content shown in Figures 23A-1 and 23A-2, the content displayed on the screen of electronic device 2 is only the content on the screen of electronic device 1.

[0339] As shown in Figure 23C, electronic device 1 projects the content displayed on its screen onto the screen of electronic device 2. In this case, for electronic device 1, its screen is display area 1, and the screen of electronic device 2 is display area 2. Unlike the content shown in Figures 23A-1, 23A-2, and 23B, the content displayed on the screen of electronic device 2 is only a portion of the content on the screen of electronic device 1.

[0340] In a multi-display area scenario, if there are multiple electronic devices, for example, if there is a display area 1 on electronic device 1 and a display area 2 on electronic device 2, it should be noted that only the content displayed on the screen of electronic device 1 may be displayed in display area 2, or some or all of the content displayed on the screen of electronic device 1 may be superimposed on the content originally displayed on the screen of electronic device 2 and displayed in display area 2.

[0341] (b) Method for generating an interface in a single device, multi-display area, and scenario. Figure 24 is a schematic diagram of another example of an interface generation method according to one embodiment of the present application.

[0342] As shown in Figure 24, the interface generation method in a single-device, multi-display-area scenario includes the following steps:

[0343] S2401: Constructs and generates a render tree after receiving a vertical signal.

[0344] For specific details, please refer to the corresponding explanation in step S901 of Figure 9. Further details are not provided in this specification.

[0345] S2402: Transfer the render tree using the cross-process method.

[0346] For specific details, please refer to the corresponding explanation in step S902 of Figure 9. Further details are not provided in this specification.

[0347] If necessary, in some embodiments of this application, the DisplayRender subprocess may directly obtain the render tree via IPC communication with the corresponding application. The UniRender process determines the correspondence between the DisplayRender subprocess and the application.

[0348] In some embodiments of this application, as needed, a set of shared memory comprising multiple shared memories may exist between all DisplayRender subprocesses and all applications, with each DisplayRender subprocess holding a handle to the multiple shared memories.

[0349] For the concept of the DisplayRender subprocess, please refer to the corresponding explanation in step S2404. Further details are not provided in this specification.

[0350] S2403: Transfers application window control information and display area information.

[0351] For specific details, please refer to the corresponding explanation in step S903 of Figure 9. Further details are not provided in this specification.

[0352] S2404: Assign render tree.

[0353] The UniRender process may create subprocesses called DisplayRender (DisplayRender1 to DisplayRenderN in Figure 24), the number of which is equal to the number of display areas, and transfer the render tree of the application displayed in the display area to DisplayRender.

[0354] For example, applications 1 and 2 are displayed in display area 1, which corresponds to subprocess DisplayRender1, and applications 3 and 4 are displayed in display area 2, which corresponds to subprocess DisplayRender2. In this case, the UniRender process transfers the render trees of application 1 and application 2 to subprocess DisplayRender1, and the render trees of application 3 and application 4 to subprocess DisplayRender2.

[0355] DisplayRender1 and DisplayRender2 may obtain a handle to the corresponding application's shared memory from the UniRender process in order to obtain the corresponding application's render tree. Alternatively, the application's render tree may be obtained from the UniRender process using other IPC communication methods such as Binder. This is not limited to the foregoing.

[0356] If necessary, in some embodiments of this application, the UniRender process may first merge the render trees of multiple applications into a target render tree, and then transfer the target render tree to the corresponding DisplayRender subprocess. The transfer of the target render tree may be achieved by multiple IPC communication schemes, such as shared memory and Binder, but is not limited herein.

[0357] If necessary, in some embodiments of this application, the UniRender process may transfer off-screen rendering instructions for a render tree and then transfer the render tree to a corresponding DisplayRender subprocess, or transfer the window control information and render tree of an application in a corresponding display area to a corresponding DisplayRender subprocess, the DisplayRender subprocess may transfer off-screen rendering instructions for a render tree and merge the multiple render trees into a target render tree.

[0358] If necessary, in some embodiments of this application, two threads, for example, an I / O thread and a render thread, may be configured for the DisplayRender subprocess. The I / O thread is responsible for receiving the render tree, and the render thread is responsible for generating the target render tree and generating a bitmap based on the target render tree.

[0359] By creating multiple DisplayRender subprocesses, the UniRender process may perform frequency division and frequency multiplication on the vertical synchronization signal (Vsync-UR) and send the vertical synchronization signal (Vsync-UR) on which frequency division and frequency multiplication are performed to different DisplayRender subprocesses, so that the different DisplayRender subprocesses generate bitmaps at different frequencies to match the refresh rates of different display areas.

[0360] Figure 25 is a schematic diagram of an example in which a UniRender process performs frequency division or frequency multiplication on a vertical synchronization signal, according to one embodiment of the present application.

[0361] As shown in Figure 25, if the refresh rates of the display areas corresponding to different DisplayRender subprocesses are different, UniRender may perform frequency division or frequency multiplication on the vertical sync signal (Vsync-UR) in order to transfer the vertical sync signal to the different DisplayRender subprocesses.

[0362] For example, suppose the refresh rate of display area 1 is 60Hz, the refresh rate of display area 2 is 30Hz, the DisplayRender subprocess corresponding to display area 1 is DisplayRender1, and the DisplayRender subprocess corresponding to display area 2 is DisplayRender2. In this case, after receiving or generating a 60Hz vertical sync signal (Vsync), the UniRender process performs frequency division to generate a 30Hz vertical sync signal (Vsync-UR) and a 60Hz vertical sync signal (Vsync-UR), transfers the 60Hz vertical sync signal (Vsync-UR) to the DisplayRender1 subprocess, and transfers the 30Hz vertical sync signal (Vsync-UR) to the DisplayRender2 subprocess.

[0363] It will be understood that frequency division or frequency multiplication is performed on the vertical synchronization signal (Vsync-UR), and as a result, the frequency at which the DisplayUniRender process receives the render tree or target render tree and the frequency at which the bitmap is generated will match the refresh frequency of the display area.

[0364] After receiving the vertical synchronization signal (Vsync-UR), the DisplayRender subprocess may begin reading the render tree, generating the target render tree, and generating the bitmap.

[0365] Note that the UniRender process receives display area information sent by the Display Manager service, determines the connection status of each display area, and may create or destroy a DisplayRender process corresponding to each display area according to the connection status of the display area.

[0366] For details on concepts such as UniRender and shared memory architecture, please refer to the explanation in step S904. Further details are not provided in this specification.

[0367] S2405: Generate a target render tree based on the acquired render tree, window control information, and display area information.

[0368] The DisplayRender subprocess merges one or more target render trees into a single target render tree and, when the application's layers are related to off-screen rendering logic, redirects off-screen rendering instructions for the application's render tree.

[0369] Note that the DisplayRender subprocess may obtain window control information and display area information within the display area from the UniRender process.

[0370] If necessary, in some embodiments of this application, the UniRender process may merge one or more render trees into a target render tree and then transfer the target render tree to a DisplayRender subprocess.

[0371] For details on concepts such as the process of generating the target render tree and transferring off-screen rendering instructions, please refer to the description of step S904. Further details are not provided in this specification.

[0372] S2406: Generates a bitmap based on the target render tree.

[0373] After generating the target render tree, the DisplayRender subprocess generates bitmaps based on the target render tree. After generating the bitmaps, the DisplayRender subprocess transfers the surface carrying the bitmaps to the UniRender process. The UniRender process then sends the contents of the surface to each display area via the DSS.

[0374] For details on the process of generating a bitmap based on the target render tree, please refer to the description of step S905. Further details are not provided in this specification.

[0375] Because the underlying graphics library calls are context-dependent, different DisplayRender subprocesses correspond to different contexts, and as a result, bitmaps can be generated in parallel for different display areas.

[0376] It will be understood that the UniRender process may generate bitmaps in parallel for each display area by creating a DisplayRender subprocess that corresponds one-to-one with each display area. In multi-display area scenarios, interface generation speed can be improved, frame freezes can be reduced, and the user experience can be improved.

[0377] Referring to the contents shown in Figures 26A and 26B, the following describes an example of a data flow process when an electronic device implements an interface generation method.

[0378] Figures 26A and 26B are schematic diagrams of an example of data flow when an electronic device implements an interface generation method according to one embodiment of this application.

[0379] As shown in Figure 26A, the electronic device has two display areas: display area 1 and display area 2. Display area 1 displays the contents of applications 1 and 3, while display area 2 displays the contents of applications 2, 3, and 4.

[0380] The UniRender process creates a DisplayRender1 subprocess corresponding to display area 1 and a DisplayRender2 subprocess corresponding to display area 2. After receiving render tree 1 for application 1, render tree 2 for application 2, render tree 3 for application 3, and render tree 4 for application 4, the UniRender process transfers render tree 1 and render tree 3 to the DisplayRender1 subprocess, and render tree 2, render tree 3, and render tree 4 to the DisplayRender2 subprocess. The UniRender process then transfers the relevant window control information and display area information to the DisplayRender subprocess.

[0381] The DisplayRender1 and DisplayRender2 subprocesses generate bitmaps separately based on the target render tree. The DisplayRender1 subprocess generates surface1 based on the display area information of display area 1, and the DisplayRender1 subprocess generates surface2 based on the display area information of display area 2. surface1 is used to carry bitmap1 generated by the DisplayRender1 subprocess, and surface2 is used to carry bitmap2 generated by the DisplayRender1 subprocess. The size of surface1 may be related to the size of display area 1, and the size of surface2 may be related to the size of display area 2.

[0382] In this embodiment of the present application, the content displayed by the application in different display areas may be the same or different. If the content displayed by the application in different display areas is the same, one render tree generated by the application is assigned to multiple DisplayRender subprocesses. If the content displayed by the application in different display areas is different, the application generates multiple different render trees and assigns the render trees to multiple DisplayRender subprocesses accordingly.

[0383] For example, in the scenario shown in Figure 26B, the sizes (e.g., resolution) of display area 1 and display area 2 are different, so the interfaces displayed by application 3 in display area 1 and display area 2 are different (e.g., the layout structure is different, but the content is the same). Therefore, the application generates render trees corresponding to the different display areas, which are render tree 31 and render tree 32, respectively. Render tree 31 is transferred to the DisplayRender1 subprocess, and render tree 32 is transferred to the DisplayRender2 subprocess.

[0384] The two threads may be configured for any DisplayRender subprocess, such as an I / O thread and a render thread. The I / O thread is responsible for receiving the render tree, while the render thread is responsible for generating the target render tree and creating a bitmap based on the target render tree.

[0385] (c) Method for generating interfaces in multi-device, multi-display area, and scenario environments. As shown in the scenarios in Figures 23A-1 and 23A-2 through 23C, the electronic device may display some or all of the on-screen content on the screen of the peer electronic device by means of projection or multi-screen coordination.

[0386] In a multi-device, multi-display-area scenario, after establishing a connection to a peer electronic device, the local electronic device may use the peer electronic device's screen as display area 2 and transfer the render trees of applications that need to be displayed in display area 2 to the peer electronic device. After receiving the render trees from the local electronic device, the peer electronic device generates a bitmap and merges the render trees of all applications displayed in display area 2 into a target render tree in order to send the bitmap for display.

[0387] Alternatively, the local electronic device may send the interface of the application displayed in display area 2 to the peer electronic device.

[0388] Figure 27 is a schematic diagram of another example of an interface generation method according to one embodiment of the present application.

[0389] As shown in Figure 27, the interface generation method provided in this embodiment of the present application includes the following steps:

[0390] S2701: Establishing a connection.

[0391] Electronic device 1 and electronic device 2 establish a communication connection using multiple methods such as Bluetooth, Wi-Fi, and HiLink. Electronic device 1 uses the screen of electronic device 2 as display area 2, and the screen of electronic device 1 is used as display area 1.

[0392] After determining that a connection has been established between electronic device 1 and electronic device 2, or after receiving projection requests, multi-screen coordination requests, etc., from the display manager service and window manager service, the UniRender process on electronic device 2 may request heap memory to store data sent by electronic device 1, such as bitmaps, used to render and generate render trees.

[0393] S2702: Determines the application displayed in display area 2.

[0394] The window manager service and display manager service on electronic device 1 determine which applications should be displayed in display area 2 and transfer the result to the UniRender1 process on electronic device 1. The UniRender1 process on electronic device 1 then obtains the render tree of the applications displayed in display area 2 via IPC communication.

[0395] For information on obtaining the application's render tree via IPC communication using the UniRender1 process, please refer to the above explanation in step S902. Further details are not provided in this specification.

[0396] S2703: Send the render tree of the application displayed in display area 2.

[0397] The UniRender1 process on electronic device 1 sends the render tree of the application displayed in display area 2 to electronic device 2.

[0398] UniRender1 may determine the start address and size of the shared memory and transfer the render tree to the heap memory created by the UniRender2 process on the electronic device 2 by using the communication connection in step S2701.

[0399] The data storage structure in heap memory may match the data storage structure in shared memory within UniRender1. For details regarding the data storage structure in heap memory and the read and write security of heap memory, please refer to the description in step S902. Further details are not provided herein.

[0400] It will be understood that by electronic device 1 sending the render tree to electronic device 2, the amount of data transmitted between electronic device 1 and electronic device 2 can be reduced, thereby reducing latency and improving interface generation speed.

[0401] S2704: Merge the render tree of the application displayed in display area 2 into the target render tree.

[0402] After receiving the vertical synchronization signal (Vsync-UR2), the UniRender2 process on electronic device 2 may merge the render tree of the application displayed in display area 2 into the target render tree, and this process may also perform the transfer of off-screen rendering instructions.

[0403] Electronic device 2 may merge the render tree sent by electronic device 1 and its own local render tree into the target render tree.

[0404] Figures 28A and 28B are schematic diagrams illustrating an example of merging a render tree into a target render tree according to one embodiment of the present application.

[0405] As shown in Figure 28A, applications 1 and 2 run on electronic device 1, the render tree generated by application 1 is render tree 1, and the render tree generated by application 2 is render tree 2. Application 3 runs on electronic device 2, and the render tree generated by application 3 is render tree 3.

[0406] The UniRender1 process on electronic device 1 determines that the interface of application 2 needs to be displayed in display area 2 and sends render tree 2 to the UniRender2 process on electronic device 2.

[0407] The UniRender1 process forwards off-screen rendering instructions separately to render tree 1 and render tree 2, and merges render tree 1 and render tree 2 into target render tree 1 to generate bitmap 1. The UniRender2 process forwards off-screen rendering instructions separately to render tree 2 and render tree 3, and merges render tree 2 and render tree 3 into target render tree 2 to generate bitmap 2.

[0408] Note that when the UniRender1 process sends data from render tree 2 to the UniRender2 process, it may send all the data from render tree 2 or the differential data from render tree 2 to the UniRender2 process. All the data from render tree 2 may be current properties, current displaylist, staging displaylist, or staging properties in shared memory, and the differential data from render tree 2 may be the draw instruction list and rendering properties of the render nodes of render tree 2 in the properties_dirty queue and displaylist_dirty queue.

[0409] As shown in Figure 28B, applications 1, 2, and 3 run on electronic device 1, and the render tree generated by application 1 is render tree 1, the render tree generated by application 2 is render tree 2, and the render tree generated by application 2 is render tree 3. Application 4 runs on electronic device 2, and the render tree generated by application 4 is render tree 4.

[0410] If the UniRender1 process on electronic device 1 determines that the interfaces of applications 2 and 3 need to be displayed in display area 2, the UniRender1 process sends render trees 2 and 3 to the UniRender2 process on electronic device 2, or, after render trees 2 and 3 are merged into target render tree 2, sends target render tree 2 to the UniRender2 process on electronic device 2.

[0411] The UniRender1 process separately redirects off-screen rendering instructions for render tree 1, render tree 2, and render tree 3, and then merges render tree 1, render tree 2, and render tree 3 into target render tree 1 to generate bitmap 1. The UniRender1 process redirects off-screen rendering instructions for render tree 2 and then merges render tree 2 into target render tree 2.

[0412] If necessary, in some embodiments of this application, the UniRender1 process may create multiple subprocesses, DisplayRender, to receive render trees separately from different applications. Specifically, as shown in Figure 28B, the UniRender1 process may create two subprocesses, DisplayRender:DisplayRender1 and DisplayRender2. The DisplayRender1 subprocess receives render tree 1, render tree 2, and render tree 3, and merges render tree 1, render tree 2, and render tree 3 into target render tree 1. The DisplayRender2 subprocess receives render tree 2 and render tree 3, merges render tree 2 and render tree 3 into target render tree 2, and sends target render tree 2 to the UniRender2 process.

[0413] The UniRender2 process forwards off-screen rendering instructions separately to render tree 2 and render tree 3, and then merges render tree 2 and render tree 3 into target render tree 2 to generate bitmap 2.

[0414] Note that in response to a vertical synchronization signal (Vsync-UR1) on electronic device 1, the UniRender1 process retrieves the render trees of one or more applications from shared memory and begins generating the target render tree. In response to a vertical synchronization signal (Vsync-UR2) on electronic device 2, UniRender2 retrieves the render trees of one or more applications from heap memory and / or shared memory and begins generating the target render tree.

[0415] If necessary, in some embodiments of this application, when the frequencies of the vertical synchronization signal (Vsync-UR1) on electronic device 1 and the vertical synchronization signal (Vsync-UR2) on electronic device 2 do not match, the UniRender on electronic device 1 may adjust the frequency of the vertical synchronization signal (Vsync-UR1) to be the same as the frequency of Vsync-UR2 via frequency division or frequency multiplication. Alternatively, the UniRender2 on electronic device 2 may adjust the frequency of the vertical synchronization signal (Vsync-UR2) to be the same as the frequency of the vertical synchronization signal (Vsync-UR1) via frequency division or frequency multiplication.

[0416] If necessary, in some embodiments of this application, multiple frequency vertical synchronization signals (Vsync-UR1) may be configured for the electronic device 1. For example, in the scenario shown in Figures 28A and 28B, the original frequency of the vertical synchronization signal (Vsync-UR) on electronic device 1 is 60 Hz, and the original frequency of the vertical synchronization signal (Vsync-UR2) on electronic device 2 is 90 Hz. After electronic device 1 decides to display the interface of application 2 on electronic device 2, the UniRender process of electronic device 1 may generate vertical synchronization signal (Vsync-UR1) frequencies of 60 Hz and 90 Hz, respectively, so that application 2 generates a render tree at a frequency of 90 Hz. In this way, the UniRender1 process may transfer the render tree to the UniRender2 process at a frequency of 90 Hz. Application 1 may continue to generate a render tree at a frequency of 60 Hz.

[0417] On any electronic device, the vertical synchronization signal (Vsync-UR) received or generated by UniRender may be the same as the vertical synchronization signal (Vsync-APP) received by the application.

[0418] S2705: Generates a bitmap based on the target render tree.

[0419] For specific details, please refer to the explanation in step S905. Further details will not be provided in this specification.

[0420] If necessary, in some embodiments of this application, the UniRender1 process of electronic device 1 may first merge one or more render trees in the display area 2 into a target render tree, generate a bitmap based on the target render tree, and transfer the bitmap to the UniRender2 process of electronic device 2.

[0421] When electronic device 2 displays the interface of application 1 on electronic device 1, the user may tap the interface of application 1 on electronic device 2 using other interaction methods such as tapping, and electronic device 2 may send the location of the tap operation performed by the user within the interface of application 1 to application 1 on electronic device 1, thereby enabling application 1 to respond correctly to the user interaction. Alternatively, since the interface of application 1 on electronic device 2 is generated by the UniRender2 process on electronic device 2 based on a render tree, and the render tree contains view position information, electronic device 2 may determine the view tapped by the user and directly send a view tap event to application 1 on electronic device 1, thereby enabling application 1 to respond correctly to the user interaction.

[0422] (3) Finally, the hardware architecture and software architecture of the electronic device provided in the embodiments of this application will be described.

[0423] Figure 29 is a schematic diagram of the hardware structure of an electronic device according to one embodiment of this application.

[0424] The embodiments will be described in detail below, using an electronic device as an example. It should be understood that the electronic device may have more or fewer components than those shown in the figure, may be a combination of two or more components, or may have different component configurations. The components shown in the figure may be implemented as hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0425] The electronic device may include a processor 110, an external memory interface 120, internal memory 121, a universal serial bus (USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, a subscriber identification module (SIM) card interface 195, and the like. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, an optical proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, and a bone conduction sensor 180M, among others.

[0426] It will be understood that the structures shown in this embodiment of the present invention do not impose any particular limitations on electronic devices. In some other embodiments of this application, the electronic devices may include more or fewer components than those shown in the figures, or some components may be combined, or some components may be divided, or different arrangements of components may be present. The components shown in the figures may be implemented by hardware, software, or a combination of software and hardware.

[0427] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent components or may be integrated into one or more processors.

[0428] The controller may be the central hub and command center of an electronic device. The controller may generate operation control signals based on instruction operation codes and time-series signals to perform instruction fetching and instruction execution control.

[0429] Further memory may be located within the processor 110 to store instructions and data. In some embodiments, the memory within the processor 110 is a cache. The memory may store instructions or data that have been used by the processor 110, or that are used periodically. When the processor 110 needs to use the instructions or data again, the instructions or data may be retrieved directly from memory. This avoids repeated access, reduces latency in the processor 110, and consequently improves system efficiency.

[0430] In some embodiments, the processor 110 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0431] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled separately to the touch sensor 180K, charger, flash, and camera 193, etc., via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K by using the I2C bus interface, and as a result, the processor 110 communicates with the touch sensor 180K by using the I2C interface to implement the touch functionality of the electronic device.

[0432] The I2S interface may be used for audio communication. The PCM interface may perform audio communication and be configured to sample, quantize, and encode analog signals. The UART interface is a universal serial data bus and is used for asynchronous communication.

[0433] The MIPI interface may be configured to connect the processor 110 to peripheral components such as a display 194 or a camera 193. The MIPI interface includes camera serial interface (CSI) and display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate with each other by using the CSI interface to implement the imaging function of the electronic device. The processor 110 communicates with the display 194 by using the DSI interface to implement the display function of the electronic device. The GPIO interface may be configured by using software. The GPIO interface may be configured as control signals or as data signals. In some embodiments, the GPIO interface may be configured to connect the processor 110 to the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. Alternatively, the GPIO interface may be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc. The SIM interface may be configured to communicate with the SIM card interface 195 in order to implement the function of sending data to or reading data from the SIM card. The USB interface 130 is an interface compliant with the USB standard specification, and may specifically be a Mini USB interface, Micro USB interface, USB Type C interface, etc.

[0434] It should be understood that the intermodal interface connection relationships shown in this embodiment of the present invention are merely illustrative examples and are not limited to the structure of the electronic device. In some other embodiments of this application, the electronic device may use an interface connection method different from that of the embodiments described above, or a combination of multiple interface connection methods.

[0435] The charge management module 140 is configured to receive a charge input from the charger. The power management module 141 is configured to connect the battery 142, the charge management module 140, and the processor 110.

[0436] The wireless communication function of an electronic device may be realized by using antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, baseband processor, etc. Antennas 1 and 2 are configured to transmit and receive electromagnetic wave signals. Each antenna in the electronic device may be configured to cover one or more communication bands. Different antennas may be multiplexed to improve antenna utilization. Mobile communication module 150 may provide a solution for wireless communication, including 2G / 3G / 4G / 5G, etc., applied to the electronic device. Mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. Mobile communication module 150 may receive electromagnetic waves using antenna 1, perform processing such as filtering and amplification on the received electromagnetic waves, and send the processed electromagnetic waves to the modem processor for demodulation. Mobile communication module 150 may further amplify the signal modulated by the modem processor and convert this signal into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be located in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be located in the same components as at least some of the modules of the processor 110.

[0437] The modem processor may include a modulator and a demodulator. The modulator is configured to modulate the low-frequency baseband signal to be transmitted into a medium / high-frequency signal. The demodulator is configured to demodulate the received electromagnetic signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs an acoustic signal by using an audio device (not limited to speaker 170A and receiver 170B, etc.) or displays an image or video by using display 194. In some embodiments, the modem processor may be a separate device. In some other embodiments, the modem processor may be separate from processor 110 and may be located in the same device as the mobile communication module 150 or other functional modules.

[0438] The wireless communication module 160 may provide a solution for wireless communication applicable to electronic devices, including wireless local area networks (WLANs) (e.g., wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSS), frequency modulation (FM), near-field communication (NFC), and infrared (IR) technology. The wireless communication module 160 may be one or more components integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering on the electromagnetic wave signal, and sends the processed signal to processor 110. The wireless communication module 160 may further receive a signal to be transmitted from processor 110, perform frequency modulation and amplification on the signal to be transmitted, and convert the signal to be transmitted into electromagnetic waves for radiation via antenna 2.

[0439] In some embodiments, in an electronic device, antenna 1 is coupled to a mobile communication module 150, and antenna 2 is coupled to a wireless communication module 160, so that the electronic device can communicate with a network and other devices according to wireless communication technology. Wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA®), time-division code division multiple access (TD-SCDMA), long-term evolution (LTE), BT, GNSS, WLAN, NFC, FM, IR technology, and the like. GNSS may include the Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Beidou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0440] The electronic device implements display functionality by using a GPU, a display 194, an application processor, and the like. The GPU is a microprocessor for image processing and is connected to the display 194 and the application processor. The GPU is configured to perform mathematical and geometric calculations and to render graphics. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0441] The display 194 is configured to display images, videos, and the like. The display 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), and the like. In some embodiments, the electronic device may include one or N displays 194, where N is a positive integer greater than 1.

[0442] The electronic device may implement the shooting function and acquire real-time video data by using an ISP, camera 193, video codec, GPU, display 194, application processor, etc.

[0443] The ISP is configured to process data fed back by the camera 193. For example, during shooting, the shutter is pressed, light is sent through the lens to the camera's photodetector, the light signal is converted into an electrical signal, the camera's photodetector sends the electrical signal to the ISP for processing, and the electrical signal is converted into a visible image. The ISP may further perform algorithmic optimizations for image noise and brightness. The ISP may further optimize parameters such as exposure and color temperature for the shooting scenario. In some embodiments, the ISP may be located on the camera 193.

[0444] The camera 193 is configured to capture still images or video. An optical image of an object is generated using a lens and projected onto a photodetector. The photodetector may be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photodetector converts the optical signal into an electrical signal and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into a standard image signal in a format such as RGB or YUV. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1.

[0445] A digital signal processor is configured to process digital signals. In addition to processing digital image signals, a digital signal processor may further process other digital signals. For example, when an electronic device selects a frequency, the digital signal processor may be configured to perform a Fourier transform or similar operation on the frequency energy.

[0446] A video codec is configured to compress or decompress digital video. An electronic device may support one or more video codecs. In this way, an electronic device may play or record video in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, and MPEG 4.

[0447] An NPU is a neural network (NN) computing processor that can rapidly process input information and continuously perform further self-learning by referencing the structure of biological neural networks, for example, by referencing the transmission modes between nerve cells in the human brain. NPUs may be used to realize intelligent cognition in electronic devices, such as image recognition, face recognition, speech recognition, and text comprehension.

[0448] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0449] Random access memory may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, where, for example, fifth-generation DDR SDRAM is commonly referred to as DDR5 SDRAM), and others.

[0450] Non-volatile memory may include disk storage devices and flash memory.

[0451] In the embodiments of this application, non-real-time video may be stored in non-volatile memory.

[0452] According to the operating principle, flash memory may include NOR FLASH, NAND FLASH, and 3D NAND FLASH. Depending on the number of potential levels of the memory unit, flash memory may include single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. According to the memory specifications, flash memory may include universal flash storage (UFS), embedded multimedia card (eMMC), etc.

[0453] Random access memory may be directly read from and written to by the processor 110, may be configured to store executable programs (e.g., machine instructions) of the operating system or other running programs, and may be further configured to store user and application data.

[0454] Non-volatile memory may also store executable program, user and application data, etc., and may be pre-loaded into random-access memory for direct reading and writing by the processor 110.

[0455] The external memory interface 120 may be configured to connect to external non-volatile memory in order to expand the storage capacity of the electronic device. The external non-volatile memory communicates with the processor 110 by using the external memory interface 120 in order to implement data storage functions. Files such as music and videos may be stored in the external non-volatile memory.

[0456] The electronic device may implement audio functions such as music playback and recording by using an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, an application processor, and the like.

[0457] The audio module 170 is configured to convert digital audio information into an analog audio signal for output, and is also configured to convert an analog audio input into a digital audio signal. The speaker 170A, also called the "loudspeaker," is configured to convert an audio electrical signal into an acoustic signal. The receiver 170B, also called the "earphone," is configured to convert an audio electrical signal into an acoustic signal. The microphone 170C, also called the "mike" or "mic," is configured to convert an acoustic signal into an electrical signal.

[0458] The headset jack 170D is configured to connect to a wired headset. The headset jack 170D may be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0459] The pressure sensor 180A is configured to sense a pressure signal and may convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A may be located on the display 194. The gyro sensor 180B may be configured to determine the motion gesture of the electronic device. The barometric pressure sensor 180C is configured to measure atmospheric pressure. The magnetic sensor 180D includes a Hall sensor. The accelerometer 180E may detect the acceleration values ​​of the electronic device in all directions (usually three axes). The distance sensor 180F is configured to measure distance. The optical proximity sensor 180G may include a photodetector such as a light-emitting diode (LED) and a photodiode. The ambient light sensor 180L is configured to sense the brightness of ambient light. The fingerprint sensor 180H is configured to take a fingerprint. Based on the characteristics of the taken fingerprint, the electronic device may implement fingerprint-based unlocking, application-locked access, fingerprint-based photography, and fingerprint-based phone answering, etc. The temperature sensor 180J is configured to detect temperature.

[0460] The touch sensor 180K is also referred to as a touch panel. The touch sensor 180K may be located on the display 194. The touch sensor 180K and the display 194 form a touchscreen, also referred to as a “touchscreen”. The touch sensor 180K is configured to detect touch operations performed on or near the touch sensor. The touch sensor may forward the detected touch operations to an application processor to determine the type of touch event. Visual outputs related to touch operations may be provided by using the display 194. In some other embodiments, the touch sensor 180K may be located on the surface of the electronic device at a different location from the display 194.

[0461] Button 190 includes a power button, volume buttons, etc. Button 190 may be a mechanical button or a touch button. The electronic device may receive button input and generate button signal inputs related to user settings and function control of the electronic device. Motor 191 may generate vibration prompts. Indicator 192 may be an indicator light and may be used to indicate the charging status or power change, or to indicate messages, missed calls, notifications, etc. SIM card interface 195 is configured to connect to a SIM card. The electronic device interacts with the network by using the SIM card to implement functions such as conversation and data communication.

[0462] Figure 30 is a schematic diagram of an example of the software structure of an electronic device according to one embodiment of this application.

[0463] In a layered architecture, software is divided into several layers, each with a distinct role and task. These layers communicate with each other through software interfaces. In some embodiments, the system is divided into four layers, from top to bottom: the application layer, the application framework layer, the system libraries, and the kernel layer.

[0464] The application layer may include a series of application packages. As shown in Figure 20, the application packages may include application programs (also referred to as applications) such as camera, gallery, calendar, phone, maps, navigation, WLAN, Bluetooth, music, video, and messaging.

[0465] The application framework layer provides an application programming interface (API) and programming framework for applications within the application layer. The application framework layer includes several predefined functions.

[0466] As shown in Figure 30, the application framework layer may include a window manager service, a display manager service, a content provider, a view system, a phone manager, a resource manager, a notification manager, a Local Profile Assistant (LPA), and the like.

[0467] The window manager service is responsible for starting, adding, and removing windows; determining which applications are displayed in windows; creating, destroying, and modifying the properties of application layers; determining whether a status bar is present; locking the screen; and capturing the screen.

[0468] The display manager service can retrieve the number and size of display areas and is responsible for starting, adding, and deleting display areas.

[0469] Content providers are configured to store and retrieve data, and to allow applications to access that data. This data may include videos, images, audio, outgoing and incoming calls, browsing history and bookmarks, and address books.

[0470] The telephone manager is configured to provide communication functions for electronic devices, such as managing call status (including answering or rejecting calls).

[0471] The resource manager provides various resources for the application, such as localized text, icons, pictures, layout files, and video files.

[0472] The notification manager may be configured to allow applications to display notification information in the status bar and to forward notification-type messages. This information may disappear automatically after a short delay without user interaction. For example, the notification manager may be configured to notify download completion, message reminders, etc. Alternatively, the notification manager may appear as notifications in the form of graphs or scrollbar text in the status bar at the top of the system, for example, notifications from applications running in the background, or notifications that appear on the screen in the form of a dialog interface. For example, the status bar may display text information, a prompt tone may be emitted, the electronic device may vibrate, and an indicator may flash.

[0473] A view system includes visual controls such as text display controls or picture display controls. A view system may be configured to build an application. A display interface may include one or more views. For example, a display interface containing an SMS notification icon may include a text display view and a picture display view.

[0474] The view system further includes a UniRender, which may receive the render trees of one or more applications. The UniRender may synchronize layer information, such as layer creation, layer destruction, and property changes, by using a window manager service. The UniRender may also synchronize display area information, such as screen size, from a display manager service.

[0475] Where necessary, in some embodiments of this application, the view system further includes a SurfaceFlinger. On an electronic device configured with a trust list, when an application does not belong to the trust list, the application's UI thread generates a render tree, the application's render thread generates a bitmap, and the bitmap is then sent to the SurfaceFlinger for layer compositing.

[0476] In some embodiments of this application, where necessary, when applications in the trusted list and applications not in the trusted list are displayed in the display area, UniRender is responsible for generating bitmaps for the applications in the trusted list. After generating the bitmaps, UniRender transfers the bitmaps to SurfaceFlinger. SurfaceFlinger then performs layer compositing on the bitmaps and other applications not in the trusted list to generate the bitmaps to be sent for display.

[0477] The runtime includes kernel libraries and virtual machines. The runtime is responsible for scheduling and managing the operating system.

[0478] The kernel library consists of two parts: the functions that need to be called using the Java language, and the kernel library itself.

[0479] The application layer and application framework layer run on a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is configured to perform functions such as object lifecycle management, stack management, thread management, security and anomaly management, and garbage collection.

[0480] The system library may include multiple functional modules, such as a surface manager, media libraries, and graphics processing libraries, the graphics processing libraries of which may include a 3D graphics processing library (e.g., OpenGL ES), a 2D graphics engine (e.g., SGL), and so on.

[0481] The surface manager manages the display subsystem and is configured to provide fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0482] The media library supports playback and recording of multiple commonly used audio and video formats, as well as still image files. The media library can support multiple audio and video encoding formats, including MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0483] A 3D graphics processing library is configured to perform 3D graphics drawing, image rendering, layer compositing, layer processing, and other similar functions. A 2D graphics engine is a rendering engine for 2D drawing.

[0484] The kernel layer is the layer between hardware and software. The kernel layer includes at least a display driver, camera driver, audio driver, sensor driver, and virtual card driver.

[0485] In embodiments of this application, one or more applications in the application layer transfer the render tree generated by the application's UI thread to the UniRender process of the view system. The UniRender process obtains window control information and display area information from the window manager service and display manager service, and then merges the render tree of the application in the display area into a target render tree. After generating the target render tree, the UniRender process generates bitmaps by calling the layer processing library to execute DrawOP operations in the target render tree's drawing instruction list. UniRender transfers the generated bitmaps to the display driver for display.

[0486] Depending on the context, the term “when” as used in the embodiments described above may be interpreted as “in the case of,” “after,” “depending on the decision,” or “depending on the detection.” Similarly, depending on the context, the phrases “when a decision is made” or “when (the stated state or event) is detected” may be interpreted as “when a decision is made,” “depending on the decision,” “when (the stated state or event) is detected,” or “depending on the detection (of the stated state or event).”

[0487] All or part of the embodiments described above may be implemented by software, hardware, firmware, or any combination thereof. When software is used to implement an embodiment, all or part of the embodiment may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded into a computer and executed, all or part of the procedures or functions according to the embodiments of this application are generated. The computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired method (e.g., coaxial cable, optical fiber, digital subscriber line) or by a wireless method (e.g., infrared, radio waves, microwave). The computer-readable storage medium may be any available medium accessible by a computer, or it may be a data storage device such as a server or data center that integrates one or more available media. The usable media may include magnetic media (e.g., floppy disks, hard disks, or magnetic tapes), optical media (e.g., DVDs), and semiconductor media (e.g., solid-state drives).

[0488] Those skilled in the art will understand that all or part of the steps of the method in the embodiments described above may be performed by a computer program that instructs the relevant hardware. The program may be stored in a computer-readable storage medium. When the program is executed, it may include the steps of the embodiments of the method described above. The aforementioned storage medium includes any medium capable of storing program code, such as ROM, random-access memory (RAM), magnetic disks, and optical disks. [Explanation of Symbols]

[0489] 110 processors 120 External memory interface 121 Internal Memory 130 Universal Serial Bus Interface 140 Charging Management Module 141 Power Management Module 142 batteries 150 Mobile Communication Modules 160 Wireless Communication Modules 170 Audio Modules 170A speaker 170B receiver 170C Microphone 170D headset jack 180 Sensor Modules 180A pressure sensor 180B Gyro Sensor 180°C barometric pressure sensor 180D Magnetic Sensor 180E Accelerometer 180F Distance Sensor 180G Optical Proximity Sensor 180H Fingerprint Sensor 180J Temperature Sensor 180K touch sensor 180L Ambient Light Sensor 180M Bone Conduction Sensor 190 buttons 191 Motor 192 Indicators 193 Camera 194 displays 195 Subscriber Identification Module Card Interface

Claims

1. An interface generation method applied to a first electronic device, wherein the first electronic device determines that the content displayed in a first display area includes an interface for a first process and an interface for a second process, and the method is A step of generating a first render tree by the first process, wherein the first render tree is used to draw the interface of the first process, A step of generating a second render tree by the second process, wherein the second render tree is used to draw the interface of the second process, A third process generates a first target interface based on the first render tree and the second render tree, wherein the first target interface includes the interface of the first process and the interface of the second process, and the first target interface is displayed in the first display area. The third process includes the step of deleting render nodes in a first target render tree based on the Z-order of the first render tree and the Z-order of the second render tree, wherein the deleted render nodes correspond to a completely occluded view. An interface generation method, including the above.

2. The third process generates a first target interface based on the first render tree and the second render tree, The third process includes the step of generating a first target render tree based on the first render tree and the second render tree, The third process includes the steps of generating the first target interface based on the first target render tree and The method according to claim 1, which specifically includes the following.

3. The third process generates a first target render tree based on the first render tree and the second render tree, The third process involves creating a root render node as the root node of the first target render tree, and using the first render tree and the second render tree as child nodes of the root render node. The method according to claim 2, which specifically includes the following.

4. The third process includes the step of deleting drawing operations in the first target render tree based on the Z-order of the first render tree and the Z-order of the second render tree, wherein the deleted drawing operations correspond to completely occluded graphics. The method according to claim 3, further comprising:

5. In a process in which the third process generates the first target interface based on the first render tree and the second render tree, the third process performs a merge or batch on a first drawing operation and a second drawing operation, wherein the first drawing operation belongs to the first render tree and the second drawing operation belongs to the second render tree. The method according to claim 1, further comprising:

6. The third process comprises the step of determining the off-screen rendering logic for the interface of the first process, wherein the off-screen rendering logic includes at least one of window rounding, color conversion, rotation, and scaling. The third process adds offscreen rendering properties to the rendering properties of the first render tree based on the offscreen rendering logic of the interface of the first process, wherein the offscreen rendering properties include at least one of rounding properties, color properties, rotation properties, and scaling properties. It further includes, The off-screen rendering properties correspond one-to-one with the off-screen rendering logic, and the off-screen rendering properties are used to modify drawing operations in the process in which the third process generates the first target interface based on the first render tree and the second render tree in order to implement the off-screen rendering logic. The method according to claim 1.

7. The reference coordinate system of the first render tree is the first coordinate system, and the reference coordinate system corresponding to the first display area is the second coordinate system, and when the first coordinate system is different from the second coordinate system, The third process includes the steps of determining a first parameter based on the first and second coordinate systems and adding the first parameter to the rendering properties of the first render tree, In the process in which the third process generates the first target interface based on the first render tree and the second render tree, the third process modifies the reference coordinate system of the first drawing operation based on the first parameters, wherein the first drawing operation belongs to the first render tree, and The method according to claim 1, further comprising:

8. An interface generation method applied to a second electronic device, wherein the second electronic device determines that the content displayed in a first display area includes an interface for a first process and an interface for a second process, and the method is A step of receiving a first render tree and a second render tree by a third process operating on the second electronic device, wherein the first render tree is generated by the first process operating on the first electronic device and is used to draw the interface of the first process, and the second render tree is generated by the second process operating on the second electronic device and is used to draw the interface of the second process, A third process generates a target interface based on the first render tree and the second render tree, wherein the target interface includes the interface of the first process and the interface of the second process, and the target interface is displayed in the first display area. The third process includes the step of deleting render nodes in a first target render tree based on the Z-order of the first render tree and the Z-order of the second render tree, wherein the deleted render nodes correspond to a completely occluded view. An interface generation method, including the above.

9. An electronic device comprising one or more processors and memory, The memory is coupled to one or more processors, the memory is configured to store computer program code, the computer program code includes computer instructions, the one or more processors call the computer instructions, and as a result the electronic device performs the method according to any one of claims 1 to 8.

10. A chip system for application to an electronic device, wherein the chip system comprises one or more processors, the processors are configured to call computer instructions, and as a result, the electronic device performs the method according to any one of claims 1 to 8.

11. A computer-readable storage medium containing instructions, wherein when the instructions are executed on an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 8.

12. A computer program comprising computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, the method according to any one of claims 1 to 8 is realized.

Images