Interface generation method and electronic device

Abstract

The application provides an interface generation method and an electronic device, and relates to the technical field of electronics. The interface generation method provided in the application generates interfaces for different display regions in parallel through multiple processes, thereby improving the interface generation rate of the electronic device under multiple display regions, avoiding interface frame dropping, and improving the user experience.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and in particular to interface generation methods and electronic devices. Background Technology

[0002] With the development of electronic technology, more and more electronic devices are becoming part of users' daily lives. Furthermore, as the resolution, size, and other parameters of electronic device screens increase, the amount of content that can be displayed on these devices also increases.

[0003] However, before displaying an application's interface, electronic devices need to expend computing and storage resources to generate the application's interface, increasing the device's power consumption. Furthermore, when there are multiple applications or windows on the screen, the device needs to expend even more computing and storage resources to render and generate the interfaces for each application or window. Summary of the Invention

[0004] This application provides an interface generation method and an electronic device. The interface generation method provided in this application improves the interface generation rate of electronic devices with multiple display areas by drawing and generating interfaces for different display areas in parallel through multiple processes, avoiding frame drops and enhancing the user experience.

[0005] In a first aspect, embodiments of this application provide an interface generation method applied to an electronic device having at least two display areas, including a first display area and a second display area, comprising: a third process receiving a first rendering tree generated by a first process, the first rendering tree being used to draw the interface of the first process to be displayed on the first display area; the third process generating a first interface based on the first rendering tree, the first interface including the interface of the first process, the first interface being used to display on the first display area; a fourth process receiving a second rendering tree generated by a second process, the second rendering tree being used to draw the interface of the second process to be displayed on the first display area; the fourth process generating a second interface based on the second rendering tree, the second interface including the interface of the second process, the second interface being used to display on the second display area.

[0006] In the above embodiments, by drawing and generating interfaces for different display areas in parallel through multiple processes, the interface generation rate of electronic devices under multiple display areas is improved, frame drops are avoided, and the user experience is enhanced.

[0007] In conjunction with some embodiments of the first aspect, in some embodiments, before the third process receives the first rendering tree generated by the first process, and before the fourth process receives the second rendering tree generated by the second process, the method further includes: a fifth process determining that the at least two display areas include the first display area and the second display area; the fifth process generating the third process and the fourth process, wherein the third process and the fourth process are child processes of the third process.

[0008] In the above embodiment, the fifth process creates a third process corresponding to the first display area and a fourth process corresponding to the second display area; the third process and the fourth process are child processes of the fifth process, and the fifth process is responsible for managing the life cycle of the third process and the fourth process, thereby achieving unified scheduling management, avoiding frame drops in the interface, and improving the user experience.

[0009] In conjunction with some embodiments of the first aspect, in some embodiments, the third process receives the first rendering tree generated by the first process, specifically including: the third process receives the first rendering tree based on inter-process communication, the inter-process communication including at least one of Binder, interface definition language AIDL, shared memory, and socket.

[0010] In the above embodiments, cross-process communication (CPC) enables the transfer of rendering tree information between different processes, allowing the third process to obtain the rendering tree of the first process, and the fourth process to obtain the rendering tree of the second process. Furthermore, CPC does not block the UI thread, avoiding the UI thread blocking caused by the slow rendering tree synchronization process in native Android workflows, thus improving the user experience.

[0011] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes: the fifth process sending a first vertical synchronization signal to the first process at a first frequency, the first vertical synchronization signal being used to instruct the first process to generate the first rendering tree, the first frequency being the refresh frequency of the first display area; and the third process receiving the first rendering tree at the first frequency.

[0012] In the above embodiments, the fifth process can adjust the rate at which the first process generates the rendering tree and the rate at which the third process receives the rendering tree based on the refresh rate of the first display area, thereby ensuring that the frequency at which the application generates the rendering tree is equal to the frequency at which the application's interface is generated.

[0013] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes: the fifth process sending a second vertical synchronization signal to the second process at a second frequency, the second vertical synchronization signal being used to instruct the second process to generate the second rendering tree, the second frequency being the refresh frequency of the second display area; and the fourth process receiving the second rendering tree at the second frequency, the second frequency being different from the first frequency.

[0014] In the above embodiments, the fifth process can adjust the rate at which the second process generates the rendering tree and the rate at which the fourth process receives the rendering tree based on the refresh rate of the second display area, thereby ensuring that the frequency at which the application generates the rendering tree and the frequency at which the application's interface is generated are equal. Furthermore, the fifth process can adjust the rendering tree generation rate of different applications based on the refresh rate of different display areas, thereby achieving frame rate adaptation, reducing the power consumption of electronic devices, and improving the user experience.

[0015] In conjunction with some embodiments of the first aspect, in some embodiments, the third process generates a first interface based on the first rendering tree and the sixth rendering tree, specifically including: the third process creates a root rendering node as the root node of the first target rendering tree, and uses the first rendering tree and the sixth rendering tree as child nodes of the root rendering node; the third process generates the first interface based on the first target rendering tree.

[0016] In the above embodiments, the interface displayed on the first display area includes the interfaces of multiple applications. The third process merges the rendering trees of the multiple applications into a target rendering tree and generates the interface based on the target rendering tree. This can reduce the number of renderings by rendering only once and without participating in layer compositing, thereby reducing the power consumption of the electronic device.

[0017] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes: the third process deleting a rendering node in the first target rendering tree based on the Z-order of the first rendering tree and the sixth rendering tree, wherein the deleted rendering node corresponds to a view that is completely obscured.

[0018] In the above embodiments, the third process can optimize the parameters in the target rendering tree, for example, by deleting the rendering nodes corresponding to views that are completely obscured on the interface, thereby reducing over-drawing.

[0019] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes: the third process deleting drawing operations in the first target rendering tree based on the Z-order of the first rendering tree and the sixth rendering tree, wherein the deleted drawing operations correspond to graphics that are completely occluded.

[0020] In the above embodiments, the third process can optimize the parameters in the target rendering tree, for example, by deleting the drawing operations corresponding to graphics that are completely obscured on the interface, thereby reducing over-drawing.

[0021] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes: during the process of the third process generating the first interface based on the first rendering tree and the sixth rendering tree, the third process performs merging or batching of the first drawing operation and the second drawing operation, wherein the first drawing operation belongs to the first rendering tree and the second drawing operation belongs to the sixth rendering tree.

[0022] In the above embodiments, the third process can merge or batch the drawing operations in the target rendering tree, thereby improving the speed of interface generation, reducing lag, reducing the power consumption of electronic devices, and thus improving the user experience.

[0023] Secondly, embodiments of this application provide an electronic device, the electronic device comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors calling the computer instructions to cause the electronic device to execute: a third process receiving a first rendering tree generated by a first process, the first rendering tree being used to draw the interface of the first process to be displayed on a first display area; the third process generating a first interface based on the first rendering tree, the first interface including the interface of the first process, the first interface being used to display on the first display area; a fourth process receiving a second rendering tree generated by a second process, the second rendering tree being used to draw the interface of the second process to be displayed on the first display area; the fourth process generating a second interface based on the second rendering tree, the second interface including the interface of the second process, the second interface being used to display on a second display area.

[0024] In the above embodiments, by drawing and generating interfaces for different display areas in parallel through multiple processes, the interface generation rate of electronic devices under multiple display areas is improved, frame drops are avoided, and the user experience is enhanced.

[0025] In conjunction with some embodiments of the second aspect, in some embodiments, the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute: a fifth process determines that the at least two display areas include the first display area and the second display area; the fifth process generates the third process and the fourth process, the third process and the fourth process being child processes of the third process.

[0026] In conjunction with some embodiments of the second aspect, in some embodiments, the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to execute: the third process receives the first rendering tree based on inter-process communication, the inter-process communication including at least one of Binder, Interface Definition Language AIDL, shared memory, and Socket.

[0027] In conjunction with some embodiments of the second aspect, in some embodiments, the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute: the fifth process sends a first vertical synchronization signal to the first process at a first frequency, the first vertical synchronization signal being used to instruct the first process to generate the first rendering tree, the first frequency being the refresh frequency of the first display area; and the third process receives the first rendering tree at the first frequency.

[0028] In conjunction with some embodiments of the second aspect, in some embodiments, the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute: the fifth process sends a second vertical synchronization signal to the second process at a second frequency, the second vertical synchronization signal being used to instruct the second process to generate the second rendering tree, the second frequency being the refresh frequency of the second display area; the fourth process receives the second rendering tree at the second frequency, the second frequency being different from the first frequency.

[0029] In conjunction with some embodiments of the second aspect, in some embodiments, the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to perform: the third process creates a root rendering node as the root node of a first target rendering tree, and uses the first rendering tree and the sixth rendering tree as child nodes of the root rendering node; the third process generates the first interface based on the first target rendering tree.

[0030] In conjunction with some embodiments of the second aspect, in some embodiments, the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute: the third process, based on the Z-order of the first and sixth rendering trees, deletes a rendering node in the first target rendering tree, the deleted rendering node corresponding to a completely obscured view.

[0031] In conjunction with some embodiments of the second aspect, in some embodiments, the one or more processors are further configured to invoke the computer instructions to cause the electronic device to perform: the third process, based on the Z-order of the first and sixth rendering trees, deletes drawing operations in the first target rendering tree, the deleted drawing operations corresponding to completely occluded graphics.

[0032] In conjunction with some embodiments of the second aspect, in some embodiments, the one or more processors are further configured to invoke the computer instructions to cause the electronic device to perform: during the process of the third process generating the first interface based on the first rendering tree and the sixth rendering tree, the third process performs a merging or batching of the first drawing operation and the second drawing operation, wherein the first drawing operation belongs to the first rendering tree and the second drawing operation belongs to the sixth rendering tree.

[0033] Thirdly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0034] Fourthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0035] Fifthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0036] It is understood that the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer program product provided in the fourth aspect, and the computer storage medium provided in the fifth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0037] Figure 1A , Figure 1B This is an exemplary schematic diagram of an electronic device interface provided in an embodiment of this application.

[0038] Figure 2 An exemplary schematic diagram illustrating the generation of bitmaps by the application provided in this application embodiment.

[0039] Figure 3 This is an exemplary schematic diagram of a rendering tree provided in an embodiment of this application.

[0040] Figure 4 This is an exemplary schematic diagram of a bitmap used as a layer in layer composition, as provided in an embodiment of this application.

[0041] Figure 5 This is an exemplary schematic diagram of layer composition provided in an embodiment of this application.

[0042] Figure 6 This is another exemplary schematic diagram illustrating layer composition using SurfaceFlinger provided in this application embodiment.

[0043] Figure 7A , Figure 7B This is another exemplary schematic diagram of layer composition provided in the embodiments of this application.

[0044] Figure 8 This is an exemplary schematic diagram of the system architecture of the interface generation method provided in the embodiments of this application.

[0045] Figure 9 This is an exemplary schematic diagram of the interface generation method provided in the embodiments of this application.

[0046] Figure 10 This is an exemplary schematic diagram of the architecture of UniRender provided in the embodiments of this application.

[0047] Figure 11 This is an exemplary schematic diagram of a shared memory transfer rendering tree provided in an embodiment of this application.

[0048] Figure 12A , Figure 12B This is an exemplary schematic diagram of the storage structure of the rendering tree in shared memory provided in an embodiment of this application.

[0049] Figure 13 This is an exemplary schematic diagram of the architecture of UniRender provided in the embodiments of this application.

[0050] Figure 14 This is an exemplary schematic diagram of an application providing an embodiment of this application writing a render tree into shared memory.

[0051] Figure 15 Another exemplary schematic diagram illustrating the application provided in this application writing a render tree into shared memory.

[0052] Figures 16A-16C This is an exemplary schematic diagram of a scenario that triggers off-screen rendering drawing logic according to an embodiment of this application.

[0053] Figure 17A , Figure 17B This is an exemplary schematic diagram of UniRender triggering the off-screen rendering instruction to move forward, provided for an embodiment of this application.

[0054] Figure 18 This is an exemplary schematic diagram illustrating the modification of rendering node attributes in the rendering tree by UniRender, as provided in an embodiment of this application.

[0055] Figure 19 This is an exemplary schematic diagram illustrating how multiple off-screen rendering instructions are processed and then combined into a single target rendering tree, as provided in an embodiment of this application.

[0056] Figure 20 This is an exemplary schematic diagram illustrating how multiple off-screen rendering instructions are processed and then combined into a single target rendering tree, as provided in an embodiment of this application.

[0057] Figure 21 This is an exemplary schematic diagram of setStaticMatrix() provided in an embodiment of this application.

[0058] Figures 22A-22C This is an exemplary schematic diagram of a multi-display area scenario provided in an embodiment of this application.

[0059] Figures 23A-23C This is another exemplary schematic diagram of a multi-display-area scenario provided in an embodiment of this application.

[0060] Figure 24 This is another exemplary schematic diagram of the interface generation method provided in the embodiments of this application.

[0061] Figure 25 This is an exemplary schematic diagram of the UniRender process frequency division and multiplication vertical synchronization signal provided in the embodiments of this application.

[0062] Figure 26A , Figure 26B This is an exemplary schematic diagram of data flow when an electronic device implements the interface generation method provided in the embodiments of this application.

[0063] Figure 27 This is another exemplary schematic diagram of the interface generation method provided in the embodiments of this application.

[0064] Figure 28A , Figure 28B This is an exemplary schematic diagram illustrating the synthesis of a target rendering tree from the rendering tree provided in the embodiments of this application.

[0065] Figure 29 This is an exemplary schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.

[0066] Figure 30 This is an exemplary schematic diagram of the software structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0067] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0068] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0069] The term "user interface (UI)" used in the following embodiments of this application refers to the medium interface through which an application or operating system interacts and exchanges information with the user. It realizes the conversion between the internal form of information and the form that the user can accept. The user interface is source code written in a specific computer language such as Java or Extensible Markup Language (XML). The interface source code is parsed and rendered on the electronic device, ultimately presenting content that the user can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be visible interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets displayed on the screen of an electronic device.

[0070] To facilitate understanding, the relevant terms and concepts involved in the embodiments of this application will be introduced below. The terminology used in the embodiments section of this invention is only used to explain specific embodiments of the invention and is not intended to limit the invention.

[0071] As the medium for interaction and information exchange between applications and users, the user interface (UI) requires the electronic device to generate the UI for the foreground application each time a vertical synchronization signal arrives. The frequency of the vertical synchronization signal is related to the refresh rate of the electronic device's screen; for example, the frequency of the vertical synchronization signal may be the same as the screen's refresh rate.

[0072] In other words, before each time an electronic device refreshes the content displayed on its screen, it is necessary to generate the interface for the foreground application so that the newly generated interface of the application can be shown to the user when the screen refreshes.

[0073] The interface displayed on an electronic device can include the interface of one or more applications. That is, the electronic device needs to generate an interface for one or more applications and combine the interfaces to obtain the combined interface displayed on the screen.

[0074] In this process, the application's interface is generated by the electronic device, requiring the application to render and generate its own bitmap, which is then passed to the SurfaceFlinger. Specifically, the application acts as a producer, drawing and generating the bitmap, which is then stored in a buffer queue provided by SurfaceFlinger. SurfaceFlinger, acting as a consumer, continuously retrieves the bitmaps generated by the application from the BufferQueue. The bitmap resides on the surface generated by the application, and this surface is then populated into the BufferQueue.

[0075] After SurfaceFlinger obtains the bitmap of the visible application, it works with the Hardware Composer (HWC) module to determine how the bitmap is composited as a layer. SurfaceFlinger can determine the visible application through the Window Manager Service (WMS).

[0076] After SurfaceFlinger obtains the off-screen rendering logic that applies to the application's window from the window management service, such as rounded corners, rotation, color changes, scaling, etc., it copies the application's bitmap to the off-screen buffer for rendering and obtains the bitmap used for layer composition through off-screen rendering.

[0077] After the application's bitmap is composited, SurfaceFlinger / HWC fills the composited bitmap (which can also be called a layer on SurfaceFlinger) into the frame buffer and passes it to the display subsystem (DSS). The DSS can then display the composited bitmap on the screen. This frame buffer can be an on-screen buffer.

[0078] (1) First, the following examples illustrate (1.1) the process of the application generating bitmaps, (1.2) the process of SurfaceFlinger / HWC compositing bitmaps, and (1.3) off-screen rendering.

[0079] (1.1) The process of the application generating a bitmap

[0080] Figure 1A , Figure 1B This is an exemplary schematic diagram of an electronic device interface provided in an embodiment of this application.

[0081] like Figure 1A As shown, the screen of the electronic device displays the interfaces of application 1 and application 2. The interface of application 1 can be referred to as the status bar. Figure 1A In this context, application 1 can be an operating system, and application 2 can be a music application.

[0082] Electronic devices displaying such Figure 1A Before the interface shown, each application needs to generate its own independent bitmap. The status bar interface is maintained by the operating system; therefore, the application that generates the status bar bitmap can be considered the operating system. Specifically, after generating bitmap 1, the operating system passes bitmap 1 to SurfaceFlinger; the music application generates its own bitmap... Figure 2 Afterwards, the position Figure 2 Passed to SurfaceFlinger. Bitmap 1 carries the image information of the status bar; bitmap 1... Figure 2 It contains image information for a music application.

[0083] Among them, SurfaceFlinger / HWC receives bitmap 1 and bitmap 2. Figure 2 Then, bitmap 1 and bitmap 2 Figure 2 Layers are used for compositing. The details of layer compositing can be found in the text description of the bitmap compositing process in SurfaceFlinger / HWC (1.2), which will not be repeated here.

[0084] For example Figure 1B As shown, the screen of the electronic device displays the interfaces of Application 1, Application 2, and Application 3. The interface of Application 1 can be referred to as the status bar. Figure 1B In this context, application 1 can be an operating system, application 2 can be a browser application, and application 3 can be a text messaging application.

[0085] Electronic devices displaying such Figure 1B In front of the interface shown, separate bitmaps for the application also need to be generated.

[0086] That is, the operating system generates bits Figure 3 Afterwards, the position Figure 3 Passed to SurfaceFlinger; SMS application generates bits Figure 4 Afterwards, the position Figure 4 Passed to SurfaceFlinger; News app generates bits Figure 5 Afterwards, the position Figure 5 Passed to SurfaceFlinger. (Bit) Figure 3 The image information of the status bar is displayed on the top; Figure 4 The image information carried by the SMS application; bit Figure 5 It contains image information from a news application.

[0087] Among them, SurfaceFlinger / HWC receives bits Figure 3 ,Bit Figure 4 and position Figure 5 Afterwards, the position Figure 3 ,Bit Figure 4 and position Figure 5 Layers are used for compositing. The details of layer compositing can be found in the text description of the bitmap compositing process in SurfaceFlinger / HWC (1.2), which will not be repeated here.

[0088] The process by which the application generates a bitmap is as follows: Figure 2 As shown.

[0089] Figure 2 An exemplary schematic diagram illustrating the generation of bitmaps by the application provided in this application embodiment.

[0090] like Figure 2 As shown, after receiving the vertical synchronization signal (Vsync-APP), the application begins to generate a bitmap. The specific steps include three steps:

[0091] ① Disable the view hierarchy, and the main thread (UI thread) traverses the application's views, determines and saves the drawing operations for each view, and records the view and the drawing operation (Draw Operation Struct, DrawOP) involved in the view into the display list of the render node in the render tree.

[0092] Among them, the view is the basic element that constitutes the application interface, and one control on the interface can correspond to one or more views.

[0093] The drawing operation, as a data structure, is used to draw graphics, such as lines, widths, rectangles, and text. At the rendering node, drawing operations are translated into API calls to image processing libraries, such as OpenGL interface calls. For example, `DrawLineOp` is a data structure containing drawing data such as line length and width.

[0094] The drawing instruction list can be a buffer that records all drawing operations or their identifiers (such as addresses and sequence numbers) included in a single frame of the application's interface. When the application has multiple windows or displays on different display areas, multiple render trees need to be generated independently, resulting in multiple drawing instruction lists corresponding to different windows and display areas.

[0095] In this embodiment, the display area can be a screen or a virtual display. A virtual screen can be the area used by the electronic device to display content on the screen during screen recording.

[0096] The render tree, generated by the UI thread, is a data structure used to generate the application's interface. It can include multiple render nodes, each containing render attributes and a list of drawing instructions. The render tree records all the information needed to generate a single frame of the application's interface.

[0097] ②The UI thread passes / synchronizes the rendering tree to the rendering thread. The rendering tree is located in the stack of the application's corresponding process and may not be distributed contiguously in physical address space.

[0098] ③ The rendering thread first acquires a hardware canvas and performs drawing operations in the rendering tree on the hardware canvas to generate a bitmap. The hardware canvas resides in the surface held by the application, which contains bitmap or other data in other formats used to store image information.

[0099] ① can be considered the construction phase, which is mainly responsible for determining the size, position, transparency, and other attributes of each view in the application. For example, the drawLine view in the view can be encapsulated into a DrawLineOp during the construction phase. It contains drawing data such as the length and width of the line, and can also contain the interface call corresponding to the drawLineOP of the underlying graphics processing library, which is used to call the underlying graphics library to generate bitmaps during the rendering phase.

[0100] Similarly, ③ can be considered as the rendering stage, which is mainly responsible for traversing the rendering nodes of the rendering tree and performing the drawing operation of each rendering node, thereby generating a bitmap on the hardware canvas. In this process, the rendering thread calls the underlying graphics processing library, such as OpenGL, and then calls the GPU to complete the rendering to generate the bitmap.

[0101] Figure 3 This is an exemplary schematic diagram of a rendering tree provided in an embodiment of this application.

[0102] The application's interface consists of multiple nested views, with parent-child relationships between them. Therefore, the parent-child relationships between rendering nodes in the rendering tree generated by traversing the views are the same as the parent-child relationships between the views. In other words, the parent-child relationships between views determine the nesting relationships between different rendering nodes, thus enabling the rendering thread to correctly render the application's interface when generating bitmaps based on the rendering tree.

[0103] A view can correspond to one or more rendering nodes, with the root view (DecorView) corresponding to the root rendering node (RootRenderNode). That is, the nesting relationship between rendering nodes corresponds to the parent-child relationship of views. Rendering nodes also include rendering properties, which are used to determine the position, size, and opacity of the view corresponding to that rendering node on the surface when rendering a bitmap.

[0104] For example, the structure of the application's interface is as follows: the application's PhoneWindow hosts the root view, and the root view's subviews are View1 and View2. Figure 2 ,See Figure 2 The subview is the view Figure 3 The structure of the render tree generated by the application's UI thread is as follows: the root render node corresponding to PhoneWindow is the root node of the render tree; the child node of the root render node is render node 0 corresponding to the root view; the child nodes of render node 0 are render node 1 corresponding to view 1 and render node 1 corresponding to view 1. Figure 2 The corresponding rendering node 2, the child nodes of rendering node 2 are the view Figure 3 The corresponding rendering node is 3.

[0105] The relationship between a view and a rendering node means that the rendering node includes all the drawing operations of the corresponding view.

[0106] After receiving the rendering tree synchronized with the UI thread, the rendering thread calls the OpenGL interface to render a bitmap on the application's own surface, and then sends the surface to SurfaceFlinger to wait for compositing and display.

[0107] It is worth noting that the build phase requires CPU computing resources, while the rendering phase requires GPU resources.

[0108] It is worth noting that if hardware acceleration is not enabled, the application completes all operations in the build and rendering stages through the UI thread, and does not need to be encapsulated as a rendering node. After traversing the application's views and performing the view drawing operations, it requests an anonymous shared memory from SurfaceFlinger and directly calls the underlying graphics library to generate a bitmap on that memory.

[0109] It is worth noting that, Figure 2 The diagram illustrates the bitmap generation process of an application when hardware acceleration is enabled. When hardware acceleration is not enabled on the electronic device, the application generates the bitmap through software rendering. This software rendering includes:

[0110] ① The view hierarchy is disabled, and the UI thread iterates through the application's views, recording the drawing operations for each view. ② The UI thread obtains a software canvas for drawing through an interface, such as Surface.lockCanvas(), and draws on this software canvas based on the saved list of drawing operations, generating a bitmap. This software canvas resides on the surface generated by the application.

[0111] The surface held by the application is allocated to it by SurfaceFlinger via binder communication. The number of surfaces held by the application can be the same as the number of its current windows (PhoneWindow).

[0112] After introducing the process of generating bitmaps by the application, the process of compositing bitmaps is described by way of example.

[0113] (1.2) The process of synthesizing bitmaps using SurfaceFlinger / HWC

[0114] SurfaceFlinger is a system service on electronic devices used to assign surfaces to applications and composite bitmaps from one or more surfaces as layers. HWC is a functional module of the Hardware Abstraction Layer (HAL) in electronic devices, responsible for compositing and display. It provides an interface to the upper-layer SurfaceFlinger and calls upon the capabilities of the underlying hardware (such as the display driver, excluding the GPU) to perform layer compositing.

[0115] Figure 4This is an exemplary schematic diagram of a bitmap used as a layer in layer composition, as provided in an embodiment of this application.

[0116] like Figure 4 As shown, the application's rendering thread generates bitmaps on the surface allocated to it by SurfaceFlinger. SurfaceFlinger maintains one or more application surfaces using a BufferQueue mechanism, meaning SurfaceFlinger can obtain bitmaps from different applications.

[0117] After SurfaceFlinger obtains bitmaps from one or more applications, it can call the GPU to combine multiple bitmaps into a single bitmap (the combination of bitmaps is called layer compositing). This compositing can also be called Client compositing or GLES compositing.

[0118] After SurfaceFlinger obtains the application's bitmap, it can use HWC to call the underlying hardware (excluding the GPU) for compositing. This compositing method is also known as Device compositing.

[0119] Client compositing requires the GPU and can composite multiple layers, performing pixel-by-pixel processing on the layers, such as linear fading.

[0120] Device compositing can composite a limited number of layers and does not support many pixel-by-pixel processing methods. Device compositing, when compositing multiple layers that do not overlap on the screen, can avoid layer compositing and instead read data from different surfaces at different locations on the display screen for display.

[0121] like Figure 4 As shown, SurfaceFlinger obtains bitmap 1 of application 1 and bitmap 2 of application 2. Figure 2 After the bitmap N of application N, bitmap 1 and bitmap 2 can be accessed. Figure 2 Perform Client synthesis to generate bits Figure 11 Secondly, SurfaceFlinger will be positioned... Figure 11 The bitmap N is synthesized by calling the underlying hardware through HWC.

[0122] Device synthesis will temporarily save the location. Figure 11 And bitmap N, and when the electronic device screen needs to display it, from the bitmap N. Figure 11 / Bitmap N takes the corresponding pixels and displays them on the screen. For example, in Figure 1B In the interface shown, bit Figure 11This is a bitmap composited from the SMS and news applications, where bitmap N is the bitmap of the status bar.

[0123] For the underlying hardware corresponding to SurfaceFlinger or HWC, each bitmap is equivalent to a layer.

[0124] The layer composition method can be determined by the underlying hardware corresponding to HWC, or by SurfaceFlinger.

[0125] For example, after acquiring the bitmap, SurfaceFlinger passes the set of layers to the underlying hardware via HWC. The underlying hardware then determines which layers are used for client compositing and which are used for device compositing. The underlying hardware labels the layers in the layer list with compositing methods and returns these methods to SurfaceFlinger. SurfaceFlinger then composites the layers labeled as GPU compositing and places the composited result into a buffer. SurfaceFlinger then passes this buffer, along with other layers labeled as Overlay compositing, to the underlying hardware via HWC, whereby the underlying hardware completes the layer compositing.

[0126] For example, secondly, SurfaceFlinger directly marks layers that trigger off-screen rendering logic during the layer compositing process, such as those involving window animations, as GPU-composited. This off-screen rendering logic includes elements like rounded corners, scaling, rotation, and color conversion—logic that the underlying hardware of HWC cannot handle.

[0127] Because the surface allocated to it by the SurfaceFlinger held by the application contains not only bitmaps generated by the application's rendering thread, but also window control information such as layer Z-order obtained from the window management service, SurfaceFlinger can obtain the layer's window control information from the surface, and thus determine whether the layer needs to be composed by the GPU. The layer Z-order determines the height order of the layers on the Z-axis, where the Z-axis is the direction perpendicular to the screen, used to calculate the vertical relationship between different layers.

[0128] Figure 5 This is an exemplary schematic diagram of layer composition provided in an embodiment of this application.

[0129] like Figure 5As shown, after SurfaceFlinger obtains the application's bitmaps, it writes each bitmap as a layer into a layer list. SurfaceFlinger passes the layer list to the underlying hardware via HWC. The underlying hardware determines the composition method for different layers based on its own capabilities and returns the result to SurfaceFlinger via HWC.

[0130] After obtaining the results returned by HWC, SurfaceFlinger can determine the compositing method for each layer in the layer list. For layers marked as GPU compositing, SurfaceFlinger performs layer compositing and passes the composited layer, along with the layers marked as overlay compositing, to the underlying hardware via HWC. The underlying hardware then performs layer compositing based on the HWC.

[0131] Figure 6 This is another exemplary schematic diagram illustrating layer composition using SurfaceFlinger provided in this application embodiment.

[0132] After obtaining multiple surfaces, SurfaceFlinger can determine the bitmaps on each surface as the compositing method for layers. Once SurfaceFlinger has multiple layers and has determined the compositing method to be GPU, it can perform client-side compositing. The compositing modes for layers can include: Mode.CLEAR (displays the top-level layer in Z-order), Mode.SRC_OVER (displays layers sequentially in Z-order), and Mode.DST_IN (displays the non-overlapping portions of the top-level layer in Z-order and the layers below it), etc.

[0133] For example, such as Figure 6 As shown, when the layer composition mode is Mode.SRC_OVER, SurfaceFlinger will combine the contents of the three layers, determine the vertical relationship of the different layers according to the Z order of the layers, and stack them onto a single layer in sequence.

[0134] The Z-order of layer 1 is a, the Z-order of layer 2 is a+1, and the Z-order of layer 3 is a+2. SurfaceFlinger will combine the contents of the three layers, completely obscuring the contents of layer 1 and layer 2, and only displaying the contents of layer 3.

[0135] In this context, complete occlusion means that due to the occlusion of layers with higher Z-order, the view on layers with lower Z-order will not be displayed at all.

[0136] All layers composited using the GPU will trigger off-screen rendering, which is used for layer compositing. The following section introduces the relevant concepts of off-screen rendering.

[0137] (1.3) Off-screen rendering

[0138] If SurfaceFlinger determines that any number of layers need to be composited using the GPU, off-screen rendering needs to be enabled to complete the layer compositing.

[0139] Off-screen rendering involves SurfaceFlinger allocating an off-screen buffer and using the GPU for image processing within that buffer. This off-screen buffer is a memory location outside the current screen buffer, and multiple layers are composited within it.

[0140] Off-screen rendering can include the following steps:

[0141] ① The bitmap in the GPU-composite layer determined by SurfaceFlinger needs to be converted into a texture and uploaded to GPU memory (i.e., off-screen buffer) or mapped to an OpenGL texture via shared memory. Then, OpenGL binds the texture (binding the texture includes binding the context).

[0142] ② Render the texture according to the window animation instructions corresponding to the layer. Merge textures from multiple applications, where layer merging is performed pixel-by-pixel rendering based on the layer's compositing mode.

[0143] ③ SurfaceFlinger obtains the rendered texture from GPU memory or directly from shared memory.

[0144] Obviously, off-screen rendering leads to context switching, which adds additional performance overhead.

[0145] Combining (1.1) the process of generating bitmaps by the application, (1.2) the process of compositing bitmaps by SurfaceFlinger / HWC, and (1.3) the content in off-screen rendering, the following is a complete introduction to the process of compositing bitmaps as layers.

[0146] Figure 7A , Figure 7B This is another exemplary schematic diagram of layer composition provided in the embodiments of this application.

[0147] like Figure 7A As shown, the UI threads of different applications independently generate their own rendering trees and pass the rendering trees to their respective rendering threads for rendering to generate bitmaps.

[0148] The rendering thread first renders and generates a bitmap based on the attributes and drawing instruction list in the rendering nodes of the rendering tree, and then composites the bitmap as a layer. When the layer compositing method is GPU compositing, SurfaceFlinger merges multiple layers into one layer through off-screen rendering.

[0149] For example, application 1 generates render tree 1, and then generates bitmap 1 based on render tree 1; application 2 generates render tree 2, and then generates bitmap 1 based on render tree 2. Figure 2 SurfaceFlinger receives bitmap 1 and bitmap 2. Figure 2 Then, for bitmap 1 and bitmap 2 Figure 2 Perform off-screen rendering to generate bitmaps Figure 5 In the off-screen rendering process, bitmap 1 is first copied to the off-screen buffer. Based on the window animation information, bitmap 1 is transformed into a bitmap in the off-screen buffer. Figure 3 Similarly, the position Figure 2 Transform to bit Figure 4 Finally, the position Figure 3 and position Figure 4 Based on the layer compositing mode, bits are generated by overlaying. Figure 5 .

[0150] like Figure 7B As shown, to achieve Figure 1B As shown in the interface, after receiving the application's bitmap, SurfaceFlinger first performs off-screen rendering on the application's bitmap involved in the window animation bitmap to complete the transformation, and then copies the transformed layer to the on-screen buffer for display.

[0151] For example, if the bitmap of application 3 needs to be scaled down, the bitmap of application 3 cannot be directly copied to the on-screen buffer for transformation, as this may affect the bitmaps of other applications. Instead, it needs to be copied to the off-screen buffer for transformation separately, and the transformed result is copied back to the on-screen buffer.

[0152] Clearly, the correct interface can only be generated by first modifying the bitmaps generated by each application and then overlaying them according to the layer composition mode.

[0153] From the moment the application's main thread generates the rendering tree until the SurfaceFlinger layer compositing is completed, the GPU needs to be started at least N+1 times because the process that calls the GPU keeps changing (from application 1 to application 2, ..., application N, SurfaceFlinger), where N is the number of applications.

[0154] Understandably, because different applications independently build and render their own bitmaps, but during layer compositing, layers with lower Z-order may be completely obscured by layers with higher Z-order, inevitably leading to overdraw of bitmaps from different applications. For example... Figure 6 Overdraw occurs in the overlapping parts of the three graphics, i.e., layer 1 and layer 2 are overdrawn, and the composited interface is not affected by layer 1 and layer 2; secondly, off-screen rendering requires the allocation of a new off-screen buffer and switching contexts, which increases the memory and computational overhead of electronic devices.

[0155] (2) Next, the interface generation method provided in the embodiments of this application will be introduced below.

[0156] The interface generation method provided in this application firstly obtains the rendering trees of one or more applications through the UniRender process, and then recombines the one or more rendering trees to generate a target rendering tree; secondly, the UniRender process renders based on the target rendering tree to directly obtain a bitmap carrying the interface image information of the one or more applications, without performing layer compositing.

[0157] First, the interface generation method provided in this application merges the target rendering trees of one or more applications into a single target rendering tree. During the generation of the target rendering tree, the UniRender process determines the off-screen rendering logic for each layer and adds or modifies the attributes of the corresponding rendering nodes in the target rendering tree according to the off-screen rendering logic, thereby enabling the UniRender process to directly generate bitmaps without performing off-screen rendering.

[0158] Secondly, the interface generation method provided in this application merges the target rendering trees of one or more applications into a single target rendering tree, eliminating the need to generate multiple bitmaps as layers for compositing. During the bitmap generation process based on the target rendering tree, the UniRender process uses the Z-order of the layers as the Z-order of the rendering tree, which can reduce the rendering nodes corresponding to views that are not displayed or do not affect display, thereby avoiding overdrawing.

[0159] Finally, the interface generation method provided in this application embodiment allows the application to avoid generating a rendering thread, instead letting the UniRender process perform unified rendering, thereby helping to improve the speed of interface rendering.

[0160] (2.1) System architecture of the interface generation method provided in the embodiments of this application

[0161] Figure 8 This is an exemplary schematic diagram of the system architecture of the interface generation method provided in the embodiments of this application.

[0162] When the application's interface updates, the application can request a vertical synchronization signal (Vsync-APP) from the UniRender process. (SurfaceFlinger does not do this.) Figure 8 As shown in the image, when SurfaceFlinger is present, an application can request a vertical synchronization signal (Vsync-APP) from SurfaceFlinger.

[0163] The vertical synchronization signal (Vsync-APP) of the UniRender process can originate from SurfaceFlinger, be generated directly from the underlying hardware of HWC (such as the screen), or be generated by a timed wake-up thread started by the UniRender process. When SurfaceFlinger is present, Vsync-APP can originate from SurfaceFlinger.

[0164] After the application receives the vertical synchronization signal (Vsync-APP), it generates a render tree and passes it to the UniRender process.

[0165] Upon receiving a vertical synchronization signal (Vsync-UR), the UniRender process merges one or more render trees to generate a target render tree. Then, the UniRender process uses the rendering engine to traverse and execute the drawing operations in the drawing instruction list of each render node in the render tree, generating a bitmap that carries image information from one or more render trees. This bitmap can reside in the screen buffer.

[0166] The difference between Vsync-UR and Vsync-APP is Vsync-Offset, which can be determined by the UniRender process. In the presence of SurfaceFlinger, Vsync-Offset can be determined by SurfaceFlinger.

[0167] After generating the bitmap, the UniRender process passes the bitmap to the display subsystem via HWC, and then displays it.

[0168] After introducing the system architecture of the interface generation method of this application, the method flow of the interface generation method provided by the embodiment of this application is described by way of example.

[0169] (2.2) Method flow of the interface generation method provided in the embodiments of this application

[0170] The following is combined with Figure 9 The interface generation method provided in the embodiments of this application is described by way of example.

[0171] Figure 9 This is an exemplary schematic diagram of the interface generation method provided in the embodiments of this application.

[0172] like Figure 9 As shown, the interface generation method provided in this application includes:

[0173] S901: After receiving the vertical synchronization signal, construct and generate the rendering tree.

[0174] When an application needs to update its interface, it can request a vertical synchronization signal (Vsync-APP) from the UniRender process. Upon receiving the Vsync signal, the application executes the `measure()`, `layout()`, and `draw()` methods on the UI thread. During the `draw()` method, the UI thread iterates through the application's views, determines the drawing instructions required for each view's rendering, and continuously records / inserts these instructions into the drawing instruction list of the corresponding rendering node for that view.

[0175] The application's interface consists of multiple nested views. The drawing instruction list corresponding to the root view (DecorView) includes the drawing instruction list entries for the root view's child views. Therefore, the nesting relationship between drawing instruction lists is the same as the nesting relationship between views, and consequently, the nesting relationship of rendering nodes is the same as the nesting relationship of views. For definitions of the rendering tree and rendering nodes, please refer to the above text. Figure 3 The corresponding text description.

[0176] Figure 10 This is an exemplary schematic diagram of constructing a rendering tree provided in an embodiment of this application.

[0177] After measuring, laying out, and drawing, the application's UI thread can obtain the parent-child structure of multiple views of the interface to be updated. During the process of traversing the views, it determines the content to be displayed on each view and the interface calls required to generate that content, such as drawCircle, drawLine, etc.

[0178] The application encapsulates drawing interface calls such as drawCircle and drawLine into corresponding DrawOp calls, such as DrawCircleOp and DrawLineOp, and stores them in the drawing instruction list. DrawLineOp is an interface call for drawing graphics from the underlying graphics library (such as OpenGL), which will be further converted into instructions to call the GPU to draw graphics.

[0179] like Figure 10As shown, the application's interface view structure is as follows: root view, and the root view's subviews are view 1 and view 2. Figure 2 ,See Figure 2 The subview is the view Figure 3 Among them, vision Figure 2 If the shape includes rectangles and circles, then it is considered... Figure 2 The drawing operations in the drawing instruction list of the corresponding rendering node 2 include: drawing rectangles, drawing circles, etc.

[0180] When the application's UI thread executes the `draw()` method, it starts from the root view and traverses all views according to their parent-child relationships. It then determines the drawing operations for each view and encapsulates these operations into `DrawOp` objects. After generating the drawing instruction list, the application's UI thread further encapsulates this list into a render tree.

[0181] A rendering node in the rendering tree includes a list of drawing instructions and rendering properties. The rendering properties determine the position, size, transparency, and other attributes of the view to be rendered by the rendering node on the surface. The list of drawing instructions determines the content of the view to be rendered by the rendering node, such as lines, rectangles, circles, etc.

[0182] The surface is requested by the application, and the application determines its size. If SurfaceFlinger exists, the application can request the surface from SurfaceFlinger; if SurfaceFlinger does not exist, it can request it from the UniRender process. SurfaceFlinger may choose not to allocate a surface to the application.

[0183] Optionally, in some embodiments of this application, the UniRender process can determine the screen refresh rate of each display area and then adjust the frequency of the vertical synchronization signal (Vsync-APP) on each display area so that the application displayed on display area 1 generates a render tree at the frequency of the screen refresh rate of display area 1.

[0184] It is worth noting that an application's UI thread can generate multiple rendering trees, such as in multi-screen, virtual screen, multi-window scenarios and other multi-display scenarios. For multi-display scenario, please refer to the text description in (2.2) Interface generation method in multi-display scenario.

[0185] S902: Cross-process transfer of render tree

[0186] After generating the render tree, the application's UI thread passes it to the UniRender process via IPC communication. The render tree resides on the stack of the application's corresponding process. The UniRender process needs to receive render trees from different applications and determine the mapping between the render tree and the application.

[0187] In this process, multiple foreground applications pass the render tree to UniRender. An application is considered a foreground application if it meets any of the following three conditions: the application has a visible activity, the application has a foreground service, or other foreground applications are associated with the application.

[0188] Since different processes do not share memory, data exchange between processes requires inter-process communication (IPC). Specifically, the application can pass the rendering tree to the UniRender process via IPC through methods such as Binder, AIDL, shared memory, and Socket, which are not limited here.

[0189] The following example uses shared memory as an example of IPC communication to illustrate how to pass the rendering tree across processes.

[0190] (a) The application writes the render tree to shared memory.

[0191] Figure 11 This is an exemplary schematic diagram of a shared memory transfer rendering tree provided in an embodiment of this application.

[0192] like Figure 11 As shown, firstly, the application requests shared memory from the UniRender process each time it starts. After receiving the shared memory request from the application, the UniRender process requests a block of shared memory and a handle to the corresponding shared memory through the Anonymous Shared Memory (Ashmem) subsystem.

[0193] Secondly, after successfully requesting shared memory from the Ashmem subsystem, the UniRender process receives a handle from the Ashmem subsystem for reading and writing physical memory. The UniRender process returns the handle to the application, allowing the application to use it to write the rendering tree into physical memory. The UniRender process can directly read this physical memory from its own process space, and thus directly read the application's rendering tree.

[0194] The shared memory can be a virtual file created in RAM through a temporary file system (tmpfs) and mapped to the user space of different processes.

[0195] Cross-process communication, such as the application requesting shared memory from the UniRender process, the UniRender process requesting shared memory from the Ashmem subsystem, and the UniRender process returning the handle to the obtained shared memory to the application, can be achieved through Binder communication.

[0196] It is understandable that the rendering tree stored on the application process's stack can be passed to the UniRender process through other IPC methods such as shared memory, and this is not a limitation here.

[0197] Optionally, in some embodiments of this application, a whitelist is configured in the local configuration file of the electronic device or on the cloud server. The whitelist stores the application's package name and other identifiers that can uniquely identify the application process. If the application is on the whitelist, the rendering tree is passed to the UniRender process; if the application is not on the whitelist, it is drawn by the UI thread software or rendered into a bitmap by the rendering thread, and then passed to UniRender for process compositing, or passed to SurfaceFlinger, where SurfaceFlinger combines the bitmap of the application that is not on the whitelist with the bitmap generated by the UniRender process.

[0198] Optionally, in some embodiments of this application, when the application has multiple layers, that is, when the application generates multiple render trees, the application can request two shared memory blocks from the UniRender process to store data for different layers, that is, to store data for different render trees.

[0199] Optionally, in some embodiments of this application, the application may request two shared memory blocks from the UniRender process for interleaved writing of the render tree. For example, the render tree corresponding to a certain frame is written to the first shared memory block, then the render tree corresponding to the next frame is written to the second shared memory block; and so on. This helps to prevent read / write conflicts from causing data of the render tree to be written to the shared memory in a timely manner when there is only one shared memory block available.

[0200] (b) Data structure for storing the rendering tree in shared memory

[0201] To further improve the efficiency of IPC communication, in this embodiment, the rendering tree is stored in shared memory as a memory tree. The data structure of the rendering tree stored in shared memory is described below as an example:

[0202] The memory tree can consist of multiple data segments, with different segments storing layer information, rendering data, etc. The following example... Figure 12A , Figure 12B The content shown is used as an example to illustrate the structure of a memory tree. The offsets of different data segments in shared memory can be fixed or variable and adaptive.

[0203] Figure 12A , Figure 12B This is an exemplary schematic diagram of the storage structure of the rendering tree in shared memory provided in an embodiment of this application.

[0204] like Figure 12A As shown, the storage structure of the rendering tree includes three data segments: the HEAD field, the MAPPING field, and the NODES field.

[0205] The HEAD field includes layerkey and rootid; the MAPPING field includes nodeid and the address corresponding to nodeid; and the NODES field includes currentproperties, stagingproperties, stagingdisplaylist, and currentdisplaylist.

[0206] Here, layerkey is the ID of the entire rendering tree as a layer; rootid is the ID of the root node of the rendering tree; nodeid is the ID of the rendering node other than the root node, and each nodeid corresponds to an address, which is the starting address of the rendering properties (renderproperties / properties) and the display list (displaylist) in the rendering tree node; stagingproperties are the rendering properties written by the application, stagingdisplaylist is the display list written by the application, currentproperties are the rendering properties read by the UniRender process, and currentdisplaylist is the display list read by the UniRender process.

[0207] It is worth noting that if we consider "stagingproperties and stagingdisplaylist" as the first set of data and "currentproperties and currentdisplaylist" as the second set of data, then the data written by the application is the first set of data, and the data written by the application next time is the second set of data, thus implementing a double buffering mechanism; similarly, the data read by the UniRender process is the first set of data, and the data read next time is the second set of data.

[0208] Optionally, in some embodiments of this application, the storage structure of the rendering tree in shared memory can be as follows: Figure 12B As shown, the NODES field can contain only one set of data, namely only the list of rendering properties and drawing instructions.

[0209] The layerkey is used by the UniRender process to obtain the application to be displayed and the layer IDs of the application that need to participate in layer compositing from the WMS before reading layer data from the shared memory via handles. The UniRender process then verifies this layer ID against the layer IDs included in the layerkey from the shared memory.

[0210] The rootid is used as the entry point of the rendering tree, storing the entry points of other rendering nodes. After obtaining the rootid, the UniRender process can read the data of the rendering tree and restore the nested structure of the rendering tree.

[0211] The values ​​of currentproperties, stagingproperties, stagingdisplaylist, and currentdisplaylist are used to: swap the values ​​of currentproperties and stagingproperties after the application has finished writing the drawing instruction list and rendering properties, and swap the values ​​of currentdisplaylist and stagingdisplaylist. The UniRender process reads the rendering properties and drawing instruction list from currentproperties and currentdisplaylist.

[0212] For example, Application 1 is an application displayed in the foreground. Application 1 can have multiple layers. That is, after receiving the vertical synchronization signal (Vsync-APP), Application 1 will generate multiple render trees, such as render tree 1 and render tree 2. The UniRender process determines from the WMS that the render tree corresponding to the layer participating in layer compositing is render tree 1.

[0213] Because the offsets of layerkey and rootid are fixed, the UniRender process can determine the address of the root node. The UniRender process finds the root node's position at the NODES end in the MAPPING segment, reads the rendering instructions for that node, and if there are child nodes, there will be corresponding DrawRenderNode instructions, which store the child node's ID. The hash value is used to find the child node's position in the MAPPING segment. For example, the parent-child relationship of rendering nodes in the rendering tree is stored in the DrawOP operations. For instance, the drawing instruction list for rendering node 2 includes several DrawOP operations and the "Draw RenderNode3" operation, thus the UniRender process can determine that rendering node 3 is a child node of rendering node 2.

[0214] Understandably, the rendering tree in shared memory still maintains the same nesting relationship as the application's views, so the UniRender process can read data starting from the root node and then read the data of the entire rendering tree.

[0215] Specifically, by dividing currentproperties, stagingproperties, stagingdisplaylist, and currentdisplaylist, the security of the data read by the UniRender process and written by the application is ensured, preventing data that is only partially written by the application from being read by the UniRender process as the latest data for rendering the application interface. For details on the security of simultaneous read and write operations on the rendering tree, please refer to the textual description of read and write operations on the rendering tree in shared memory in section (2.3) below; it will not be repeated here.

[0216] Optionally, in some embodiments of this application, the size of the three data segments can be fixed. That is, when the UniRender process requests shared memory from the Ashmem subsystem, the obtained shared memory is of size (a+b+c). Here, the positions from the starting address (physical address) to the starting address + a are for filling the HEAD field; the positions from the starting address + a+1 to the starting address + a+b are for filling the MAPPING field; and the positions from the starting address + a+b+1 to the starting address + a+b+c are for filling the NODES field.

[0217] Understandably, with the three data segments of fixed size, the UniRender process can determine the start of each data segment based on the fixed offset, and then find the Mapping segment. The Mapping segment stores the offset of the rendering node of the rendering tree in the NODES field, and then finds the data of each rendering node.

[0218] Optionally, in some embodiments of this application, when the sizes of the three data segments are fixed, if the rendering node written by the application exceeds the size of b, a second shared memory block is requested from the UniRender process. The second shared memory block can have the same format as the first shared memory block, and the NODES field of the second shared memory block continues to store the drawing instruction list and rendering node attributes of the rendering node. The HEAD field and / or MAPPING field of the second shared memory block can be empty or may not exist. That is, in some embodiments of this application, the second shared memory block only includes the NODES field.

[0219] (c) UniRender reads the rendering tree from shared memory

[0220] (c.1) UniRender Architecture

[0221] Figure 13 This is an exemplary schematic diagram of the architecture of UniRender provided in the embodiments of this application.

[0222] like Figure 13 As shown, the UniRender provided in this application may include four parts, namely NodeManager, LayerManager, DisplayManager, and UniRenderCore.

[0223] NodeManager is the node management module within the UniRender process, responsible for receiving the rendering tree and other data sent by the application. The composition of the target rendering tree can be referred to the textual description in step S903.

[0224] LayerManager is the layer management module within the UniRender process, responsible for synchronizing layer information from the Window Manager Service (WMS), such as layer creation, destruction, and attribute changes. A bitmap is equivalent to one layer.

[0225] Among them, DisplayerManager is the display device management module within the UniRender process, which is responsible for synchronizing display device information, such as screen size, from the Display Manager Service (DMS).

[0226] UniRenderCore is the rendering management module within the UniRender process, responsible for: establishing corresponding rendering nodes for each layer; receiving rendering trees corresponding to different applications maintained in NodeManager, removing layer information instructions for the application from LayerManager and inserting them into rendering nodes; merging the rendering trees corresponding to all visible layers of the active display devices maintained in DisplayManager; traversing the merged rendering tree for each display area, and generating bitmaps on the buffer allocated by UniRender, etc.

[0227] (c.2) UniRender reads the rendering tree

[0228] The UniRender process first identifies the applications displayed on each display area from the DMS and WMS; these applications are those participating in layer compositing. The UniRender process can further utilize a whitelist to determine the applications performing layer compositing within the UniRender process. Specifically, the UniRender process uses the WMS to determine the layer ID for each application.

[0229] The DisplayerManager in the UniRender process is responsible for communicating with the DMS, and the LayerManager in the UniRender process is responsible for communicating with the WMS.

[0230] Because the UniRender process stores a handle to the shared memory corresponding to the application, UniRender can determine the shared memory corresponding to that application after identifying the application participating in layer compositing. UniRender then reads the render tree from the shared memory using this handle.

[0231] The NodeManager in the UniRender process is responsible for managing the handles to the shared memory and reading the rendering tree from the shared memory.

[0232] The process by which the UniRender process reads the render tree from shared memory includes:

[0233] First, the UniRender process reads the layerkey from the starting address of shared memory and verifies the layer ID. Specifically, the UniRender process compares the layer ID determined by the WMS with the layer ID determined from the layerkey. If they match, the UniRender process reads the rendering tree starting from the root node (rootid).

[0234] Secondly, after finding the address of the root node, the UniRender process determines the starting address of the root node in the NODES segment from the address field in the MAPPING segment, and then begins reading the drawing instruction list and rendering attributes of that rendering node. If the root node has child nodes, the entry points for the child nodes, such as the DrawRenderNode instruction, are stored in the root node's drawing instruction list. Since the DrawRenderNode instruction includes the child node's ID, the UniRender process uses a hash operation to find the corresponding nodeid in the MAPPING segment, thereby determining the location of the child node's drawing instruction list and rendering attributes in the NODES segment, and then reading these attributes.

[0235] (d) Reading and writing the rendering tree in shared memory

[0236] The rendering tree, located in shared memory, can be read and written by two or more processes. To reduce or avoid read-write conflicts that could lead to errors in the rendering tree data, inter-process synchronization locks can be configured to ensure the safety of reading and writing the rendering tree.

[0237] The following is combined with Figure 14 , Figure 15 The content shown exemplifies the process of reading and writing the rendering tree.

[0238] Figure 14 This is an exemplary schematic diagram of an application providing an embodiment of this application writing a render tree into shared memory.

[0239] Each application maintains at least one lock variable A to prevent the application and UniRender from simultaneously reading and writing to the shared memory. UniRender obtains the status (held or released) of the lock variable held by different applications via IPC communication.

[0240] like Figure 14 As shown, ① First, when the application needs to update the interface, it requests and receives the vertical synchronization signal (Vsync-APP), then obtains the lock variable A of the shared memory corresponding to this application, and calculates and updates the attributes and drawing instruction list of the rendering node starting from the root node.

[0241] ② Secondly, the application writes the updated properties and drawing instruction list of the rendering node into the stagingproperties and stagingdisplaylist data segments of the NODES segment in shared memory, and adds the changed rendering node ID to the properties_dirty queue and displaylist_dirty queue, which are stored in the shared memory management class singleton on the application side.

[0242] Understandably, the properties_dirty queue and the displaylist_dirty queue mark the rendered nodes that have changed, which can enable differential updates to the rendered tree.

[0243] Optionally, in some embodiments of this application, the properties_dirty queue and displaylist_dirty queue may not be saved, thus enabling a full update of the render tree.

[0244] ③ Next, the application copies the stagingproperties segment of the corresponding rendering node from the properties_dirty queue to the currentproperties segment. The application swaps the draw_pointer and record_pointer of the corresponding rendering node from the displaylist_dirty queue, that is, copies the stagingdisplaylist segment of the corresponding rendering node from the displaylist_dirty queue to the currentdisplaylist; or the application copies the stagingdisplaylist segment to the currentdisplaylist.

[0245] Understandably, in response to the current vertical synchronization signal (Vsync-APP), the application only changes the data of the rendering node corresponding to displaylist_dirty compared to the previous vertical synchronization signal (Vsync-APP). The application swaps the draw_pointer and record_pointer of the corresponding rendering node in the displaylist_dirty queue, thereby achieving differential updates to currentdisplaylist.

[0246] Understandably, the application can copy the stagingdisplaylist segment to the currentdisplaylist to achieve a full update, that is, write all the data of the rendering tree generated by the application in response to the vertical synchronization signal (Vsync-APP) directly into shared memory, which is relatively simple to implement.

[0247] Optionally, in some embodiments of this application, without saving the properties_dirty queue and the displaylist_dirty queue, the stagingproperties segment of all rendering nodes of the application is copied to the currentproperties segment, and the stagingdisplaylist segment is copied to the currentdisplaylist segment. This copying can be achieved by changing the position of the pointers.

[0248] ④ Next, the application transmits the information that lock variable A has been released to the UniRender process via IPC communication.

[0249] ⑤ Again, the UniRender process holds lock variable A.

[0250] ⑥ Finally, corresponding to ③, the UniRender process reads the current displaylist and current properties from shared memory, or reads the stagingdisplaylist segment of the corresponding rendering node from the displaylist_dirty queue and copies it to the current displaylist.

[0251] After reading the data, the UniRender process can release lock variable A and notify the application that lock variable A has been released. Then, when the next vertical synchronization signal (Vsync-APP) arrives, it holds lock variable A and writes the render tree into shared memory. At this point, the roles of the staging data segment and the current data segment are reversed. Finally, the UniRender process reads the stagingdisplaylist and stagingproperties segments, implementing a "double buffering" mechanism to ensure the robustness of the generated interface.

[0252] Optionally, in some embodiments of this application, when a lock variable exists between the application and the UniRender process, the NODES field may include only stagingdisplaylist and stagingproperties, or only currentdisplaylist and currentproperties. In this case, the application and the UniRender process achieve read-write safety through the lock variable, ensuring that the UniRender process reads the correct rendering tree.

[0253] Optionally, in some embodiments of this application, each application may hold more lock variables, for example, each application holds lock variable A and lock variable B. Furthermore, after an application releases lock A, it does not need to wait for UniRender to release lock A; after holding lock variable B, it directly writes the render tree data to the shared memory upon receiving the next vertical synchronization signal (Vsync-APP).

[0254] The holding, releasing, and inter-process synchronization of lock variable B are the same as those of lock variable A. For details, please refer to [link / reference needed]. Figure 14 The textual description shown will not be repeated here.

[0255] Figure 15 Another exemplary schematic diagram illustrating the application provided in this application writing a render tree into shared memory.

[0256] like Figure 15 As shown, when the application holds a lock variable A, upon receiving the vertical synchronization signal 1 (Vsync-APP), the application holds the lock variable A and writes the generated render tree into shared memory during the holding phase. After the application writes the render tree into shared memory, it releases the lock variable A. Once UniRender determines that the lock variable A has been released by the application, it holds the lock variable A and begins reading the render tree generated by the application in response to the vertical synchronization signal 1 from shared memory.

[0257] While the UniRender process holds lock variable A, the application receives vertical synchronization signal 2 (Vsync-APP) after vertical synchronization signal 1. Because lock variable A is held by the UniRender process, the application cannot write the rendering tree to shared memory in a timely manner and needs to wait for a period of time until the application determines that the UniRender process releases lock A.

[0258] However, it is quite obvious that in Figure 13 The UniRender process reads the currentdisplaylist and currentproperties fields, while the application writes to the stagingdisplaylist and stagingproperties fields. Concurrent reading and writing of different fields in shared memory can be achieved by increasing the number of lock variables.

[0259] like Figure 15As shown, when the application holds two lock variables B, upon receiving the vertical synchronization signal 1 (Vsync-APP), the application acquires lock variable A and writes the generated render tree into shared memory during the holding phase. After the application writes the render tree into shared memory, it releases lock variable A. Once the UniRender process determines that lock variable A has been released by the application, it acquires the lock variable and begins reading the render tree generated by the application in response to the vertical synchronization signal 1 from shared memory.

[0260] While the UniRender process holds lock variable A, the application receives vertical synchronization signal 2 (Vsync-APP) after vertical synchronization signal 1. At this time, it holds lock variable B, and the application can write the rendering tree to shared memory in a timely manner without waiting for a period of time until the application determines that the UniRender process releases lock A.

[0261] Correspondingly, after determining that lock variable B has been released by the application, the UniRender process holds lock variable B and begins to read the rendering tree generated by the application in response to vertical synchronization signal 2 from shared memory.

[0262] It is worth noting that the number of lock variables held by the application can be related to the contents of the NODES field in shared memory, or it can be related to the value of Vsync-offset.

[0263] For example, if we consider `currentdisplaylist` and `currentproperties` as the first group, and `stagingdisplaylist` and `stagingproperties` as the second group, then we can configure two synchronization lock variables in the application and the UniRender process, each corresponding to one of the two groups of data. Similarly, if the `NODES` field contains three groups of data, then we can configure three synchronization lock variables in the application and the UniRender process.

[0264] In this system, each lock variable corresponds to a set of data. For example, if lock variable A corresponds to currentdisplaylist and currentproperties, then the change of lock variable A from a held state to a released state indicates that the application has successfully updated the data of currentdisplaylist and currentproperties in shared memory, and the data of currentdisplaylist and currentproperties can be read by the UniRender process. Alternatively, the change of lock variable A from a held state to a released state indicates that the UniRender process has completed reading the data of currentdisplaylist and currentproperties from shared memory, and the data of currentdisplaylist and currentproperties can be updated by the application.

[0265] The number of lock variables can be related to the value of Vsync-offset.

[0266] That is, it can be related to the difference between the vertical synchronization signal (Vsync-APP) and the vertical synchronization signal (Vsync-UR), Vsync-offset. If Vsync-offset is large, lock variables do not need to be configured. Without configuring lock variables, the UniRender process reads the render tree from shared memory after receiving the vertical synchronization signal (Vsync-UR). Because Vsync-offset is large, the application has already completely written the render tree into shared memory when the UniRender process reads it.

[0267] S903: Transmits application window control information and display area information.

[0268] In the UniRender process, LayerManager obtains window control information of one or more applications from the window management service, and then, in conjunction with the layers of the one or more applications obtained in step S802, determines whether there is drawing logic that triggers off-screen rendering on any application's layer.

[0269] The UniRender process can also obtain the Z-order between layers from different applications. The Z-order refers to the vertical order of the different layers along the Z-axis.

[0270] The DisplayManager in the UniRender process obtains display area information from the display management service, including the size of the display device. Based on this information, the UniRender process determines the size of the allocated surface, which is used to hold the bitmap generated by UniRender based on the target rendering tree. The process of generating the bitmap based on the target rendering tree can be found in the textual description of step S905, and will not be repeated here.

[0271] The instructions involved in off-screen rendering include instructions such as rounded corners, scaling, rotation, and color conversion; the definition of off-screen rendering can be found in the text description of off-screen rendering in (1.3) above, and will not be repeated here.

[0272] The following examples illustrate drawing logic that triggers off-screen rendering, such as rounded corners, scaling, rotation, and color conversion.

[0273] Figures 16A-16C This is an exemplary schematic diagram of a scenario that triggers off-screen rendering drawing logic according to an embodiment of this application.

[0274] Responding to user interactions, such as gestures on a full-screen display (e.g., swiping up from the bottom to the center of the screen) or tapping the multitasking controls in the bottom navigation bar, electronic devices display a multitasking interface, such as... Figure 16A As shown.

[0275] In this case, the gallery application generates bitmaps that are the same size as the screen, while in other cases... Figure 16A The gallery application's interface in the multitasking interface shown is a scaled-down, rounded-corner version.

[0276] like Figure 16B As shown, in small window mode, the electronic device's interface displays views of multiple applications. Some of these applications' interfaces are located in floating windows.

[0277] For example, the interface of an electronic device displays the interfaces of a news application and a text messaging application, with the text messaging application's interface located in a rendered floating window.

[0278] In one example, the bitmap generated by the SMS application is the same size as the screen, while in another example... Figure 16B In the small window mode shown, the SMS application interface is scaled down and rounded.

[0279] like Figure 16CAs shown, in response to the user clicking the reading application icon on the desktop application, the electronic device launches the reading application. During the launch process, the main interface (the interface corresponding to the Main Activity) or the starting window of the reading application continuously enlarges.

[0280] The bitmap corresponding to the main interface or launch window of the reading application is the screen size, and the zoom effect is achieved by adjusting the scaling ratio. Secondly, the bitmap corresponding to the main interface or launch window of the reading application is a bitmap with rounded corners, which is then displayed on the electronic device's interface.

[0281] Understandably, in Figures 16A-16C In the scenarios shown, and in many other scenarios, rounded corners, scaling, rotation, and color conversion are needed to generate interfaces that better suit users' visual habits and improve user experience.

[0282] It is worth noting that, in Figures 16A-16C In the scenario shown, and in many other scenarios, the UniRender process receives window control information for each application's layers from the window management service, and then determines whether each application's layers need to be rounded, scaled, rotated, or have their colors changed.

[0283] Optionally, in some embodiments of this application, the UniRender process may first obtain window control information of one or more applications from the window management service, and then obtain the rendering tree of the one or more applications. That is, the timing of steps S902 and S903 can be interchanged.

[0284] S904: Generate a target rendering tree based on the obtained rendering tree, window control information, and display area information.

[0285] First, after the UniRender process receives the rendering tree generated by one or more applications, it responds to the received vertical synchronization signal and window control information, and then determines from the window control information whether each application's layer has the logic to trigger off-screen rendering.

[0286] Specifically, when the application is displayed on the display area of ​​a local electronic device, the window control information can come from the local window management service; when the application is displayed on the display area of ​​another electronic device, the window control information can come from the window management service of the other electronic device.

[0287] If the UniRender process determines that any application's layer contains logic that triggers off-screen rendering, it converts this logic into off-screen rendering instructions and then applies these instructions to the properties of the corresponding render nodes in the render tree. For ease of description, this process of converting off-screen rendering logic into off-screen rendering instructions and applying them to the properties of the corresponding render nodes in the render tree can be simply referred to as "off-screen rendering instruction forwarding."

[0288] Then, the UniRender process moves forward the off-screen rendering trigger instructions for one or more render trees, and for each display area, merges the render trees corresponding to the visible layers of each display area to generate a target render tree. That is, the number of target render trees can be related to the number of display areas.

[0289] The UniRender process can either start reading the render tree from shared memory after receiving the vertical synchronization signal (Vsync-UR), and then move off-screen rendering instructions forward and composite the render trees after acquiring multiple render trees, or it can start reading the render tree from shared memory while holding a lock variable, and then move off-screen rendering instructions forward and composite the render trees upon receiving the vertical synchronization signal (Vsync-UR).

[0290] The following sections describe the process of triggering the off-screen rendering instruction to move forward and the process of generating the target rendering tree.

[0291] (a) The process of moving the instruction that triggers off-screen rendering forward.

[0292] First, the UniRender process obtains the window control information of each application layer and determines whether any layer's window control information contains drawing logic that triggers off-screen rendering. If the UniRender process determines that the window control information of all application layers in a display area does not contain drawing logic that triggers off-screen rendering, it can directly combine one or more render trees into a target render tree. If the UniRender process determines that the window control information of any application layer in a display area contains drawing logic that triggers off-screen rendering, it first moves the off-screen rendering instruction forward, and then combines multiple render trees into a target render tree.

[0293] The process of triggering the off-screen rendering instruction to move forward:

[0294] The UniRender process first determines the drawing logic that triggers off-screen rendering from the window control information, and then converts this logic into instructions that can be configured into the rendering properties of the rendering node (or drawing logic instruction). After determining the binding relationship between the layer and the rendering tree, the UniRender process updates the off-screen rendering instructions into the rendering properties of the corresponding rendering node.

[0295] Specifically, if the rendering node has corresponding scaling, rounding, color transformation, and transformation attributes, then the parameters from the scaling, rounding, color transformation, and transformation instructions in the off-screen rendering command will be assigned to the scaling, rounding, color transformation, and transformation attributes in the rendering node's attributes. If the rendering node does not have corresponding scaling, rounding, color transformation, and transformation attributes, then these attributes will be added to the rendering node, and the parameters from the scaling, rounding, color transformation, and transformation instructions in the off-screen rendering command will be assigned to the scaling, rounding, color transformation, and transformation attributes in the rendering node's attributes.

[0296] Figure 17A , Figure 17B This is an exemplary schematic diagram of UniRender triggering the off-screen rendering instruction to move forward, provided for an embodiment of this application.

[0297] like Figure 17A As shown, after the UI threads of Application 1 and Application 2 independently generate render trees and independently pass the render trees to the UniRender process via IPC communication, the UniRender process determines the off-screen rendering instructions for each application's layers from the window management service. The render tree generated by Application 1 is called Render Tree 1, and the render tree generated by Application 2 is called Render Tree 2.

[0298] UniRender updates the properties of rendering nodes in different rendering trees independently. That is, the UniRender process updates the properties of rendering nodes in rendering tree 1, and UniRender updates the properties of rendering nodes in rendering tree 2.

[0299] If the off-screen rendering instruction applies to the entire layer, the UniRender process will assign the parameters in the off-screen rendering instruction to the root node of the rendering tree corresponding to the layer, or directly to all rendering nodes of the rendering tree.

[0300] In the case where the UniRender process assigns the parameters in the off-screen rendering instruction to the root node of the rendering tree corresponding to the layer, the UniRender process will automatically configure these attributes of the rendering node to the rendering attributes of the child nodes of the root node when generating a bitmap based on the rendering tree.

[0301] For example, if the child node of rendering node 1 is rendering node 2, then when the rendering property of rendering node 1 is configured with the rotation property (transform), the UniRender process will configure the same rotation property (transform) to the rendering property of RenderNode 2 when generating the rendering tree.

[0302] like Figure 17B As shown, before the off-screen rendering instruction is moved forward, after SurfaceFlinger receives the bitmap corresponding to the rendering tree generated by the application, it needs to perform different processing on the bitmaps of different applications based on the window control information, and then complete the layer compositing.

[0303] After moving the off-screen rendering instructions forward in the application's render tree, the UniRender process assigns the parameters from the off-screen rendering instructions to the render tree's rendering properties. This allows the transformed bitmap to be drawn directly on the canvas during the bitmap generation process based on the render tree. Since it's not necessary to process bitmaps from different applications differently, bitmaps from different applications can be drawn sequentially on a single on-screen buffer.

[0304] The UniRender process assigns parameters from the off-screen rendering instructions to the rendering properties of the render tree, such as... Figure 18 As shown.

[0305] Figure 18 This is an exemplary schematic diagram illustrating the modification of rendering node attributes in the rendering tree by UniRender, as provided in an embodiment of this application.

[0306] like Figure 18 As shown, the UniRender process modifies the properties of the root node of the rendering tree by adding commands to increase the properties of the rendering node. Specifically, the command for the scaling property is `setScale()`, the command for the rounded corner property is `outline().setRoundRect()`, the command for the color conversion property is `setColorFilter()`, and the command for the rotation property is `setLeftTopRightBottom()`, etc.

[0307] In step S905, which follows, when the UniRender process generates a bitmap based on the rendering tree, it modifies the drawing operations in the drawing instruction list according to the rendering attributes, thereby generating a scaled, rounded, color-converted, and rotated bitmap. For details, please refer to the text description in step S905; it will not be repeated here.

[0308] Optionally, in some embodiments of this application, the instructions in the attributes of the root node of the rendering tree may also include instructions such as setStaticMatrix(). Since the application UI thread generates the rendering tree based on the surface requested from SurfaceFlinger, in order to transform the reference frame when UniRender generates a bitmap based on the rendering tree in step S905, the instruction setStaticMatrix() is configured in the root node of the rendering tree during the process of moving the instruction that triggers off-screen rendering forward. The specific content of the instruction setStaticMatrix() can be found in the textual description of step S905 below, and will not be repeated here.

[0309] (b) The process of generating the target rendering tree

[0310] After the UniRender process performs off-screen rendering instruction forwarding on all render trees, it obtains one or more processed render trees.

[0311] If there is only one processed render tree, then that processed render tree is the target render tree; if there is more than one processed render tree, the multiple processed render trees are combined into a single target render tree.

[0312] Figure 19 This is an exemplary schematic diagram illustrating how multiple off-screen rendering instructions are processed and then combined into a single target rendering tree, as provided in an embodiment of this application.

[0313] like Figure 19 As shown, the UniRender process can create a new root rendering node, such as... Figure 19 The new root rendering node is created, and the processed rendering trees of multiple applications are used as child nodes of the root rendering node. That is, the root rendering node can store the parent-child relationship of the processed rendering trees of different applications, which allows the UniRender process to traverse the target rendering tree from the root rendering node to find the processed rendering trees of different applications.

[0314] Figure 20 This is an exemplary schematic diagram illustrating how multiple off-screen rendering instructions are processed and then combined into a single target rendering tree, as provided in an embodiment of this application.

[0315] The UniRender process can first determine the Z-order of layers corresponding to different applications based on the window management service, that is, determine the top and bottom occlusion relationships between application layers. Then, during the generation of the target rendering tree, the rendering nodes corresponding to completely occluded views are deleted, thereby reducing the amount of computation in the bitmap generation process in step S905 and improving the bitmap generation speed.

[0316] For example, such as Figure 20 As shown, the rendering tree generated by application 1 corresponds to layer 1, and the rendering tree generated by application 2 corresponds to layer 2. The Z-order of layer 1 is higher than that of layer 2. The rendering tree corresponding to layer 1 consists of root rendering node 1, rendering node 1, rendering node 2, and rendering node 3. Rendering node 1 and rendering node 2 are child rendering nodes of root rendering node 1, and rendering node 3 is a child rendering node of rendering node 2. The rendering tree corresponding to layer 2 consists of root rendering node 2, rendering node 4, rendering node 5, and rendering node 6. Rendering node 4 and rendering node 5 are child rendering nodes of the new root rendering node, and rendering node 6 is a child rendering node of rendering node 5.

[0317] The UniRender process can traverse the rendering child nodes of the rendering tree corresponding to layer 1 and the rendering child nodes of the rendering tree corresponding to layer 2, determine the position of the view corresponding to each rendering node in the surface (as allocated by the UniRender process), and combine the layer Z-order of layer 1 and the Z-order of layer 2 to determine the completely occluded view and the rendering node of the completely occluded view.

[0318] For example, UniRender can determine that the view corresponding to rendering node 6 in the rendering tree of layer 2 is completely obscured, and then delete rendering node 6.

[0319] After deleting the rendering node corresponding to a completely obscured view, the UniRender process can proceed as follows: Figure 18 The content shown merges all the rendered trees after the off-screen rendering instructions are moved forward into the target rendered tree.

[0320] Optionally, in some embodiments of this application, UniRender can also traverse rendering nodes and optimize the parameters of the target rendering tree at the granularity of DrawOP drawing operations. For example, DrawOP drawing operations that do not affect the interface can be deleted, where "does not affect" can mean that the graphics drawn by the DrawOP drawing operations are not displayed on the interface; or, for different applications, the position of DrawOP operations in the drawing nodes can be modified so that the same type of DrawOP drawing operations can be executed together, such as modifying the DrawOP drawing operation of rendering node 2 of application 1 to the drawing instruction list of rendering node 1 of application 2, which is not limited here.

[0321] Optionally, in some embodiments of this application, it may also be carried out in accordance with... Figure 18 The content shown merges the rendering trees of multiple triggered off-screen rendering instructions into one rendering tree, then identifies the completely obscured view and deletes the rendering node corresponding to the completely obscured view to generate the target rendering tree.

[0322] Optionally, in some embodiments of this application, for partially obscured views, the rendering nodes corresponding to those views can also be clipped using the clip command. This clipping can be performed before or after compositing the target rendering tree.

[0323] Optionally, in some embodiments of this application, after step S904 and before step S905, after the UniRender process generates the target rendering tree, it decapsulates the drawing instruction list of the RenderNode to obtain a series of DrawOPs, and then performs DrawOP operations on the target rendering tree as a whole to perform batch and merge operations, so that in step S805, a target rendering tree that can generate bitmaps can be obtained with less computation.

[0324] Understandably, in Figure 16A , Figure 16B In the scenarios shown and others, by merging multiple rendering trees and optimizing the merged rendering tree, over-drawing can be avoided or reduced, thereby increasing the speed of bitmap generation in step S905, saving power consumption of electronic devices, and improving the user experience.

[0325] It is worth noting that the UniRender process can also optimize other parameters of the target render tree during the generation process, which will not be limited here.

[0326] S905: Generate bitmap based on target rendering tree

[0327] After obtaining the target render tree, the UniRender process allocates a surface to the target render tree. The UniRender process then generates a bitmap on that surface based on the target render tree. This bitmap corresponds to the composited interface of one or more applications.

[0328] When an electronic device has multiple display areas, the surface can be bound to one of the display areas, and the surface can be the same size as the display area bound to the surface.

[0329] The UniRender process generates bitmaps based on the target render tree within this surface, including:

[0330] First, the UniRender process traverses the target rendering tree starting from the root node, and it can traverse the child nodes of the root node in various ways.

[0331] The UniRender process can traverse different layers under the root node according to the Z-order of the layers. The UniRender process can traverse different layers in descending Z-order or ascending Z-order.

[0332] For example, in Figure 20 In the target rendering tree shown, the Z-order of layer 1 is higher than that of layer 2. Therefore, the UniRender process can first traverse the rendering tree corresponding to layer 1, that is, first traverse the rendering tree composed of root rendering node 1, rendering node 1, rendering node 2, and rendering node 3, and then traverse the rendering tree composed of root rendering node 2, rendering node 4, and rendering node 5. Similarly, the UniRender process can first traverse the rendering tree corresponding to layer 2, and then traverse the rendering tree corresponding to layer 1.

[0333] It is worth noting that when the UniRender process traverses the layers from high to low in Z order, when executing the drawing operations in the drawing instruction list, if the layers are composited in an occlusion relationship, the UniRender process can only draw in the areas that are not being drawn, which can reduce over-drawing.

[0334] It is worth noting that when the UniRender process traverses the layers from low to high in Z order, the UniRender process executes the drawing operations in the drawing instruction list in the rendering tree corresponding to different layers in turn to generate bitmaps.

[0335] Secondly, the UniRender process modifies the drawing operations in the drawing instruction list according to the rendering properties, executes the drawing operations, and then generates a bitmap.

[0336] When the UniRender process iterates through each rendering node, it first reads the node's rendering properties. If the rendering properties include scaling, rounded corners, color conversion, rotation, etc., the UniRender process modifies the parameters of the drawing operations in the drawing instruction list based on these properties, and then executes the drawing operations in the modified drawing instruction list. If the rendering properties do not include scaling, rounded corners, color conversion, rotation, etc., UniRender directly executes the drawing operations in the drawing instruction list.

[0337] The following example illustrates how the UniRender process modifies the parameters of drawing operations in the drawing instruction list based on rendering attributes.

[0338] For example, if the rendering node's rendering properties include `setscale(0.5)`, which means scaling down to 0.5 times the original size, and the drawing instruction list includes `drawCircle(x0, y0, 5)`, then the UniRender process will transform the drawing operation `drawCircle(x0, y0, 5)` into `drawCircle(x0, y0, 2.5)`. In `drawCircle()`, the first parameter is the X-coordinate of the circle's center, the second parameter is the Y-coordinate of the circle's center, and the third parameter is the radius of the circle.

[0339] Considering that the list of drawing instructions generated by the application's UI thread may be related to the size of the surface requested by the application itself, in this embodiment of the application, the UniRender process configures setStaticMatrix() in the properties of the rendering node when traversing the rendering tree node to transform the reference coordinate system for drawing and rendering.

[0340] The UniRender process can determine the size of the reference coordinate system or reference surface when each application generates a list of drawing instructions through WMS, and thus determine the parameters of setStaticMatrix().

[0341] Figure 21 This is an exemplary schematic diagram of setStaticMatrix() provided in an embodiment of this application.

[0342] After determining the surface coordinate system referenced by the application to generate the drawing instruction list, the UniRender process determines the coordinate transformation matrix Transformation based on the coordinate system of the surface held by the UniRender process.

[0343]

[0344] Among them, scale x Scaling along the x-axis, skew x For twisting / skewing along the x-axis, translate x This represents a translation along the x-axis; scale y Scaling along the y-axis, skew y For twisting / skewing along the y-axis, translate y Translation along the y-axis.

[0345] After executing the `setStaticMatrix()` instruction, the UniRender process calculates the coordinate transformation matrix `Transformation` and applies it to each drawing operation in the drawing instruction list. For example... Figure 21 As shown, when the application's UI thread generates a rendering tree, the reference coordinate system is not the coordinate system on the surface held by the UniRender process. The UniRender process determines the coordinate transformation matrix Transformation through the setStaticMatrix() instruction, and transforms the reference coordinate system when drawing the interface on the surface based on the drawing instruction list, so as to draw the interface correctly.

[0346] The preceding text mainly introduced the specific process of implementing the interface generation method provided in the embodiments of this application in a single display area scenario. The following mainly introduces the specific process of implementing the interface generation method provided in the embodiments of this application in a multi-display area scenario.

[0347] (2.3) Interface generation method in multi-display area scenarios

[0348] First, we will introduce a scenario with multiple display areas as an example.

[0349] (a) Multi-display area scenario

[0350] In this embodiment, the display area can be a physical screen or a virtual screen. A virtual screen can be the area used by the electronic device to display content on the screen during screen recording.

[0351] Figures 22A-22C This is an exemplary schematic diagram of a multi-display area scenario provided in an embodiment of this application.

[0352] In electronic devices with multiple screens, each screen represents a display area. For example... Figure 22AAs shown, the main screen of the electronic device serves as display area 1, displaying desktop applications, while the secondary screen serves as display area 2, displaying music applications.

[0353] In situations where an electronic device's screen can have multiple states, each state can consist of one or more display areas. For example... Figure 22B As shown, when the screen of an electronic device is folded, each part of the screen can be a display area; when the screen of an electronic device is unfolded, the left half of the screen and the right half of the screen can each be a display area.

[0354] like Figure 22C As shown, when screen recording is enabled, the electronic device's screen is one display area, and the virtual screen displaying the recorded content is another. The content displayed on both display areas can be the same.

[0355] Figures 23A-23C This is another exemplary schematic diagram of a multi-display-area scenario provided in an embodiment of this application.

[0356] like Figure 23A As shown, electronic device 1 shares the content displayed on its screen with the screen of electronic device 2. In this case, for electronic device 1, its screen is display area 1, and for electronic device 2, its screen is display area 2. The content displayed on the screen of electronic device 2 includes the content displayed on the screen of electronic device 1.

[0357] like Figure 23B As shown, electronic device 1 projects the content displayed on its screen onto the screen of electronic device 2. In this case, for electronic device 1, its screen is display area 1, and for electronic device 2, its screen is display area 2. Figure 23A The difference is that the content displayed on the screen of electronic device 2 is only the content displayed on the screen of electronic device 1.

[0358] like Figure 23C As shown, electronic device 1 projects the content displayed on its screen onto the screen of electronic device 2. In this case, for electronic device 1, its screen is display area 1, and for electronic device 2, its screen is display area 2. Figure 23A , Figure 23B The difference is that the content displayed on the screen of electronic device 2 is a portion of the content displayed on the screen of electronic device 1.

[0359] It is worth noting that in multi-display-area scenarios, and in the case of multiple electronic devices, such as electronic device 1 having display area 1 and electronic device 2 having display area 2, display area 2 can display only the content displayed on the screen of electronic device 1; or, display area 2 can display part or all of the content displayed on the screen of electronic device 1 and the content originally displayed on the screen of electronic device 2, etc.

[0360] (b) Interface generation method in single-device multi-display-area scenarios

[0361] Figure 24 This is another exemplary schematic diagram of the interface generation method provided in the embodiments of this application.

[0362] like Figure 24 As shown, the interface generation methods in a single-device, multi-display-area scenario include:

[0363] S2401: After receiving the vertical signal, construct and generate the rendering tree.

[0364] For details, please refer to [link / reference]. Figure 9 The textual description of step S901 will not be repeated here.

[0365] S2402: Cross-process transfer of render tree

[0366] For details, please refer to [link / reference]. Figure 9 The textual description of step S902 is not repeated here.

[0367] Optionally, in some embodiments of this application, the DisplayRender subprocess may obtain the rendering tree directly through IPC communication with the corresponding application. The UniRender process determines the correspondence between the DisplayRender subprocess and the application.

[0368] Optionally, in some embodiments of this application, there may be a shared memory set consisting of multiple shared memory blocks between all DisplayRender subprocesses and all applications, with each DisplayRender subprocess holding a handle to the multiple shared memory blocks.

[0369] The concept of the DisplayRender subprocess can be found in the corresponding text description in step S2404, and will not be repeated here.

[0370] S2403: Transmits application window control information and display area information.

[0371] For details, please refer to [link / reference]. Figure 9 The textual description of step S903 is not repeated here.

[0372] S2404: Assign Render Tree

[0373] The UniRender process can create a number of child processes, DisplayRender, equal to the number of display areas. Figure 24 The middle part is DisplayRender1 to DisplayRenderN), and the rendering tree of the application displayed on the display area is passed to DisplayRender.

[0374] For example, application 1 and application 2 are displayed on display area 1, which corresponds to the child process DisplayRender1; application 3 and application 4 are displayed on display area 2, which corresponds to the child process DisplayRender2. Then, the UniRender process passes the rendering trees of application 1 and application 2 to the child process DisplayRender1, and passes the rendering trees of application 3 and application 4 to the child process DisplayRender2.

[0375] DisplayRender1 and DisplayRender2 can obtain handles to the shared memory of the corresponding application from the UniRender process, thereby obtaining the rendering tree of the corresponding application. Alternatively, they can obtain the application's rendering tree from the UniRender process through other IPC communication methods such as Binder; this is not limited to these methods.

[0376] Optionally, in some embodiments of this application, the UniRender process may first combine the rendering trees of multiple applications into a target rendering tree, and then pass the target rendering tree to the corresponding DisplayRender subprocess. The passing of the target rendering tree can be achieved through various IPC communication methods such as shared memory and Binder, and is not limited thereto.

[0377] Optionally, in some embodiments of this application, the UniRender process can perform off-screen rendering instruction forward processing on the rendering tree before passing it to the corresponding DisplayRender subprocess, or it can pass the window control information of the application on the corresponding display area and the rendering tree to the corresponding DisplayRender subprocess, which then performs off-screen rendering instruction forward processing on the rendering tree and combines multiple rendering trees into a target rendering tree.

[0378] Optionally, in some embodiments of this application, the DisplayRender subprocess can be configured with two threads, such as an I / O thread and a rendering thread. The I / O thread is responsible for receiving the rendering tree, and the rendering thread is responsible for generating the target rendering tree and generating a bitmap based on the target rendering tree.

[0379] Understandably, the UniRender process creates multiple DisplayRender subprocesses, which can divide and multiply the vertical synchronization signal (Vsync-UR), and then pass the divided and multiplied Vsync-UR to different DisplayRender subprocesses. This allows different DisplayRender subprocesses to generate bitmaps at different frequencies to match the refresh rate of different display areas.

[0380] Figure 25 This is an exemplary schematic diagram of the UniRender process frequency division and multiplication vertical synchronization signal provided in the embodiments of this application.

[0381] like Figure 25 As shown, when the refresh rates of the display areas corresponding to different DisplayRender subprocesses are different, UniRender can divide or multiply the vertical synchronization signal (Vsync-UR) and then forward it to different DisplayRender subprocesses.

[0382] For example, if the refresh rate of display area 1 is 60Hz and the refresh rate of display area 2 is 30Hz, and the DisplayRender subprocess corresponding to display area 1 is DisplayRender1, and the DisplayRender subprocess corresponding to display area 2 is DisplayRender2, then after the UniRender process receives the 60Hz vertical sync signal (Vsync), or after the UniRender process generates the 60Hz vertical sync signal (Vsync), it divides the frequencies to generate a 30Hz vertical sync signal (Vsync-UR) and a 60Hz vertical sync signal (Vsync-UR). The 60Hz vertical sync signal (Vsync-UR) is then passed to the DisplayRender1 subprocess, and the 30Hz vertical sync signal (Vsync-UR) is passed to the DisplayRender2 subprocess.

[0383] It is understandable that by dividing or multiplying the vertical synchronization signal (Vsync-UR), the frequency at which the DisplayUniRender process receives the rendering tree or target rendering tree, and the frequency at which it generates the bitmap, can be made consistent with the refresh frequency of the display area.

[0384] Upon receiving the vertical synchronization signal (Vsync-UR), the DisplayRender subprocess can begin reading the rendering tree, generating the target rendering tree, and generating the bitmap.

[0385] It is worth noting that the UniRender process receives display area information from the display management service, determines the connection status of each display area, and then creates or destroys the DisplayRender process corresponding to each display area based on the connection status of the display area.

[0386] The concepts of UniRender's architecture and shared memory can be found in the text description in step S904, and will not be repeated here.

[0387] S2405: Generate a target rendering tree based on the obtained rendering tree, window control information, and display area information.

[0388] The DisplayRender subprocess combines one or more target render trees into a single target render tree, and, if the application's layers involve off-screen rendering logic, moves the off-screen rendering instructions forward for the application's render tree.

[0389] It is worth noting that the DisplayRender subprocess can obtain window control information and display area information on the display area from the UniRender process.

[0390] Optionally, in some embodiments of this application, the UniRender process may combine one or more render trees into a target render tree before passing it to the DisplayRender subprocess.

[0391] The process of generating the target rendering tree and the forward movement of off-screen rendering instructions can be found in the text description of step S904, and will not be repeated here.

[0392] S2406: Generating a bitmap based on the target rendering tree

[0393] After generating the target rendering tree, the DisplayRender subprocess generates a bitmap based on the target rendering tree. After generating the bitmap, the DisplayRender subprocess passes the surface carrying the bitmap to the UniRender process, which then sends the content of the surface to each display area via the DSS.

[0394] The process of generating a bitmap based on the target rendering tree can be referred to in the text description of step S905, and will not be repeated here.

[0395] Because the calls to the underlying graphics library depend on the context, different DisplayRender subprocesses correspond to different contexts, thus allowing bitmaps to be generated in parallel for different display areas.

[0396] Understandably, the UniRender process can generate bitmaps for each display area in parallel by creating DisplayRender subprocesses that correspond one-to-one with the display area. In multi-display-area scenarios, this can improve the speed of interface generation, reduce lag, and enhance the user experience.

[0397] The following is combined with Figure 26A , Figure 26B The content shown exemplifies the data flow process when an electronic device implements an interface generation method.

[0398] Figure 26A , Figure 26B This is an exemplary schematic diagram of data flow when an electronic device implements the interface generation method provided in the embodiments of this application.

[0399] like Figure 26A As shown, the electronic device has two display areas, namely display area 1 and display area 2. Display area 1 displays the content of application 1 and application 3; display area 2 displays the content of application 2, application 3, and application 4.

[0400] The UniRender process creates a DisplayRender1 child process corresponding to display area 1 and a DisplayRender2 child process corresponding to display area 2. After receiving rendering tree 1 from application 1, rendering tree 2 from application 2, rendering tree 3 from application 3, and rendering tree 4 from application 4, the UniRender process passes rendering tree 1 and rendering tree 3 to the DisplayRender1 child process, and rendering trees 2, 3, and 4 to the DisplayRender2 child process. The UniRender process also passes relevant window control information and display area information to the DisplayRender child processes.

[0401] The DisplayRender1 and DisplayRender2 subprocesses independently generate bitmaps based on the target rendering tree. The DisplayRender1 subprocess generates surface1 based on the display area information of display area 1, and the DisplayRender2 subprocess generates surface2 based on the display area information of display area 2. Surface1 is used to hold bitmap 1 generated by the DisplayRender1 subprocess; surface2 is used to hold bitmap 1 generated by the DisplayRender1 subprocess. Figure 2 The size of surface1 can be related to the size of display area 1, and the size of surface2 can be related to the size of display area 2.

[0402] In this embodiment, the content displayed by the application on different display areas can be the same or different. When the content displayed by the application is the same on different display areas, one rendering tree generated by the application will be assigned to multiple DisplayRender subprocesses; when the content displayed by the application is different on different display areas, the application will generate multiple different rendering trees, which will be assigned to multiple DisplayRender subprocesses accordingly.

[0403] For example, in Figure 26B As shown, because display area 1 and display area 2 have different sizes (e.g., resolutions), application 3 will display different interfaces on display area 1 and display area 2 (e.g., different layout structures, but the same content). Therefore, the application will generate rendering trees corresponding to different display areas, namely rendering tree 31 and rendering tree 32. Rendering tree 31 will be passed to the DisplayRender1 subprocess, and rendering tree 32 will be passed to the DisplayRender2 subprocess.

[0404] Each DisplayRender subprocess can be configured with two threads, such as an I / O thread and a rendering thread. The I / O thread is responsible for receiving the rendering tree, and the rendering thread is responsible for generating the target rendering tree and generating a bitmap based on the target rendering tree.

[0405] (c) Interface generation method in multi-device, multi-display-area scenarios

[0406] like Figures 23A-23C As shown in the scenario, electronic devices can display part or all of the content on their screens on the screen of the other electronic device through methods such as screen mirroring and multi-screen collaboration.

[0407] In a multi-device, multi-display-area scenario, after establishing a connection with the peer electronic device, the local electronic device can use the screen of the peer electronic device as display area 2, and pass the rendering tree of the application to be displayed on display area 2 to the peer electronic device. After receiving the rendering tree from the local electronic device, the peer electronic device will combine the rendering trees of all applications displayed on display area 2 into a target rendering tree, and then generate a bitmap and send it for display.

[0408] Alternatively, the local electronic device can send the interface of the application displayed on display area 2 to the remote electronic device.

[0409] Figure 27 This is another exemplary schematic diagram of the interface generation method provided in the embodiments of this application.

[0410] like Figure 27 As shown, the interface generation method provided in this application includes:

[0411] S2701: Establish connection

[0412] Electronic device 1 and electronic device 2 establish a communication connection through various methods such as Bluetooth, Wi-Fi, and HiLink. Electronic device 1 uses the screen of electronic device 2 as display area 2, while electronic device 1's own screen serves as display area 1.

[0413] After determining that electronic devices 1 and 2 have established a connection, or after receiving requests from display management services and window management services for screen mirroring, multi-screen collaboration, etc., the UniRender process on electronic device 2 can request to allocate a heap memory to store the data sent by electronic device 1 for rendering and generating bitmaps, such as the rendering tree.

[0414] S2702: Determine the application displayed on display area 2.

[0415] The window management service and display management service on electronic device 1 determine the application to be displayed on display area 2 and pass the result to the UniRender1 process on electronic device 1. Then, the UniRender1 process on electronic device 1 obtains the rendering tree of the application to be displayed on display area 2 through IPC communication.

[0416] The UniRender1 process obtains the application's rendering tree through IPC communication, as described in step S902 above, and will not be repeated here.

[0417] S2703: Send the rendering tree of the application displayed on display area 2.

[0418] The UniRender1 process of electronic device 1 sends the rendering tree of the application displayed on display area 2 to electronic device 2.

[0419] UniRender1 can determine the starting address and size of the shared memory, and through the communication connection in step S2701, it passes the rendering tree to the already allocated heap memory of the UniRender2 process of the electronic device 2.

[0420] The storage structure of data in the heap memory can be consistent with the storage structure of data in the shared memory on UniRender1. The storage structure of data in the heap memory and the read / write security of the heap memory can be found in the textual description in step S902, and will not be repeated here.

[0421] It is understandable that sending the rendering tree from electronic device 1 to electronic device 2 can reduce the amount of data transmitted between electronic device 1 and electronic device 2, thereby reducing latency and increasing the speed of interface generation.

[0422] S2704: Composite the application's render tree displayed on display area 2 into the target render tree.

[0423] After receiving the vertical synchronization signal (Vsync-UR2), the UniRender2 process of electronic device 2 will synthesize the rendering tree of the application displayed on display area 2 into the target rendering tree, and in the process, it can complete the forwarding of off-screen rendering instructions.

[0424] Among them, electronic device 2 can combine the rendering tree sent by electronic device 1 and the local rendering tree of electronic device 2 to form a target rendering tree.

[0425] Figure 28A , Figure 28B This is an exemplary schematic diagram illustrating the synthesis of a target rendering tree from the rendering tree provided in the embodiments of this application.

[0426] like Figure 28A As shown, electronic device 1 runs application 1 and application 2. The rendering tree generated by application 1 is called rendering tree 1, and the rendering tree generated by application 2 is called rendering tree 2. Electronic device 2 runs application 3, and the rendering tree generated by application 3 is called rendering tree 3.

[0427] If the UniRender1 process on electronic device 1 determines that the interface of application 2 should be displayed on display area 2, it sends rendering tree 2 to the UniRender2 process on electronic device 2.

[0428] The UniRender1 process moves off-screen rendering instructions forward from render tree 1 and render tree 2 respectively, and combines them into target render tree 1, generating bitmap 1. The UniRender2 process moves off-screen rendering instructions forward from render tree 2 and render tree 3 respectively, and combines them into target render tree 2, generating bitmap 1. Figure 2 .

[0429] It's worth noting that when the UniRender1 process sends rendering tree 2 data to the UniRender2 process, it can send either all of the rendering tree 2 data or a subset of it. The all-inclusive rendering tree 2 data can be current properties, current displaylist, or staging displaylist and staging properties stored in shared memory. The subset of rendering tree 2 data can be the list of drawing instructions and rendering attributes of the rendering nodes in the properties_dirty queue and displaylist_dirty queue.

[0430] like Figure 28B As shown, electronic device 1 runs application 1, application 2, and application 3. The rendering tree generated by application 1 is called rendering tree 1, the rendering tree generated by application 2 is called rendering tree 2, and the rendering tree generated by application 3 is called rendering tree 3. Electronic device 2 runs application 4, and the rendering tree generated by application 4 is called rendering tree 4.

[0431] If the UniRender1 process on electronic device 1 determines that the interfaces of application 2 and application 3 should be displayed on display area 2, then it sends rendering tree 2 and rendering tree 3 to the UniRender2 process of electronic device 2, or it combines rendering tree 2 and rendering tree 3 into target rendering tree 2 and then sends target rendering tree 2 to the UniRender2 process of electronic device 2.

[0432] The UniRender1 process moves off-screen rendering instructions forward for render tree 1, render tree 2, and render tree 3 respectively, and combines them into target render tree 1 to generate bitmap 1; the UniRender1 process moves off-screen rendering instructions forward for render tree 2 respectively, and combines them into target render tree 2.

[0433] Optionally, in some embodiments of this application, the UniRender1 process can create multiple child processes DisplayRender, each receiving and processing rendering trees from different applications. That is, in Figure 28BAs shown, the UniRender1 process can create two child processes, DisplayRender, namely DisplayRender1 and DisplayRender2. The DisplayRender1 child process receives render tree 1, render tree 2, and render tree 3, and composites them into target render tree 1; the DisplayRender2 child process receives render tree 2 and render tree 3, composites them into target render tree 2, and sends target render tree 2 to the UniRender2 process.

[0434] The UniRender2 process moves off-screen rendering instructions forward for render tree 2 and render tree 3 respectively, and combines them into target render tree 2, generating bit... Figure 2 .

[0435] It is worth noting that the UniRender1 process, in response to the vertical synchronization signal (Vsvnc-UR1) on electronic device 1, retrieves the render trees of one or more applications from shared memory and begins generating the target render tree. UniRender2, in response to the vertical synchronization signal (Vsync-UR2) on electronic device 2, retrieves the render trees of one or more applications from heap memory and / or shared memory and begins generating the target render tree.

[0436] Optionally, in some embodiments of this application, when the frequencies of the vertical synchronization signal (Vsync-UR1) on electronic device 1 and the vertical synchronization signal (Vsync-UR2) on electronic device 2 are inconsistent, UniRender on electronic device 1 can adjust the frequency of the vertical synchronization signal (Vsync-UR1) to be the same as that of Vsync-UR2 by frequency division or frequency multiplication; or UniRender2 on electronic device 2 can adjust the frequency of the vertical synchronization signal (Vsync-UR2) to be the same as that of the vertical synchronization signal (Vsync-UR1) by frequency division or frequency multiplication.

[0437] Optionally, in some embodiments of this application, the electronic device 1 may be configured with multiple frequencies of vertical synchronization signals (Vsync-UR1). For example, in Figure 28A , Figure 28BIn the scenario shown, the original vertical sync signal (Vsync-UR) frequency on electronic device 1 is 60Hz, and the original vertical sync signal (Vsync-UR2) frequency on electronic device 2 is 90Hz. After electronic device 1 determines that the interface of application 2 will be displayed on electronic device 2, the UniRender process of electronic device 1 can generate vertical sync signal (Vsync-UR1) frequencies of 60Hz and 90Hz respectively, causing application 2 to generate a render tree at a frequency of 90Hz. Then, the UniRender1 process can pass the render tree to the UniRender2 process at a frequency of 90Hz. Meanwhile, application 1 can still generate a render tree at a frequency of 60Hz.

[0438] In particular, on any electronic device, the vertical synchronization signal (Vsync-UR) received or generated by UniRender can be the same as the vertical synchronization signal (Vsync-APP) received by the application.

[0439] S2705: Generating a bitmap based on the target rendering tree

[0440] For details, please refer to the text description in step S905, which will not be repeated here.

[0441] Optionally, in some embodiments of this application, the UniRender1 process of electronic device 1 may first combine one or more rendering trees on display area 2 into a target rendering tree, then generate a bitmap based on the target rendering tree, and pass the bitmap to the UniRender2 process of electronic device 2.

[0442] It's worth noting that when electronic device 2 displays the interface of application 1 on electronic device 1, and the user clicks on the interface of application 1 on electronic device 2 through clicks or other interactive methods, electronic device 2 can send the location of the user's click within the interface of application 1 to application 1 on electronic device 1, allowing application 1 to correctly respond to the user's interaction. Alternatively, since the interface of application 1 on electronic device 2 is generated by the UniRender2 process on electronic device 2 based on a rendering tree, and the rendering tree contains view position information, electronic device 2 can determine the view clicked by the user and directly send the view click event to application 1 on electronic device 1, enabling application 1 to correctly respond to the user's interaction.

[0443] (3) Finally, the hardware architecture and software architecture of the electronic device provided in the embodiments of this application are introduced.

[0444] Figure 29 This is an exemplary schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.

[0445] The following describes the embodiments using an electronic device as an example. It should be understood that the electronic device may have more or fewer components than shown in the figures, may combine two or more components, or may have different component configurations. The various components shown in the figures can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0446] The electronic device may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0447] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0448] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0449] The controller can serve as the nerve center and command center of an electronic device. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0450] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0451] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0452] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device.

[0453] The I2S interface can be used for audio communication. The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. The UART interface is a universal serial data bus used for asynchronous communication.

[0454] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to achieve the electronic device's shooting function. The processor 110 and the display screen 194 communicate via the DSI interface to achieve the electronic device's display function. The GPIO interface can be configured via software. The GPIO interface can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc. The SIM interface can be used to communicate with the SIM card interface 195 to achieve the function of transferring data to or reading data from the SIM card. USB interface 130 is an interface that conforms to the USB standard specification, specifically it can be a Mini USB interface, Micro USB interface, USB Type C interface, etc.

[0455] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0456] The charging management module 140 is used to receive charging input from the charger. The power management module 141 is used to connect the battery 142, and the charging management module 140 is connected to the processor 110.

[0457] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. Mobile communication module 150 can provide solutions for wireless communication applications in electronic devices, including 2G / 3G / 4G / 5G. Mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. Mobile communication module 150 can receive electromagnetic waves through antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to modem processor for demodulation. Mobile communication module 150 can also amplify the signal modulated by modem processor and convert it into electromagnetic waves for radiation through antenna 1. In some embodiments, at least some functional modules of mobile communication module 150 can be disposed in processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the same device as at least some modules of the processor 110.

[0458] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0459] The wireless communication module 160 can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0460] In some embodiments, antenna 1 of the electronic device is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling the electronic device to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0461] Electronic devices implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0462] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include one or N displays 194, where N is a positive integer greater than 1.

[0463] Electronic devices can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor, thereby acquiring real-time video data.

[0464] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise and brightness. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0465] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1.

[0466] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when an electronic device is selecting a frequency, a DSP can perform a Fourier transform on the frequency energy.

[0467] Video codecs are used to compress or decompress digital video. Electronic devices can support one or more video codecs. This allows the electronic device to play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0468] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, voice recognition, and text understanding.

[0469] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0470] Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, fifth generation DDR SDRAM is generally called DDR5 SDRAM), etc.

[0471] Non-volatile memory can include disk storage devices and flash memory.

[0472] In this embodiment, the non-real-time video may be located in non-volatile memory.

[0473] Flash memory can be classified according to its operating principle, including NOR FLASH, NAND FLASH, 3D NAND FLASH, etc.; according to the level of the storage cell, including single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc.; and according to the storage specification, including universal flash storage (UFS) and embedded multimedia card (eMMC), etc.

[0474] The random access memory can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0475] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110.

[0476] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0477] Electronic devices can implement audio functions such as music playback and recording through audio modules 170, speakers 170A, receivers 170B, microphones 170C, headphone jacks 170D, and application processors.

[0478] Audio module 170 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Speaker 170A, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. Receiver 170B, also called a "handpiece," is used to convert audio electrical signals into sound signals. Microphone 170C, also called a "microphone" or "microphone unit," is used to convert sound signals into electrical signals.

[0479] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0480] A pressure sensor 180A is used to sense pressure signals and can convert them into electrical signals. In some embodiments, the pressure sensor 180A may be located on a display screen 194. A gyroscope sensor 180B can be used to determine the motion posture of the electronic device. A barometric pressure sensor 180C is used to measure barometric pressure. A magnetic sensor 180D includes a Hall effect sensor. An accelerometer sensor 180E can detect the magnitude of acceleration of the electronic device in various directions (typically three axes). A distance sensor 180F is used to measure distance. A proximity sensor 180G may include, for example, a light-emitting diode (LED) and a photodetector, such as a photodiode. An ambient light sensor 180L is used to sense ambient light intensity. A fingerprint sensor 180H is used to collect fingerprints. The electronic device can utilize the collected fingerprint characteristics to achieve fingerprint unlocking, access app locks, fingerprint photography, fingerprint answering of calls, etc. A temperature sensor 180J is used to detect temperature.

[0481] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of the electronic device, in a different position than display screen 194.

[0482] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. The electronic device can receive button input and generate key signal inputs related to user settings and function control. Motor 191 can generate vibration alerts. Indicator 192 can be an indicator light, used to indicate charging status, battery level changes, messages, missed calls, notifications, etc. SIM card interface 195 is used to connect a SIM card. The electronic device interacts with the network through the SIM card to achieve functions such as calls and data communication.

[0483] Figure 30This is an exemplary schematic diagram of the software structure of the electronic device provided in the embodiments of this application.

[0484] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into four layers, from top to bottom: the application layer, the application framework layer, the system library layer, and the kernel layer.

[0485] The application layer can include a series of application packages. For example... Figure 20 As shown, the application package may include applications (also known as apps) such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0486] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0487] like Figure 30 As shown, the application framework layer may include window management services, display management services, content providers, view systems, phone managers, resource managers, notification managers, local profile management assistants (Local Profile Assistant, LPA), etc.

[0488] The window management service is responsible for starting, adding, and deleting windows. It can determine the application displayed on the window, as well as the creation, destruction, and attribute changes of the application's layers. It can also determine whether there is a status bar, lock the screen, and capture the screen.

[0489] The display management service can obtain the number and size of display areas, and is responsible for starting, adding, and deleting display areas.

[0490] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0491] A phone manager is used to provide communication functions for electronic devices. For example, it manages call status (including connection and disconnection).

[0492] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0493] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog-style notifications on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0494] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0495] The view system also includes UniRender, which can receive the rendering trees of one or more applications. UniRender can synchronize layer information, such as layer creation, destruction, and attribute changes, through the window management service. UniRender can also synchronize display area information, such as screen size, from the display management service.

[0496] Optionally, in some embodiments of this application, the view system further includes SurfaceFlinger. On an electronic device configured with a whitelist, when an application is not on the whitelist, the application's UI thread generates a render tree, the application's rendering thread generates a bitmap, and then SurfaceFlinger performs layer composition.

[0497] Optionally, in some embodiments of this application, when both whitelisted and non-whitelisted applications are displayed on the display area, the generation of bitmaps for whitelisted applications is handled by UniRender. After generating the bitmap, UniRender passes it to SurfaceFlinger, which then composites the bitmap with other non-whitelisted applications to generate a bitmap for display.

[0498] The runtime includes the core libraries and the virtual machine. The runtime is responsible for the scheduling and management of the operating system.

[0499] The core library consists of two parts: one part contains the functionalities that the Java language needs to call, and the other part is the core library itself.

[0500] The application layer and application framework layer run in a virtual machine. The virtual machine executes the application layer and application framework layer's iava files as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0501] System libraries can include multiple functional modules. For example: surface manager, media libraries, and graphics processing libraries, where the graphics processing libraries include: 3D graphics processing libraries (e.g., OpenGLES), 2D graphics engines (e.g., SGL), etc.

[0502] The Surface Manager is used to manage the display subsystem and provides the fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0503] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0504] 3D graphics processing libraries are used to implement 3D graphics drawing, image rendering, layer compositing, and layer processing. 2D graphics engines are drawing engines for 2D graphics.

[0505] The kernel layer is the layer between hardware and software. The kernel layer includes at least the display driver, camera driver, audio driver, sensor driver, and virtual card driver.

[0506] In this embodiment, one or more applications in the application layer pass their respective UI thread-generated render trees to the UniRender process of the view system. The UniRender process obtains window control information and display area information from the window management service and the display management service, and then composites the render trees of the applications on a display area into a target render tree. After generating the target render tree, the UniRender process calls the layer processing library to execute the DrawOP operation in the drawing instruction list of the target render tree, thereby generating a bitmap. UniRender then passes the generated bitmap to the display driver for display.

[0507] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0508] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0509] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An interface generation method, applied to an electronic device, the electronic device having at least two display areas, the at least two display areas including a first display area and a second display area, characterized in that, include: The third process receives the first rendering tree generated by the first process, and the first rendering tree is used to draw the interface of the first process to be displayed on the first display area; The third process generates a first interface based on the first rendering tree. The first interface includes the interface of the first process and is used to display on the first display area. The fourth process receives the second rendering tree generated by the second process. The second rendering tree is used to draw the interface of the second process to be displayed on the second display area. The fourth process, the third process, the second process and the first process are all different. The fourth process generates a second interface based on the second rendering tree. The second interface includes the interface of the second process and is used to display on the second display area.

2. The method according to claim 1, characterized in that, Before the third process receives the first rendering tree generated by the first process, and before the fourth process receives the second rendering tree generated by the second process, the method further includes: The fifth process determines that the at least two display areas include the first display area and the second display area; The fifth process generates the third process and the fourth process, and the third process and the fourth process are child processes of the fifth process.

3. The method according to claim 1 or 2, characterized in that, The third process receives the first render tree generated by the first process, specifically including: The third process receives the first rendering tree based on inter-process communication, which includes at least one of Binder, Interface Definition Language (AIDL), shared memory, and Socket.

4. The method according to claim 2, characterized in that, The method further includes: The fifth process sends a first vertical synchronization signal to the first process at a first frequency. The first vertical synchronization signal is used to instruct the first process to generate the first rendering tree. The first frequency is the refresh frequency of the first display area. The third process receives the first rendering tree at the first frequency.

5. The method according to claim 4, characterized in that, The method further includes: The fifth process sends a second vertical synchronization signal to the second process at a second frequency. The second vertical synchronization signal is used to instruct the second process to generate the second rendering tree. The second frequency is the refresh frequency of the second display area. The fourth process receives the second render tree at the second frequency, which is different from the first frequency.

6. The method according to claim 1, characterized in that, The method further includes: the third process receiving the sixth rendering tree generated by the sixth process; The third process generates a first interface based on the first rendering tree, specifically including: the third process generates a first interface based on the first rendering tree and the sixth rendering tree, and the first interface also includes the interface corresponding to the sixth process.

7. The method according to claim 6, characterized in that, The third process generates the first interface based on the first rendering tree and the sixth rendering tree, specifically including: The third process creates a root rendering node as the root node of the first target rendering tree, and uses the first rendering tree and the sixth rendering tree as child nodes of the root rendering node; The third process generates the first interface based on the first target rendering tree.

8. The method according to claim 7, characterized in that, The method further includes: The third process deletes rendering nodes in the first target rendering tree based on the Z-order of the first rendering tree and the sixth rendering tree. The deleted rendering nodes correspond to views that are completely obscured.

9. The method according to claim 7, characterized in that, The method further includes: The third process, based on the Z-order of the first and sixth rendering trees, deletes drawing operations in the first target rendering tree. The deleted drawing operations correspond to graphics that are completely occluded.

10. The method according to claim 6, characterized in that, The method further includes: During the process of the third process generating the first interface based on the first rendering tree and the sixth rendering tree, the third process merges or batches the first drawing operation and the second drawing operation. The first drawing operation belongs to the first rendering tree, and the second drawing operation belongs to the sixth rendering tree.

11. An electronic device, characterized in that, The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-10.

12. A chip system applied to an electronic device, the chip system comprising one or more processors, the processors being configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-10.

13. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-10.

14. A computer program product comprising computer-readable instructions that, when executed by one or more processors, implement the method as described in any one of claims 1-10.

Images