Method, device, equipment and product for processing user interface

By employing multi-surface rendering and dynamic switching of rendering contexts, the stability and scalability issues of traditional user interface tree rendering in complex scenes are resolved, achieving stable display of independent subtrees and consistency across viewports.

CN121501400APending Publication Date: 2026-02-10BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202511642532.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-11-10
Publication Date
2026-02-10

AI Technical Summary

Technical Problem

Traditional user interface tree rendering uses a single surface rendering path, which leads to high coupling between rendering and compositing in complex scenes, limiting stability and scalability.

Method used

A multi-surface rendering method is adopted to display independent or critical subtrees on independent surfaces, and the atomicity of the environment context and the consistency across viewports are ensured by dynamically switching the rendering context.

Benefits of technology

It achieves stability and consistency across multiple surfaces in rendering, reduces the risk of animation desynchronization and tearing, and improves the consistency of overlay semantics and input method avoidance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121501400A_ABST
    Figure CN121501400A_ABST
Patent Text Reader

Abstract

The invention relates to a method, a device, equipment and a product for processing a user interface. The method includes determining a first node based on a node tree of a user interface. The method further includes determining a first sub-tree of the first node in the node tree based on the first node. In addition, the method also includes drawing a first node and a first sub-tree on the first surface. Moreover, the method also includes drawing at least a portion of the other nodes in the node tree, except the nodes of the first sub-tree, on a second surface, the first surface being different from the second surface. Through the method, multi-surface rendering can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of computers, and more particularly, to a method, apparatus, device and computer program product for processing a user interface. BACKGROUND

[0002] Conventional user interface (UI) tree rendering usually adopts a single surface rendering path, i.e., all nodes of the UI tree are uniformly drawn and composed in the same rendering target, and the screen display is controlled by the root view. In this way, when complex scenarios such as cross-border overlay, input method avoidance, scroll linkage, etc. are involved, the rendering and composition are highly coupled, and the stability and extensibility are limited. SUMMARY

[0003] In an embodiment of the present disclosure, a method, apparatus, electronic device and computer program product for processing a user interface are provided.

[0004] In a first aspect of the present disclosure, a method for processing a user interface is provided. The method includes determining a first node based on a node tree of the user interface. The method also includes determining a first subtree of the first node in the node tree based on the first node. In addition, the method further includes drawing the first node and the first subtree on a first surface. And, the method also includes drawing at least a portion of other nodes in the node tree except for the nodes of the first subtree on a second surface, the first surface being different from the second surface.

[0005] In a second aspect of the present disclosure, an apparatus for processing a user interface is provided. The apparatus includes a first determining module configured to determine a first node based on a node tree of the user interface. The apparatus also includes a second determining module configured to determine a first subtree of the first node in the node tree based on the first node. In addition, the apparatus further includes a first drawing module configured to draw the first node and the first subtree on a first surface. And, the apparatus also includes a second drawing module configured to draw at least a portion of other nodes in the node tree except for the nodes of the first subtree on a second surface, the first surface being different from the second surface.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes a processor. The electronic device also includes a memory coupled with the processor, the memory having stored therein instructions that, when executed by the processor, cause the electronic device to perform the method according to the first aspect of the present disclosure.

[0007] In a fourth aspect of the present disclosure, a computer program product is provided, which includes computer executable instructions, wherein the computer executable instructions are executed by a processor to implement the method according to the first aspect of the present disclosure.

[0008] In a fifth aspect of the present disclosure, a computer readable storage medium is provided. The computer readable storage medium has stored thereon computer executable instructions, wherein the computer executable instructions are executed by a processor to implement the method according to the first aspect.

[0009] The summary is intended to provide a simplified form of the concepts selected to introduce the subject matter, which will be further described below in the detailed description. The summary is not intended to identify key or essential features of the claimed subject matter or limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and other features, aspects and advantages of various embodiments of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, like reference numerals refer to like elements, wherein:

[0011] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of the present disclosure can be implemented is shown;

[0012] Figure 2 A flowchart of a method 200 for processing a user interface of some embodiments of the present disclosure is shown;

[0013] Figure 3 A schematic flowchart of a method 300 for hierarchical recording or rendering according to some embodiments of the present disclosure is shown;

[0014] Figure 4 A block diagram of a system 400 for user interface processing according to some embodiments of the present disclosure is shown;

[0015] Figure 5 A timing and flowchart of a rendering method 500 based on an Android platform according to one or more embodiments of the present disclosure is shown;

[0016] Figure 6 A general flowchart of a method 600 for rendering based on providing a shared image base according to one or more embodiments of the present disclosure is shown;

[0017] Figure 7 A block diagram of an apparatus 700 for processing a user interface according to some embodiments of the present disclosure is shown; and

[0018] Figure 8 A block diagram of a device 800 capable of implementing various embodiments of the present disclosure is shown.

[0019] In all the drawings, like reference numerals refer to like elements throughout. DETAILED DESCRIPTION

[0020] It can be understood that the data (including but not limited to the data itself, the acquisition or use of the data) involved in the technical solutions of the present disclosure should comply with the requirements of the corresponding laws and regulations and relevant provisions.

[0021] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type of personal information involved in the present disclosure, the use range, the use scenario, etc. should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0022] For example, when receiving the active request of the user, the prompt information is sent to the user to explicitly prompt the user that the operation requested to be executed will need to acquire and use the personal information of the user. Thus, the user can voluntarily choose whether to provide the personal information to the software or hardware such as the electronic device, the application program, the server or the storage medium, etc. that executes the operation of the technical solutions of the present disclosure according to the prompt information.

[0023] As an optional but non-limiting implementation manner, in response to receiving the active request of the user, the manner of sending the prompt information to the user may, for example, be the manner of a pop-up window, and the prompt information may, for example, be presented in the form of text in the pop-up window. In addition, the pop-up window may, for example, also carry a selection control for the user to select “agree” or “disagree” to provide the personal information to the electronic device.

[0024] It can be understood that the above notification and acquisition of the authorization of the user are only illustrative, and do not limit the implementation manners of the present disclosure, and other manners that meet the relevant laws and regulations can also be applied to the implementation manners of the present disclosure.

[0025] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, but rather, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the scope of protection of the present disclosure.

[0026] In the description of the embodiments of the present disclosure, the term “comprising” and similar terms are to be understood as open-ended including, i.e., “including but not limited to”. The term “based on” is to be understood as “at least partially based on”. The term “one embodiment” or “the embodiment” is to be understood as “at least one embodiment”. The terms “first”, “second”, etc. can refer to different or same objects unless explicitly stated otherwise. Other explicit and implicit definitions can also be included below.

[0027] As described above, the traditional user interface tree rendering usually takes a single surface rendering path, i.e., all nodes of the UI tree are uniformly drawn and composited in the same rendering target, and the screen display is controlled by the root view. In complex scenarios involving at least one of cross-border overlay, input method avoidance, or scroll linkage, the rendering and compositing are highly coupled, and the stability and extensibility are greatly limited.

[0028] To this end, one or more embodiments of the present disclosure propose a multi-surface rendering method. In another aspect, one or more embodiments of the present disclosure also propose a method of dynamic switching of rendering contexts. For example, one or more embodiments of the present disclosure can switch a UI subtree that needs to be independently rendered to a dedicated target surface or window for drawing in the same frame period, restore the original context after completion, and stably combine with the main surface in the compositing stage. For another example, one or more embodiments of the present disclosure can realize autonomous display on the screen across windows while maintaining the declarative semantics of the single view tree, effectively reducing the risk of animation asynchronization and tearing, and improving the consistency and predictability of overlay semantics and input method avoidance.

[0029] In another specific embodiment, the present disclosure provides a method for processing a user interface. The method includes determining a first node based on a node tree of the user interface. The method also includes determining a first subtree of the first node in the node tree based on the first node. And the method further includes drawing the first node and the first subtree on a first surface, and the method still further includes drawing at least a portion of nodes other than the first subtree in the node tree on a second surface, the first surface being different from the second surface.

[0030] In this way, one or more embodiments of the present disclosure achieve good results. For example, on the one hand, one or more embodiments of the present disclosure can perform multi-surface rendering to allow independent or critical subtrees to be displayed independently on independent surfaces. On the other hand, one or more embodiments of the present disclosure can also perform dynamic switching of rendering environment contexts, where the switching and restoration of the environment context are atomic and do not cause mutual interference between different surfaces. In another aspect, one or more embodiments of the present disclosure can also achieve stable cross-surface compositing, predictable overlay and rendering order, and better cross-window consistency, and other beneficial technical effects.

[0031] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of the present disclosure can be implemented is shown. As shown in FIG. 1, the environment 100 includes a device 102, a server 104, and a network 106. Figure 1As shown, environment 100 includes a graphics application 102, a surface rendering component 104 provided in one or more embodiments of this disclosure, and a graphics interface or graphics environment 106. Within environment 100, the graphics application 102 can run on the graphics interface or graphics environment 106. In the graphics application 102, users can create user interfaces using materials such as text, images, videos, or live photos. For example, the graphics application 102 can be a content publishing platform, a video creation platform, a video editing platform, or a social platform. The surface rendering component 104, provided in one or more embodiments of this disclosure, calls relevant functions of the graphics interface or graphics environment 106 based on the UI tree obtained from the graphics application 102 to perform surface rendering and form the corresponding user interface. Furthermore, the graphics application 102 can display the relevant user interface on the display screen 108.

[0032] It should be understood that, for the sake of brevity, Figure 1 Only a specific number of graphics applications 102, graphics environments 106, and sub-surface rendering components 104 are shown in this specification, but it is not intended to limit the number of graphics applications 102, graphics environments 106, and sub-surface rendering components 104. In other implementations, any number of graphics applications 102, graphics environments 106, and sub-surface rendering components 104 suitable for this disclosure may be included, thereby adapting to different platforms.

[0033] In this way, one or more embodiments of this disclosure achieve good results. For example, one or more embodiments of this disclosure can perform multi-surface rendering, allowing independent or critical subtrees to be displayed independently on independent surfaces. On the other hand, one or more embodiments of this disclosure can also perform dynamic switching of rendering environment contexts, wherein the switching and restoration of environment contexts are atomic and do not cause mutual interference between different surfaces. Furthermore, one or more embodiments of this disclosure can also achieve beneficial technical effects such as stable cross-surface compositing, predictable coverage and rendering order, and better cross-viewpoint consistency.

[0034] The following will combine Figures 2 to 8 The process according to embodiments of this disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and not intended to limit the scope of this disclosure. It should be understood that the embodiments described below may also include additional actions not shown and / or actions shown may be omitted, and the scope of this disclosure is not limited in this respect.

[0035] Figure 2A flowchart illustrating a method 200 for processing a user interface according to some embodiments of the present disclosure is shown. Method 200 can be executed by a processing device. For example, method 200 can be executed by a computing processing device. The computing processing device may include, for example, a smartphone, tablet computer, laptop computer, desktop computer, workstation, quantum computer, or smart wearable device, etc.

[0036] In box 202, a first node is determined based on the node tree of the user interface. In some embodiments, the first node may include a child window or the like that needs to be rendered independently. In some embodiments, determining the first node based on the node tree of the user interface may include determining the first node using a depth-first search strategy. In some embodiments, the node tree of the user interface may be described by JavaScript or the like.

[0037] In box 204, based on the first node, the first subtree of the first node in the node tree is determined. In some embodiments, method 200 may further include, after determining the first node and its first subtree, returning to the root node and again using a depth-first search strategy to determine whether there are other independent nodes that need to be drawn independently. In some embodiments, method 200 may also switch the environment context in response to completing the determination of the first subtree of the first node, so as to complete the drawing of the first node and the first subtree on the first surface. In some embodiments, switching the environment context may include calling the graphics interface corresponding to the first surface.

[0038] In block 206, a first node and a first subtree are drawn on a first surface. In some embodiments, method 200 further includes creating a new container layer for the first surface. In some embodiments, method 200 further includes invoking a graphical interface corresponding to the first surface. In some embodiments, invoking the graphical interface corresponding to the first surface may include replacing the graphical interface of the environment context with the graphical interface corresponding to the first surface, the graphical interface including a recording interface and a drawing interface. In some embodiments, drawing the first node and the first subtree on the first surface may include recording the nodes of the first node and the first subtree through the recording interface in the graphical interface, and drawing the nodes of the first node and the first subtree on the first surface based on the recorded nodes of the first node and the first subtree through the drawing interface in the graphical interface. In some embodiments, drawing the first node and the first subtree on the first surface may further include invoking at least one of a start write method or an end write method, wherein the start write method represents the start of drawing the first node and the first subtree on the first surface, and the end write method represents the end of drawing the first node and the first subtree on the first surface.

[0039] In block 208, at least a portion of the nodes in the node tree other than the nodes of the first subtree are drawn on the second surface, where the first surface differs from the second surface. In some embodiments, in response to completing the drawing of the first node and the first subtree on the first surface, the original context is restored to complete the drawing of at least a portion of the nodes in the node tree other than the nodes of the first subtree on the second surface. In some embodiments, switching the context or restoring the original context is limited to a frame period. In some embodiments, method 200 may further include combining the first surface and the second surface, and displaying the combined first surface and the second surface. In some embodiments, combining the first surface and the second surface may include receiving the first surface and the second surface frame by frame via a frame queue, and combining the first surface and the second surface frame by frame.

[0040] In some embodiments, method 200 may further include determining a second node based on a node tree of the user interface. In some embodiments, method 200 may further include determining a second subtree of the second node based on the second node. In some embodiments, method 200 may further include drawing the second node and the second subtree on a third surface. Furthermore, method 200 may further include switching the environment context in response to completing the determination of the second subtree of the second node, to complete drawing the second node and the second subtree on the third surface. Additionally, in some embodiments, method 200 may further include restoring the original environment context in response to completing the drawing of the second node and the second subtree on the third surface. In some embodiments, method 200 may further include overlaying the user interfaces corresponding to the first surface, the second surface, and the third surface, as well as the status bar in the user interface, based on predetermined hierarchical rules. In some embodiments, method 200 may further include displaying the first surface, the second surface, and the third surface in the same window or multiple independent windows.

[0041] In this way, one or more embodiments of method 200 achieve good results. For example, one or more embodiments of method 200 can perform multi-surface rendering, allowing independent or critical subtrees to be displayed independently on independent surfaces. On the other hand, one or more embodiments of method 200 can also perform dynamic switching of rendering environment contexts, wherein the switching and restoration of environment contexts are atomic and do not cause mutual interference between different surfaces. Furthermore, one or more embodiments of method 200 can also achieve beneficial technical effects such as stable cross-surface compositing, predictable overlay and rendering order, and better cross-viewpoint consistency.

[0042] As described above, one or more embodiments of this disclosure provide methods for rendering a single-view tree across multiple target surfaces or windows. Additionally, one or more embodiments of this disclosure provide methods for dynamically switching rendering contexts, and one or more embodiments of this disclosure provide methods for stable compositing across surfaces. These methods involve layered collaboration at the rendering level, including recording layers and compositors, shared image bases and target surfaces, platform window semantic adaptation, or unified event chains. The following sections will provide a detailed description of the rendering-related aspects.

[0043] Figure 3 A schematic flowchart of a method 300 for layered recording or drawing according to some embodiments of the present disclosure is shown. Figure 3 In this process, a UI tree 301 of the user interface is obtained. In some embodiments, the UI tree 301 of the user interface may include a root node 303, child root nodes 305 and 307 corresponding to independent windows, etc. Child root nodes 305 and 307 and their corresponding leaf nodes can constitute the subtrees corresponding to child root nodes 305 and 307. Based on the UI tree 301 of the user interface, a layer tree 309 containing layer subtrees 311 and 313 corresponding to each independent child window can be determined by, for example, a depth-first strategy. In some embodiments, method 300 can draw layer subtree 311 onto a first surface 319. In some embodiments, method 300 can draw layer subtree 311 onto a second surface 317. In some embodiments, method 300 can draw nodes in layer tree 309 other than the nodes in layer subtrees 311 and 313 onto a main surface 315. In some embodiments, the first surface 319, the second surface 317, and the main surface 315 are different surfaces. The first surface 319 and the second surface 317 are used for independent layer subtrees to be displayed independently on the screen, thereby separating them from the main rendering surface to reduce coupling.

[0044] In some implementations, method 300 may further include switching the environment context in response to completing the determination of the layer subtree 311 of node 305, to complete drawing the layer subtree 311 on the first surface 319. In some implementations, method 300 may further include restoring the original environment context in response to completing the drawing of the layer subtree 311 on the first surface 319, to complete drawing subsequent surfaces. In some implementations, method 300 may further include creating a new container layer for the first surface 319. In some implementations, switching the environment context may include calling the graphics interface corresponding to the first surface 319. In some implementations, calling the graphics interface corresponding to the first surface 319 may include replacing the graphics interface of the environment context with the graphics interface corresponding to the first surface 319, such as a recording interface graphics library (GL) and a drawing interface Canvas. In some implementations, method 300 may further include recording nodes of the subtree 311 via the recording interface GL in the graphics interface, and drawing the subtree 311 on the first surface 319 based on the recorded nodes of the subtree 311 via the drawing interface Canvas in the graphics interface. In some implementations, method 300 may further include calling a start write method, which indicates the start of drawing the first node and the first subtree on the first surface, or calling an end write method, which indicates the end of drawing the first node and the first subtree on the first surface. In some implementations, switching the environment context is limited to completion within a frame period. Similarly, it is also possible to draw the subtree 311 onto the second surface 317, and to draw the nodes in the subtree 309 other than the nodes of the subtrees 311 and 313 onto the main surface 315. In some implementations, method 300 may further include, in response to completing the drawing of layer subtrees 311 and 313 on the first surface 319 and the second surface 317, restoring the original context to complete drawing all nodes in the layer tree 309 except for the nodes of layer subtrees 311 and 313 onto the main surface 315. This completes the recording and drawing of the key subtree and other nodes of the UI tree, and then atomically restores the context.

[0045] In some implementations, method 300 may further include combining the first surface 319, the second surface 317, and the main surface 315 via a synthesizer, and displaying the combined first surface 319, the second surface 317, and the main surface 315. In some implementations, combining the first surface 319, the second surface 317, and the main surface 315 may include receiving the first surface 319, the second surface 317, and the main surface 315 frame by frame via a frame queue, and combining the first surface 319, the second surface 317, and the main surface 315 frame by frame. In some implementations, method 300 may further include stably combining the main surface with independent target surfaces during the synthesis stage via a synthesizer, ensuring presentation order and anti-tearing strategy.

[0046] In this way, one or more embodiments of method 300 achieve good results. For example, one or more embodiments of method 300 can perform multi-surface rendering, allowing independent or critical subtrees to be displayed independently on independent surfaces. On the other hand, one or more embodiments of method 300 can also perform dynamic switching of rendering environment contexts, wherein the switching and restoration of environment contexts are atomic and do not cause mutual interference between different surfaces. Furthermore, one or more embodiments of method 300 can also achieve beneficial technical effects such as stable cross-surface compositing, predictable overlay and rendering order, and better cross-viewpoint consistency.

[0047] Figure 4 A schematic framework diagram of a system 400 for user interface processing according to some embodiments of the present disclosure is shown. Figure 4 In this system 400, there are platform layer implementation (subsystem) 407, framework implementation (subsystem) 417, and engine implementation (subsystem) 429. Platform layer implementation (subsystem) 407 refers to the specific structure and method of implementing one or more embodiments of this disclosure on a platform. Platform layer implementation (subsystem) 407 includes a platform full-screen window, such as a Dialog on an Android platform or a UIAlertController on an information operating system. Platform layer implementation (subsystem) 407 interacts with the PlatformRecordingTarget method 419 and the OverlayViewNG method 421 in framework implementation (subsystem) 417. The PlatformRecordingTarget method module 419 is used for platform recording targets, while the OverlayViewNG method module 421 is used for overlay view rendering, etc. The interaction between platform layer implementation (subsystem) 407 and framework implementation (subsystem) 417 can be accomplished through the recording target plugin 415 and the overlay platform plugin 413 of framework implementation (subsystem) 417. The information recorded by the target plugin 415 may include platform layer surface information or surface texture. Additionally, the information recorded by overriding the platform plugin 413 may include UI parameters or events, user touch events, etc.

[0048] In some implementations, the OverlayViewNG method module 421 can be specifically processed by the engine implementation (module) 429. The engine implementation (module) 429 is used to render various surfaces and ultimately form the user interface. The engine implementation (module) 429 mainly includes the RenderRecordingObject method module 425, the Clay:SubwindowLayer method module 431, and the flutter:SubwindowLayer method module 435. In some implementations, the RenderRecordingObject method module 425 can be used to render general recorded objects (such as the main surface), the Clay:SubwindowLayer method module 431 is used to handle the rendering of sub-window layers (such as the first surface, the second surface, etc.), and the flutter:SubwindowLayer method module 435 can be used to handle the jitter of sub-window layers during compositing. The RenderRecordingObject method module 425, the Clay:SubwindowLayer method module 431, and the flutter:SubwindowLayer method module 435 all obtain the processing objects from the Shared_img_backing shared image bases 423, 427, or 433, respectively.

[0049] In some implementations, engine implementation (module) 429 can combine multiple surfaces. In some implementations, engine implementation (module) 429 can also receive and combine individual surfaces frame by frame via a frame queue. In some implementations, system 400 can also overlay multiple surfaces based on predetermined hierarchy rules, or display multiple surfaces in the same window or multiple independent windows on multiple platforms. In some implementations, multiple surfaces include child windows that need to be rendered independently and the main interface in the user interface. In some implementations, system 400 can also map hierarchy rules to platform window types to achieve alignment such as overlay and input method avoidance.

[0050] In this way, one or more embodiments of system 400 achieve good results. For example, one or more embodiments of method 400 can perform multi-surface rendering, allowing independent or critical subtrees to be displayed independently on independent surfaces. On the other hand, one or more embodiments of system 400 can also perform dynamic switching of rendering environment contexts, wherein the switching and restoration of environment contexts are atomic and do not cause mutual interference between different surfaces. Furthermore, one or more embodiments of system 400 can also achieve beneficial technical effects such as stable cross-surface compositing, predictable overlay and rendering order, and better cross-viewpoint consistency.

[0051] In some implementations, from a rendering perspective, one or more embodiments of this disclosure establish a continuous data and control flow through the SharedImgBacking method during the rendering build phase, using upper-layer declared overlay rules (e.g., hierarchy, display state). The SharedImgBacking method provides a shared image base, traversing between rendering subtrees and layer trees, providing a unified framework for cross-tree / cross-context drawing. At the start of a frame, the recording layer switches to the image interface or environment GL / Canvas of an independent target before drawing the layer subtree. Upon completion of drawing, the original context is restored, and the compositor combines the first and second surfaces with the main surface during the rendering phase. Additionally, the platform adaptation layer implements overlay / avoidance strategies based on the hierarchy mapping and binds the event acquisition window.

[0052] Figure 5 A timing and flow diagram of a rendering method 500 based on the Android platform according to one or more embodiments of the present disclosure is shown. Figure 5 In the code, rendering method 500 is based on the Android platform's `Tasm:CreateView` 501. Rendering method 500 also involves the following methods: `ViewFactory` (view factory) 502, `OverlayViewNG` (overlay view rendering container) 503, `OverlayPlatformPlugin` (overlay platform plugin) 505, `OverlayNGDialog` (overlay rendering dialog) 507, `RecordingTextureView.java` (Java-based recording texture view) 509, `RenderRecordingContentObject` (rendering recorded content object) 511, `clay:SubWindowLayer` (composition: subwindow layer) 513, and `flutter:SubWindowLayer` (anti-shaking: subwindow layer) 515. Figure 5 In the middle, at 517, an instance of OverlayViewNG (overlay view rendering container) 503 can be generated based on ViewFactory 502. Generally speaking, the instance of OverlayViewNG (overlay view rendering container) 503 is based on C++, so it is necessary to further generate the Java-based OverlayPlatformPlugin plugin 505 at 519.

[0053] At point 521, the Java-based OverlayPlatformPlugin plugin 505 is required to generate a dialog instance for the OverlayNGDialog method 507. At point 523, a texture view is generated based on the dialog instance of OverlayNGDialog method 507. At point 525, the surface texture of the generated texture view is updated via RecordingTextureView.java 509, forming an updated surface texture object, which is returned to the OverlayPlatformPlugin plugin 505 at point 527. The OverlayPlatformPlugin plugin 505 then notifies the OverlayViewNG 503 at point 529 that the updated surface texture is ready. The OverlayViewNG 503 then notifies the RenderRecordingContentObject method 511 at point 533 to set the SharedImgBacking method to provide a shared image base. At 531, the prepared surface texture is made visible, and at 535, the RenderRecordingContentObject method (rendering the recorded content object) is instructed to begin drawing preparations at 511.

[0054] At points 537 and 539, the `RenderRecordingContentObject` method (rendering the recorded content object) 511 adds a surface layer and sets a shared image base. At points 541 and 543, based on the added surface layer and the set shared image base, a frame is further added and the shared image base is set even further. At points 545, 547, 549, 551, and 553, the `flutter:SubWindowLayer` method (debouncing: sub-window layer) 515 runs the `SharedImgBacking:BeginWrite` method, changes the rendering environment context, and begins recording and drawing the sub-window. After recording and drawing are complete, the original rendering environment context is restored, and the `SharedImgBacking:EndWrite` method is run to end the drawing process. This completes the entire process of layered user interface drawing.

[0055] Therefore, it is evident that one or more embodiments of method 500 achieve good results through this approach. For example, one or more embodiments of method 500 can perform multi-surface rendering, allowing independent or critical subtrees to be displayed independently on separate surfaces. Furthermore, one or more embodiments of method 500 can also dynamically switch rendering environment contexts, wherein the switching and restoration of environment contexts are atomic, without causing interference between different surfaces. Additionally, one or more embodiments of method 500 can also achieve beneficial technical effects such as stable cross-surface compositing, predictable coverage and rendering order, and better cross-viewpoint consistency.

[0056] On the other hand, one or more embodiments of this disclosure can perform context switching before drawing sub-windows, ensuring that critical subtrees are displayed on independent targets without affecting the composition of the main surface, and can use the SharedImgBacking::BeginWrite / EndWrite methods to wrap the drawing, ensuring the atomicity of EGL / GL switching and Canvas restoration. In addition, the mapping between hierarchy rules and platform window types is fixed with overlay semantics, and dynamic adjustment is prohibited in the display state to avoid jitter. Figure 6 This illustrates a general flowchart of a method 600 for rendering based on providing a shared image pedestal according to one or more embodiments of the present disclosure. Figure 6 In method 600, the textures to be drawn can be prepared at 601, 603 and 605 respectively, and the recording base can be set. The shared recording base is set by the RecordingRender::SetContainerLayer(Recordinglayer) method.

[0057] At points 607 and 609, method 600 uses the `PageView::Paint` and `PaintingContext::RepaintCompositedChildGetContainerLayer` methods to obtain the child window that needs to be drawn in the container layer and switches the rendering context. At point 611, it checks if it's overlay rendering. If so, it uses the `PaintingContext::PushRecordingLayer` method at point 613 to push the recording layer to set a shared recording base to set the initial frame. At point 615, it sets the initial frame using the `PageView::BeginFrame` method. At point 617, it sets the shared recording base again using the `RecordingLayer::AddRecordingLayerToFrame` method to prepare for drawing and rendering the child window. At point 619, it begins operations on the RasterRunner to finally complete the drawing of the child window. At points 621 and 623, based on RasterRunner, the `Rasterizer::OnDraw` method is used to begin drawing the child window using the `RecordingLayer::Paint` method. At point 625, the `SharedImgBacking::BeginWrite` method indicates the start of drawing the child window. At point 627, the actual drawing of the child window takes place. Finally, at point 629, the `SharedImgBacking::EndWrite` method indicates the end of drawing the child window, thus ensuring the atomicity or independence of the child window drawing.

[0058] In summary, context switching between the initial render and frame-by-frame rendering is crucial for stability. During the initial render, a shared base is established to maintain continuity; during frame-by-frame rendering, the atomicity of switching and restoring is ensured to avoid animation desynchronization and tearing. This is achieved by creating a new container layer and switching the GL context and Canvas context in the rendering context to the GL context and Canvas context corresponding to the child window before drawing the child window. The original context is then restored after the child window is drawn.

[0059] Therefore, it is evident that one or more embodiments of method 600 achieve good results through this approach. For example, one or more embodiments of method 600 can perform multi-surface rendering, allowing independent or critical subtrees to be displayed independently on separate surfaces. Furthermore, one or more embodiments of method 600 can also dynamically switch rendering environment contexts, wherein the switching and restoration of environment contexts are atomic, without causing interference between different surfaces. Additionally, one or more embodiments of method 600 can also achieve beneficial technical effects such as stable cross-surface compositing, predictable overlay and rendering order, and better cross-viewpoint consistency.

[0060] In one or more embodiments of this disclosure, during the initial rendering and frame-by-frame drawing, the GL context of the drawing context is switched with the surface corresponding to the independent target (e.g., the child window) pointed to by the SkCanvas before drawing the child window, and the original context is restored after drawing is completed. When constructing the rendering tree, SharedImgBacking is passed through, and finally, the switching between the EGL context and Canvas is controlled within the recording layer through BeginWrite / EndWrite to ensure the atomicity and consistency of drawing.

[0061] In one or more embodiments of this disclosure, frame scheduling can follow a "switch-record-restore-compositing" approach. After the BeginFrame method is triggered, the recording layer completes the subtree drawing on an independent target and restores the context before the EndFrame, handing it over to the compositor for unified compositing. By limiting context switching and the write window to within the frame period, cross-frame drift that causes animation desynchronization is avoided. Double buffering can be used if necessary to reduce tearing.

[0062] In one or more embodiments of this disclosure, frame-by-frame synchronization assurance can achieve predictable beats by switching and resuming only within a single frame. Frame-by-frame synchronization assurance can also reduce synthesis tearing by employing double buffering and vertical synchronization strategies for independent targets. Furthermore, frame-by-frame synchronization assurance can ensure presentation order and cross-window consistency through frame queue delivery between the recording layer and the compositor.

[0063] In one or more embodiments of this disclosure, overlays are expressed as levels and mapped to platform window types to align system coverage and input method avoidance strategies. Generally, dynamic adjustment of levels is not allowed in the visible state; changes only occur in the hidden state to avoid window system jitter. Table 1 shows some examples of level rules as follows: Table 1

[0064] In one or more embodiments of this disclosure, cross-platform rendering backend adaptation uses SharedImgBacking as the carrying abstraction, combined with the target switching interface of RecordingLayer, to maintain the same context switching and write semantics when replacing the rendering backend, reducing migration costs and ensuring consistency. In some embodiments, regarding compatibility recommendations (rendering and window semantics), on the one hand, layer adjustments should be performed in the hidden state, and changes should be minimized during display to ensure stable coverage relationships; on the other hand, for input method avoidance, a combination strategy of nothing + smart-scroll is recommended to reduce jitter caused by content translation.

[0065] In some embodiments, a rendering configuration that binds layers and targets can bind independent targets and layer mappings to a configuration-driven rendering pipeline, thereby decoupling coverage semantics from independent on-screen display. For example, this configuration establishes a SharedImgBacking through-link during the build phase; during frame-by-frame rendering, the recording layer switches to the surface before drawing the sub-window, and resumes this state after drawing is complete, with the compositor outputting according to the composition_order.

[0066] In some embodiments, for rendering scheduling and frame-by-frame display (or presentation on the screen), the recording layer can execute `BeginWrite` to switch the EGL / GL context of the target surface before the start of each frame, and execute `EndWrite` to restore the rendering after the subtree is drawn, ensuring animation consistency with atomicity. This is suitable for complex scrolling and overlapping animation scenarios. After recording, playback can be performed on different surfaces to achieve the result of rendering different subtree results separately on different surfaces.

[0067] Figure 7 A block diagram of an apparatus 700 for processing a user interface according to some embodiments of the present disclosure is shown. Figure 7 As shown, the device 700 includes a first determining module 702 configured to determine a first node based on a node tree of the user interface. The device 700 also includes a second determining module 704 configured to determine a first subtree of the first node in the node tree based on the first node. Furthermore, the device 700 includes a first drawing module 706 configured to draw the first node and the first subtree on a first surface. The device 700 also includes a second drawing module 708 configured to draw at least a portion of the nodes in the node tree other than the nodes of the first subtree on a second surface, where the first surface is different from the second surface.

[0068] It is understandable that one or more embodiments of device 700 achieve good results in this way. For example, one or more embodiments of device 700 are capable of multi-surface rendering, allowing independent or critical subtrees to be displayed independently on independent surfaces.

[0069] Figure 8 A block diagram of a device 800 capable of implementing various embodiments of the present disclosure is shown. (See diagram for example.) Figure 8 As shown, device 800 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or loaded from storage unit 808 into random access memory (RAM) 803. The RAM 803 can also store various programs and data required for the operation of device 800. The CPU / GPU 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804. Although not shown in... Figure 8 As shown, device 800 may also include a coprocessor.

[0070] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0071] The various methods or processes described above can be executed by CPU / GPU 801. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU / GPU 801, one or more steps or actions in the methods or processes described above can be performed.

[0072] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0073] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0074] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0075] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to execute the computer-readable program instructions, thereby implementing various aspects of this disclosure.

[0076] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0077] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0078] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0079] This disclosure also provides the following examples:

[0080] Example 1. A method for processing a user interface, comprising:

[0081] Based on the node tree of the user interface, determine the first node;

[0082] Based on the first node, determine the first subtree of the first node in the node tree;

[0083] Draw the first node and the first subtree on the first surface; and

[0084] At least a portion of the nodes in the node tree other than the nodes of the first subtree are drawn on the second surface, where the first surface is different from the second surface.

[0085] Example 2. The method described in Example 1 further includes:

[0086] In response to the completion of determining the first subtree of the first node, switch the environment context to complete drawing the first node and the first subtree on the first surface.

[0087] Example 3. The method described in Examples 1-2 further includes:

[0088] In response to the completion of drawing the first node and the first subtree on the first surface, the original environment context is restored to complete the drawing of at least some of the nodes of the node tree on the second surface, excluding the nodes of the first subtree.

[0089] Example 4. The method described in Examples 1-3, wherein switching the environment context or restoring the original environment context is limited to being completed within a frame period.

[0090] Example 5. The method described in Examples 1-4 further includes:

[0091] Create a new container layer for the first surface.

[0092] Example 6. The method described in Examples 1-5, wherein switching the environment context includes:

[0093] Call the graphics interface corresponding to the first surface.

[0094] Example 7. The method described in Examples 1-6, wherein calling the graphics interface corresponding to the first surface includes:

[0095] Replace the graphical interface of the environment context with the graphical interface corresponding to the first surface. The graphical interface includes a recording interface and a drawing interface.

[0096] Example 8. According to the method described in Examples 1-7, drawing the first node and the first subtree on the first surface includes:

[0097] Record the first node and the nodes of the first subtree using the recording interface in the graphical interface; and

[0098] Based on the recorded first node and the nodes of the first subtree, draw the first node and the nodes of the first subtree on the first surface through the drawing interface in the graphics interface.

[0099] Example 9. The method according to Examples 1-8, wherein drawing the first node and the first subtree on the first surface further includes at least one of the following:

[0100] Calling the `startWrite` method indicates that the first node and the first subtree are being drawn on the first surface; or

[0101] Call the end-write method, which indicates the end of drawing the first node and the first subtree on the first surface.

[0102] Example 10. The method according to Examples 1-9 further includes:

[0103] Combining the first and second surfaces; and

[0104] Showing the first and second surfaces after combination.

[0105] Example 11. The method according to Examples 1-10, wherein combining the first surface and the second surface includes:

[0106] The first and second surfaces are received frame by frame through a frame queue; and

[0107] The first and second surfaces are combined frame by frame.

[0108] Example 12. The method according to Examples 1-11 further includes:

[0109] Determine the second node based on the node tree of the user interface;

[0110] Based on the second node, determine the second subtree of the second node;

[0111] Draw the second node and the second subtree on the third surface.

[0112] Example 13. The method according to Examples 1-12 further includes:

[0113] In response to the completion of determining the second subtree of the second node, switch the environment context to complete drawing the second node and the second subtree on the third surface.

[0114] Example 14. The method according to Examples 1-13 further includes:

[0115] In response to the completion of drawing the second node and the second subtree on the third surface, restore the original environment context.

[0116] Example 15. The method described according to Examples 1-14 further includes any one of the following:

[0117] Based on predetermined hierarchical rules, the user interfaces corresponding to the first, second, and third surfaces, as well as the status bar within the user interface, are displayed in overlay; or

[0118] The first surface, the second surface, and the third surface can be displayed in the same window or multiple independent windows.

[0119] Example 16. The method according to Examples 1-15, wherein the first node or the second node includes a sub-window that needs to be presented independently, and the second surface corresponds to the main interface in the user interface.

[0120] Example 17. According to the method described in Examples 1-16, determining the first node based on the node tree of the user interface includes:

[0121] The node tree based on the user interface determines the first node using a depth-first search strategy.

[0122] Example 18. An apparatus for processing a user interface, comprising:

[0123] The first determining module is configured to determine the first node based on the node tree of the user interface;

[0124] The second determining module is configured to determine the first subtree of the first node in the node tree based on the first node;

[0125] A first drawing module is configured to draw the first node and the first subtree on a first surface; and

[0126] The second drawing module is configured to draw at least a portion of the nodes in the node tree other than the nodes of the first subtree on a second surface, the first surface being different from the second surface.

[0127] Example 19. An electronic device comprising:

[0128] Processor; and

[0129] A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of Examples 1 to 17.

[0130] Example 20. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of Examples 1 to 17.

[0131] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for processing a user interface, comprising: Based on the node tree of the user interface, determine the first node; Based on the first node, determine the first subtree of the first node in the node tree; Draw the first node and the first subtree on the first surface; as well as At least a portion of the nodes in the node tree other than those in the first subtree are drawn on a second surface, the first surface being different from the second surface.

2. The method according to claim 1, further comprising: In response to completing the determination of the first subtree of the first node, the environment context is switched to complete the drawing of the first node and the first subtree on the first surface.

3. The method according to claim 2, further comprising: In response to completing the drawing of the first node and the first subtree on the first surface, the original environment context is restored to complete the drawing of at least some of the nodes of the node tree other than the nodes of the first subtree on the second surface.

4. The method according to claim 3, wherein the switching of the environment context or the restoration of the original environment context is completed within a frame period.

5. The method according to claim 1, further comprising: Create a new container layer for the first surface.

6. The method of claim 2, wherein the switching environment context includes: Call the graphics interface corresponding to the first surface.

7. The method according to claim 6, wherein calling the graphics interface corresponding to the first surface includes: The graphical interface of the environment context is replaced with a graphical interface corresponding to the first surface, the graphical interface including a recording interface and a drawing interface.

8. The method of claim 7, wherein drawing the first node and the first subtree on the first surface comprises: Record the first node and the nodes of the first subtree through the recording interface in the graphical interface; as well as Based on the recorded first node and the nodes of the first subtree, the first node and the nodes of the first subtree are drawn on the first surface through the drawing interface in the graphics interface.

9. The method of claim 8, wherein drawing the first node and the first subtree on the first surface further comprises at least one of the following: Call the start-write method, which indicates that the first node and the first subtree are to be drawn on the first surface; or Call the end-write method, which indicates the end of drawing the first node and the first subtree on the first surface.

10. The method of claim 1, further comprising: Combine the first surface and the second surface; as well as The combined first and second surfaces are shown.

11. The method of claim 10, wherein combining the first surface and the second surface comprises: The first and second surfaces are received frame by frame through a frame queue. as well as The first surface and the second surface are combined frame by frame.

12. The method according to claim 1, further comprising: Based on the node tree of the user interface, determine the second node; Based on the second node, determine the second subtree of the second node; Draw the second node and the second subtree on the third surface.

13. The method of claim 12, further comprising: In response to the completion of determining the second subtree of the second node, the environment context is switched to complete the drawing of the second node and the second subtree on the third surface.

14. The method of claim 13, further comprising: In response to the completion of drawing the second node and the second subtree on the third surface, restore the original environment context.

15. The method of claim 12, further comprising any one of the following: Based on predetermined hierarchical rules, the user interfaces corresponding to the first, second, and third surfaces, as well as the status bar within those user interfaces, are displayed in overlay; or The first surface, the second surface, and the third surface can be displayed in the same window or multiple independent windows.

16. The method of claim 12, wherein the first node or the second node includes a sub-window that needs to be presented independently, and the second surface corresponds to the main interface in the user interface.

17. The method of claim 1, wherein determining the first node based on the user interface-based node tree comprises: The first node is determined based on the node tree of the user interface using a depth-first search strategy.

18. An apparatus for processing a user interface, include:. The first determining module is configured to determine the first node based on the node tree of the user interface; The second determining module is configured to determine the first subtree of the first node in the node tree based on the first node; The first drawing module is configured to draw the first node and the first subtree on the first surface; as well as The second drawing module is configured to draw at least a portion of the nodes in the node tree other than the nodes of the first subtree on a second surface, the first surface being different from the second surface.

19. An electronic device comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1 to 17.

20. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of claims 1 to 17.