Rendering method, apparatus for rendering frame on basis of rendering service, and electronic device

Abstract

The present application provides a rendering method, an apparatus for rendering a frame on the basis of a rendering service, and an electronic device. The rendering method comprises: a rendering thread synchronizes drawing information of a first changed node, wherein the first changed node is a node whose drawing information of a next frame of an application changes relative to drawing information of a current frame of the application, and the drawing information comprises rendering attribute information and a drawing instruction; the rendering thread replays a drawing instruction of a first target node on a first canvas, wherein the first target node is determined on the basis of the first changed node, and the first target node corresponds to an area to be rendered of the next frame; and an execution thread performs rendering on the first target node on the basis of the drawing information of the first target node to obtain the next frame. The technical solution of the embodiments of the present application can reduce rendering time and reduce the possibility of picture freezing.

Description

Rendering method, device for rendering frames based on rendering service, and electronic device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on February 28, 2024, with application number 202410223629.6 and application name “Rendering method, device and electronic device for rendering frames based on rendering service”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of graphics processing, and more specifically, to a rendering method, a device for rendering frames based on a rendering service, and an electronic device. Background Art

[0003] 2D graphics display is a key factor influencing the smoothness of the user experience on electronic devices. However, as electronic devices display increasingly richer content, the rendering overhead is increasing. When electronic devices render complex scenes or under heavy rendering load, they may experience display interface lag and frame skipping, resulting in a poor user experience. Summary of the Invention

[0004] The present application provides a rendering method, a device for rendering frames based on a rendering service, and an electronic device, which can reduce the time consumed in rendering and improve the rendering speed.

[0005] In a first aspect, a rendering method is provided, which is applied to an electronic device, and the method includes: a rendering thread synchronizing drawing information of a first change node, where the first change node is a node whose drawing information of the next frame of the application changes relative to the current frame of the application, and the drawing information includes rendering attribute information and drawing instructions; the rendering thread replays the drawing instructions of a first target node on a first canvas, where the first target node is determined based on the first change node, and the first target node corresponds to an area to be rendered of the next frame; and an execution thread performs rendering on the first target node based on the drawing information of the first target node to obtain the next frame.

[0006] In this embodiment, the rendering thread updates the drawing information of the first changed node and replays the drawing instructions, while the execution thread performs the specific rendering process, such as calling the GPU and / or CPU for rendering. This allows the rendering process to be completed in parallel by different threads. While the previous frame's rendering is in progress, the rendering thread can update the changed nodes of the next frame and replay the drawing instructions. Compared to the above process where both processes are executed by a single thread, this can reduce the processing time of a single frame when rendering complex scenes or under high rendering load, thereby reducing the perception of image lag.

[0007] In combination with the first aspect, in certain implementations of the first aspect, before the rendering thread synchronizes the drawing information of the first changed node, the method further includes: the rendering thread replays the drawing instructions of the second target node on the second canvas, and the second target node is determined based on the node whose drawing information of the current frame of the application has changed relative to the previous frame of the application, wherein the first canvas and the second canvas are created based on different canvas contexts.

[0008] When the rendering thread and execution thread process the previous frame, they also synchronously change the node's drawing information, replay drawing instructions, and perform rendering. To avoid conflicts between multiple threads accessing drawing instructions from adjacent frames, any two or more adjacent frames can use different canvas contexts. For example, any two, three, four, or more consecutive frames can use different canvas contexts, allowing the drawing instructions of two, three, four, or more adjacent frames to be managed independently.

[0009] The canvas contexts of the current frame and the next frame are different. In addition to the two consecutive frames, there may be more frames adjacent to these two frames that use different canvas contexts from these two frames. This application does not limit this.

[0010] In one embodiment, N rotating canvas contexts may be used, and each frame may be sequentially bound to the N rotating contexts with N as a period.

[0011] In combination with the first aspect, in certain implementations of the first aspect, before the rendering thread synchronizes the drawing information of the first changed node, the method further includes: the rendering thread replays the drawing instructions of the second target node on the second canvas, and the second target node is determined based on the node whose drawing information of the current frame of the application has changed relative to the previous frame of the application, wherein the first canvas and the second canvas are created based on the same canvas context, and the drawing instructions of the first target node and the drawing instructions of the second target node are stored in different memories corresponding to the canvas context.

[0012] In this embodiment, the canvas context used by each frame can be the same, but any two or more adjacent frames can use different memories of the same canvas context. For example, any two, three, four, or more consecutive frames can use different memories of the same canvas context, so that the drawing instructions of the two, three, four, or more adjacent frames can be managed independently. For example, N rotating memories can be obtained from the memory corresponding to the canvas context, so that the drawing instructions of the N adjacent frames are stored in the N memories in sequence.

[0013] In combination with the first aspect, in certain implementations of the first aspect, before the rendering thread plays back the drawing instructions of the first target node on the first canvas, the method further includes: the rendering thread obtains a first cache, which is used to store the rendering result of the next frame; the rendering thread determines, based on the first cache, to execute the step of playing back the drawing instructions of the first target node on the first canvas created based on the first canvas context.

[0014] In this embodiment, the number of rotating buffers (rotating buffers) is equal to the number of rotating canvas contexts and they are associated one-to-one. Therefore, when the rendering thread obtains a certain rotating buffer, it can determine the canvas context and canvas resources used for the next frame based on the corresponding relationship. In this way, after using the canvas context corresponding to the rotating buffer to create a canvas, there is no need to repeatedly create the canvas when the same buffer is obtained subsequently, thereby reducing overhead.

[0015] In combination with the first aspect, in certain implementations of the first aspect, before the rendering thread plays back the drawing instructions of the first target node on the first canvas, the method also includes: the rendering thread unbinding the shared resources bound to the canvas context of the current frame, the shared resources including at least one of the following: texture resources, text resources, and path resources; when the shared resources are needed to perform rendering on the next frame, the method also includes: the rendering thread binding the shared resources to the canvas context of the next frame.

[0016] In this embodiment, the shared resource bound to the canvas context can be unbound before the rendering thread plays back the target drawing instructions for any frame, so that when the shared resource is needed in the next frame, the unbound shared resource can be directly bound to the canvas context of the next frame without waiting.

[0017] Optionally, the method also includes: when the first canvas plays back the drawing command of the first target node, the rendering thread determines whether the next frame needs to use the shared resources bound to the canvas context of the current frame; if the next frame needs to use the shared resources bound to the canvas context of the current frame, the rendering thread unbinds the shared resources from the canvas context of the current frame; and the rendering thread binds the shared resources to the canvas context of the next frame.

[0018] In this way, when the rendering thread plays back the target drawing instructions for the next frame, it can determine whether the next frame needs to use the shared resources bound to the current frame. If it needs to be used, the shared resources are unbound from the canvas context of the current frame and then bound to the canvas context of the next frame. Since some frames may not need to use the shared resources of the previous frame, unbinding only when it is confirmed that they will be used can reduce the number of unbinding times and reduce the processing overhead of the rendering thread.

[0019] In combination with the first aspect, in certain implementations of the first aspect, before the rendering thread plays back the drawing instructions of the first target node on the first canvas, the method also includes: the rendering thread hands over the shared resources of the current frame to the global context for management, and the shared resources include at least one of the following: texture resources, text resources, and path resources. The global context is a context that can be accessed by the canvas contexts of all frames.

[0020] Since the global context is accessible to all canvas contexts, entrusting shared resources to the global context for management eliminates the need for complex unbinding and unbinding operations on shared resources, reducing the complexity of shared resource management.

[0021] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: when the rendering thread determines that the shared resource needs to be released, the rendering thread submits the release task of the shared resource to the execution thread or submits it to the execution thread through the global context; the execution thread releases the shared resource when the shared resource is not in use.

[0022] For example, the shared resource release process can be triggered when the memory space occupied by the shared resource exceeds a preset threshold (for example, the memory space is full) or when the shared resource used in a certain frame is no longer needed within a preset time after the frame (for example, the shared resource used in the current frame will not be needed in the next frame). The shared resource release process can be triggered by the rendering thread, but can be specifically executed by the execution thread, thereby avoiding resource access errors caused by the execution thread directly releasing the shared resource it is using during rendering.

[0023] In combination with the first aspect, in certain implementations of the first aspect, the application is an associated application of a user interface UI thread, and the method further includes: the UI thread determines that the interface of the associated application changes in the next frame relative to the current frame, and obtains the drawing information of the first change node of the associated application; the UI thread passes the drawing information of the first change node to the rendering thread.

[0024] The rendering architecture used in the embodiment of the present application can be a unified rendering architecture or a separate rendering architecture. For the unified rendering architecture, the rendering thread will obtain the drawing information of the first change node from the UI threads of different applications, and determine the first target node when rendering the entire frame based on the window Z-level relationship and the first change node of the application to which the different UI threads belong. For the separate rendering architecture, the rendering thread and the UI thread are threads belonging to the same application, and the rendering threads and execution threads belonging to different applications process the rendering of each application separately. The first target node of the application is determined based on the first change node of each application and the rendering is executed. The final rendering result is submitted to the surface synthesizer for synthesis and display.

[0025] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: in response to the termination of the first application, the rendering thread submits a drawing instruction resource release task of the node of the first application to the execution thread, where the drawing instruction resource of the node of the first application is a resource that stores the drawing instructions of the node of the first application; after the execution thread performs rendering on the first target node according to the drawing information of the first target node, the execution thread releases the drawing instruction resource of the node of the first application.

[0026] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the rendering thread submits a task of releasing the first surface resource and / or the first image resource to the execution thread, the first surface resource corresponds to some nodes in the first target node, and the first image resource corresponds to some nodes in the first target node; the execution thread releases the first surface resource and / or the first image resource after rendering the first target node according to the drawing information of the first target node according to the task of releasing the first surface resource and / or the first image resource.

[0027] When it is necessary to release any of the drawing instruction resources, the first surface resources, and the first image resources, the rendering thread can submit the corresponding task to the execution thread, so that the execution thread can release the corresponding resources when executing the current rendering task, so that the execution of the execution thread's rendering task will not be affected when the resources are released.

[0028] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a synthesis and display thread synthesizes and displays the rendering result of the next frame, and the synthesis and display thread and the rendering thread belong to different threads of the same process.

[0029] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the synthesis and display thread synthesizes the next frame rendering results of the rendering thread and the execution thread belonging to different applications to obtain the next frame; and the display thread displays the next frame.

[0030] The composite display thread may be a surface flinger.

[0031] In one embodiment, the synthesis display thread and the display thread may be the same thread.

[0032] In a second aspect, a device for rendering frames based on a rendering service is provided, comprising: an application and a rendering service communicating through inter-process communication, the rendering service comprising a rendering thread and an execution thread, wherein the rendering thread is used to: synchronize drawing information of a first change node of the application, the first change node being a node whose drawing information of the next frame of the application changes relative to the current frame of the application, the drawing information comprising rendering attribute information and drawing instructions; replaying drawing instructions of a first target node on a first canvas, the first target node being determined based on the first change node, the first target node corresponding to an area to be rendered of the next frame; the execution thread performing rendering on the first target node based on the drawing information of the first target node to obtain the next frame.

[0033] In combination with the second aspect, in certain implementations of the second aspect, the rendering thread is further used to play back drawing instructions of a second target node on a second canvas, where the second target node is determined based on a node whose drawing information of a current frame of the application has changed relative to a previous frame of the application, wherein the first canvas and the second canvas are created based on different canvas contexts.

[0034] In combination with the second aspect, in certain implementations of the second aspect, the rendering thread is further used to play back the drawing instructions of the second target node on the second canvas, where the second target node is determined based on the node whose drawing information of the current frame of the application has changed relative to the previous frame of the application, wherein the first canvas and the second canvas are created based on the same canvas context, and the drawing instructions of the first target node and the drawing instructions of the second target node are stored in different memories corresponding to the canvas context.

[0035] In combination with the second aspect, in certain implementations of the second aspect, the rendering thread is further used to: obtain a first cache, which is used to store the rendering result of the next frame; and determine, based on the first cache, to execute the drawing instruction step of replaying the first target node on the first canvas created based on the first canvas context.

[0036] In combination with the second aspect, in certain implementations of the second aspect, the rendering thread is further used to: unbind the shared resources bound to the canvas context of the current frame, the shared resources including at least one of the following: texture resources, text resources, and path resources; and bind the shared resources to the canvas context of the next frame when the shared resources are needed to perform rendering on the next frame.

[0037] In combination with the second aspect, in certain implementations of the second aspect, the rendering thread is also used to hand over the shared resources of the current frame to the global context for management, and the shared resources include at least one of the following: texture resources, text resources, and path resources. The global context is a context that can be accessed by the canvas contexts of all frames.

[0038] In combination with the second aspect, in certain implementations of the second aspect, when the rendering thread determines that the shared resource needs to be released, the rendering thread is further used to submit the release task of the shared resource to the execution thread or submit it to the execution thread through the global context; the execution thread is also used to release the shared resource when the shared resource is not in use.

[0039] In combination with the second aspect, in certain implementations of the second aspect, in response to application termination, the rendering thread is further used to submit a drawing instruction resource release task for the application's node to the execution thread, where the drawing instruction resource for the application's node is a resource that stores the drawing instructions for the application's node; the execution thread is further used to release the drawing instruction resource for the application's node after rendering the first target node according to the drawing information of the first target node.

[0040] In combination with the second aspect, in certain implementations of the second aspect, the rendering thread is also used to submit a task of releasing the first surface resource and / or the first image resource to the execution thread, the first surface resource corresponding to some nodes in the first target node, and the first image resource corresponding to some nodes in the first target node; the execution thread is also used to release the first surface resource and / or the first image resource after rendering the first target node according to the drawing information of the first target node according to the task of releasing the first surface resource and / or the first image resource.

[0041] In a third aspect, an electronic device is provided, comprising: one or more processors; one or more memories; and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions that, when executed by the one or more processors, enable the electronic device to perform the method described in the first aspect or any one of the implementations of the first aspect.

[0042] In a fourth aspect, a computer-readable storage medium is provided, wherein a program or instruction is stored in the storage medium. When the program or instruction is executed, the method described in the first aspect or any one of the implementation methods of the first aspect is implemented.

[0043] In a fifth aspect, a chip is provided, comprising: one or more processors; one or more memories, wherein the one or more processors are coupled to the one or more memories and are used to read and execute instructions in the one or more memories to execute the method described in the first aspect or any one of the implementations of the first aspect.

[0044] In a sixth aspect, a computer program product is provided, comprising: a computer program code, which, when executed, implements the method described in the first aspect or any one of the implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] FIG1 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.

[0046] FIG2 is a structural block diagram of an electronic device provided in an embodiment of the present application.

[0047] FIG3 is a schematic flow chart of a method for drawing an interface.

[0048] 4A and 4B are operation sequence diagrams when the rendering thread executes various steps.

[0049] FIG5 is a schematic flowchart of a rendering method provided in an embodiment of the present application.

[0050] FIG6 is a timing diagram of the execution process of each thread when the rendering method provided in an embodiment of the present application is executed.

[0051] 7A and 7B are schematic diagrams of context binding of the rendering method provided in an embodiment of the present application.

[0052] FIG8 is a schematic diagram of processing texture resources when executing the rendering method provided in an embodiment of the present application.

[0053] FIG9 is a schematic block diagram of an apparatus for rendering frames based on a rendering service according to an embodiment of the present application.

[0054] FIG10 is a schematic block diagram of another apparatus for rendering frames based on a rendering service provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The technical solutions of this application will be described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, rather than all the embodiments.

[0056] In the description of the embodiments of this application, unless otherwise specified, " / " represents or. For example, A / B can represent A or B. "And / or" in this article is simply a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of this application, "plurality" or "a plurality" means two or more than two.

[0057] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0058] The terms used in the following examples are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and appended claims of this application, the singular expressions "a," "an," "said," "above," "the," and "this" are intended to include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the following examples of this application, "at least one," "one or more" refer to one, two, or more than two.

[0059] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "one embodiment," "some embodiments," "another embodiment," and "other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically stated. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically stated.

[0060] In order to facilitate understanding of the embodiments of the present application, a brief explanation of the terms involved in the embodiments of the present application is first given.

[0061] 1. Rendering

[0062] Rendering is the process of generating an image from a model using software. A model is a description of a three-dimensional object or virtual scene that is strictly defined using language or data structures. It includes information such as geometry, viewpoint, texture, lighting, and shading.

[0063] 2. User interface (UI)

[0064] UI, also known as the human-computer interaction interface, is primarily used for information exchange. UI controls are the components that make up the UI. For example, UI controls can include button controls, input box controls, slider controls, GUI switching button controls, progress bar controls, status switch controls, text controls, vertical layout controls, horizontal layout controls, grid layout controls, and more.

[0065] It should be noted that in the embodiments of the present application, UI controls are simply referred to as controls.

[0066] 3. Control Tree

[0067] The control tree is also called the "window tree" or "view tree." It consists of a root node and at least one branch. Each branch consists of at least one intermediate node and a tail node. On each branch, the current node and the node directly connected to it constitute a parent-child relationship. That is, the current node is the parent of the next node, and the next node is the child of the current node.

[0068] 4. Render Tree

[0069] The render tree consists of a series of render objects, which correspond roughly one-to-one to the widget tree. Each render object contains layout information (such as size and position), style attributes such as color and font. The main purpose of the render tree is to determine the specific presentation of each element on the page, and ultimately draw it to the screen.

[0070] For example, FIG1 shows a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.

[0071] As shown in FIG1 , electronic device 100 may include a central processing unit (CPU) 101, a graphics processing unit (GPU) 102, and a display device 103. Optionally, electronic device 100 may also include at least one communication bus, memory, etc. (not shown in FIG1 ) for enabling connection and communication between various components and storing various instructions.

[0072] It should be understood that the various components in the electronic device 100 may also be coupled via other connectors, which may include various interfaces, transmission lines, or buses. The various components in the electronic device 100 may also be connected in a radial manner centered around the central processing unit 101. In various embodiments of the present application, coupling refers to mutual electrical connection or communication, including direct connection or indirect connection through other devices.

[0073] There are many ways to connect the CPU 101 and the GPU 102, and they are not limited to the way shown in Figure 1. The CPU 101 and the GPU 102 in the electronic device 100 can be located on the same chip or on separate chips.

[0074] The following briefly introduces the functions of the CPU 101 , GPU 102 , and display device 103 .

[0075] CPU 101: Used to run an operating system 105 and an application 107. Application 107 can be a graphics application, such as a game, video player, etc. Operating system 105 provides a system graphics library interface, through which operating system 105 receives an instruction stream (e.g., rendering instructions) sent by application 107 for rendering graphics or image frames. Drivers provided by operating system 105, such as a graphics library user-mode driver and / or a graphics library kernel-mode driver, generate an instruction stream and the required related rendering data that can be recognized by the rendering pipeline in GPU 102. System graphics libraries include, but are not limited to, system graphics libraries such as OpenGL ES (Open Graphics Library for Embedded Systems), the Kronos Platform Graphics Interface, or Vulkan (a cross-platform drawing application program interface). The instruction stream contains a series of instructions, which are generally calls to the system graphics library interface.

[0076] Optionally, the CPU 101 may include at least one of the following types of processors: an application processor, one or more microprocessors, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor.

[0077] The CPU 101 may also include some hardware accelerators, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or an integrated circuit for implementing logical operations. The CPU 101 may be coupled to one or more data buses for transmitting data and instructions between the various components of the electronic device 100.

[0078] GPU 102: is used to receive the graphics instruction stream sent by CPU 101, generate a rendering target through a rendering pipeline, and display the rendering target on the display device 103 through the layer synthesis display module of the operating system.

[0079] Alternatively, GPU 102 may include a general-purpose GPU that executes software, such as a GPU or other type of dedicated graphics processing unit.

[0080] Display device 103 : used to display various images generated by electronic device 100 , which may be a graphical user interface (GUI) of an operating system or image data (including still images and video data) processed by GPU 102 .

[0081] Optionally, the display device 103 may include any suitable type of display screen, such as a liquid crystal display (LCD), a plasma display, or an organic light-emitting diode (OLED) display.

[0082] In one embodiment, the display device 103 and the electronic device 100 may be different devices. For example, the display device 103 may be connected to the electronic device 100 wirelessly or by wire, so that the display device 103 can obtain and display the content to be displayed by the electronic device 100.

[0083] The rendering pipeline is a series of operations that the GPU 102 performs sequentially when rendering graphics or image frames. Typical operations include vertex processing, primitive processing, rasterization, fragment processing, etc.

[0084] The software system of the electronic device 100 shown in FIG1 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture.

[0085] The embodiment of the present application does not limit the software system. For example, the software system can be system, System or System, etc.

[0086] 2 , the software structure of the electronic device 100 is exemplarily described by taking a layered architecture system as an example.

[0087] FIG2 is a block diagram of the software structure of an electronic device 100 according to an embodiment of the present application. A layered architecture divides software into several layers, each with clear roles and division of labor. Layers communicate with each other through software interfaces. As shown in FIG2 , the software system may include a 2D graphical user interface (GUI) application layer, a system desktop layer, a GUI framework layer, an operating system (OS) window, a driver layer, and a kernel layer. The hardware layer may communicate with the kernel layer through an interface.

[0088] The GUI framework layer provides a drawing interface for user interface (UI) controls for use by the 2D GUI application layer. For example, the GUI framework layer may include a 2D rendering library, which is generally used in conjunction with the system graphics library of the driver layer. For example, the 2D rendering library may include Skia, Qt, or GTK.

[0089] The OS window layer includes multiple functional modules. As shown in Figure 2, the OS window layer includes a UI drawer, a window manager, and a compositor. The UI drawer draws UI controls. The window manager manages window programs, such as obtaining the display screen size. The compositor synthesizes UI controls for display.

[0090] The driver layer is an interface layer for 2D image rendering, and includes a system graphics library for generating rendering instructions. For example, the system graphics library can be OpenGL (Open Graphics Library) or Vulkan.

[0091] The kernel layer is a layer between hardware and software. The kernel layer can provide drivers for the electronic device 100 and enable communication between the system and various hardware components, such as display screens, cameras, memory, keyboards, wireless networks, audio equipment, power supplies, etc.

[0092] The hardware layer may include multiple processors, such as the CPU and GPU shown in Figure 2. The CPU may be used to generate rendering instructions and send the generated rendering instructions to the GPU, which is used to execute the rendering instructions.

[0093] It should be understood that the electronic device 100 to which the embodiments of the present application are applicable may not include some of the components shown in FIG. 1 and FIG. 2 , for example, may not include a GPU.

[0094] Figure 3 is a schematic flow chart of a method for drawing an interface. As shown in Figure 3, the process of drawing an interface includes steps executed by the UI thread and the rendering thread. The UI thread performs the following steps: event response, layout measurement, and drawing instruction recording. This allows the UI thread to obtain drawing information for changed nodes, which can be synchronized to the rendering thread. The rendering thread then performs steps: updating drawing information, replaying drawing instructions, and issuing drawing instructions. Finally, the rendering thread calls the GPU to perform rendering (drawing instruction execution).

[0095] Event response is the process of handling user input or other events and calculating animations and special effects. Layout measurement is the process of re-measuring the new page layout, such as measuring window size, position, and other properties. Drawing instruction recording is the process of generating drawing instructions corresponding to each node in the control tree.

[0096] Updating drawing information may include updating the drawing attribute information and corresponding drawing instructions of changed nodes in the render tree. Playing back drawing instructions may include creating a canvas and playing back drawing instructions on the canvas through a rendering thread. Sending drawing instructions may include converting drawing instructions into instructions that the GPU can understand and execute. GPU execution may include executing the drawing instructions to complete the rendering. After the rendering process is completed, the rendering process may enter the compositing and displaying thread to composite and display the rendering results.

[0097] Figure 4 is a schematic diagram of the operation sequence of each step in the execution process of the rendering thread. As shown in Figure 4A, at time t1, in response to the vertical synchronization (Vsync) signal, the rendering thread performs preprocessing, drawing instruction playback and drawing instruction issuance and execution on the image frame F1, wherein the preprocessing may include the process of the rendering thread updating the drawing information, and may also include steps such as dirty area calculation and determination, and the drawing instruction playback includes the process of the rendering thread replaying the drawing instruction on the canvas. After the rendering thread is processed, the synthesis and display thread synthesizes and displays the image frame F1. At time t2, in response to the Vsync signal, the rendering thread performs the corresponding steps on the image frame F2. Afterwards, the synthesis and display thread displays the image frame F2. At time t3, the rendering thread performs the corresponding steps on the image frame F3. At subsequent moments, the corresponding steps can be performed similarly to realize the rendering process of the electronic device display interface, which will not be repeated here.

[0098] As shown in Figure 4B, if the rendering thread of a certain image frame (for example, image frame F2) takes too long to execute rendering (for example, when the drawing instruction execution process is heavily loaded or when rendering a complex scene), so that the execution time of the rendering thread exceeds the time of one frame (for example, when the refresh rate is 60Hz, the length of one frame is 16.6ms), it will directly affect the processing time of the next frame (F3), resulting in frame loss or display freeze, affecting the user experience.

[0099] The embodiment of the present application provides a rendering method that can reduce the possibility of image display freezes and improve the user experience.

[0100] As shown in FIG5 , the method includes:

[0101] S510 , the rendering thread synchronizes drawing information of a first changed node. The first changed node is a node whose drawing information of the next frame of the application changes relative to the current frame of the application. The drawing information includes rendering attribute information and drawing instructions.

[0102] S520: The rendering thread replays the drawing instruction of the first target node on the first canvas. The first target node is determined according to the first change node, and the first target node corresponds to the area to be rendered in the next frame.

[0103] S530: The execution thread performs rendering on the first target node according to the drawing information of the first target node to obtain a next frame.

[0104] In an embodiment of the present application, the rendering thread and the execution thread can be different threads, so that the synchronization of the rendering attribute information of the first change node, the playback of the drawing instructions and the rendering execution process can be executed concurrently by two threads. The rendering thread does not need to wait for the end of the rendering execution process of the current frame and can update the drawing information of the first change node of the next frame and replay the drawing instructions, thereby reducing the rendering time. When the rendering load is large or a complex picture is rendered, it is not easy to cause events such as screen freeze or frame loss, thereby improving the user experience.

[0105] The rendering method provided in the embodiment of the present application can be applied to scenarios where an electronic device needs to refresh the interface displayed on the screen. For example, when a user inputs an interface displayed on the screen of the electronic device, for example, when a user clicks or slides a control on the interface, causing the interface to change, the interface may need to be refreshed. Alternatively, the change in the interface displayed on the screen of the electronic device may not be triggered by user input, for example, when the status bar time changes, the interface may need to be refreshed.

[0106] When an interface change event is detected, a drawing signal may be periodically generated. The drawing signal may be a vertical synchronization (Vsync) signal. The period of the drawing signal may be related to the screen refresh rate of the electronic device. For example, when the screen refresh rate is 60Hz, the period of the drawing signal may be 16.6ms.

[0107] In S510 , in one embodiment, the rendering thread may be triggered by a drawing signal to synchronize drawing information of the first changed node.

[0108] A first-changed node is one whose drawing information for the next frame of an application changes relative to the current frame. This drawing information can include rendering attribute information and drawing instructions. This rendering attribute information can include the transparency, color, size, background, and position of the control corresponding to the node. It can also include the text, images, and graphics displayed by the control. A drawing instruction for a node can include one or more drawing operations, such as drawing a line, a rectangle, or text.

[0109] That is, the first changed node is a node whose rendering attribute information and / or drawing instructions of the next frame of the application have changed relative to the current frame of the application. For example, the first changed node may include a node that is newly added, a node that is deleted, or a node whose rendering attribute information has changed relative to the current frame.

[0110] The relationship between different controls on the user interface can be represented by a control tree or a rendering tree. The control tree or the rendering tree includes multiple nodes, which have corresponding relationships with the controls. The first changing node is a node on the control tree or the rendering tree.

[0111] The rendering thread can maintain a render tree corresponding to the application. Each node in the render tree includes rendering attribute information and drawing instructions. The drawing instructions can be stored in a drawing instruction list, which can be stored in shared memory. The drawing instruction list records all drawing operations on the interface or their identifiers, such as addresses and sequence numbers. When the screen interface of the electronic device needs to be refreshed, the rendering thread can obtain the node with changed drawing information and update the drawing information of the first changed node in the render tree by traversing the render tree.

[0112] Each UI thread can correspond to an application, which can be called the associated application of the UI thread. Before the rendering thread synchronizes the drawing information of the first changed node, the UI thread can determine whether the next frame of the associated application's interface has changed relative to the current frame, obtain the drawing information of the first changed node of the associated application, and then the UI thread can pass the drawing information of the first changed node to the rendering thread.

[0113] The UI thread can be called the main thread of the application. The UI thread of each application can measure, layout and record instructions when the interface of the application changes, so as to obtain the attribute information of the changed nodes of the application and generate drawing instructions to facilitate drawing the next frame page of the current application. Specifically, the attribute information of each control can be obtained by measuring and laying out the controls of the next frame page. The size of the control can be obtained by measuring the control, for example, it can be measured by the measure function. The position of the control can be obtained by laying out the control, for example, it can be laid out by the layout function and the view function. In addition, it is possible to check whether the content of the control has changed, and set animations and special effects. For the next frame page to be drawn, one or more drawing operations (drawing instructions) required to draw the next frame can be determined by drawing recording. The UI thread can determine the node that has changed on the control tree of the display interface (the first changed node), so that the drawing information of the corresponding node can be passed to the rendering thread, for example, by sending a drawframe command to the rendering thread, and the drawframe command is stored in the task queue (taskqueue) of the rendering thread. Therefore, the rendering thread can traverse the rendering tree according to the first changed node drawing information, update the drawing information of the first changed node in the rendering tree, and determine the dirty area that needs to be rendered in the next frame.

[0114] The rendering method provided in the embodiments of the present application can be applied in different rendering architectures, and can be exemplarily applied in a unified rendering architecture or a separate rendering architecture.

[0115] In a unified rendering architecture, an application may include a UI thread, but the rendering thread and the execution thread may be threads of the operating system, rather than threads running in the process corresponding to the application. In this way, when the UI threads of different applications determine that the next frame of the interface of the associated application has changed relative to the current frame, they can obtain the drawing information of the first changed node of the associated application and pass it to the rendering thread. For example, the UI thread of the application can complete data interaction with the rendering thread through inter-process communication (IPC). The application program can implement IPC communication through methods such as Binder, AIDL, shared memory, Socket, etc., which are not limited here. Therefore, the rendering thread can update the drawing information by traversing the rendering tree nodes of the corresponding application according to the drawing information passed by the UI threads of different applications.

[0116] In a separate rendering architecture, an application can include a rendering thread, an execution thread, and a UI thread. Different applications independently perform rendering steps when the display interface changes. Thus, when an application's UI thread determines that the next frame of the application has changed relative to the current frame, it can obtain the drawing information of the first changed node of the application and pass it to the rendering thread. The rendering thread can then traverse the corresponding rendering tree nodes of the application and update the node's drawing information to facilitate subsequent rendering execution.

[0117] In addition to updating the drawing information for the first change node, the rendering thread can also determine a first target area based on the first change node. This target area, referred to as a dirty area, is the area to be rendered in the next frame. This dirty area can be a single area, such as a bounding rectangle encompassing all controls corresponding to the first change node. Alternatively, the dirty area can be multiple rectangular areas, each of which can be the bounding rectangle of one or more controls corresponding to the first change node. This allows subsequent execution threads to render only the dirty area during the rendering process, reducing rendering overhead.

[0118] Some or all nodes corresponding to the controls in the first target area may be referred to as first target nodes.

[0119] Exemplarily, under a unified rendering architecture, the rendering thread will determine the dirty area and the first target node based on the first change nodes of multiple applications. Since different applications have a stacking relationship, the first target node finally determined may include the first change nodes of different applications, and may also include nodes other than the first change node. Although some nodes may not change, these nodes are included in the dirty area when determining the bounding rectangle of all nodes. In some cases, the first change nodes of some applications may not be included in the first target node. For example, a change node of an application may be blocked by the windows of other applications, and the area corresponding to the node will not be displayed on the interface. Therefore, when playing back the drawing instruction, the playback of the change node can be skipped. In other words, some nodes included in the dirty area are the first target nodes, and these some nodes are the nodes located in the top-level window.

[0120] Alternatively, in a separate rendering architecture, the rendering thread of each application determines the dirty area and the first target node of each application based on the first change node of each application. In this case, the first target node may include the first change node of the application, or may include other nodes in addition to these change nodes.

[0121] At the same time, after the rendering thread obtains the drawing instruction of the first target node, it can create a first canvas bound to the next frame to be rendered and initialize the canvas resources, and replay the drawing instruction of the first target node on the canvas, thereby recording the drawing information of the first target node on the canvas context of the canvas.

[0122] After the drawing instructions are played back on the canvas, during the rendering execution process, the CPU and / or GPU can perform rendering according to the drawing information of the canvas context.

[0123] The execution thread is a different thread from the rendering thread. The execution thread in S530 performs rendering on the first target node according to the drawing information of the first target node, which can be understood as a process including the execution thread sending the drawing instructions to the GPU / CPU and calling the GPU / CPU to specifically perform the rendering. When the rendering thread finishes playing back the instructions on the canvas, the execution thread can send the drawing instructions of the first target node to the GPU / CPU, that is, converting the drawing instructions of the first target node into instructions that the GPU / CPU can understand and process, so that the execution thread can call the GPU / CPU to render the first target node according to the drawing information of the first target node, that is, to render the control corresponding to the first target node, so as to obtain the next frame.

[0124] In the unified rendering architecture, the rendering thread and the execution thread perform drawing information updates, drawing instruction playback, and specific rendering operations for the interface of the entire electronic device. After completion, the rendering results can be synthesized and displayed by the synthesis and display thread. The final display can be executed by the synthesis and display thread or by a separate display thread, which is not limited in this application.

[0125] In a separate rendering architecture, each application's rendering thread and execution thread only synchronize the drawing information and rendering of the nodes in the render tree of the application to which it belongs (associated influence). The rendering results of the execution thread can be passed to the synthesis and display thread (such as the surface compositor SurfaceFlinger). The synthesis and display thread can unify the rendering results of multiple application execution threads into the next frame of image and display it on the electronic device's display. Display can be performed by the surface compositor or by a separate thread, and this application does not limit this.

[0126] Figure 6 shows a timing diagram of the execution process of each thread when the rendering method of an embodiment of the present application is executed. The preprocessing process in Figure 6 may include the step of updating the drawing information of the changed node, and may also include the steps executed by the rendering thread such as the determination of the dirty area. After the preprocessing is completed, the rendering thread can create a canvas and replay the drawing instructions of the first target node on the canvas. After that, the execution thread can issue the drawing instructions and execute the specific rendering process (referred to as the drawing instruction issuance execution in Figure 6). Taking the execution thread calling the GPU to perform rendering as an example, the execution thread can first convert the drawing instructions into instructions executable by the GPU. These commands instruct the GPU how to draw text, shapes, images and other visual content on the screen. After that, the execution thread can call the GPU to execute the specific rendering process. The synthesis and display thread can synthesize the rendering results for display after the execution thread is completed. The synthesis can be a synthesis of software rendering results, a synthesis of hardware rendering results, or a synthesis of software rendering and hardware rendering results.

[0127] As shown in Figure 6, in response to the first vertical synchronization (Vsync1) signal, at time t1, the rendering thread can pre-process frame F1. After receiving the Vsync1 signal, the rendering thread can determine whether the UI thread has synchronized the corresponding data (drawing information of the changed nodes). Once the corresponding data is obtained, it can perform pre-processing and playback of drawing instructions based on this data, and synchronize the processing results to the execution thread. At time t4, the execution thread calls the CPU and / or GPU to perform rendering and drawing operations on frame F1 based on the data synchronized by the rendering thread. During the rendering process of frame F1, at time t2, in response to the Vsync1 signal, the rendering thread pre-processes and playbacks drawing instructions for frame F2 based on the data synchronized by the UI thread. At time t7, the synthesis and display thread, in response to the second vertical synchronization (Vsync2) signal, obtains the rendering results of the execution thread and synthesizes the rendering results for display. At time t5, the execution thread obtains the processing results of frame F2 by the rendering thread and performs rendering on frame F2. Subsequently, triggered by different signals, the rendering thread, execution thread, and synthesis and display thread sequentially process each frame, thus ensuring the continuous rendering process.

[0128] It should be understood that FIG6 shows two vertical synchronization signals, and the two vertical synchronization signals trigger the rendering thread and the synthesis display thread to perform their respective operations at different times. In fact, the number of vertical synchronization signals is not limited to two, and the timing of the two vertical synchronization signals can also overlap. For example, the vertical synchronization signal Vsync1 can be used to trigger the two threads to perform the corresponding steps. In this way, the synthesis display thread can send the display when it receives the first Vsync1 signal after the execution thread executes the drawing instruction. The total processing time of one frame will be extended to a certain extent compared with the embodiment shown in FIG6. For another example, the execution of the drawing instruction can also be triggered by the vertical synchronization signal. In this way, the total processing time of one frame will be further extended compared with the above two embodiments.

[0129] As can be seen from Figure 6, since the drawing instruction issuance and execution process is configured as an independent execution thread, even if the rendering overhead is large, the issuance and execution of the drawing instruction does not affect the rendering thread's execution of the next frame's preprocessing and drawing instruction playback. As a result, the entire rendering process can be pipelined and concurrent. As long as the total time of preprocessing and drawing instruction playback does not exceed one frame time, and the drawing instruction execution time does not exceed one frame time, there will be no problems such as screen freeze and frame loss, thereby improving the user experience.

[0130] The rendering architecture in Figure 6 is a unified rendering architecture, with the rendering and execution threads being independent of the application being rendered. The synthesis and display thread performs the synthesis and display of the rendering results. When using a separate rendering architecture, in addition to the application's rendering and execution threads, a synthesis and display thread and a display thread may also be included. The synthesis and display thread synthesizes and displays the rendering results of different applications, and the display thread displays the synthesized results on the electronic device's display.

[0131] In addition, the UI thread is not shown in Figure 6. When a separate rendering architecture is used, the execution of the rendering thread may not be directly triggered by the vertical synchronization signal, but the UI thread may be directly triggered by the vertical synchronization signal. When the UI thread is executed, the rendering thread can be notified through inter-process communication, so that the rendering thread continues to process (update the drawing information of the changed node and replay the drawing instructions). In this case, when the total processing time of the UI thread and the rendering thread, the processing time of the execution thread, the processing time of the synthesis and display thread, and the processing time of the display thread are all less than the time length of one frame, the screen displayed by the electronic device will not be stuck. Since the rendering execution process is executed by an independent thread, the possibility of screen freezes and frame drops can still be reduced, thereby improving the user experience.

[0132] In the embodiments of the present application, when the rendering thread plays back instructions on the canvas, it needs to create a canvas, and the canvas needs to be bound to a canvas context. The canvas context is associated with the canvas, and the canvas context allows graphics, images, text, etc. to be drawn on the canvas. When the getContext method is called, a context object of a specific type can be returned.

[0133] The previous frame's drawing instructions are issued and executed, and the next frame's drawing instructions are played back. Both require access to the canvas context. Since this embodiment uses two threads for drawing instruction playback and issuing and executing them, respectively, it's possible that the issuance and execution of drawing instructions for two different frames and the playback of drawing instructions may overlap. If the canvas contexts of the two frames are the same, an access conflict will occur. In this case, the canvases of the two frames need to be bound to different canvas contexts.

[0134] Referring to Figure 7A, two adjacent frames use a single canvas context. During the playback of frame F1's drawing instructions, the canvas context records frame F1's drawing instructions (ops), which are managed by different opchains (opchain1, opchain2, opchain3, etc.). Different opchains can correspond to different memory locations within the canvas context. If the execution of frame F1's drawing instructions takes a long time to execute, and they overlap with the playback of frame F2's drawing instructions, and the canvases of both frames are bound to the same canvas context, the execution thread will need to access the canvas context when executing the drawing instructions for frame F1. Simultaneously, the rendering thread will also need the canvas context when replaying the drawing instructions for frame F2, resulting in conflicting access to the same canvas context between the rendering thread and the execution thread.

[0135] To solve the above problem, as shown in FIG7B , in one embodiment, two adjacent frames can be bound to different canvas contexts, for example, odd-numbered frames are bound to canvas context 1, and even-numbered frames are bound to canvas context 2. Assuming that F1 and F2 are two adjacent frames, frame F1 is bound to canvas context 1, and frame F2 is bound to canvas context 2. Thus, when the execution thread issues a drawing instruction for frame F1, it accesses canvas context 1, and when the rendering thread replays the drawing instruction for frame F2, it accesses canvas context 2. Moreover, when the execution thread completes rendering and drawing frame F1, it can unbind the canvas of frame F1 from canvas context 1, and bind canvas context 1 before replaying the drawing instruction for the next frame after frame F2. In this way, there will be no access conflict between the two adjacent frames.

[0136] When the rendering or execution thread processing is further extended, more canvas contexts can be used for rotation. This means that the canvases corresponding to two or more adjacent frames are created based on different canvas contexts. For example, three or more canvas contexts can be alternately bound to canvases from different frames to avoid access conflicts or read and write conflicts between different threads.

[0137] Optionally, even if the canvases of different frames are created based on the same canvas context, the target drawing instructions of at least two consecutive adjacent frames can also be managed independently, that is, the drawing instructions of at least two consecutive adjacent frames can be stored in different memories corresponding to the same canvas context (this memory can be called a rotating memory). Taking the number of rotating memories as 3 as an example, for example, the drawing instructions of the mth frame are stored in memory 1, the drawing instructions of the m+1th frame are stored in memory 2, the drawing instructions of the m+2th frame are stored in memory 3, the drawing instructions of the m+4th frame are stored in memory 1... The drawing instructions of subsequent frames can be stored alternately in memory 1, memory 2 and memory 3, which will not be repeated here.

[0138] Before the rendering thread creates a canvas, it needs to request a buffer from the buffer queue so that the rendering results of the execution thread can be drawn into the buffer. In the embodiments of the present application, the buffer working mechanism can be a rotating buffer mechanism, such as a double-buffer rotating mechanism, a triple-buffer rotating mechanism, a quad-buffer rotating mechanism, etc. The embodiments of the present application do not limit the buffer rotating mechanism.

[0139] In one embodiment, the canvas context may use a carousel canvas context mechanism. The number of carousel canvas contexts may be equal to the number of carousel buffers, and the carousel canvas contexts and the carousel buffers may have a one-to-one correspondence. Taking the example that the number of rotating canvas contexts and rotating buffers are both 3, when the rendering thread processes the first frame and obtains rotating buffer 1, it can determine the canvas context 1 corresponding to rotating buffer 1 based on rotating buffer 1, and create canvas 1 based on the canvas context. When the rendering thread processes the second frame and obtains rotating buffer 2, it can determine the corresponding canvas context 2 based on rotating buffer 2, and create canvas 2 based on the canvas context. When the rendering thread processes the third frame and obtains rotating buffer 3, it can determine the corresponding canvas context 3 based on rotating buffer 3, and create canvas 3 based on canvas context 3. When the rendering thread processes the fourth frame and obtains rotating buffer 1, canvas 1 has been created when processing the first frame, and the canvas is bound to canvas context 1. Canvas context 1 and canvas 1 can be used directly...Subsequent frames can determine the corresponding canvas context based on the obtained rotating buffer, so that each frame can be bound to each canvas context in turn, and the canvas does not need to be created repeatedly, saving processing overhead.

[0140] When rendering different frames, the previous and subsequent frames may use the same shared resources, such as texture resources, text (TXT) resources, and path resources. Texture resources are image resources that can be any type of image file (JPEG, PNG, BMP, etc.), or image data obtained from video frames, canvas content, or other generated data sources. Path resources can be a collection of drawing instructions, such as a collection of drawing instructions for basic primitives, such as points, lines, and surfaces.

[0141] Taking the shared resource as a texture resource as an example, the texture resource can be uploaded to the GPU before use, so that the GPU can call the texture resource. In an embodiment of the present application, when creating a canvas, it involves the creation and uploading of texture resources (uploaded to the GPU), and the texture resources are bound to the canvas context. However, in an embodiment of the present application, the canvas contexts of two adjacent frames are different. As shown in Figure 8, when the texture resource is bound to the canvas context of the previous frame (F1), the next frame (F2) cannot use the same texture resource, or the same texture resource needs to be uploaded repeatedly, resulting in resource waste or access errors.

[0142] In order to solve this problem, in an embodiment of the present application, the time when the texture resource is bound to the canvas context can be changed. Specifically, the rendering thread can upload the texture resources required for the current frame to the GPU when the drawing instructions of the current frame are played back, and bind the texture resources to the canvas context of the current frame, and unbind the texture resources from the canvas context of the current frame before the next frame instruction is played back (for example, during the preprocessing process). If the same texture resources need to be used during the execution of the next frame drawing instruction, the texture resources can be bound to the canvas context of the next frame before the drawing instruction playback is executed. If the next frame does not need to use the texture resources of the current frame, the rendering thread can determine that the texture resources of the current frame need to be released when the drawing instructions of the next frame are played back, and the rendering thread can submit the texture resource release task to the execution thread. The execution thread can release the texture resources after completing the rendering execution process of the current frame, so as not to affect the rendering execution process of the current frame.

[0143] It should be understood that the release of texture resources is not limited to being triggered by the next frame not using the texture resources used by the current frame. It can also be triggered after an interval of one frame or other frames not using the texture resources of the current frame, or when the storage space occupied by the texture resources is greater than a preset threshold, etc. This application does not limit this.

[0144] In the above embodiment, for any frame, the texture resources and canvas context of the previous frame will be unbound before the drawing instructions of the frame are played back, so that when the frame needs to use the texture resources of the previous frame, the texture resources can be directly bound to reduce the waiting time overhead.

[0145] Alternatively, unbinding texture resources can be deferred until the drawing instructions for the current frame are played back. In this case, the rendering thread can first determine whether the current frame needs to use the texture resources of the previous frame. If so, the texture resources can be unbound. If not, the texture resources do not need to be unbound, thus reducing the overhead caused by the unbinding operation.

[0146] In this way, texture resources can be easily shared without having to upload the same texture resources repeatedly, and the release process of texture resources will not affect the rendering process.

[0147] In one embodiment, the texture resource may not be bound to the canvas context of any frame, but may be managed by a global context (globalContext). The global context may be a context that is accessible to contexts of all frames.

[0148] For example, the texture resource used for rendering the current frame can be handed over to the global context for management. If the rendering thread determines that the next frame still needs to use the texture resource when executing the drawing instruction playback for the next frame, the storage location of the texture resource can be obtained through the global context. Thus, through the global context, the rendering thread can call the GPU to access the shared resource through the corresponding interface to query whether the corresponding texture resource exists and can use the texture resource. If the texture resource exists, the texture resource does not need to be uploaded, and the GPU directly uses the texture resource through the interface. When the texture resource does not exist, the texture resource can be uploaded so that the GPU can use the texture resource when rendering.

[0149] When using the global context mechanism to manage shared resources, the task of releasing shared resources can also be submitted by the rendering thread to the execution thread through the global context, so that the execution thread can release the texture resource when it determines that the rendering execution process of the current frame no longer needs the texture resource.

[0150] In the embodiment of the present application, in addition to shared resources such as texture resources, resources such as surface resources, image resources, and drawing instruction resources may also need to be released.

[0151] Surface resources are used to create surfaces. They typically correspond to a two-dimensional image storage area containing pixel information. They can be part of the screen display or a frame buffer, which corresponds to a portion of the first target node. The GPU performs rendering operations on the surface, drawing images or 3D scenes. When the rendering thread processes a frame, it obtains the rendering environment context, creates a surface based on the context, and then creates a canvas based on the surface (the canvas context corresponds to the rendering environment context). When the playback of the frame is completed, the rendering thread can trigger the release of the surface resources. Since the rendering execution process of the frame has a certain delay compared to the playback phase, in order to prevent the rendering thread from directly releasing the surface resources and affecting the execution thread's rendering execution process of the frame, the rendering thread can submit a release command to the execution thread when it determines that the resource needs to be released. This allows the execution thread to release the surface resources after the rendering is completed, for example, by calling the release() method to close and release the surface.

[0152] Image resources can also include bitmap resources, pixelmap resources, rawdata resources, etc. Image resources can also correspond to some nodes in the first target node of the next frame. Usually, the corresponding resources are obtained and generated during the instruction playback process, or the resources can be synchronized from the UI thread to the rendering thread. After the instruction playback is completed, the rendering thread can trigger the release of the corresponding resources. The release task can be submitted to the execution thread, so that the execution thread releases the resources after completing the rendering task of the frame, avoiding affecting the rendering execution process of the frame.

[0153] Optionally, the image resource may also correspond to all nodes in the first target node of the next frame. For example, the image resource used in a screen recording scene or an animation special effects scene may correspond to all nodes in the first target node.

[0154] Drawing instruction resources are resources that store drawing instructions for each node in the render tree or control tree. These resources are typically stored in shared memory, making it easier for rendering and execution threads to access the corresponding resources. Each application has a corresponding control tree and render tree. When the application ends, these resources can be released. The rendering thread can have a mapping relationship between each application and the corresponding node. When the application ends, it can trigger the release of the drawing instruction resources for the corresponding node. This release task can also be submitted to the execution thread, so that the execution thread can release the corresponding resources after the rendering execution ends.

[0155] Figure 9 shows a schematic block diagram of an apparatus 900 for rendering frames based on a rendering service, provided in an embodiment of the present application. As shown in Figure 9, the apparatus for rendering frames based on a rendering service includes a preprocessing module 910, a playback module 920, and an execution module 930. The preprocessing module 910 and the playback module 920 are modules in the rendering thread, and the execution module 930 is a module in the execution thread. The rendering thread and the execution thread are different threads.

[0156] Specifically, the preprocessing module 910 is used to synchronize the drawing information of the first change node, which is the node whose drawing information of the next frame of the application changes relative to the current frame of the application, and the drawing information includes rendering attribute information and drawing instructions; the playback module 920 is used to replay the drawing instructions of the first target node on the first canvas, and the target node is determined based on the first change node, and the first target node corresponds to the area to be rendered in the next frame; the execution module 930 is used to perform rendering on the first target node according to the drawing information of the first target node to obtain the next frame.

[0157] In one embodiment, the playback module 920 is further used to: play back the drawing instructions of the second target node on the second canvas, where the second target node is determined based on the node whose drawing information of the current frame of the application has changed relative to the previous frame of the application, wherein the first canvas and the second canvas are created based on different canvas contexts.

[0158] In one embodiment, the playback module 920 is further used to: replay the drawing instructions of the second target node on the second canvas, where the second target node is determined based on the node whose drawing information of the current frame of the application has changed relative to the previous frame of the application, wherein the first canvas and the second canvas are created based on the same canvas context, and the drawing instructions of the first target node and the drawing instructions of the second target node are stored in different memories corresponding to the canvas context.

[0159] In one embodiment, the device for rendering frames based on a rendering service further includes an acquisition module for acquiring a first cache, where the first cache is used to store the rendering result of the next frame; the playback module 920 is further used to determine, based on the first cache, the steps of replaying the drawing instructions of the first target node on the first canvas created based on the first canvas context.

[0160] In one embodiment, the preprocessing module 910 is further used to unbind the shared resources bound to the canvas context of the current frame, where the shared resources include at least one of the following: texture resources, text resources, and path resources; when the shared resources are needed to render the next frame, the playback module 920 is further used to bind the shared resources to the canvas context of the next frame.

[0161] In one embodiment, the playback module 920 is also used to submit a drawing instruction resource release task of the node of the first application to the execution thread in response to the termination of the first application. The drawing instruction resource of the node of the first application is a resource that stores the drawing instructions of the node of the first application; the execution module 930 is also used to release the drawing instruction resource of the node of the first application after rendering the first target node according to the drawing information of the first target node.

[0162] In one embodiment, the playback module 920 is also used to submit a task of releasing the first surface resource and / or the first image resource to the execution thread, where the first surface resource corresponds to some nodes in the first target node, and the first image resource corresponds to some nodes in the first target node; the execution module 930 is also used to release the first surface resource and / or the first image resource after rendering the first target node according to the drawing information of the first target node according to the task of releasing the first surface resource and / or the first image resource.

[0163] In one embodiment, the playback module 920 is further used to: hand over the shared resources of the current frame to the global context for management, and the shared resources include at least one of the following: texture resources, text resources and path resources. The global context is a context that can be accessed by the canvas contexts of all frames.

[0164] In one embodiment, the playback module 920 is further used to: submit the release task of the shared resource to the execution thread or submit it to the execution thread through the global context when it is determined that the shared resource needs to be released; the execution module is also used to release the shared resource after the rendering of the current frame is completed.

[0165] In one embodiment, the device also includes an interface management module for at least one application, wherein each interface management module is used to: determine whether the next frame of the interface of the associated application has changed relative to the current frame, obtain the drawing information of the first change node of the associated application; and synchronize the drawing information of the first change node to the rendering thread.

[0166] In one embodiment, the preprocessing module 910 is also used to: in response to the termination of the first application, submit a drawing instruction resource release task of the node of the first application to the execution module 930, where the drawing instruction resource of the node of the first application is a resource that stores the drawing instructions of the node of the first application; the execution module 930 is also used to release the drawing instruction resource of the node of the first application after rendering the first target node according to the drawing information of the first target node.

[0167] In one embodiment, the preprocessing module 910 is also used to submit a task of releasing the first surface resource and / or the first image resource to the execution module 930, where the first surface resource corresponds to some nodes in the first target node, and the first image resource corresponds to some nodes in the first target node; the execution module 930 is also used to release the first surface resource and / or the first image resource after rendering the first target node according to the drawing information of the first target node according to the task of releasing the first surface resource and / or the first image resource.

[0168] In one embodiment, the device further includes a synthesis and display module configured to synthesize and display the rendering result of the next frame.

[0169] In one embodiment, the device further includes: a synthesis and display module for synthesizing the next frame rendering results belonging to different applications to obtain the next frame; and a display module for displaying the next frame.

[0170] FIG10 shows a schematic structural diagram of another apparatus 1000 for rendering frames based on a rendering service provided in an embodiment of the present application.

[0171] As shown in FIG10 , the apparatus 1000 for rendering frames based on a rendering service includes one or more processors 1010 and one or more memories 1020. The one or more memories 1020 store one or more computer programs, each of which includes instructions. When the one or more processors 1010 execute the instructions, the apparatus 1000 performs the method described in the embodiment of FIG5 .

[0172] The present application provides a computer program product that, when executed on an electronic device, causes the electronic device to execute the method described in the embodiment of FIG5 . The implementation principle and technical effects are similar to those of the embodiment described in FIG5 , and are not further described here.

[0173] The present embodiment provides a computer-readable storage medium containing instructions that, when executed on an electronic device, cause the electronic device to execute the method described in the embodiment of FIG5 . The implementation principles and technical effects are similar and will not be further described here.

[0174] The present embodiment provides a chip for executing instructions. When the chip is running, it executes the method described in the embodiment of Figure 5. The implementation principles and technical effects are similar and will not be described in detail here.

[0175] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0176] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0177] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0178] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0179] In addition, the functional modules in the various embodiments of the present application may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0180] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0181] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A rendering method, applied to an electronic device, characterized in that: The method comprises: The rendering thread synchronizes drawing information of a first changed node, where the first changed node is a node whose drawing information of a next frame of the application changes relative to the drawing information of a current frame of the application, and the drawing information includes rendering attribute information and drawing instructions; The rendering thread replays a drawing instruction of a first target node on a first canvas, where the first target node is determined according to the first change node, and the first target node corresponds to an area to be rendered in a next frame; The execution thread performs rendering on the first target node according to the drawing information of the first target node to obtain the next frame.

2. The method according to claim 1, characterized in that Before the rendering thread synchronizes the drawing information of the first changed node, the method further includes: The rendering thread plays back drawing instructions of a second target node on a second canvas, where the second target node is determined based on a node whose drawing information of a current frame of the application has changed relative to that of a previous frame of the application, wherein the first canvas and the second canvas are created based on different canvas contexts.

3. The method according to claim 1, characterized in that Before the rendering thread synchronizes the drawing information of the first changed node, the method further includes: The rendering thread plays back the drawing instructions of the second target node on the second canvas, where the second target node is determined based on a node whose drawing information of the current frame of the application has changed relative to the previous frame of the application, wherein the first canvas and the second canvas are created based on the same canvas context, and the drawing instructions of the first target node and the drawing instructions of the second target node are stored in different memories corresponding to the canvas context.

4. The method according to any one of claims 1 to 3, characterized in that Before the rendering thread plays back the drawing instruction of the first target node on the first canvas, the method further includes: The rendering thread obtains a first cache, where the first cache is used to store a rendering result of the next frame; The rendering thread determines, based on the first cache, to execute a step of replaying the drawing instruction of the first target node on a first canvas created based on a first canvas context.

5. The method according to any one of claims 1 to 4, characterized in that Before the rendering thread plays back the drawing instruction of the first target node on the first canvas, the method further includes: the rendering thread unbinding the shared resource bound to the canvas context of the current frame, the shared resource including at least one of the following: a texture resource, a text resource, and a path resource; In a case where the shared resource is required to be used for rendering the next frame, the method further includes: the rendering thread binding the shared resource to the canvas context of the next frame.

6. The method according to any one of claims 1 to 4, characterized in that Before the rendering thread plays back the drawing instruction of the first target node on the first canvas, the method further includes: The rendering thread hands over the shared resources of the current frame to the global context for management. The shared resources include at least one of the following: texture resources, text resources, and path resources. The global context is a context that can be accessed by canvas contexts of all frames.

7. The method according to claim 5 or 6, characterized in that The method further comprises: When the rendering thread determines that the shared resource needs to be released, the rendering thread submits the task of releasing the shared resource to the execution thread or submits it to the execution thread through the global context; The execution thread releases the shared resource when the shared resource is not in use.

8. The method according to any one of claims 1 to 7, characterized in that The application is an associated application of a user interface UI thread, and the method further includes: The UI thread determines that the interface of the associated application changes in the next frame relative to the current frame, and obtains drawing information of a first changed node of the associated application; The UI thread transmits the drawing information of the first changed node to the rendering thread.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: In response to termination of the first application, the rendering thread submits a drawing instruction resource release task of the node of the first application to the execution thread, where the drawing instruction resource of the node of the first application is a resource storing the drawing instruction of the node of the first application; After rendering the first target node according to the drawing information of the first target node, the execution thread releases the drawing instruction resource of the node of the first application.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: The rendering thread submits a task of releasing a first surface resource and / or a first image resource to the execution thread, where the first surface resource corresponds to some nodes in the first target node, and the first image resource corresponds to some nodes in the first target node; The execution thread releases the first surface resource and / or the first image resource after rendering the first target node according to the drawing information of the first target node according to the task of releasing the first surface resource and / or the first image resource.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: The synthesis and display thread synthesizes and displays the rendering result of the next frame, and the synthesis and display thread and the rendering thread belong to different threads of the same process.

12. The method according to any one of claims 1 to 10, characterized in that The method further comprises: The synthesis display thread synthesizes the next frame rendering results of the rendering threads and the execution thread belonging to different applications to obtain the next frame; The display thread displays the next frame.

13. A device for rendering frames based on a rendering service, characterized in that: include: An application and a rendering service communicating via inter-process communication, wherein the rendering service includes a rendering thread and an execution thread. The rendering thread is used to: Synchronizing drawing information of a first changed node of the application, where the first changed node is a node where drawing information of a next frame of the application changes relative to the current frame of the application, the drawing information including rendering attribute information and drawing instructions; replaying a drawing instruction of a first target node on a first canvas, where the first target node is determined according to the first change node, and the first target node corresponds to an area to be rendered in a next frame; The execution thread performs rendering on the first target node according to the drawing information of the first target node to obtain the next frame.

14. The device according to claim 13, characterized in that The rendering thread is further used to play back drawing instructions of a second target node on a second canvas, where the second target node is determined based on a node whose drawing information of a current frame of the application has changed relative to a previous frame of the application, wherein the first canvas and the second canvas are created based on different canvas contexts.

15. The device according to claim 13, characterized in that The rendering thread is further used to play back the drawing instructions of a second target node on a second canvas, where the second target node is determined based on a node whose drawing information of a current frame of the application has changed relative to a previous frame of the application, wherein the first canvas and the second canvas are created based on the same canvas context, and the drawing instructions of the first target node and the drawing instructions of the second target node are stored in different memories corresponding to the canvas context.

16. The device according to any one of claims 13 to 15, characterized in that The rendering thread is also used to: Obtaining a first cache, where the first cache is used to store the rendering result of the next frame; According to the first cache, it is determined to execute a step of replaying the drawing instruction of the first target node on a first canvas created based on a first canvas context.

17. The device according to any one of claims 13 to 16, characterized in that The rendering thread is also used to: Unbinding a shared resource bound to the canvas context of the current frame, wherein the shared resource includes at least one of the following: a texture resource, a text resource, and a path resource; In a case where the shared resource is required to be used for rendering the next frame, the shared resource is bound to the canvas context of the next frame.

18. The device according to any one of claims 13 to 16, characterized in that The rendering thread is further configured to hand over the shared resources of the current frame to a global context for management. The shared resources include at least one of the following: texture resources, text resources, and path resources. The global context is a context accessible to canvas contexts of all frames.

19. The device according to claim 17 or 18, characterized in that In a case where the rendering thread determines that the shared resource needs to be released, the rendering thread is further configured to submit a task of releasing the shared resource to the execution thread or submit it to the execution thread through the global context; The execution thread is further configured to release the shared resource when the shared resource is not in use.

20. The device according to any one of claims 13 to 19, characterized in that In response to the application being terminated, the rendering thread is further configured to submit a drawing instruction resource release task of the application node to the execution thread, where the drawing instruction resource of the application node is a resource storing the drawing instruction of the application node; The execution thread is further configured to release the drawing instruction resources of the node of the application after rendering the first target node according to the drawing information of the first target node.

21. The device according to any one of claims 13 to 20, characterized in that The rendering thread is further configured to submit a task of releasing a first surface resource and / or a first image resource to the execution thread, where the first surface resource corresponds to some nodes in the first target node, and the first image resource corresponds to some nodes in the first target node; The execution thread is further configured to release the first surface resource and / or the first image resource after rendering the first target node according to the drawing information of the first target node according to the task of releasing the first surface resource and / or the first image resource.

22. An electronic device, characterized in that: include: one or more processors; one or more memories; and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 12.

23. A computer-readable storage medium, characterized in that The storage medium stores a program or instruction, and when the program or instruction is executed, the method according to any one of claims 1 to 12 is implemented.

24. A chip, characterized in that: include: one or more processors; One or more memories, the one or more processors being coupled to the one or more memories and configured to read and execute instructions in the one or more memories to perform the method according to any one of claims 1 to 12.

25. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is executed, the method according to any one of claims 1 to 12 is implemented.

Images