Interface generation method and electronic equipment

Abstract

The invention discloses an interface generation method and electronic equipment, and relates to the technical field of electronics. The interface generation method provided by the invention comprises the steps of splitting a rendering tree into a plurality of sub-rendering trees, converting the plurality of sub-rendering trees into a rendering instruction in parallel through a plurality of threads, and further generating an interface based on the rendering instruction, so that the time delay of converting the rendering tree into the rendering instruction can be reduced, and frame dropping and jamming of the interface are avoided.

Description

Interface generation method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on October 19, 2022, with application number 202211281435.9, and priority to the Chinese patent application entitled “Interface Generation Method and Electronic Device”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of electronic technology, and in particular to an interface generation method and an electronic device. Background Art

[0003] With the development of technology, the resolution and refresh rate of the screens of electronic devices are getting higher and higher. Among them, the resolution of the screen affects the pixels contained in a frame of interface, and the refresh rate affects the time to generate a frame of interface.

[0004] Before the electronic device displays the first frame interface, the electronic device needs to expend computing resources to generate the first frame interface; before the electronic device displays the second frame interface, the electronic device needs to expend computing resources again to generate the second frame interface.

[0005] If the electronic device fails to generate the second frame interface in time, the content displayed on the electronic device's screen will freeze. To ensure that the second frame interface can be generated in time, the electronic device often increases the CPU's operating frequency to improve the electronic device's computing power, which in turn leads to higher energy consumption for the electronic device to generate a frame interface, reducing the energy efficiency of the interface generation.

[0006] Summary of the Invention

[0007] The present invention provides an interface generation method and electronic device. The interface generation method provided by the present invention splits a rendering tree into multiple sub-rendering trees, and then converts the multiple sub-rendering trees into rendering instructions in parallel, thereby reducing the time it takes to convert the sub-rendering trees into rendering instructions, thereby reducing the time it takes to generate the interface and avoiding interface freezes, frame drops, etc.

[0008] In a first aspect, an embodiment of the present application provides an interface generation method, which is applied to an electronic device running a first application, the method comprising: the electronic device generating a first rendering tree, the first rendering tree including drawing operations for generating a frame interface of the first application; the electronic device splitting the first rendering tree to obtain N sub-rendering trees, where N is greater than 1; the electronic device converting the N sub-rendering trees into first rendering instructions in parallel, the rendering instructions being instructions in a rendering engine, an image processing library, or a GPU driver; and the electronic device generating a frame interface of the first application based on the first rendering instructions.

[0009] In the above embodiment, the rendering tree is split into multiple sub-rendering trees, and then the multiple sub-rendering trees are converted into rendering instructions in parallel, which reduces the time it takes for the sub-rendering trees to be converted into rendering instructions, thereby reducing the time it takes to generate the interface and avoiding interface freezes, frame drops, etc.

[0010] In combination with some embodiments of the first aspect, in some embodiments, the electronic device splits the first rendering tree to obtain N sub-rendering trees, specifically including: the electronic device determines a first task amount, where the first task amount is the task amount of the first rendering tree, and the first task amount is used to indicate the time consumption or computational amount of converting the first rendering tree into the first rendering instruction; in response to the electronic device determining that the first task amount is greater than a first threshold, the electronic device splits the first rendering tree to obtain the N sub-rendering trees.

[0011] In the above embodiment, considering that the time required to convert a rendering tree into rendering instructions is longer when the amount of tasks of the rendering tree is greater than a threshold, the rendering tree is split into multiple sub-rendering trees in this case to reduce the time required. When the amount of tasks of the rendering tree is less than the threshold, the rendering tree may not be split.

[0012] In combination with some embodiments of the first aspect, in some embodiments, after the electronic device determines the first task amount, the method also includes: the electronic device determines M based on the first task amount and the first threshold, the M is less than or equal to the N, and the M is an integer greater than or equal to the ratio of the first task amount to the first threshold; the electronic device determines the integer greater than or equal to the M as the N.

[0013] In the above embodiment, splitting a rendering tree may first determine the minimum number of sub-rendering trees that need to be split, and then determine the method of splitting the rendering tree. The number of sub-rendering trees to be split may be determined by a threshold and the amount of work for the rendering tree.

[0014] In combination with some embodiments of the first aspect, in some embodiments, the electronic device determines the first task amount, specifically including: the electronic device determines the first task amount by determining the task amount of drawing operations in the first rendering tree.

[0015] In the above embodiment, the first task amount can be determined in various ways.

[0016] In combination with some embodiments of the first aspect, in some embodiments, the N sub-rendering trees include a second rendering tree and a third rendering tree, a difference between the second task volume and the third task volume is less than a difference threshold, the second task volume is the task volume of the second rendering tree, and the second task volume is used to measure the time or computational amount of converting the second rendering tree into rendering instructions, and the third task volume is the task volume of the third rendering tree, and the third task volume is used to measure the time or computational amount of converting the third rendering tree into rendering instructions.

[0017] In the above embodiment, after determining the minimum number of sub-rendering trees that need to be split and splitting the corresponding number of sub-rendering trees, the sub-rendering trees can be further split so that the task volume of each sub-rendering tree is less than a threshold.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, splitting, by the electronic device, the first rendering tree to obtain N sub-rendering trees specifically includes: determining, by the electronic device, that a root rendering node of the first rendering tree has N child nodes, where the child nodes are rendering nodes directly connected to the rendering node; and splitting, by the electronic device, the first rendering tree into the N sub-rendering trees.

[0019] In the above embodiment, the rendering tree may be split according to its data structure to obtain multiple rendering subtrees.

[0020] In combination with some embodiments of the first aspect, in some embodiments, the electronic device splits the first rendering tree to obtain N sub-rendering trees, specifically including: the electronic device divides the interface of the first application into N areas; the electronic device splits the first rendering tree based on the N areas to obtain N sub-rendering trees, and the N sub-rendering trees correspond one-to-one to the N areas.

[0021] In the above embodiment, the interface may be divided into N areas first, and then the rendering tree may be split according to the areas to obtain N sub-rendering trees.

[0022] In combination with some embodiments of the first aspect, in some embodiments, the electronic device splits the first rendering tree to obtain N sub-rendering trees, specifically including: the electronic device determines a first task amount, where the first task amount is the task amount of the first rendering tree, the first task amount is used to measure the time or computational amount of converting the first rendering tree into rendering instructions, and the first task amount is greater than a first threshold; the electronic device determines that a root rendering node of the first rendering tree has K child nodes, where K is less than N; the electronic device splits the first rendering tree into K sub-rendering trees; after the electronic device determines that the task amount of a fourth rendering tree is greater than the first threshold, the electronic device splits the fourth rendering tree to obtain N-K+1 rendering sub-trees, where the K sub-rendering trees include the fourth rendering tree, and the task amounts of the N rendering sub-trees are all less than the first threshold.

[0023] In the above embodiment, the rendering tree and the rendering subtrees can be split so that the task volume of each rendering subtree is less than the threshold, thereby ensuring that the time for converting the rendering subtree into rendering instructions will not timeout, thereby ensuring timely generation of the interface.

[0024] In combination with some embodiments of the first aspect, in some embodiments, the electronic device converts the N sub-rendering trees into a first rendering instruction in parallel, specifically including: the electronic device fills the instructions converted from the N sub-rendering trees into N buffers through N threads respectively; the electronic device submits the instructions of the N buffers to the first buffer, and the instructions in the first buffer are the first rendering instructions.

[0025] In the above embodiment, after converting multiple rendering subtrees into multiple rendering instructions, the rendering instructions need to be merged into a buffer, and then the buffer is submitted to the GPU to drive the GPU to generate an interface.

[0026] In a second aspect, an embodiment of the present application provides an interface generation method, which is applied to an electronic device running a first application, the method comprising: the electronic device generates a first rendering tree through the first process, the first rendering tree including drawing operations for generating a one-frame interface of the first process; the electronic device splits the first rendering tree into a second rendering tree and a third rendering tree, the second rendering tree including some drawing operations in the first rendering tree, the third rendering tree including some drawing operations in the first rendering tree, and the second rendering tree and the third rendering tree are different; the electronic device converts the second rendering tree into a first rendering instruction through a first thread, the first rendering instruction is stored in a first buffer, and the rendering instruction is an instruction in a rendering engine, an image processing library, or a GPU driver; the electronic device converts the third rendering tree into a second rendering instruction through a second thread, the second instruction is stored in a second buffer; the electronic device generates a one-frame interface of the first process based on the first rendering instruction and the second rendering instruction.

[0027] In the above embodiment, the electronic device can split the rendering tree generated by the process into multiple rendering trees, and generate multiple sets of rendering instructions through different threads, and finally generate a frame of interface based on the multiple sets of rendering instructions. Since the threads are parallel, the time required to convert the rendering tree into rendering instructions can be reduced.

[0028] In combination with some embodiments of the second aspect, in some embodiments, the electronic device determines a first task amount, where the first task amount is the task amount of the first rendering tree, and the first task amount is used to measure the time or computational amount of converting the first rendering tree into rendering instructions, and the first task amount is greater than a first threshold.

[0029] In the above embodiment, considering that the time required to convert a rendering tree into rendering instructions is longer when the amount of tasks of the rendering tree is greater than a threshold, the rendering tree is split into multiple sub-rendering trees in this case to reduce the time required. When the amount of tasks of the rendering tree is less than the threshold, the rendering tree may not be split.

[0030] In combination with some embodiments of the second aspect, in some embodiments, the electronic device generates a frame interface of the first process based on the first instruction and the second instruction, specifically including: the first rendering instruction is located in the first buffer held by the first thread, the second rendering instruction is located in the second buffer held by the second thread, and the electronic device submits the instructions in the first buffer and the rendering instructions in the second buffer to the third buffer; the electronic device generates a frame interface of the first process based on the third buffer.

[0031] In the above embodiment, the electronic device may submit rendering instructions in multiple buffers to one buffer, thereby driving the GPU to generate an interface.

[0032] In combination with some embodiments of the second aspect, in some embodiments, the third buffer is the second buffer, or the third buffer is the first buffer.

[0033] In a third aspect, an embodiment of the present application provides an interface generation method, which is applied to an electronic device running a first application, the method comprising: the electronic device generates a first rendering tree, the first rendering tree including drawing operations for generating a frame interface of the first application; the electronic device determines a first task amount, the first task amount being the task amount of the first rendering tree, the first task amount being used to measure the time or computational amount of converting the first rendering tree into rendering instructions, the first task amount being greater than a first threshold, the rendering instructions being instructions in a rendering engine, an image processing library, or a GPU driver; if the first task amount is greater than the first threshold, the electronic device configures the CPU operating frequency to change from a first frequency to a second frequency, the second frequency being higher than the first frequency; the electronic device generates a frame interface of the first application based on the first rendering tree; during the process of the electronic device generating the frame interface of the first application, the electronic device operates at the second frequency.

[0034] In the above embodiment, the electronic device can determine whether to adjust the CPU frequency based on the relationship between the task volume of the rendering tree and the threshold. If the task volume of the rendering tree is greater than the threshold, the electronic device operates at a higher frequency so that the rendering tree can be converted into rendering instructions in a timely manner.

[0035] In combination with some embodiments of the third aspect, in some embodiments, the electronic device generates a frame interface of the first application based on the first rendering tree, specifically including: the electronic device splits the first rendering tree to obtain N sub-rendering trees, where N is an integer greater than 1; the electronic device converts the N sub-rendering trees into first rendering instructions in parallel; and the electronic device generates a frame interface of the first application based on the first rendering instructions.

[0036] In the above embodiment, the electronic device may further split the rendering tree into multiple sub-rendering trees, and convert the multiple sub-rendering trees into rendering instructions in parallel to increase the duration of converting the rendering tree into rendering instructions.

[0037] In a fourth aspect, an embodiment of the present application provides an interface generation method, which is applied to an electronic device running a first application, the method comprising: the electronic device generates a first rendering tree, the first rendering tree including drawing operations for generating a frame interface of the first application; the electronic device traverses different parts of the first rendering tree through multiple different threads to generate a first rendering instruction, the rendering instruction is an instruction in a rendering engine, an image processing library or a GPU driver; the electronic device generates a frame interface of the first application based on the first rendering instruction.

[0038] In the above embodiment, the electronic device traverses the rendering tree in different orders and converts the rendering tree into rendering instructions in parallel through multiple threads, thereby reducing the latency of converting the rendering tree into rendering instructions and ensuring that the electronic device can generate the interface in a timely manner.

[0039] In combination with some embodiments of the fourth aspect, in some embodiments, before the electronic device traverses the first rendering tree in different orders through multiple different threads to generate a first instruction, the method also includes: the electronic device determines a first task amount, which is the task amount of the first rendering tree, and the first task amount is used to measure the time or computational amount of converting the first rendering tree into the first rendering instruction; the electronic device determines that the first task amount is greater than a first threshold.

[0040] In the above embodiment, considering that the time required to convert the rendering tree into rendering instructions is long when the amount of rendering tree tasks is greater than a threshold, in this case, the rendering tree is converted into rendering instructions in parallel through multiple threads, thereby reducing the conversion time.

[0041] In combination with some embodiments of the fourth aspect, in some embodiments, the electronic device traverses the first rendering tree in different orders through multiple different threads to generate a first instruction, specifically including: the electronic device traverses the first part of the first rendering tree through the first thread, and saves the generated second rendering instruction in the first buffer; the electronic device traverses the second part of the first rendering tree through the second thread, and saves the generated third rendering instruction in the second buffer; the electronic device submits the rendering instruction in the first buffer and the rendering instruction in the second buffer to the third buffer to obtain the first rendering instruction.

[0042] In the above embodiment, rendering instructions generated by conversion of different threads may be located in different buffers, and finally the rendering instructions in different buffers are submitted to the same buffer, thereby driving the GPU to generate an interface.

[0043] In combination with some embodiments of the fourth aspect, in some embodiments, the electronic device submits the rendering instructions in the first buffer and the rendering instructions in the second buffer to the third buffer to obtain the first rendering instruction, specifically including: the rendering instructions in the first buffer include the second rendering instructions and the third rendering instructions, and the instructions in the second buffer include the fourth rendering instructions; the instructions in the third buffer are arranged in the following order: the second rendering instruction, the fourth rendering instruction, and the third rendering instruction.

[0044] In the above embodiment, when rendering instructions in different buffers are submitted to the same buffer, the order of the rendering instructions can be adjusted, thereby restoring the dependency of the rendering nodes.

[0045] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code comprising computer instructions, the one or more processors calling the computer instructions to enable the electronic device to execute: the electronic device generates a first rendering tree, the first rendering tree comprising a drawing operation for generating a frame interface of the first application; the electronic device splits the first rendering tree into N sub-rendering trees, where N is greater than 1; the electronic device converts the N sub-rendering trees into a first rendering instruction in parallel, the rendering instruction being an instruction in a rendering engine, an image processing library, or a GPU driver; the electronic device generates a frame interface of the first application based on the first rendering instruction.

[0046] In combination with some embodiments of the fifth aspect, in some embodiments, the one or more processors are specifically used to call the computer instructions to cause the electronic device to execute: the electronic device determines a first task amount, where the first task amount is the task amount of the first rendering tree, and the first task amount is used to indicate the time consumption or computational amount of converting the first rendering tree into the first rendering instruction; in response to the electronic device determining that the first task amount is greater than a first threshold, the electronic device splits the first rendering tree to obtain the N sub-rendering trees.

[0047] In combination with some embodiments of the fifth aspect, in some embodiments, the one or more processors are also used to call the computer instructions to enable the electronic device to execute: the electronic device determines M based on the first task amount and the first threshold, and the M is less than or equal to the N, and the M is an integer greater than or equal to the ratio of the first task amount to the first threshold; the electronic device determines the integer greater than or equal to the M as the N.

[0048] In combination with some embodiments of the fifth aspect, in some embodiments, the one or more processors are specifically used to call the computer instruction to enable the electronic device to execute: the electronic device determines the first task amount by determining the task amount of the drawing operation in the first rendering tree.

[0049] In combination with some embodiments of the fifth aspect, in some embodiments, the N sub-rendering trees include a second rendering tree and a third rendering tree, a difference between the second task volume and the third task volume is less than a difference threshold, the second task volume is the task volume of the second rendering tree, and the second task volume is used to measure the time or computational amount of converting the second rendering tree into rendering instructions, and the third task volume is the task volume of the third rendering tree, and the third task volume is used to measure the time or computational amount of converting the third rendering tree into rendering instructions.

[0050] In conjunction with some embodiments of the fifth aspect, in some embodiments, the one or more processors are specifically configured to call the computer instructions to cause the electronic device to execute: determining, by the electronic device, that the root rendering node of the first rendering tree has N child nodes, where the child nodes are rendering nodes directly connected to the rendering node; and splitting, by the electronic device, the first rendering tree into N child rendering trees.

[0051] In combination with some embodiments of the fifth aspect, in some embodiments, the one or more processors are specifically used to call the computer instructions to cause the electronic device to execute: the electronic device divides the interface of the first application into N areas; the electronic device splits the first rendering tree based on the N areas to obtain N sub-rendering trees, and the N sub-rendering trees correspond one-to-one to the N areas.

[0052] In conjunction with some embodiments of the fifth aspect, in some embodiments, the one or more processors are specifically configured to call the computer instructions to cause the electronic device to execute: the electronic device determines a first task amount, where the first task amount is the task amount of the first rendering tree, and the first task amount is used to measure the time or computational amount of converting the first rendering tree into rendering instructions, and the first task amount is greater than a first threshold; the electronic device determines that a root rendering node of the first rendering tree has K child nodes, where K is less than N; the electronic device splits the first rendering tree into K sub-rendering trees; and after the electronic device determines that the task amount of the fifth rendering tree is greater than the first threshold, the electronic device splits the fifth rendering tree into N-K+1 rendering sub-trees, where the K sub-rendering trees include the fifth rendering tree, and the task amounts of the N rendering sub-trees are all less than the first threshold.

[0053] In combination with some embodiments of the fifth aspect, in some embodiments, the one or more processors are specifically used to call the computer instruction to enable the electronic device to execute: the electronic device fills the instructions converted from the N sub-rendering trees into N buffers through N threads respectively; the electronic device submits the instructions of the N buffers to the first buffer, and the instructions in the first buffer are the first rendering instructions.

[0054] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors calling the computer instructions to enable the electronic device to execute: the electronic device generates a first rendering tree through the first process, the first rendering tree including drawing operations for generating a one-frame interface of the first process; the electronic device splits the first rendering tree into a second rendering tree and a third rendering tree, the second rendering tree including part of the drawing operations in the first rendering tree, the third rendering tree including part of the drawing operations in the first rendering tree, and the second rendering tree and the third rendering tree are different; the electronic device converts the second rendering tree into a first rendering instruction through a first thread, the first rendering instruction is stored in a first buffer, and the rendering instruction is an instruction in a rendering engine, an image processing library, or a GPU driver; the electronic device converts the third rendering tree into a second rendering instruction through a second thread, the second instruction is stored in a second buffer; the electronic device generates a one-frame interface of the first process based on the first rendering instruction and the second rendering instruction.

[0055] In combination with some embodiments of the sixth aspect, in some embodiments, the one or more processors are further used to call the computer instruction to cause the electronic device to execute: the electronic device determines a first task amount, the first task amount is the task amount of the first rendering tree, the first task amount is used to measure the time or computational amount of converting the first rendering tree into rendering instructions, and the first task amount is greater than a first threshold.

[0056] In combination with some embodiments of the sixth aspect, in some embodiments, the one or more processors are also used to call the computer instructions to enable the electronic device to execute: the first rendering instruction is located in the first buffer held by the first thread, the second rendering instruction is located in the second buffer held by the second thread, and the electronic device submits the instructions in the first buffer and the rendering instructions in the second buffer to the third buffer; the electronic device generates a frame interface of the first process based on the third buffer.

[0057] In combination with some embodiments of the sixth aspect, in some embodiments, the third buffer is the second buffer, or the third buffer is the first buffer.

[0058] In a seventh aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors calling the computer instructions to enable the electronic device to execute: the electronic device generates a first rendering tree, the first rendering tree including a drawing operation for generating a one-frame interface of the first application; the electronic device determines a first task amount, the first task amount being the task amount of the first rendering tree, the first task amount being used to measure the time or computational amount of converting the first rendering tree into rendering instructions, the first task amount being greater than a first threshold, the rendering instructions being instructions in a rendering engine, an image processing library or a GPU driver; if the first task amount is greater than the first threshold, the electronic device configures the CPU operating frequency to change from the first frequency to a second frequency, the second frequency being higher than the first frequency; the electronic device generates a one-frame interface of the first application based on the first rendering tree; in the process of the electronic device generating the one-frame interface of the first application, the electronic device operates at the second frequency.

[0059] In combination with some embodiments of the seventh aspect, in some embodiments, the one or more processors are specifically used to call the computer instructions to enable the electronic device to execute: the electronic device splits the first rendering tree to obtain N sub-rendering trees, where N is an integer greater than 1; the electronic device converts the N sub-rendering trees into first rendering instructions in parallel; and the electronic device generates a frame interface of the first application based on the first rendering instructions.

[0060] In an eighth aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code comprising computer instructions, the one or more processors calling the computer instructions to enable the electronic device to execute: the electronic device generates a first rendering tree, the first rendering tree comprising drawing operations for generating a frame interface of the first application; the electronic device traverses different parts of the first rendering tree through multiple different threads to generate a first rendering instruction, the rendering instruction being an instruction in a rendering engine, an image processing library or a GPU driver; the electronic device generates a frame interface of the first application based on the first rendering instruction.

[0061] In combination with some embodiments of the eighth aspect, in some embodiments, the one or more processors are further used to call the computer instruction to cause the electronic device to execute: the electronic device determines a first task amount, the first task amount is the task amount of the first rendering tree, and the first task amount is used to measure the time or computational amount of converting the first rendering tree into the first rendering instruction; the electronic device determines that the first task amount is greater than a first threshold.

[0062] In combination with some embodiments of the eighth aspect, in some embodiments, the one or more processors are specifically used to call the computer instructions to enable the electronic device to execute: the electronic device traverses the first part of the first rendering tree through a first thread, and saves the generated second rendering instructions in a first buffer; the electronic device traverses the second part of the first rendering tree through a second thread, and saves the generated third rendering instructions in a second buffer; the electronic device submits the rendering instructions in the first buffer and the rendering instructions in the second buffer to the third buffer to obtain the first rendering instructions.

[0063] In combination with some embodiments of the eighth aspect, in some embodiments, the one or more processors are specifically used to call the computer instructions to enable the electronic device to execute: the rendering instructions of the first buffer include the second rendering instructions and the third rendering instructions, and the instructions of the second buffer include the fourth rendering instructions; the instructions of the third buffer are arranged in the following order: the second rendering instruction, the fourth rendering instruction, and the third rendering instruction.

[0064] In the ninth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions to enable the electronic device to execute the method described in the first aspect, the second aspect, the third aspect, the fourth aspect, any possible implementation of the first aspect, any possible implementation of the second aspect, any possible implementation of the third aspect, and any possible implementation of the fourth aspect.

[0065] In the tenth aspect, an embodiment of the present application provides a computer program product comprising instructions. When the above-mentioned computer program product is run on an electronic device, the above-mentioned electronic device executes the method described in the first aspect, the second aspect, the third aspect, the fourth aspect, any possible implementation of the first aspect, any possible implementation of the second aspect, any possible implementation of the third aspect, and any possible implementation of the fourth aspect.

[0066] In the eleventh aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions. When the instructions are executed on an electronic device, the electronic device executes the method described in the first aspect, the second aspect, the third aspect, the fourth aspect, any possible implementation of the first aspect, any possible implementation of the second aspect, any possible implementation of the third aspect, and any possible implementation of the fourth aspect.

[0067] It is understandable that the electronic devices provided in the fifth, sixth, seventh, and eighth aspects, the chip system provided in the ninth aspect, the computer program product provided in the tenth aspect, and the computer storage medium provided in the eleventh aspect are all used to execute the methods provided in the embodiments of the present application. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] FIG1 is an exemplary schematic diagram of an application generating a bitmap according to an embodiment of the present application;

[0069] FIG2 is an exemplary schematic diagram of a process of generating an interface for an electronic device according to an embodiment of the present application;

[0070] FIG3 is an exemplary schematic diagram of an interface generation method provided in an embodiment of the present application;

[0071] FIG4 is an exemplary schematic diagram of determining the task amount of a rendering tree according to an embodiment of the present application;

[0072] FIG5A and FIG5B are exemplary schematic diagrams of a split rendering tree provided in an embodiment of the present application;

[0073] FIG6A and FIG6B are exemplary schematic diagrams of a split rendering tree provided in an embodiment of the present application;

[0074] FIG7 is another exemplary schematic diagram of a split rendering tree provided in an embodiment of the present application;

[0075] FIG8 is an exemplary schematic diagram of converting a rendering tree into GPU instructions in parallel according to an embodiment of the present application;

[0076] 9A and 9B are another exemplary schematic diagram of a process of generating an interface for an electronic device according to an embodiment of the present application;

[0077] FIG10 is another exemplary schematic diagram of the process of the interface generation method provided in an embodiment of the present application;

[0078] FIG11 is an exemplary schematic diagram of adjusting CPU computing power based on the workload of a rendering tree according to an embodiment of the present application;

[0079] FIG12 is another exemplary schematic diagram of the interface generation method provided in an embodiment of the present application;

[0080] FIG13 is an exemplary schematic diagram of a rendering thread traversing a rendering tree in different orders according to an embodiment of the present application;

[0081] FIG14 is an exemplary schematic diagram of submitting a GPU instruction to an instruction queue according to an embodiment of the present application;

[0082] FIG15 is an exemplary schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0083] FIG16 is an exemplary schematic diagram of the software structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0084] The terms used in the following examples of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and appended claims of this application, the singular expressions "a," "an," "the," "above," "the," and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0085] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0086] The term "user interface (UI)" in the following embodiments of this application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content that the user can recognize. The commonly used form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0087] For ease of understanding, the following first introduces the relevant terms and concepts involved in the embodiments of this application. The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0088] The interface serves as a medium for interaction and information exchange between applications and users. Each time a vertical synchronization signal arrives, the electronic device generates the interface for the foreground application. The frequency of the vertical synchronization signal is related to the refresh rate of the electronic device's screen; for example, the frequency of the vertical synchronization signal can be the same as the refresh rate of the electronic device's screen.

[0089] That is, each time before the electronic device refreshes the content displayed on the screen, it needs to generate the interface of the application for the foreground application, so as to show the newly generated interface of the application to the user when the screen is refreshed.

[0090] For an electronic device to generate an application interface, the application must render and generate a bitmap, passing the bitmap to the surface compositor (SurfaceFlinger). That is, the application, as a producer, draws and generates a bitmap, storing it in a buffer queue (BufferQueue) provided by the surface compositor; the surface compositor, as a consumer, continuously retrieves the bitmap generated by the application from the BufferQueue. The bitmap is located on a surface generated by the application, which is then filled into the BufferQueue.

[0091] After the surface compositor obtains the bitmap of the visible application, the surface compositor and the hardware compositing strategy module (HWC) determine how to composite the bitmap as a layer.

[0092] After the surface compositor and / or hardware compositing strategy module performs bitmap synthesis, it places the synthesized bitmap into a frame buffer and passes it to the display subsystem (DSS). The DSS then displays the synthesized bitmap on the screen. The frame buffer can be an on-screen buffer. Bitmaps on the surface compositor are also called layers.

[0093] The process of the application generating a bitmap is shown in Figure 1 below.

[0094] FIG1 is an exemplary schematic diagram of an application generating a bitmap provided in an embodiment of the present application.

[0095] As shown in FIG1 , after receiving the vertical synchronization signal (Vsync), the application starts to generate a bitmap. The specific steps can be divided into three steps, namely step S101 , step S102 and step S103 .

[0096] S101: The main thread traverses the views of the application and saves the drawing operation of each view into a newly generated rendering tree.

[0097] The main thread (UI thread) invalidates the view hierarchy. The UI thread traverses the application's views by calling measure(), layout(), and draw() methods, determining and saving the drawing operations for each view. It then records the view and its associated drawing operations (e.g., drawline) into the display list of the render node in the render tree. The data stored in the display list can be a drawing operation structure (DrawOP or DrawListOP).

[0098] Among them, the view is the basic element that constitutes the application interface, and a control on the interface can correspond to one or more views.

[0099] Optionally, in some embodiments of the present application, within a drawing method call, the application's UI thread also reads the content carried by the view into memory. For example, an image carried by an image view (imageview) or text carried by a text view (textview). Alternatively, within a drawing method call, the application's UI thread determines the operation of reading the content carried by the view into memory and records it in a drawing instruction list. The drawing operation structure in the drawing instruction list can also be called a drawing instruction.

[0100] Among them, the drawing operation structure is a data structure used to draw graphics, such as drawing lines, drawing rectangles, drawing text, etc. When the rendering node is traversed, the drawing operation structure will be converted into API calls of the image processing library, interface calls in the OpenGLES library, Vulkan library, and Metal library through the rendering engine. For example, in the rendering engine (Skia library), drawline will be encapsulated as DrawLineOp. DrawLineOp is a data structure that contains drawing data such as line length, width and other information. DrawLineOp will be further encapsulated into interface calls in the OpenGLES library, Vulkan library, and Metal library, and then obtain GPU instructions. In the following text, the interface calls in the Skia library, the OpenGLES library interface calls, the Vulkan library interface calls and / or the Metal library interface calls are collectively referred to as rendering instructions. That is, the rendering tree will be converted into rendering instructions by the rendering thread, and then further converted into GPU instructions that the GPU can recognize and process. The OpenGL ES, Vulkan, and Metal libraries are collectively referred to as image processing libraries or graphics rendering libraries. During the generation of a single frame, electronic devices generate rendering instructions using the OpenGL ES, Vulkan, or Metal libraries. Image processing libraries provide graphics rendering APIs and driver support.

[0101] Among them, DrawOP can be stored in the stack of the application in a chain data structure.

[0102] The drawing instruction list can be a buffer that records all drawing operation structures or identifiers of all drawing operations, such as addresses and sequence numbers, included in a frame of the application interface. When an application has multiple windows or displays on different display areas, multiple render trees corresponding to the multiple windows need to be generated independently.

[0103] The render tree is a data structure generated by the UI thread and used to generate the application interface. The render tree can include multiple render nodes, each of which contains rendering attributes and a list of drawing instructions. The render tree records some or all of the information required to generate a single frame of the application interface.

[0104] Optionally, in some embodiments of the present application, the UI thread may only traverse the dirty regions (also known as regions that need to be redrawn) of the view to generate a differential render tree. After the differential render tree is passed / synchronized to the rendering thread, the rendering thread can determine the render tree to use for rendering the current frame's interface by comparing the differential render tree with the render tree used for rendering the previous frame.

[0105] S102: The main thread synchronizes the rendering tree to the rendering thread. The rendering tree is located in the stack of the application.

[0106] The UI thread passes / synchronizes the render tree to the render thread, where the render tree is located in the stack of the process corresponding to the application.

[0107] S103: The rendering thread executes the drawing instructions in the rendering tree to generate a bitmap.

[0108] The rendering thread first obtains a hardware canvas (HardwareCanvas) and performs drawing operations in the render tree on the hardware canvas to generate a bitmap. The hardware canvas is located on a surface owned by the application, which carries a bitmap or other format of data used to store image information.

[0109] S104: The rendering thread sends the surface carrying the bitmap to the surface compositor.

[0110] The rendering thread sends the generated bitmap to the surface compositor through the surface to participate in layer synthesis.

[0111] Step S101 can be considered the construction phase, which is primarily responsible for determining the size, position, transparency, and other attributes of each view in the application. For example, the drawLine function in a view can be encapsulated into a DrawLineOp during construction, which contains drawing data such as the length and width of the line. It can also include an interface call corresponding to the DrawLineOp of the underlying graphics processing library, which is used to call the underlying graphics library to generate a bitmap during the rendering phase.

[0112] Similarly, step S103 can be considered as the rendering stage, which is mainly responsible for traversing the rendering nodes of the rendering tree and performing the drawing operation of each rendering node, thereby generating a bitmap on the hardware canvas. In this process, the rendering thread calls the underlying graphics processing library, such as the OpenGLES library, Vulkan library, Metal library, etc., and then calls the GPU to complete the rendering to generate the bitmap.

[0113] FIG2 is an exemplary schematic diagram of a process of generating an interface for an electronic device provided in an embodiment of the present application.

[0114] As shown in FIG2 , the process of generating the first frame interface is steps ①, ②, ③, ④, and ⑤ in FIG2 ; the process of generating the second frame interface is steps ⑥, ⑦, and ⑤ in FIG2 .

[0115] The process of an electronic device generating a first frame interface includes: after the UI thread of the application receives the vertical synchronization signal, the UI thread of the application generates a rendering tree, as shown in ① in Figure 2; after receiving the rendering tree, the rendering thread of the application needs to convert the drawing instruction list and rendering attributes in the rendering tree into GPU instructions, such as Vulkan library instructions, OpenGLES library instructions, Metal instructions and other instructions that the GPU driver can recognize and process, as shown in ② in Figure 2; after receiving the GPU instructions, the GPU or GPU driver generates a bitmap, as shown in ③ in Figure 2; the surface compositor and / or hardware synthesis strategy module receives the bitmap and performs layer synthesis on the bitmap as a layer, as shown in ④ in Figure 2; the display subsystem receives the synthesized bitmap from the surface compositor and / or hardware synthesis strategy module, and then sends it for display, as shown in ⑤ in Figure 2.

[0116] The process of the electronic device generating the second frame interface includes: after the UI thread of the application receives the vertical synchronization signal, the UI thread of the application generates a rendering tree, as shown in ⑥ in Figure 2; after receiving the rendering tree, the rendering thread of the application needs to convert the drawing instruction list and rendering attributes in the rendering tree into GPU instructions, such as Vulkan library instructions, OpenGLES library instructions, Metal instructions and other instructions that the GPU driver can recognize and process, as shown in ⑦ in Figure 2.

[0117] However, in the process of the electronic device generating the second frame interface, since the rendering thread takes a long time to convert the drawing instruction list and rendering attributes in the rendering tree into instructions that the GPU can recognize and process, the bitmap cannot be passed to the surface compositor and / or hardware synthesis strategy module in time. After receiving the vertical synchronization signal, the surface compositor and / or hardware synthesis strategy module will perform layer synthesis and then send it for display. However, since the surface compositor and / or hardware synthesis strategy module does not receive the interface of the application in the second frame interface, the surface compositor and / or hardware synthesis strategy module will use the interface of the application in the first frame interface as the interface of the application in the second frame interface to perform layer synthesis and then send it for display, as shown in ⑤ in Figure 2.

[0118] Obviously, since the rendering thread fails to convert the rendering attributes and drawing instruction lists in the render tree into GPU instructions in a timely manner, the GPU cannot generate the bitmap in time, which causes the application interface to freeze.

[0119] Among them, in different scenarios, the reason why the rendering thread cannot convert the rendering tree into GPU instructions in time can be summarized as: the mismatch between the load of the rendering tree corresponding to the interface of the application to be generated and the computing power of the electronic device. The load of the rendering tree can be expressed in many ways, which are not limited here, for example, the amount of calculation required to convert the rendering tree into GPU instructions, the memory usage of the rendering tree, etc. Among them, the matching of the load of the rendering tree and the computing power of the electronic device means that when the computing power of the electronic device is within a certain range, the rendering tree can always be converted into GPU instructions within a preset time; on the contrary, when the time it takes for the rendering tree to be converted into GPU instructions exceeds the preset time, the interface of the application will freeze or drop frames. At this time, the load of the rendering tree and the computing power of the electronic device do not match.

[0120] Optionally, in some embodiments of the present application, the computing power of the electronic device may be increased by increasing the frequency of the CPU, thereby enabling the rendering thread to always convert the rendering tree into GPU instructions in a timely manner.

[0121] However, increasing the CPU frequency increases the power consumption of electronic devices, reducing the energy efficiency of generating a single frame of interface. Furthermore, before the rendering thread converts the render tree into GPU instructions, it cannot determine the load of converting the render tree into GPU instructions and cannot select an appropriate CPU frequency.

[0122] Based on this, embodiments of the present application provide an interface generation method and an electronic device.

[0123] The interface generation method provided in the embodiment of the present application can have a built-in load scoring model on the electronic device. During the process of the UI thread of the application generating the rendering tree, or after the UI thread of the application generates the rendering tree, the UI thread or rendering thread or unified rendering process of the application can determine the load of the rendering tree converted into GPU instructions based on the load scoring model, and the electronic device can further select an appropriate CPU frequency based on the load.

[0124] Among them, the unified rendering process (unirender) is a process independent of the application, which is used to receive the rendering tree generated by the UI threads of different applications. Among them, the application and the unified rendering process complete data interaction through inter-process communication (IPC). For the specific content of the unified rendering process, please refer to the patent application with application number 2021114105136 and the invention name “Interface Generation Method and Electronic Device” and the patent application with application number 202111410643X and the invention name “Interface Generation Method and Electronic Device”, which will not be repeated here.

[0125] It can be understood that the interface generation method provided in the embodiment of the present application determines the load of the rendering tree converted into GPU instructions through a load scoring model, and then selects the appropriate CPU frequency, so that the rendering thread or unified rendering process can execute the conversion of GPU instructions in a timely manner, while reducing the power consumption increase caused by increasing the CPU frequency.

[0126] Optionally, in some embodiments of the present application, the interface generation method provided in the embodiments of the present application can split the rendering tree of the application, and the multiple split rendering trees are converted into GPU instructions by different threads respectively.

[0127] Optionally, in some embodiments of the present application, the interface generation method provided in the embodiments of the present application can modify the order of traversing the rendering tree so that multiple threads can traverse a rendering tree simultaneously to generate GPU instructions.

[0128] It is understandable that the interface generation method provided in the embodiment of the present application reduces the time it takes to convert the rendering tree into GPU instructions through multi-threaded parallelism, thereby reducing the probability of interface freezes. Moreover, without considering the overhead of multi-threaded parallelism, if the low-frequency energy efficiency of the CPU of the electronic device is relatively high, the rendering tree can be converted into GPU instructions through multi-threaded parallelism by reducing the frequency, thereby improving the energy efficiency of generating a frame of interface without increasing the time it takes to generate a frame of interface.

[0129] The following is an illustrative introduction to the interface generation method and electronic device provided in the embodiments of the present application in conjunction with the content shown in FIG3 .

[0130] FIG3 is an exemplary schematic diagram of the process of the interface generation method provided in an embodiment of the present application.

[0131] S301: After receiving a vertical synchronization signal, the UI thread of the application generates a rendering tree corresponding to the current frame interface, and determines the task amount of the rendering tree during the process of generating the rendering tree.

[0132] The rendering tree corresponding to the current frame interface generated by the UI thread of the application can be referred to the text description in Figure 1 above, which will not be repeated here.

[0133] When generating a render tree, the UI thread of the application can determine the workload of the render tree by determining the workload of each drawing operation or drawing operation structure. The workload of the render tree represents the workload of converting the render tree into GPU instructions during the generation of the current frame's interface.

[0134] In particular, when the CPU computing power is the same, such as the frequency, the amount of rendering tree tasks is positively correlated with the time it takes to convert the rendering tree into GPU instructions.

[0135] Since the more complex the GPU instructions, the longer it takes to convert the render tree into GPU instructions; and the more complex the GPU instructions, the longer it takes for the GPU to render and generate the bitmap. Therefore, it can be considered that the workload of the render tree is positively correlated with the time it takes to generate the current frame interface.

[0136] The load scoring model may be a task model table as shown below.

[0137] A task model table may be stored locally in the electronic device or in a cloud accessible to the electronic device. The task model table includes task quantity scores corresponding to different drawing operation structures. The task model table is shown in Tables 1 and 2 below.

[0138] Table 1 and Table 2 are exemplary schematic tables of a task model table provided in an embodiment of the present application.

[0139] Table 1

[0140] As shown in Table 1, the task model table stores the correspondence between drawing operations or drawing operation structures and task amounts. For example, the task amount corresponding to DrawRect(parameter 1, parameter 2) is F1(parameter 1, parameter 2), the task amount corresponding to DrawImage(parameter 3, parameter 4) is F2(parameter 3, parameter 4), and the task amount corresponding to ClipRect(parameter 5) is F3(parameter 5). F1(), F2(), and F3() are the task amount calculation functions corresponding to different drawing operations or drawing operation structures.

[0141] Optionally, in some embodiments of the present application, the task model table may store the correspondence between the drawing operation or drawing operation structure and the time consumed in converting the drawing operation or drawing operation structure into GPU instructions under different CPU computing capabilities, as shown in Table 2.

[0142] Table 2

[0143] As shown in Table 2, the time consumed by DrawRect(parameter 1, parameter 2) is T1(CPU parameter, parameter 1, parameter 2), the time consumed by DrawImage(parameter 3, parameter 4) is T2(CPU parameter, parameter 3, parameter 4), and the time consumed by ClipRect(parameter 5) is T3(CPU parameter, parameter 5). T1(), T2(), and T3() are the time calculation functions for different drawing operations or drawing operation structures.

[0144] The task model table can be generated by offline testing. For example, developers of terminal manufacturers can test each drawing operation or drawing operation structure and record the time consumption or task amount to generate the task model table.

[0145] Alternatively, the task model table can be generated and updated online. For example, when a user uses an electronic device, the operating system on the electronic device records the time or amount of tasks for different drawing operations or drawing operation structures in the process of generating the interface, and then records and updates the task model table in real time. In this case, due to the different conditions of electronic devices of the same model, the task model tables of electronic devices of the same model can be different, thereby more accurately evaluating the load of a frame interface. The condition of the electronic device may include the degree of aging of the electronic device, etc., which is not limited here.

[0146] Optionally, in some embodiments of the present application, rendering attributes are also involved in the calculation of the task amount. This is because in the subsequent interface call process converted into the image processing library, the rendering attributes of the rendering node will also act on the drawing operations in the rendering node's drawing instruction list, thereby affecting the converted GPU instructions.

[0147] Optionally, in some embodiments of the present application, the input parameters of the drawing operation may not be considered. Different drawing operations correspond to different task amounts. The same drawing operation, ie, the input parameters of the drawing operation may be different, but correspond to the same task amount.

[0148] Optionally, in some embodiments of the present application, after the UI thread of the application determines the workload of the rendering tree, the workload of the rendering tree can be saved as a separate parameter, and the separate parameter and the rendering tree can be passed to the rendering thread or the unified rendering process.

[0149] Optionally, in some embodiments of the present application, the UI thread of the application determines the task amount of each rendering node during the process of generating the rendering tree, and the task amount of each rendering node can be saved in the rendering node, as shown in Figure 4. The parameter used to save the task amount of the rendering node can be called a task parameter.

[0150] FIG4 is an exemplary schematic diagram of determining the task amount of a rendering tree provided by an embodiment of the present application.

[0151] As shown in FIG4 , the view structure corresponding to the current frame interface of the application is as follows: the subviews of the root view (view container 0) are view container 1 and view container 2, view container 1 has several subviews, and the subviews of view container 2 include view 22 .

[0152] As the application's UI thread traverses the view, it generates a render tree. During this process, the application's UI thread determines the task load for each render node based on the task model table. The application's UI thread then stores the task load for each render node in the corresponding render node's task parameters. For example, if the application's UI thread has already traversed the root view and view container 1, the generated render tree includes the root render node and render node 1, with task parameters added to the root render node and render node 1. In Figure 4, the root render node is the render node corresponding to the root view, and render node 1 is the render node corresponding to view container 1.

[0153] After the application's UI thread completes traversing the view, each render node in the render tree includes a task parameter that stores the amount of work for that render node. In subsequent processing, the UI thread, render thread, or render process can determine the amount of work for the render node based on the task parameter in the render node.

[0154] Optionally, in some embodiments of the present application, the task parameters of the root render node of a render tree may store not only the task load of the render node but also the task load of the entire render tree. Furthermore, for any render node in the render tree, the task parameters of the render node may store not only the task load of the render node but also the task load of any child render tree with the render node as its root.

[0155] Optionally, in some embodiments of the present application, the UI thread can determine the workload of the differential rendering tree and store the workload of the differential rendering tree in a task parameter. After the differential rendering tree is passed to the rendering thread or unified rendering process, the rendering thread or unified rendering process generates a rendering tree for the current frame's interface based on the differential rendering tree and the rendering tree for the previous frame's interface, and then determines the rendering tree for the current frame's interface. The rendering thread or unified rendering process then determines the workload of the rendering tree for the current frame's interface.

[0156] S302: When the task volume of the rendering tree is greater than a task volume threshold, the application program splits the rendering tree based on the task volume to obtain multiple sub-rendering trees.

[0157] If the task parameters of a render node are used to determine the workload of a render tree, as shown in Table 1 above, the application's UI thread, rendering thread, or unified rendering process determines whether the workload of the render tree is greater than a task threshold of 1. If the workload of the render tree is greater than the task threshold of 1, the application's UI thread, rendering thread, or unified rendering process splits the render tree into multiple sub-render trees. The multiple sub-render trees must meet the following condition: the workload of any of the multiple sub-render trees is less than the task threshold of 1.

[0158] Among them, the task volume threshold and the task volume threshold 1 can both be called the first threshold.

[0159] Alternatively, if the task parameters of the rendering node are used to determine the time it takes to convert a render tree into GPU instructions, as shown in Table 2 above, the application's UI thread, rendering thread, or unified rendering process determines whether the render tree's task load is greater than time threshold 1. If the render tree's task load is greater than time threshold 1, the application's UI thread, rendering thread, or unified rendering process splits the render tree into multiple sub-rendering trees. The multiple sub-rendering trees after splitting must meet the following condition: the time it takes for any of the multiple sub-rendering trees to convert into GPU instructions is less than time threshold 1. Time threshold 1 may be related to the screen refresh rate; for example, the higher the screen refresh rate, the smaller the time threshold 1.

[0160] The following text will use the workload of the render tree as an example. The workload of the render tree and the time it takes to convert the render tree into GPU instructions have the same meaning, so they can be used interchangeably.

[0161] Optionally, in some embodiments of the present application, after the UI thread of the application splits the rendering tree to obtain multiple sub-rendering trees, the data transmitted or synchronized by the UI thread of the application to the rendering thread or the unified rendering process are multiple sub-rendering trees.

[0162] Optionally, in some embodiments of the present application, splitting the rendering tree may take into account the concurrency overhead of threads and / or the dependencies between rendering nodes. The dependency of the rendering nodes refers to the need for the drawing operation of the child node to be executed after the drawing operation of the parent node due to the parent-child relationship of the rendering nodes. The parent-child relationship of the rendering nodes may affect whether the interface can be generated correctly. This is because when the drawing operation of the parent node and the drawing operation in the child node operate on the same pixel point, the drawing operation of the parent node needs to be executed before the drawing operation in the child node. On the contrary, when the drawing operation of the parent node and the drawing operation in the child node do not operate on the same pixel point, the drawing operation of the parent rendering node is independent of the drawing operation in the child node and has no dependency relationship.

[0163] Optionally, in some embodiments of the present application, without considering the overhead of thread concurrency, the rendering tree is split so that the workload (or time consumption) of any sub-rendering tree is less than a workload threshold 1 (or a time threshold 1). The overhead of thread concurrency can also be characterized by the workload or time consumption.

[0164] Optionally, in some embodiments of the present application, taking into account the overhead of thread concurrency, the rendering tree is split so that the sum of the workload (or time consumption) of any sub-rendering tree and the workload (or time consumption) of thread concurrency is less than a workload threshold 1 (or a time threshold 1).

[0165] Optionally, in some embodiments of the present application, if the render tree's workload is determined to be greater than a workload threshold, the operating system may adjust the CPU's computing power, such as by adjusting the CPU's operating frequency. Alternatively, if the render tree's conversion time to GPU instructions exceeds a time threshold, the operating system may adjust the CPU's computing power, such as by adjusting the CPU's operating frequency. The manner in which the CPU frequency is adjusted based on the render tree's workload can be found in the textual descriptions corresponding to Figures 10 and 11 below and will not be further elaborated here.

[0166] Here are some examples of how to split the render tree:

[0167] (1) Splitting method when the task volume of the sub-rendering tree is less than the task volume threshold.

[0168] FIG5A and FIG5B are exemplary schematic diagrams of splitting a rendering tree provided by an embodiment of the present application.

[0169] S501: Determine whether the task volume of the rendering tree is greater than a task volume threshold 1.

[0170] The UI thread, rendering thread or unified rendering process of the application determines whether the task volume of the rendering tree is greater than the task volume threshold 1. If so, step S502 is executed; if not, the process ends.

[0171] S502: Divide the root rendering node into N sub-rendering trees according to the dependency relationship between the root rendering node and its sub-nodes.

[0172] If the root render node of a render tree has N child nodes, the render tree is split into N child render trees. The child nodes of the root render node can each serve as the root node of the N child render trees. If one of the N child render trees has a root render node, the root render node serves as the root node of the child render tree. A child node is a node directly connected to the root node. In a render tree, the root node is the root render node.

[0173] S503: Determine whether the task amount of each sub-rendering tree in all sub-rendering trees is less than or equal to a task amount threshold 1.

[0174] The UI thread, rendering thread, or unified rendering process of the application determines whether the task volume of the split sub-rendering tree is greater than the task volume threshold 1. If so, step S502 is executed; if not, the process ends.

[0175] Optionally, in some embodiments of the present application, in consideration of the thread parallel overhead, the workload of each sub-rendering tree needs to be added with the overhead of one thread parallel.

[0176] S504: Split the sub-rendering tree whose task amount is greater than the task amount threshold 1.

[0177] The method of splitting the sub-rendering tree may refer to step S502 or the text description in FIG5B , which will not be repeated here.

[0178] If the task volume threshold 1 is 100, as shown in FIG5B , the child nodes of the root rendering node of the rendering tree with a task volume of 200 are rendering node 51 and rendering node 52; the child nodes of rendering node 51 are rendering node 512 and rendering node 513; the child nodes of rendering node 52 are rendering node 521, rendering node 522, and rendering node 523; and the child node of rendering node 522 is rendering node 5221.

[0179] After the first split, a sub-rendering tree with a task size of 40 and a sub-rendering tree with a task size of 160 are obtained. The root node of the sub-rendering tree with a task size of 40 is the root rendering node, the child node of the root rendering node is rendering node 51, and the child nodes of rendering node 51 are rendering nodes 512 and 513. The root node of the rendering tree with a task size of 160 is rendering node 52, the child nodes of rendering node 52 are rendering nodes 521, 522, and 523, and the child node of rendering node 522 is rendering node 5221.

[0180] Since the task volume of the rendering tree with a task volume of 160 is greater than the task volume threshold 1, a second split is performed.

[0181] After the second split, a rendering tree with a task load of 70, a rendering tree with a task load of 50, and a rendering tree with a task load of 40 are obtained. The root node of the rendering tree with a task load of 70 is rendering node 52, and its child node is rendering node 512. The root node of the rendering tree with a task load of 50 is rendering node 522, and its child node is rendering node 5221. The root node of the rendering tree with a task load of 40 is rendering node 523.

[0182] After the second split, four sub-render trees are obtained, each with a task load less than a task load threshold of 1. Furthermore, the dependencies between the rendering nodes in each of the four sub-render trees are preserved. After the four sub-render trees are converted into GPU instructions, the complete GPU instructions can be restored by simply submitting the corresponding GPU instructions to the command buffer in the command queue. For details, refer to the description of step S303 below and will not be repeated here.

[0183] Preserving the dependencies of render nodes means that, since no new parent-child relationships are created between render nodes, the order of GPU instructions within each child render tree remains unchanged. Furthermore, after submitting the GPU instructions to the command queue, the order of all GPU instructions in the command queue remains unchanged. It is understood that splitting a child render tree when its workload is less than the workload threshold ensures that, during the subsequent conversion of the render tree into GPU instructions, no single thread's workload exceeds the workload threshold, thereby preventing issues such as frame drops and interface freezes.

[0184] (2) Load balancing splitting method.

[0185] FIG6A and FIG6B are exemplary schematic diagrams of a split rendering tree provided in an embodiment of the present application.

[0186] S601: Determine whether the task volume of the rendering tree is greater than a task volume threshold 1.

[0187] The UI thread, rendering thread or unified rendering process of the application determines whether the task volume of the rendering tree is greater than the task volume threshold 1. If so, step S602 is executed; if not, the process ends.

[0188] S602: Divide the root rendering node into N sub-rendering trees according to the dependency relationship between the root rendering node and its sub-nodes.

[0189] If the root render node has N child nodes, the render tree is split into N render trees. The root render node's child nodes can each serve as the root node of each of the N child render trees. If one of the N child render trees has a root render node, the root render node serves as the root node of the child render tree. Child nodes are render nodes directly connected to the root node.

[0190] S603: Determine whether the task amount of each sub-rendering tree in all sub-rendering trees satisfies the constraint relationship.

[0191] The UI thread, rendering thread or unified rendering process of the application determines whether the task volume of the sub-rendering tree meets the constraint relationship. If so, the process ends; if not, step S604 is executed.

[0192] Optionally, in some embodiments of the present application, the minimum number of sub-render trees required can be determined based on the render tree's task load and task load threshold 1. For example, if the render tree's task load is 200 and task load threshold 1 is 30, the minimum number of sub-render trees required is determined to be 7. After splitting to obtain 7 render trees, each sub-render tree is determined to satisfy the constraints shown below. If the constraints are not satisfied, the sub-render tree is further split; if the constraints are satisfied, the sub-render tree is no longer split.

[0193] For example, let the workload of the rendering tree be load total , the task volume threshold 1 is load threshold , the thread concurrency overhead is cost, and the task threshold of the sub-rendering tree is Where i is the i-th child rendering tree, and N is the number of concurrent threads (the number of child rendering trees), then the constraint relationship needs to be satisfied and

[0194] Among them, without considering the thread concurrency overhead, the constraint relationship is and

[0195] If load threshold =80, load total =200, N=5, ε=10, then That is, the task volume of all sub-rendering trees needs to be between 30 and 50 to meet the constraint.

[0196] If N=2, first adjust the size of N (the number of rendering trees) so that Then judge whether it satisfies or

[0197] Optionally, in some embodiments of the present application, Determine the splitting and sorting of the rendering tree. threshold =80, load total =200, then we can Calculate N, where is the ceiling function.

[0198] S604: Move the rendering node or further split the sub-rendering tree.

[0199] The UI thread, rendering thread, or unified rendering process of an application can move rendering nodes from a sub-render tree with a high workload to a sub-render tree with a low workload by moving rendering nodes. Moving rendering nodes may break the dependencies between rendering nodes.

[0200] Alternatively, the application's UI thread, rendering thread, or unified rendering process can split off high-volume sub-render trees.

[0201] FIG6B is the same as the rendering tree in FIG5B , and according to the method shown in FIG6A , if load threshold =80, load total =200, ε=11, when N=4, the constraint relationship is Since there is a sub-rendering tree with a task volume of 70 in the second split rendering tree, the constraint condition is not met. The rendering node needs to be moved or the sub-render tree needs to be split. The task volume of the rendering node 52 is 40 and the task volume of the rendering node 521 is 30. The constraint condition cannot be met by moving the rendering node. So we further split the sub-rendering tree with a task size of 70.

[0202] When N=5, The task amounts of the split sub-rendering trees are 40, 40, 30, 50, and 40 respectively, which meet the constraints.

[0203] It is understandable that the load-balanced render tree splitting method balances the workload of different split sub-render trees, which can avoid the short board effect when converting the render tree to GPU instructions. That is, it avoids the time taken to convert the render tree to GPU instructions being extended due to the excessive workload of individual sub-render trees.

[0204] Optionally, in some embodiments of the present application, in the process of splitting a rendering tree or splitting a sub-rendering tree, rendering nodes with sibling relationships may be split preferentially.

[0205] Optionally, in some embodiments of the present application, during the process of splitting a rendering tree or splitting a sub-rendering tree, if the rendering attribute of a rendering node is empty, the drawing operation of the rendering node may be split into different sub-rendering trees.

[0206] Optionally, in some embodiments of the present application, during the process of splitting a render tree or a sub-render tree, if a render node does not depend on any other render node, the render node can be arbitrarily moved to a different render subtree. For example, in Figure 5B , if the root render node is a transparent render node, then render node 51 does not depend on any other render node, and render node 52 does not depend on any other render node.

[0207] (3) Display the splitting method of the area division.

[0208] FIG. 7 is another exemplary schematic diagram of splitting a rendering tree provided by an embodiment of the present application.

[0209] The interface to be generated by the application and displayed in the next frame is interface 701. Therein, the application window can be divided into mutually non-blocking areas in a variety of ways, and the content displayed in different areas is the display content corresponding to different sub-rendering trees after the split.

[0210] For example, interface 701 (excluding the status bar) is divided into area 1 and area 2. The view corresponding to area 1 includes view container 1 and its child nodes; the view corresponding to area 2 includes view container 2 and its child nodes, such as view 22. When generating the render tree, the application's UI thread can directly generate two child render trees: child render tree 1 corresponding to area 1 and child render tree 2 corresponding to area 2.

[0211] Alternatively, after the UI thread of the application generates the rendering tree, the rendering tree is divided into sub-rendering tree 1 and sub-rendering tree 2 according to the division of the regions.

[0212] (4) Split DrawOP

[0213] Optionally, in some embodiments of the present application, after the rendering thread or the rendering sub-thread of the unified rendering process traverses the rendering tree to obtain the chain-stored DrawOP, the DrawOP can be directly split.

[0214] S303: The application creates multiple rendering threads and converts multiple sub-rendering trees into GPU instructions in parallel.

[0215] The application creates multiple rendering threads, or the unified rendering process creates multiple sub-rendering threads, and then converts multiple sub-rendering trees into GPU instructions in parallel.

[0216] The number of rendering threads is consistent with the number of sub-rendering trees after the splitting, or the number of sub-rendering threads is consistent with the number of sub-rendering trees after the splitting.

[0217] FIG8 is an exemplary schematic diagram of converting a rendering tree into GPU instructions in parallel according to an embodiment of the present application.

[0218] As shown in FIG8 , after the rendering tree is split, sub-rendering tree 1 to sub-rendering tree N are obtained.

[0219] Different threads, such as the rendering thread or child rendering thread, traverse different sub-render trees to obtain GPU instructions. The unified rendering process, the application's UI thread, or the rendering thread can also request N command buffers from the command buffer pool, as shown in Figure 8, from command buffer 1 to command buffer N. These N command buffers are used to store GPU instructions corresponding to different sub-render trees.

[0220] Optionally, in some embodiments of the present application, when traversing the rendering tree, the thread first encapsulates the drawing operation into a drawing operation structure (for example, DrawOP), and then converts it into GPU instructions, such as interface calls in the OpenGLES library, Vulkan library, and Metal library.

[0221] After different threads receive GPU instructions, they sequentially submit the GPU instructions in each command buffer to a buffer in the command queue (e.g., the primary buffer). This order is the same as the order in which the render nodes in the render tree are traversed. That is, if the render nodes in child render tree 1 are traversed earlier than the render nodes in child render tree N, the GPU instructions in command buffer 1 in the primary buffer will be executed first, and the GPU instructions in command buffer 2 in the primary buffer will be executed later. Submitting data from different command buffers to the buffer in the command queue can be achieved in a variety of ways, which are not limited here.

[0222] Optionally, in some embodiments of the present application, when the electronic device converts the render tree into GPU instructions through the Vulkan library, the GPU instructions in command buffer 1 or command buffer N can be submitted to the primary buffer in the instruction queue by calling the vkQueueSubmit method. Since data synchronization from multiple command buffers to the buffer in the instruction queue is involved, the electronic device can achieve synchronization through a semaphore.

[0223] Optionally, in some embodiments of the present application, the electronic device may also move data from multiple command buffers to a buffer in the instruction queue through pointer operations.

[0224] Optionally, in some embodiments of the present application, the electronic device may also move the data of multiple command buffers to the buffer in the instruction queue by copying.

[0225] Optionally, in some embodiments of the present application, command buffer 1 may be a buffer in an instruction queue, such as a Primarily Buffer, and GPU instructions in other command buffers need to be submitted to command buffer 1.

[0226] Alternatively, in some embodiments of the present application, instead of requesting N command buffers from the command buffer pool, N address ranges can be divided in the primary buffer to carry GPU instructions corresponding to different sub-render trees. For example, 0x000000-0x0000FF is the address range corresponding to the first command buffer, and 0x000100-0x0001FF is the address range corresponding to the second command buffer.

[0227] S304: After the GPU executes the GPU instruction to generate a bitmap, the surface compositor and / or the hardware compositing strategy module performs layer compositing on the bitmap for display.

[0228] Among them, after the bitmap is generated, the bitmap will be obtained by the surface synthesizer and / or the hardware synthesis strategy module, and then the parameters will be used for layer synthesis. The synthesized bitmap will be obtained by the display subsystem for display.

[0229] After the electronic device executes the interface generation method shown in FIG. 3 , the process of the electronic device generating the interface is shown in FIG. 9A and FIG. 9B .

[0230] 9A and 9B are another exemplary schematic diagram of the process of generating an interface of an electronic device provided in an embodiment of the present application.

[0231] Comparing the contents shown in Figure 9A and Figure 2, the same contents as shown in Figure 9A and Figure 2 are not repeated here. In Figure 9A ⑦, since the electronic device converts the rendering tree into GPU instructions through two rendering threads (such as rendering thread 1 and rendering thread 2 in Figure 9A), the latency of converting the rendering tree into GPU instructions is reduced; the GPU can receive the GPU instructions in a timely manner to generate a bitmap, as shown in Figure 9A ⑧; then, the GPU can pass the generated bitmap to the surface synthesizer and / or the hardware synthesis strategy module, as shown in Figure 9A ⑨; finally, the display subsystem can send the bitmap after the participating layers are synthesized for display, as shown in Figure 9A ⑩, and there will be no interface freeze.

[0232] The difference between Figure 9A and Figure 9B is that the process of converting the render tree into GPU instructions can be performed by the rendering sub-thread of the unified rendering process, as shown in Figure 9B. For example, the rendering sub-threads of the unified rendering process can be rendering sub-thread 1 and rendering sub-thread 2 in Figure 9B. The contents that are common between Figure 9A and Figure 9B are not repeated here.

[0233] The above describes the process of the interface generation method in one embodiment of the present application in conjunction with the content shown in Figure 3. The following describes the process of other interface generation methods that are different from the content shown in Figure 3.

[0234] Optionally, in some embodiments of the present application, after determining the workload of the rendering tree, the rendering tree may not be split, but the computing power of the CPU may be provided to reduce the time consumption of converting the rendering tree into GPU instructions.

[0235] FIG10 is another exemplary schematic diagram of the process of the interface generation method provided in an embodiment of the present application.

[0236] S1001: After receiving a vertical synchronization signal, the UI thread of the application generates a rendering tree corresponding to the current frame interface, and determines the task amount of the rendering tree during the process of generating the rendering tree.

[0237] The content of step S1001 can refer to the text description of step S301 above, which will not be repeated here.

[0238] S1002: When the task volume of the rendering tree is greater than a task volume threshold, adjust the computing capacity of the CPU based on the task volume of the rendering tree.

[0239] If the render tree's workload exceeds the threshold, the application's UI thread, rendering thread, and unified rendering process pass the workload to the operating system. The operating system then adjusts the CPU's computing power based on the workload, for example, by adjusting the CPU frequency.

[0240] FIG11 is an exemplary schematic diagram of adjusting CPU computing power based on the workload of a rendering tree according to an embodiment of the present application.

[0241] For example, as shown in FIG11 , when the task volume of the rendering tree is 0-100, the task volume of the rendering tree may not be sent to the operating system, or the operating system may not adjust the CPU frequency after receiving the task volume of the rendering tree. When the task volume of the rendering tree is 101-200, the operating system adjusts the CPU frequency to frequency 1 after receiving the task volume of the rendering tree. When the task volume of the rendering tree is 201-300, the operating system adjusts the CPU frequency to frequency 2 after receiving the task volume of the rendering tree, where frequency 2 is higher than frequency 1.

[0242] After the render tree is converted into GPU instructions, the operating system adjusts the CPU's computing power to its original state, such as the default frequency in Figure 11.

[0243] Optionally, in some embodiments of the present application, after splitting the render tree and traversing the render tree via multiple threads, the CPU frequency can be reduced. For example, when the render tree task load is between 101 and 200, the operating system, upon receiving the render tree task load, determines to process the render tree in parallel via three threads and adjusts the CPU frequency to frequency 3. When the render tree task load is between 201 and 300, the operating system, upon receiving the render tree task load, determines to process the render tree in parallel via five threads and adjusts the CPU frequency to frequency 4, where frequency 3 is lower than frequency 4.

[0244] It is understandable that by adjusting the computing power of the CPU, the time it takes to convert the rendering tree into GPU instructions can be reduced, thereby reducing the probability of interface freezes and frame drops.

[0245] S1003: After converting the rendering tree into GPU instructions, a bitmap is generated, and the surface compositor and / or the hardware compositing strategy module performs layer compositing on the bitmap for display.

[0246] The content of step S1003 can be referred to FIG. 1 and the text description in step 304 , which will not be repeated here.

[0247] Optionally, in some embodiments of the present application, after determining the workload of the rendering tree, the rendering tree may not be split. Instead, the rendering tree may be traversed by multiple threads in different traversal orders, thereby reducing the time consumed in converting the rendering tree into GPU instructions.

[0248] FIG12 is another exemplary schematic diagram of the interface generation method provided in an embodiment of the present application.

[0249] S1201: After receiving the vertical synchronization signal, the UI thread of the application generates a rendering tree corresponding to the current frame interface, and determines the task amount of the rendering tree during the process of generating the rendering tree.

[0250] Among them, step S1201 can refer to the text description in step S301 and will not be repeated here.

[0251] S1202: When the task amount of the rendering tree is greater than the task amount threshold, multiple rendering threads traverse the rendering tree in parallel in different orders and convert the rendering tree into GPU instructions.

[0252] The number of rendering threads can be determined by the workload of the rendering tree. For example, let the workload of the rendering tree be load total , the task volume threshold is load threshold , the number of rendering threads is N, then in Different rendering threads traverse the rendering tree in different orders to generate GPU instructions, as shown in Figures 13 and 14.

[0253] Optionally, in some implementations of the present application, the number of rendering threads may also be a preconfigured fixed value.

[0254] FIG13 is an exemplary schematic diagram of a rendering thread traversing a rendering tree in different orders according to an embodiment of the present application.

[0255] As shown in FIG13 , the root node of the render tree is the root render node, the child nodes of the root render node are render node 1 and render node 2, the child nodes of render node 1 are render node 11 and render node 12, the child nodes of render node 2 are render node 21, render node 22, and render node 23, and the child node of render node 22 is render node 221.

[0256] First, thread 1 traverses in the following order: root render node, render node 1, render node 11, and render node 12. Thread 2 traverses in the following order: render node 2, render node 21, render node 22, render node 23, and render node 221.

[0257] Next, when thread 1 has traversed the root rendering node, rendering node 1, rendering node 11, and rendering node 12, thread 2 traverses to rendering node 21.

[0258] Optionally, in some embodiments of the present application, after thread 1 has traversed render node 1, render node 11, and render node 12, thread 2 traverses to render node 21, and then thread 2 continues to traverse render node 22, render node 23, and render node 221. Then, as shown in FIG8 , the GPU instructions generated by different threads traversing the render nodes are located in different command buffers and need to be submitted to the buffer of the instruction queue in sequence.

[0259] Optionally, in some embodiments of the present application, after thread 1 has traversed render nodes 1, 11, and 12, thread 2 traverses to render node 21. Thread 2 then traverses render node 22, 23, and 221, respectively, while thread 1 traverses render node 23, 22, and 221, respectively. In this manner, thread 2 traverses render node 23, while thread 1 traverses render node 22 and 221. Then, as shown in FIG14 , GPU instructions from different command buffers are submitted to the buffer of the command queue.

[0260] FIG14 is an exemplary schematic diagram of submitting GPU instructions to an instruction queue provided in an embodiment of the present application.

[0261] Since render node 23 is traversed by thread 2 and the GPU instructions corresponding to render node 23 are saved in command buffer 2, in order to reconstruct the dependency relationship of the render nodes in the render tree, the order of different GPU instructions needs to be adjusted during the process of submitting the GPU instructions in command buffer 1 and command buffer 2 to the command queue.

[0262] For example, first, GPU instructions corresponding to the root render node, render node 1, render node 11, and render node 12 are submitted from command buffer 2 to the buffer in the command queue; then, all GPU instructions in command buffer 1 are submitted to the buffer in the command queue; finally, the remaining GPU instructions in command buffer 2 (i.e., the GPU instructions corresponding to render node 23) are submitted to the buffer in the command queue.

[0263] Alternatively, in some embodiments of the present application, all GPU instructions in command buffer 2 and command buffer 1 may be submitted sequentially to the command queue's buffer. In this case, the dependencies of the render nodes in the render tree may not be reconstructed. If render node 23 has a dependency on render node 2, the dependencies of the render nodes in the render tree are not reconstructed; if render node 23 does not have a dependency on render node 2, the dependencies of the render nodes in the render tree are reconstructed.

[0264] S1203: After the GPU executes the GPU instruction to generate a bitmap, the surface compositor and / or the hardware compositing strategy module performs layer compositing on the bitmap for display.

[0265] The content of step S1203 may refer to the text description of step S304 above, which will not be repeated here.

[0266] The following describes the hardware structure and software architecture of the electronic device provided in the embodiments of the present application.

[0267] FIG15 is an exemplary schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application.

[0268] The electronic device may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, an in-vehicle device, a smart home device and / or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific type of the electronic device.

[0269] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0270] It is understood that the structures illustrated in the embodiments of the present invention do not constitute specific limitations on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0271] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0272] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0273] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0274] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0275] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C busses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, and the like via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K via the I2C interface, enabling communication between the processor 110 and the touch sensor 180K via the I2C bus interface, thereby implementing the touch function of the electronic device.

[0276] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of answering calls through a Bluetooth headset.

[0277] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0278] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, enabling the function of playing music through Bluetooth headphones.

[0279] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the electronic device's camera function. The processor 110 and the display 194 communicate via the DSI interface to implement the electronic device's display function.

[0280] The GPIO interface can be configured via software. The GPIO interface can be configured as either a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, display 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0281] USB port 130 is an interface that complies with USB standards and may be a Mini USB port, a Micro USB port, a USB Type-C port, or the like. USB port 130 can be used to connect a charger to charge an electronic device, transfer data between the electronic device and peripherals, connect headphones to play audio, and connect other electronic devices, such as augmented reality devices.

[0282] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present invention is only a schematic illustration and does not constitute a structural limitation of the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0283] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the electronic device's wireless charging coil. While charging the battery 142, the charging management module 140 can also power the electronic device through the power management module 141.

[0284] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0285] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0286] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in an electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0287] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to electronic devices. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and perform filtering, amplification and other processing on the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0288] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0289] The wireless communication module 160 can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, perform frequency modulation on it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0290] In some embodiments, the antenna 1 of the electronic device is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation system (SBAS).

[0291] The electronic device implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0292] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, the electronic device can include one or N display screens 194, where N is a positive integer greater than one.

[0293] The electronic device can realize the shooting function through the ISP, camera 193, video codec, GPU, display 194 and application processor.

[0294] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise and brightness. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0295] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0296] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when an electronic device selects a frequency, the DSP performs a Fourier transform on the frequency energy.

[0297] Video codecs are used to compress or decompress digital video. Electronic devices may support one or more video codecs. This allows them to play or record videos in a variety of encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0298] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU enables intelligent cognitive applications in electronic devices, such as image recognition, face recognition, speech recognition, and text comprehension.

[0299] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0300] Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation of DDR SDRAM is generally referred to as DDR5 SDRAM), etc.

[0301] Non-volatile memory may include disk storage devices and flash memory.

[0302] Flash memory can be divided into NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operating principle; single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. according to the storage cell potential level; universal flash storage (UFS) and embedded multi media card (eMMC) can be divided into UFS and eMMC according to the storage specification.

[0303] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data.

[0304] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110 .

[0305] The external memory interface 120 can be used to connect to an external non-volatile memory device to expand the storage capacity of the electronic device. The external non-volatile memory device communicates with the processor 110 via the external memory interface 120 to implement data storage. For example, files such as music and videos can be stored in the external non-volatile memory device.

[0306] The electronic device can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0307] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0308] The speaker 170A, also called a "speaker," is used to convert audio electrical signals into sound signals. The electronic device can listen to music or make hands-free calls through the speaker 170A.

[0309] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device receives a call or voice message, the voice can be heard by placing the receiver 170B close to the human ear.

[0310] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device can be provided with at least one microphone 170C. In other embodiments, the electronic device can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device can also be provided with three, four or more microphones 170C to realize sound signal collection, noise reduction, and identification of sound sources, and realize directional recording function, etc.

[0311] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0312] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. The electronic device determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 194, the electronic device detects the touch intensity based on pressure sensor 180A. The electronic device can also calculate the touch location based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, an instruction to create a new short message is executed.

[0313] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device. In some embodiments, the angular velocity of the electronic device around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device's shake, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shake of the electronic device through reverse motion to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0314] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device calculates the altitude using the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

[0315] The magnetic sensor 180D includes a Hall sensor. The electronic device can use the magnetic sensor 180D to detect the opening and closing of a flip case. In some embodiments, when the electronic device is a flip phone, the electronic device can detect the opening and closing of the flip cover based on the magnetic sensor 180D. Based on the detected opening and closing status of the case or flip cover, features such as automatic unlocking of the flip cover can be configured.

[0316] The accelerometer 180E can detect the magnitude of an electronic device's acceleration in all directions (generally three axes). When the electronic device is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0317] Distance sensor 180F is used to measure distance. The electronic device can measure distance using infrared or laser. In some embodiments, when shooting a scene, the electronic device can use distance sensor 180F to measure distance to achieve fast focus.

[0318] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device emits infrared light outward through the light emitting diode. The electronic device uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device. When insufficient reflected light is detected, the electronic device can determine that there is no object near the electronic device. The electronic device can use the proximity light sensor 180G to detect when the user holds the electronic device close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0319] The ambient light sensor 180L senses ambient light brightness. The electronic device can adaptively adjust the brightness of the display screen 194 based on the perceived ambient light. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking photos. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the electronic device is in a pocket to prevent accidental touches.

[0320] Fingerprint sensor 180H is used to collect fingerprints. Electronic devices can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering calls, etc.

[0321] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device uses the temperature detected by the temperature sensor 180J to implement a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device heats the battery 142 to prevent the electronic device from shutting down abnormally due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device boosts the output voltage of the battery 142 to prevent abnormal shutdown due to low temperature.

[0322] The touch sensor 180K is also referred to as a "touch-sensitive device." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also referred to as a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device, at a location different from that of the display screen 194.

[0323] The bone conduction sensor 180M can obtain vibration signals. In some embodiments, the bone conduction sensor 180M can obtain vibration signals from the vibrating bones of the human body. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulse signals. In some embodiments, the bone conduction sensor 180M can also be set in headphones to form bone conduction headphones. The audio module 170 can parse out voice signals based on the vibration signals of the vibrating bones of the human body obtained by the bone conduction sensor 180M to implement voice functions. The application processor can parse heart rate information based on the blood pressure pulse signals obtained by the bone conduction sensor 180M to implement heart rate detection functions.

[0324] Keys 190 include a power button, a volume button, and the like. Keys 190 may be mechanical keys or touch-sensitive keys. The electronic device may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device.

[0325] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0326] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0327] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device by inserting it into or removing it from the SIM card interface 195. The electronic device can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. Electronic devices interact with the network through SIM cards to implement functions such as calls and data communications. In some embodiments, the electronic device uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device and cannot be separated from the electronic device.

[0328] FIG16 is an exemplary schematic diagram of the software structure of the electronic device provided in an embodiment of the present application.

[0329] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device.

[0330] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0331] The application layer can include a series of application packages.

[0332] As shown in FIG16 , the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

[0333] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0334] As shown in FIG16 , the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

[0335] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0336] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0337] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0338] The phone manager is used to provide communication functions for electronic devices, such as call status management (including answering, hanging up, etc.).

[0339] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0340] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0341] Optionally, in some embodiments of the present application, the view system includes a rendering tree task amount estimation module, which determines the rendering tree task amount during or after the rendering tree is generated.

[0342] Optionally, in some embodiments of the present application, the view system includes a render tree splitting module that can split the render tree in different ways, wherein different render trees are traversed by different threads to generate GPU instructions.

[0343] Optionally, in some embodiments of the present application, the CPU scheduling module may adjust the computing power of the CPU based on the workload of the rendering tree, such as adjusting the frequency of the CPU.

[0344] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

[0345] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0346] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0347] The system library can include multiple functional modules, such as the browser engine (WebKit), rendering engine, surface compositor, hardware compositing strategy module, media library, image processing library (such as OpenGL ES), rendering engine (such as Skia library), etc.

[0348] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0349] The image processing library is used to implement 3D graphics drawing, image rendering, etc.

[0350] A rendering engine is a drawing engine for 2D graphics.

[0351] The kernel layer is the layer between hardware and software. The kernel layer includes display drivers, camera drivers, audio drivers, sensor drivers, etc.

[0352] Optionally, in some embodiments of the present application, the display subsystem includes a display driver.

[0353] As used in the above embodiments, the term “when…” may be interpreted to mean “if…” or “after…” or “in response to determining…” or “in response to detecting…”, depending on the context. Similarly, the phrases “upon determining…” or “if (stated condition or event) is detected” may be interpreted to mean “if determining…” or “in response to determining…” or “upon detecting (stated condition or event)” or “in response to detecting (stated condition or event)”, depending on the context.

[0354] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk).

[0355] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for generating an interface, characterized in that: Applied to an electronic device running a first application, the method includes: The electronic device generates a first rendering tree, where the first rendering tree includes a drawing operation for generating a frame interface of the first application; Splitting, by the electronic device, the first rendering tree into N sub-rendering trees, where N is greater than 1; The electronic device converts the N sub-rendering trees into first rendering instructions in parallel, where the rendering instructions are instructions in a rendering engine, an image processing library, or a GPU driver; The electronic device generates a frame interface of the first application based on the first rendering instruction.

2. The method according to claim 1, characterized in that The electronic device splits the first rendering tree into N sub-rendering trees, specifically including: The electronic device determines a first task amount, where the first task amount is the task amount of the first rendering tree, and the first task amount is used to indicate the time consumption or the amount of calculation required to convert the first rendering tree into the first rendering instruction; In response to the electronic device determining that the first task amount is greater than a first threshold, the electronic device splits the first rendering tree to obtain the N sub-rendering trees.

3. The method according to claim 2, characterized in that After the electronic device determines the first task amount, the method further includes: The electronic device determines M based on the first task amount and the first threshold, where M is less than or equal to N and is an integer greater than or equal to the ratio of the first task amount to the first threshold; The electronic device determines an integer greater than or equal to M as N.

4. The method according to claim 2, characterized in that The electronic device determines the first task amount, specifically including: The electronic device determines the first task amount by determining a task amount of drawing operations in the first rendering tree.

5. The method according to any one of claims 1 to 4, characterized in that The N sub-rendering trees include a second rendering tree and a third rendering tree. A difference between a second task volume and a third task volume is less than a difference threshold. The second task volume is the task volume of the second rendering tree, and the second task volume is used to measure the time or computational amount of converting the second rendering tree into rendering instructions. The third task volume is the task volume of the third rendering tree, and the third task volume is used to measure the time or computational amount of converting the third rendering tree into rendering instructions.

6. The method according to claim 1 or 2, characterized in that The electronic device splits the first rendering tree into N sub-rendering trees, specifically including: The electronic device determines that a root rendering node of the first rendering tree has N child nodes, where the child nodes are rendering nodes directly connected to the rendering node; The electronic device splits the first rendering tree into N sub-rendering trees.

7. The method according to claim 1 or 2, characterized in that The electronic device splits the first rendering tree into N sub-rendering trees, specifically including: The electronic device divides the interface of the first application into N areas; The electronic device splits the first rendering tree into N sub-rendering trees based on the N regions, and the N sub-rendering trees correspond one-to-one to the N regions.

8. The method according to claim 1, characterized in that The electronic device splits the first rendering tree into N sub-rendering trees, specifically including: The electronic device determines a first task amount, where the first task amount is the task amount of the first rendering tree, and the first task amount is used to measure the time or computational effort of converting the first rendering tree into rendering instructions, and the first task amount is greater than a first threshold; Determining, by the electronic device, that a root rendering node of the first rendering tree has K child nodes, where K is less than N; Splitting, by the electronic device, the first rendering tree into K sub-rendering trees; After the electronic device determines that the workload of the fourth rendering tree is greater than the first threshold, the electronic device splits the fourth rendering tree into N-K+1 rendering subtrees, the K sub-rendering trees include the fourth rendering tree, and the workload of the N rendering subtrees is less than the first threshold. Threshold.

9. The method according to any one of claims 1 to 7, characterized in that The electronic device converts the N sub-rendering trees into first rendering instructions in parallel, specifically including: The electronic device fills the instructions converted from the N sub-rendering trees into N buffers respectively through N threads; The electronic device submits the N buffered instructions to a first buffer, where the instructions in the first buffer are the first rendering instructions.

10. An interface generation method, characterized in that: Applied to an electronic device, where a first process is running on the electronic device, the method includes: The electronic device generates a first rendering tree through the first process, where the first rendering tree includes a drawing operation for generating a frame interface of the first process; The electronic device splits the first rendering tree into a second rendering tree and a third rendering tree, wherein the second rendering tree includes some drawing operations in the first rendering tree, and the third rendering tree includes some drawing operations in the first rendering tree, and the second rendering tree and the third rendering tree are different; The electronic device converts the second rendering tree into a first rendering instruction through a first thread, wherein the first rendering instruction is stored in a first buffer, and the rendering instruction is an instruction in a rendering engine, an image processing library, or a GPU driver; The electronic device converts the third rendering tree into a second rendering instruction through a second thread, and the second instruction is stored in a second buffer; The electronic device generates a frame interface of the first process based on the first rendering instruction and the second rendering instruction.

11. The method according to claim 10, characterized in that The method further comprises: The electronic device determines a first task amount, where the first task amount is the task amount of the first rendering tree, and the first task amount is used to measure the time consumption or computation amount of converting the first rendering tree into rendering instructions. The first task amount is greater than a first threshold.

12. The method according to claim 10 or 11, characterized in that The electronic device generates a frame interface of the first process based on the first instruction and the second instruction, specifically including: The first rendering instruction is located in a first buffer held by the first thread, the second rendering instruction is located in a second buffer held by the second thread, and the electronic device submits the instructions in the first buffer and the rendering instructions in the second buffer to a third buffer; The electronic device generates a frame interface of the first process based on the third buffer.

13. The method according to claim 12, characterized in that The third buffer is the second buffer, or the third buffer is the first buffer.

14. An interface generation method, characterized in that: Applied to an electronic device, where a first application is running on the electronic device, the method includes: The electronic device generates a first rendering tree, where the first rendering tree includes a drawing operation for generating a frame interface of the first application; The electronic device determines a first task amount, where the first task amount is the task amount of the first rendering tree, and the first task amount is used to measure the time or computational effort of converting the first rendering tree into rendering instructions. The first task amount is greater than a first threshold, and the rendering instructions are instructions in a rendering engine, an image processing library, or a GPU driver. If the first task amount is greater than the first threshold, the operating frequency of the CPU configured in the electronic device is changed from the first frequency to a second frequency, where the second frequency is higher than the first frequency; The electronic device generates a frame interface of the first application based on the first rendering tree; During the process of the electronic device generating a frame interface of the first application, the electronic device operates at the second frequency.

15. The method according to claim 14, wherein the electronic device generates a frame interface of the first application based on the first rendering tree, specifically comprising: The electronic device splits the first rendering tree into N sub-rendering trees, where N is an integer greater than 1; The electronic device converts the N sub-rendering trees into first rendering instructions in parallel; The electronic device generates a frame interface of the first application based on the first rendering instruction.

16. An interface generation method, characterized in that: Applied to an electronic device, where a first application is running on the electronic device, the method includes: The electronic device generates a first rendering tree, where the first rendering tree includes a drawing operation for generating a frame interface of the first application; The electronic device traverses different parts of the first rendering tree through multiple different threads to generate first rendering instructions, where the rendering instructions are instructions in a rendering engine, an image processing library, or a GPU driver; The electronic device generates a frame interface of the first application based on the first rendering instruction.

17. The method according to claim 16, characterized in that Before the electronic device traverses the first rendering tree in different orders through multiple different threads to generate a first instruction, the method further includes: The electronic device determines a first task amount, where the first task amount is the task amount of the first rendering tree, and the first task amount is used to measure the time consumption or computation amount of converting the first rendering tree into the first rendering instruction; The electronic device determines that the first task amount is greater than a first threshold.

18. The method according to claim 16 or 17, characterized in that The electronic device traverses the first rendering tree in different orders through multiple different threads to generate a first instruction, specifically including: The electronic device traverses the first part of the first rendering tree through a first thread, and saves the generated second rendering instruction in a first buffer; The electronic device traverses the second part of the first rendering tree through a second thread, and saves the generated third rendering instruction in a second buffer; The electronic device submits the rendering instructions in the first buffer and the rendering instructions in the second buffer to a third buffer to obtain the first rendering instructions.

19. The method according to claim 18, characterized in that The electronic device submitting the rendering instruction in the first buffer and the rendering instruction in the second buffer to the third buffer to obtain the first rendering instruction specifically includes: The rendering instructions in the first buffer include a second rendering instruction and a third rendering instruction, and the instructions in the second buffer include a fourth rendering instruction; The instructions in the third buffer are arranged in the following order: the second rendering instruction, the fourth rendering instruction, and the third rendering instruction.

20. An electronic device, characterized in that: The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, and is configured to store computer program codes, where the computer program codes include computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 19.

21. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method as described in any one of claims 1-19.

22. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 19.

23. A computer program product comprising instructions, characterized in that When the computer program product is run on an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 19.

Images