Image frame generation method, electronic equipment, chip system and storage medium

Abstract

The invention discloses an image frame generation method, electronic equipment, a chip system and a storage medium, and relates to the technical field of electronics. The method is applied to the electronic equipment, a unified rendering process of the electronic equipment comprises a main thread and a rendering thread, and the method comprises the following steps: the main thread generates a first preprocessing result of a first image frame after receiving a first vertical synchronization signal; the main thread synchronizes the first preprocessing result to the rendering thread; the rendering thread generates a first image frame according to the first preprocessing result; in the process that the rendering thread generates the first image frame according to the first preprocessing result, after the main thread receives the second vertical synchronization signal, a second preprocessing result of the second image frame is generated, and the second preprocessing result is used for generating the second image frame. According to the technical scheme provided by the embodiment, the generation rate of the image frame can be improved, the situation of frame dropping or lagging in the display process of the electronic equipment is reduced, and the user experience is improved.

Description

[0001] This application is a divisional application. The application number of the original application is 202410697901.4, the original application date is May 30, 2024, and the name of the invention is "Image frame generation method, electronic device, chip system and storage medium". The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of electronic technology, and in particular to a method for generating an image frame, an electronic device, a chip system and a storage medium. Background Art

[0003] With the development of computer technology, electronic devices support running one or more applications at the same time in the foreground, namely foreground applications. Since each foreground application needs to present its user interface on the screen, but the display area of ​​the electronic device screen is limited, the image frame actually displayed by the electronic device is the result of merging the user interfaces of each foreground application.

[0004] Currently, electronic devices can use unified rendering to merge the user interfaces of various foreground applications into one image frame for display. However, within a unified rendering cycle, the unified rendering process UniRender needs to perform many operations, such as updating rendering nodes, traversing rendering nodes to generate image frames, and sending image frames for display. Since the rendering process of UniRender is relatively complex and the rate of generating image frames is slow, electronic devices may drop frames or freeze during the display process, resulting in poor user experience. Summary of the invention

[0005] The present application provides an image frame generation method, an electronic device, a chip system and a storage medium, which are used to solve the problem in the prior art that the unified rendering process generates image frames at a slow rate, resulting in frame drops or freezes in the display process of the electronic device.

[0006] In order to achieve the above purpose, this application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a method for generating an image frame, which is applied to an electronic device, wherein a unified rendering process of the electronic device includes a main thread and a rendering thread, and the method includes: after the main thread receives a first vertical synchronization signal, generating a first preprocessing result of a first image frame; the main thread synchronizes the first preprocessing result to the rendering thread; the rendering thread generates the first image frame according to the first preprocessing result; in the process of the rendering thread generating the first image frame according to the first preprocessing result, after the main thread receives a second vertical synchronization signal, generating a second preprocessing result of a second image frame, and the second preprocessing result is used to generate a second image frame.

[0008] The first vertical synchronization signal and the second vertical synchronization signal are signals with the same content that are sent at a fixed frequency. The first image frame and the second image frame are two different image frames that are displayed sequentially on the screen.

[0009] In this embodiment, during the unified rendering process of the electronic device, the main thread processes the second image frame and the rendering thread processes the first image frame in parallel. Therefore, the method provided in this embodiment can increase the image frame generation rate, reduce frame drops or freezes during the display process of the electronic device, and improve the user experience.

[0010] In some embodiments, the first preprocessing result is a first drawing instruction list of a first image frame; wherein the first drawing instruction list is an incremental drawing instruction list or a full drawing instruction list, wherein the incremental drawing instruction list includes drawing instructions of newly added rendering nodes in a unified rendering tree of the first image frame, and / or, drawing instructions of rendering nodes whose information has been changed; the full drawing instruction list includes drawing instructions of all rendering nodes in a unified rendering tree of the first image frame, wherein the unified rendering tree is a rendering tree obtained by merging rendering trees of all applications performing unified rendering.

[0011] It should be noted that when the main thread generates a full drawing instruction list, it does not need to perform dirty area calculation, that is, it does not need to determine the drawing instructions of the newly added rendering nodes in the unified rendering tree and / or the drawing instructions of the rendering nodes with changed information, and the generation process is relatively simple. The incremental drawing instruction list contains fewer drawing operations, which can reduce the workload of subsequent rendering threads and improve the work efficiency of rendering threads.

[0012] In some embodiments, the first image frame includes a first part from a first application and a second part from a second application. After the main thread receives a first vertical synchronization signal, a first preprocessing result of the first image frame is generated, including: after the main thread receives the first vertical synchronization signal, the first sub-thread is used to determine a first drawing instruction sub-list corresponding to the first part; after the main thread receives the first vertical synchronization signal, the second sub-thread is used to determine a second drawing instruction sub-list corresponding to the second part; wherein the first sub-thread and the second sub-thread run in parallel; the main thread generates a first drawing instruction list for the first image frame according to the first drawing instruction sub-list and the second drawing instruction sub-list.

[0013] In this embodiment, the main thread uses multiple sub-threads to process different parts of the first image frame in parallel, which can improve the efficiency of generating the first drawing instruction list of the first image frame, thereby improving the efficiency of generating the image frame.

[0014] In some embodiments, a rendering thread generates a first image frame according to a first preprocessing result, including: the rendering thread uses a first rendering sub-thread to obtain a first drawing result according to a first part of a first drawing instruction list; the rendering thread uses a second rendering sub-thread to obtain a second drawing result according to a second part of the first drawing instruction list; wherein the first rendering sub-thread and the second rendering sub-thread run in parallel; the rendering thread generates a first image frame according to the first drawing result and the second drawing result.

[0015] For example, the rendering thread may split the first drawing instruction list into N sublists, such as the first part and the second part, according to the number of off-screen frame buffer objects (FBOs). Subsequently, N rendering sub-threads, such as the first rendering sub-thread and the second rendering sub-thread, are determined to process the N sub-lists respectively, and finally obtain the first image frame.

[0016] It should be noted that the first part and the second part of the first drawing instruction list may also correspond to the first drawing instruction sub-list and the second drawing instruction sub-list in the main thread respectively, which is not limited in this embodiment.

[0017] In this embodiment, the rendering thread uses multiple rendering sub-threads to process different parts of the first drawing instruction list in parallel, so that the drawing operations in the drawing instruction list can be quickly executed to generate the first image frame.

[0018] In some embodiments, the rendering thread generates the first image frame according to the first drawing result and the second drawing result, including: when the first drawing instruction list is an incremental drawing instruction list, the rendering thread generates the first image frame according to the first drawing result and the second drawing result, as well as the drawing result of the previous image frame. For example, the rendering thread uses the first drawing result and the second drawing result to replace the corresponding parts of the drawing result of the previous image frame, so as to obtain the first image frame.

[0019] In this embodiment, the incremental rendering method is used to generate the first image frame, which can reduce the amount of data processed by the main thread and the rendering thread and increase the rate of generating image frames.

[0020] In some embodiments, the first drawing instruction list is an incremental drawing instruction list, and after the main thread receives the first vertical synchronization signal, it generates a first preprocessing result of the first image frame, including: after the main thread receives the first vertical synchronization signal, it receives node change information of a rendering tree of at least one application; the main thread generates an incremental drawing instruction list for the first image frame according to the node change information.

[0021] For example, the main thread updates the unified rendering tree according to the node change information of the application, traverses the unified rendering tree to perform dirty area calculations, and copies the dirty area drawing instructions to the incremental drawing instruction list.

[0022] In this embodiment, in the process of generating an incremental drawing instruction list according to node update information, the main thread has a smaller amount of computation, and the incremental drawing instruction list generated by the main thread contains fewer drawing instructions, which can improve the efficiency of generating image frames.

[0023] In some embodiments, the first drawing instruction list is a full drawing instruction list, and after the main thread receives the first vertical synchronization signal, it generates a first preprocessing result of the first image frame, including: after the main thread receives the first vertical synchronization signal, it receives a rendering tree of at least one application; the main thread generates a full drawing instruction list for the first image frame according to the rendering tree of at least one application.

[0024] For example, after receiving the rendering trees of each application, the main thread merges the rendering trees of each application to obtain a unified rendering tree. Subsequently, each rendering node in the unified rendering tree is recorded as a drawing instruction to a cache area to form a full drawing instruction list.

[0025] In some embodiments, the first preprocessing result is a unified rendering tree of the first image frame, wherein the unified rendering tree is a rendering tree obtained by merging rendering trees of all applications performing unified rendering.

[0026] In some embodiments, the rendering thread generates the first image frame according to the first preprocessing result, including: after the rendering thread receives the first preprocessing result, the rendering thread generates the first image frame according to the first preprocessing result.

[0027] In this embodiment, the main thread can quickly start the rendering thread through the first preprocessing result (such as the first drawing instruction list), thereby increasing the generation rate of image frames.

[0028] In some embodiments, the rendering thread generates the first image frame according to the first preprocessing result, including: after the rendering thread receives the second vertical synchronization signal, the rendering thread generates the first image frame according to the first preprocessing result.

[0029] In this embodiment, both the main thread and the rendering thread are started by the VSync signal, which can reduce the impact of the main thread on the rendering thread and improve the independence of the main thread and the rendering thread.

[0030] In some embodiments, the main thread synchronizes the first preprocessing result to the rendering thread, including: the main thread sends the first preprocessing result to the rendering thread; or, the main thread sends address information of the first preprocessing result to the rendering thread, wherein the address information is used to access the first preprocessing result; or, the main thread transfers a pointer to the address of the first preprocessing result to the rendering thread.

[0031] It can be understood that compared with copying the first preprocessing result or the address information of the first preprocessing result and sending it to the rendering thread, the time consumption of the data transmission method of pointer transfer is only the time consumption of pointer assignment, and the process is more convenient and efficient.

[0032] In a second aspect, this embodiment provides an electronic device, the electronic device comprising: one or more processors and a memory. The memory is coupled to the one or more processors. The memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the method shown in the first aspect above.

[0033] In a third aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device. The chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method shown in the first aspect above.

[0034] In a fourth aspect, this embodiment provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method shown in the first aspect above is implemented.

[0035] In a fifth aspect, this embodiment provides a computer program product, which includes a computer program. When the computer program is executed by an electronic device, the electronic device performs the method shown in the first aspect above.

[0036] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic diagram of a user interface provided in an embodiment of the present application;

[0038] Figure 2 is a structural schematic diagram of an electronic device provided by this embodiment;

[0039] Figure 3 is a schematic diagram of the software architecture of an electronic device provided by an embodiment of the present application;

[0040] Figure 4is a schematic diagram of a unified rendering process provided by an embodiment of the present application;

[0041] Figure 5 is a schematic diagram of a unified rendering abnormal scene provided by an embodiment of the present application;

[0042] Figure 6 is a schematic diagram of a main thread and a rendering thread included in a unified rendering process provided by an embodiment of the present application;

[0043] Figure 7 is a schematic diagram of a main thread and a rendering thread included in a unified rendering process provided by another embodiment of the present application;

[0044] Figure 8 This is a schematic diagram of a process in which a main thread updates a unified rendering tree, provided by an embodiment of the present application;

[0045] Fig. 9 is a schematic diagram of node relationships in a unified rendering tree provided by an embodiment of the present application;

[0046] Fig.10 is a schematic diagram of a marked rendering node provided by an embodiment of the present application;

[0047] Fig.11 is a schematic diagram of a data structure of a drawing operation in a rendering thread provided by an embodiment of the present application;

[0048] Fig.12 It is a schematic diagram of instructions in the main thread and rendering thread provided in an embodiment of the present application;

[0049] Fig.13 It is a schematic diagram of a rendering thread converting a data format of a drawing operation provided by an embodiment of the present application;

[0050] Fig.14 is a schematic diagram of a data format of a rendering thread converting a drawing operation provided by another embodiment of the present application;

[0051] Fig.15 is a schematic flow chart of a method for generating an image frame provided by an embodiment of the present application;

[0052] Fig.16 It is a schematic diagram of the operation flow of the main thread and the rendering thread provided by an embodiment of the present application;

[0053] Fig.17 is a schematic flow chart of a method for generating an image frame provided by another embodiment of the present application;

[0054] Fig.18 It is a schematic diagram of the operation flow of the main thread and the rendering thread provided by another embodiment of the present application;

[0055] Fig.19 is a schematic diagram of a frame rate for generating image frames provided by an embodiment of the present application;

[0056] Fig. 20 is a schematic flow chart of a method for generating an image frame provided by yet another embodiment of the present application;

[0057] Fig.21 This is a schematic diagram of the operation flow of the main thread and the rendering thread provided by another embodiment of the present application;

[0058] Fig. 22 is a schematic diagram of a frame rate of generating image frames provided by another embodiment of the present application;

[0059] Fig.23 is a flowchart of a main thread generating a first drawing instruction list provided by an embodiment of the present application;

[0060] Fig.24 is a schematic diagram of a process in which a main thread generates a first drawing instruction list provided by an embodiment of the present application;

[0061] Fig.25 is a flowchart of a rendering thread generating a first image frame according to a first drawing instruction list provided by an embodiment of the present application;

[0062] Fig.26 is a schematic diagram of a process in which a rendering thread generates a first image frame according to a first drawing instruction list provided by an embodiment of the present application;

[0063] Fig. 27 It is a schematic diagram of the structure of the chip system provided in the embodiment of the present application. DETAILED DESCRIPTION

[0064] It should be understood that in the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a way to describe the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0065] In this embodiment, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0066] To facilitate understanding, the relevant terms and concepts involved in the embodiments of the present application are first introduced below.

[0067] (1) User Interface

[0068] The user interface (UI) is an interface for interaction and information exchange between applications and operating systems and users, and is used to achieve the conversion between the internal form of information and the form acceptable to users. In other words, the user interface is the interface source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the electronic device and finally presented as content that the user can recognize. In short, the existence of the user interface makes the interaction between users and electronic devices more convenient.

[0069] (2) Graphical User Interface

[0070] Graphical user interface (GUI) is a commonly used form of user interface, that is, a user interface related to computer operation that is displayed in a graphical manner. GUI can display visual interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. on the screen of an electronic device.

[0071] (3) Display interface

[0072] In this embodiment, the display interface may also be referred to as an image frame or a screen interface, specifically referring to the content presented on the screen of the electronic device. When the electronic device has multiple foreground applications, the display interface is the result of synthesizing the user interfaces of the multiple foreground applications.

[0073] (4) Vertical synchronization signal

[0074] The vertical synchronization (VSync) signal is a concept in computer graphics that is used to control the output and display of image frames. Its main function is to eliminate screen tearing, that is, when the refresh rate of computer graphics is not synchronized with that of the display, the two parts of the image above and below the screen will refresh at different times. Specifically, when the display has finished displaying an image frame and is about to display the next image frame, the display controller will send a VSync signal. The frequency of the VSync signal is related to the refresh rate of the electronic device. For example, the frequency of the VSync signal is the same as the refresh rate of the electronic device screen.

[0075] VSync signals usually include software VSync signals and hardware VSync signals, and software VSync signals and hardware VSync signals usually maintain periodic synchronization. For example, software VSync signals include VSync-APP and VSync-RS. Among them, VSync-APP is used to trigger the rendering process of the application. VSync-RS is used to trigger the unified rendering process of the rendering service (RS). Hardware VSync signals include VSync-HW, which is used to trigger the display refresh process of the display.

[0076] (5) Rendering Tree

[0077] A render tree is a data structure used to generate the user interface of an application. It records part or all of the information for generating a frame of the application's user interface. When an application has multiple windows or is displayed in different display areas (such as different screens), multiple render trees need to be generated independently.

[0078] Each render tree can include multiple render nodes, each of which includes render attributes and a draw operation struct (DrawOP) list. The render attributes include the position, size, transparency and other attribute information of the view to be rendered by the render node on the surface. The draw operation list records the draw operations and the drawing order of each draw operation. The draw operation is used to determine the content of the view to be rendered by the render node, such as lines, rectangles, circles, etc.

[0079] The technical solution provided in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0080] With the development of computer technology, electronic devices support running one or more applications (application, App) in the foreground at the same time. For the convenience of description, the application running in the foreground is referred to as the foreground application in this embodiment. Since each foreground application needs to present its user interface on the screen, but the display area of ​​the electronic device screen is limited, the content (i.e., image frame) actually displayed by the electronic device is the display effect of the user interface of each foreground application synthesized according to specific rules. It should be noted that the specific rules can be determined according to the system configuration of the electronic device or the user operation, and this embodiment does not limit this.

[0081] For example Figure 1As shown, the foreground applications of the electronic device include a status bar, an information application and a video application. In the image frame of the electronic device, the user interface of the status bar is located at the top of the image frame; the user interfaces of the information application and the video application are displayed in split screen according to user instructions, and occupy the upper and lower halves of the remaining image frame respectively.

[0082] When the foreground application is running, its user interface usually needs to change constantly, including changing the displayed content and the display position of the displayed content according to the configuration of the application itself or according to user operations. Figure 1 For example, Figure 1 The status bar in the app needs to refresh the user interface based on the cellular signal type, wireless fidelity (Wi-Fi) connection status, battery level, time, etc. Information applications need to refresh the user interface based on user input operations, received information, etc. Video applications need to refresh the user interface based on the video frame to be played or the user's operation.

[0083] Since the image frame of the electronic device is the display effect of the user interface of each foreground application, when the user interface of the foreground application changes, the image frame of the electronic device also needs to change accordingly. Therefore, before refreshing the image frame each time, the electronic device needs to generate a new image frame according to the running status of all foreground applications.

[0084] The following is an exemplary description of the process of an electronic device generating an image frame.

[0085] In this embodiment, the electronic device includes a terminal device, which may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device may be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiments of the present application do not limit the specific technology and specific device form adopted by the terminal device.

[0086] Figure 2 2 is a schematic diagram of the structure of an electronic device provided in this embodiment. The electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, an air pressure sensor 280C, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 280F, a proximity light sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, etc.

[0087] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0088] For example, when the electronic device 200 is a mobile phone or a tablet computer, it may include all the components shown in the figure, or may include only some of the components shown in the figure.

[0089] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0090] The controller may be the nerve center and command center of the electronic device 200. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0091] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0092] The charging management module 240 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 240 may receive charging input from a wired charger through the USB interface 230. In some wireless charging embodiments, the charging management module 240 may receive wireless charging input through a wireless charging coil of the electronic device 200. While the charging management module 240 is charging the battery 242, it may also power the electronic device through the power management module 241.

[0093] The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, and supplies power to the processor 210, the internal memory 221, the external memory, the display screen 294, the camera 293, and the wireless communication module 260. The power management module 241 can also be used to monitor parameters such as battery capacity, battery cycle number, and battery health status (leakage, impedance).

[0094] In some other embodiments, the power management module 241 may also be disposed in the processor 210. In some other embodiments, the power management module 241 and the charging management module 240 may also be disposed in the same device.

[0095] The wireless communication function of the electronic device 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor and the baseband processor.

[0096] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0097] The mobile communication module 250 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 200. The mobile communication module 250 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0098] In some embodiments, at least some functional modules of the mobile communication module 250 may be disposed in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 250 and at least some modules of the processor 210 may be disposed in the same device.

[0099] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 270A, a receiver 270B, etc.), or displays an image or video through a display screen 294. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 210 and be set in the same device as the mobile communication module 250 or other functional modules.

[0100] The wireless communication module 260 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 200. The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 210. The wireless communication module 260 can also receive the signal to be sent from the processor 210, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0101] In some embodiments, the antenna 1 of the electronic device 200 is coupled to the mobile communication module 250, and the antenna 2 is coupled to the wireless communication module 260, so that the electronic device 200 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. GNSS may include the global positioning system (GPS), the global navigation satellite system (GLONASS), the Beidou navigation satellite system (BDS), the quasi-zenith satellite system (QZSS) and / or the satellite based augmentation system (SBAS).

[0102] The electronic device 200 implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0103] The display screen 294 is used to display images, videos, etc. For example, the image frame in the embodiment of the present application. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 200 may include 1 or N display screens 294, where N is a positive integer greater than 1.

[0104] The electronic device 200 can realize the shooting function through ISP, camera 293, video codec, GPU, display screen 294 and application processor.

[0105] The ISP is used to process the data fed back by the camera 293. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 293.

[0106] The camera 293 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 200 may include 1 or N cameras 293, where N is a positive integer greater than 1.

[0107] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 200 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0108] Video codecs are used to compress or decompress digital videos. The electronic device 200 may support one or more video codecs. Thus, the electronic device 200 may play or record videos in a variety of coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0109] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the electronic device 200 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0110] In an embodiment of the present application, the NPU or other processors may be used to perform operations such as analyzing and processing images in a video stored in the electronic device 200 .

[0111] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0112] The internal memory 221 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 210 executes various functional applications and data processing of the electronic device 200 by running the instructions stored in the internal memory 221. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area may store data created during the use of the electronic device 200 (such as audio data, a phone book, etc.).

[0113] In addition, the internal memory 221 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0114] The electronic device 200 can implement audio functions through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone interface 270D, and the application processor.

[0115] The audio module 270 is used to convert digital audio signals into analog audio signals for output, and is also used to convert analog audio inputs into digital audio signals. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be arranged in the processor 210, or some functional modules of the audio module 270 can be arranged in the processor 210.

[0116] The speaker 270A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 200 can listen to music or listen to hands-free calls through the speaker 270A. For example, the speaker can play the comparison analysis results provided in the embodiment of the present application.

[0117] The receiver 270B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 200 receives a call or voice message, the voice can be received by placing the receiver 270B close to the human ear.

[0118] Microphone 270C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 270C to input the sound signal into the microphone 270C. The electronic device 200 can be provided with at least one microphone 270C. In other embodiments, the electronic device 200 can be provided with two microphones 270C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 200 can also be provided with three, four or more microphones 270C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0119] The earphone interface 270D is used to connect a wired earphone and can be a USB interface 230 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0120] The key 290 includes a power key, a volume key, etc. The key 290 can be a mechanical key or a touch key. The electronic device 200 can receive key input and generate key signal input related to user settings and function control of the electronic device 200.

[0121] Motor 291 can generate vibration prompts. Motor 291 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 294, motor 291 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0122] Indicator 292 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0123] The SIM card interface 295 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 200 by inserting it into the SIM card interface 295 or pulling it out from the SIM card interface 295. The electronic device 200 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 295 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 295 can also be compatible with different types of SIM cards. The SIM card interface 295 can also be compatible with external memory cards. The electronic device 200 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 200 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 200 and cannot be separated from the electronic device 200.

[0124] Figure 3 Schematic diagram of the software architecture of an electronic device provided by an embodiment of the present application. Figure 3 As shown, the layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the software architecture of the electronic device is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the system library layer, and the kernel layer.

[0125] The application layer can include a series of application packages. Figure 3 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, music, desktop, gallery, call, navigation, Bluetooth, and video, etc.

[0126] The application process of the application includes at least a main thread, which may also be called a UI thread, and the main thread is used to generate a rendering tree for the application according to the running status of the application. Exemplarily, if the application supports off-screen rendering, the application process also includes a rendering thread, which is used to generate a user interface according to the rendering tree, and the user interface usually exists in the form of a bitmap inside the computer.

[0127] In some embodiments, if a foreground application uses unified rendering (UniRender), then when each VSync signal (such as VSync-App) arrives, the main thread of the foreground application generates a rendering tree according to the running status of the foreground application, and sends the rendering tree to the application framework layer for further processing.

[0128] In other embodiments, if a foreground application uses off-screen rendering, then when each VSync signal arrives, the main thread of the foreground application first generates a rendering tree, and then the rendering thread of the foreground application draws the user interface according to the rendering tree, and sends the generated user interface to the application framework layer for further processing.

[0129] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. For example Figure 3 As shown, the application framework layer may include window management services, display management services, content providers, view systems, phone managers, resource managers, notification managers, and the like.

[0130] The window management service is responsible for starting, adding, and deleting windows. It can determine the application displayed on the window and the creation, destruction, and property changes of the application's layer, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0131] The display management service can obtain the number and size of display areas, and is responsible for starting, adding, and deleting display areas.

[0132] Content providers are used to store and retrieve data and make it accessible to applications. This data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0133] The phone manager is used to provide communication functions for electronic devices, such as the management of call status (including answering, hanging up, etc.).

[0134] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, etc.

[0135] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. In addition, the notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialogue interface. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0136] The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. An image frame can be composed of one or more views. For example, a view that includes a text notification icon can include a view that displays text and a view that displays an image.

[0137] In this implementation, the view system also includes a functional module for unified rendering: RenderService (RS). RenderServic can receive the rendering trees of one or more foreground applications by running a unified rendering process (UniRender), and merge the one or more rendering trees into a unified rendering tree. In addition, UniRender can also synchronize layer information, such as layer creation, destruction, property changes, etc., through the window management service, and can also synchronize display area information, such as screen size, from the display management service.

[0138] It should be noted that in this embodiment, the unified rendering process (UniRender) and the application process are independent processes. Among them, the application and the unified rendering process complete data interaction through inter-process communication (IPC). For the specific content of the unified rendering process, please refer to the patent application with application number 2021114105136 and the invention name “Interface Generation Method and Electronic Device” and the patent application with application number 202111410643X and the invention name “Interface Generation Method and Electronic Device”, which will not be repeated here.

[0139] Optionally, in some embodiments of the present application, the view system further includes a surface synthesizer (SurfaceFlinger). On an electronic device configured with a whitelist, if the foreground application does not belong to the whitelist, after the main thread of the foreground application generates a rendering tree, the rendering thread of the foreground application generates a user interface, which is then handed over to SurfaceFlinger for layer synthesis to generate an image frame for display.

[0140] Optionally, in some embodiments of the present application, when the foreground application includes both applications in the whitelist and applications that are not in the whitelist, UniRender is responsible for generating the user interface for the applications in the whitelist. After UniRender generates the user interface, it passes the user interface to SurfaceFlinger. SurfaceFlinger then performs layer synthesis of the user interface and the user interfaces of other applications that are not in the whitelist to generate an image frame for display.

[0141] The system library can include multiple functional modules, such as surface manager, media libraries, graphics processing library, etc. The details are as follows.

[0142] The surface manager is used to manage the display subsystem and is capable of fusing two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0143] The media library supports playback and recording of many common audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video coding formats, such as moving picture expert group 4 (MPEG4), H.264, moving picture experts group audiolayer III (MP3), advanced audio coding (AAC), adaptive multi-rate (AMR), joint photographic experts group (JPEG), lossless bitmap graphics format (portable network graphics, PNG), etc.

[0144] The graphics processing library includes a 3D graphics processing library, a 2D graphics engine, etc. Among them, the 3D graphics processing library is used to implement 3D graphics drawing, image rendering, layer synthesis and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0145] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, sensor driver, audio driver and virtual card driver.

[0146] Based on the hardware and software architecture of the electronic device shown in the above embodiment, the electronic device can generate image frames by off-screen rendering or unified rendering. However, compared with off-screen rendering, unified rendering can reduce the number of rendering times, reduce the power consumption of the electronic device, and improve the user experience. Therefore, currently, unified rendering is generally used to generate image frames.

[0147] In some embodiments, the process of generating an image frame by a unified rendering method in an electronic device is as follows: Figure 4 shown.

[0148] See also Figure 4 As shown, taking the foreground application including application 1 to application 4 as an example, the process of the electronic device generating an image frame by unified rendering specifically includes the following processes (1) to (4).

[0149] (1) The foreground application sends the render tree to RenderService.

[0150] When each VSync signal arrives, each foreground application uses its main thread to generate a render tree based on its running status, and sends the render tree to RenderService across processes. For example, Application 1 to Application 4 use their respective main threads to generate Render Tree 1 to Render Tree 4, and send them to RenderService.

[0151] Exemplarily, the process of generating a rendering tree by the main thread of the foreground application includes: invalidating the view hierarchy of the foreground application. The view hierarchy is used to represent the parent-child relationship between views in the user interface. Subsequently, the main thread traverses the views of the foreground application, re-layouts each view, determines the attribute information and drawing operation DrawOP of each view, and uses the attribute information and drawing operation DrawOP of each view as a rendering node to form a rendering tree.

[0152] It should be noted that view is the basic element that constitutes the user interface of the application. A control on the user interface can correspond to one or more views. The user interface that the application needs to display is composed of multiple nested views. Different views have a parent-child relationship, so the parent-child relationship between the rendering nodes of the rendering tree generated by the main thread traversing the view is the same as the parent-child relationship of the view. That is, the parent-child relationship between views determines the nesting relationship between different rendering nodes, thereby ensuring that the rendering thread (such as the rendering thread of the application, or the rendering thread of the unified rendering process) can correctly render the correct user interface when generating the interface based on the rendering tree.

[0153] (2) RenderService merges the various render trees into a unified render tree.

[0154] RenderService uses UniRender to receive the rendering trees sent by each foreground application, and obtains window control information and display area information from the window management service and display management service, and then combines the rendering trees of the foreground applications in the same display area into a unified rendering tree based on the window control information and display area information. Among them, the window control information includes the Z-order of the window, and the display area information includes the identification information of the display area.

[0155] In some embodiments, UniRender may generate a new root rendering node, and use the original root rendering root nodes of each rendering tree as child nodes of the same level of the new rendering root node, thereby generating a unified rendering tree. Figure 4 As shown, UniRender generates a new root rendering node, and uses the original root rendering nodes of rendering trees 1 to 4 as child nodes of the same level of the new rendering root node, thereby generating a unified rendering tree.

[0156] Optionally, in the process of generating the unified rendering tree, UniRender may first determine the upper and lower masking relationships between the layers of the foreground application according to the Z-order of the layers corresponding to the foreground application, and then delete the rendering nodes corresponding to the completely masked views in the process of generating the unified rendering tree.

[0157] (3) RenderService generates image frames based on the unified rendering tree.

[0158] After generating the unified rendering tree, UniRender of RenderService calls the layer processing library, traverses each rendering node in turn, performs the drawing operations of each rendering node in the unified rendering tree in the unified cache, and generates an image frame. It should be noted that the display information of the image frame is carried on the bitmap, and the display of the electronic device can present the image frame on the screen based on the bitmap.

[0159] (4) RenderService sends the image frame to the display subsystem.

[0160] UniRender of RenderService stores the bitmap of the image frame into a frame buffer, and the display subsystem (DSS) obtains the bitmap from the frame buffer and displays the image frame on the screen according to the bitmap. Exemplarily, the frame buffer may be an on-screen buffer.

[0161] Through the above processes (1) to (4), the electronic device can combine the user interfaces of various foreground applications into image frames for display through unified rendering. However, in the above unified rendering process, UniRender needs to perform more operations within a rendering cycle, and the rendering process is relatively complex, resulting in low rendering efficiency.

[0162] For example Figure 5 As shown in the figure, when the refresh rate of the electronic device is 120fps (frames per second), the time interval between two adjacent VSync signals is only 8.3ms, that is, a rendering cycle is only 8.3ms. In a rendering cycle, UniRender needs to perform operations such as generating a unified rendering tree, generating image frames according to the unified rendering tree, and sending the image frames to the display to complete the rendering of one image frame.

[0163] It is understandable that due to the short unified rendering cycle and the complex rendering process of UniRender, UniRender may not be able to complete the rendering work within a unified rendering cycle, resulting in the failure to send the bitmap of the image frame to the display in time, which in turn causes the electronic device to drop frames or jam during the display process, resulting in a poor user experience.

[0164] To this end, the present application provides a method for generating an image frame, which is used to increase the rate of generating image frames in a unified rendering mode, thereby improving the situation of frame dropping or freezing during the display process of the electronic device and improving the user experience.

[0165] In this embodiment, see Figure 6 As shown, the unified rendering process UniRender includes a main thread and a rendering thread, which share the tasks that UniRender needs to perform. The main thread preprocesses information in the process of generating image frames and synchronizes the preprocessing results to the rendering thread. The rendering thread generates image frames based on the preprocessing results of the main thread.

[0166] It is worth noting that the main thread supports serial processing of different image frames, and the rendering thread also supports serial processing of different image frames. In addition, the main thread and the rendering thread can also run in parallel when processing different image frames.

[0167] In some embodiments of the present application, see Figure 7 As shown, the tasks performed by the main thread and rendering thread of UniRender specifically include the following contents (i) and (ii).

[0168] (I) UniRender's main thread

[0169] In this embodiment, the main thread of UniRender (hereinafter referred to as the main thread) is used to update the unified rendering tree according to the application message, and generate a drawing instruction list of the image frame according to the updated unified rendering tree. Specifically, it includes the following contents (1) to (2).

[0170] (1) Update the unified rendering tree according to application messages.

[0171] An application message refers to a message sent by at least one foreground application, which may be a unified rendering tree of the foreground application, or node change information of at least one rendering node in the unified rendering tree of the foreground application. Exemplarily, the node change information may be updated attribute information and drawing operations of the rendering node, or animation step information of the rendering node, etc. The animation step information includes movement information of the rendering node between multiple user interfaces, such as the starting position, step rate, and end position, etc.

[0172] In some embodiments, the foreground application generates a rendering tree upon receiving a VSync signal and sends the rendering tree to the main thread. Based on this, the main thread may update the unified rendering tree according to the application message by receiving and merging the rendering trees sent by each foreground application to obtain a new unified rendering tree.

[0173] In other embodiments, since in most cases, the change of the user interface of two adjacent frames of the foreground application is usually not large, or even unchanged, the foreground application can determine the changed rendering node (i.e., the changed rendering node) after receiving the VSync signal, and only send the node change information of the changed rendering node to the main thread to reduce the amount of data interacted across processes and save power consumption of electronic devices. Based on this, the process of the main thread updating the unified rendering tree according to the application message can be: receiving the node change information sent by each foreground application, and updating the unified rendering tree according to the node change information. For example, based on the unified rendering tree of the previous frame of the user interface, add a rendering node, update the rendering properties and drawing operations of the rendering node, or delete the rendering node, etc.

[0174] In an example, the unified rendering tree in the main thread is as follows Figure 8 As shown in (a) in the figure, the unified rendering tree includes a root rendering node 0, and the child nodes of the root rendering node 0 include rendering node 1, rendering node 5, and rendering node 7. The child nodes of rendering node 1 include rendering node 2 and rendering node 4. The child nodes of rendering node 2 include rendering node 3. The child nodes of rendering node 5 include rendering node 6. The child nodes of rendering node 7 include rendering node 8. The child nodes of rendering node 8 include rendering node 9. Assume that the main thread receives node change information about rendering node 2, rendering node 8, and rendering node 9 from the application, wherein the node change information of rendering node 2 is: the background color is changed to color A, and the node change information of rendering node 8 is: the node is deleted. The node change information of rendering node 9 is: the image scale is changed to scale B. Then, the unified rendering tree updated by the main thread is as follows: Figure 8 As shown in (b) in .

[0175] It should be noted that in the rendering tree, the parent node includes the child nodes, or the drawing operation of the parent node includes the drawing operation of the child node. Therefore, when the main thread updates the target rendering node, it needs to traverse the upper nodes of the target rendering node in descending order from the root node of the unified rendering tree, and then find the target rendering node, and then update the node information of the target rendering node.

[0176] by Figure 8 For example, the unified rendering tree shown in (a) in FIG. 1 is used as an example. The node relationship between the unified rendering tree can be seen in FIG. Fig. 9 As shown. Specifically, it includes the following contents: the drawing operation of the root rendering node 0 (i.e., RenderNodeDrawOp No. 0) includes the drawing operations of rendering nodes 1, 5 and 7 (i.e., RenderNodeDrawOp No. 1, 5 and 7). The drawing operation of rendering node 1 (i.e., RenderNodeDrawOp No. 1) includes the drawing operations of rendering nodes 2 and 4 (i.e., RenderNodeDrawOp No. 2 and 4). The drawing operation of rendering node 2 (i.e., RenderNodeDrawOp No. 2) includes the drawing operation of rendering node 3 (i.e., RenderNodeDrawOp No. 3). The drawing operation of rendering node 5 (i.e., RenderNodeDrawOp No. 5) includes the drawing operation of rendering node 6 (i.e., RenderNodeDrawOp No. 6). The drawing operation of rendering node 7 (i.e., RenderNodeDrawOp No. 7) includes the drawing operation of rendering node 8 (i.e., RenderNodeDrawOp No. 8). The drawing operation of rendering node 8 (i.e., RenderNodeDrawOp No. 8) includes the drawing operation of rendering node 9 (i.e., RenderNodeDrawOp No. 9).

[0177] Based on this, when the main thread updates Render Node 2, it needs to find Render Node 1 from Render Node 0, and then find Render Node 2 in Render Node 1 and update its node information. Similarly, when the main thread deletes Render Node 8, it needs to find Render Node 7 from Render Node 0, and then delete Render Node 8 from Render Node 7.

[0178] Optionally, when updating the unified rendering tree, the main thread can also mark the changed rendering node and all its parent rendering nodes for subsequent processing. Figure 8 Taking the updated unified rendering tree shown in (b) of FIG. 1 as an example, the main thread can mark rendering node 2, its upper-level rendering node 1 and rendering node 0, and rendering node 7. The main thread does not mark the deleted rendering nodes. For example, rendering node 8 and rendering node 9 are not marked.

[0179] It should be noted that in a rendering tree (including a unified rendering tree), the calling interface of a child node is in the parent node. Therefore, in order to find a changed rendering node in the process of traversing rendering nodes, the main thread needs to mark all higher-level rendering nodes of the changed rendering node.

[0180] (2) Generate a drawing instruction list based on the updated unified rendering tree.

[0181] In some embodiments, the main thread traverses each rendering node to copy each rendering node as a drawing operation to the drawing instruction list (DrawCmdList) of the unified rendering tree. That is, the main thread traverses the unified rendering tree in full to generate a full drawing instruction list of the unified rendering tree. It can be understood that the full traversal process takes a long time, and there are many drawing operations in the full drawing instruction list, resulting in a long drawing process of subsequent image frames.

[0182] In other embodiments, the main thread traverses the updated unified rendering tree through the dirty region stack technology, and completes the dirty region calculation and the copying of the dirty region drawing operation through one traversal to generate an incremental drawing instruction list.

[0183] The main thread performs dirty area calculation, which means that the main thread determines the rendering nodes in the unified rendering tree that need to be redrawn due to changes. Figure 8 Taking the updated unified rendering tree shown in (b) as an example, the main thread determines through dirty area calculation that its dirty area includes rendering node 2 and rendering node 7. Among them, rendering node 7 becomes a dirty area because it loses its child nodes (i.e., rendering node 8 and rendering node 9), which causes rendering node 7 to need to be redrawn.

[0184] In addition, copying the dirty area drawing operation includes copying the drawing operation corresponding to the dirty area to the drawing instruction list. Figure 7 Taking the update process shown in the figure as an example, since only the background of Rendering Node 2 needs to be redrawn, the main thread copies the drawing operation list (background DrawOp List) of the updated background of Rendering Node 2 to the drawing instruction list of the main thread. In addition, since Rendering Node 7 loses its child nodes and is exposed on the image frame again, the entire Rendering Node 7 is copied as a drawing operation to the drawing instruction list of the main thread.

[0185] It should be noted that, in general, the unified rendering process directly traverses the rendering nodes of the unified rendering tree to draw image frames, but in this embodiment, in order to realize the parallel operation of the main thread and the rendering thread, it is necessary to copy the rendering node to be redrawn, for example, copy the rendering node as a whole to the drawing instruction list. It can be understood that after the rendering node is copied as a whole to the drawing instruction list, the operation of the subsequent rendering thread will no longer depend on the unified rendering tree, and the rendering thread can run normally even if the rendering nodes in the unified rendering tree are modified.

[0186] Optionally, in order to improve the efficiency of the main thread in generating the drawing instruction list, the main thread may only traverse the marked rendering nodes to generate the drawing instruction list. Figure 8 The updated unified rendering tree shown in (b) of FIG. 1 has marked rendering nodes such as Fig.10 As shown, it includes rendering node 0, rendering node 1, rendering node 2 and rendering node 7. Therefore, the main thread only needs to Fig.10 The rendering nodes in are traversed to generate a drawing instruction list.

[0187] It should be noted that in the above process, the marked rendering nodes are used to determine the rendering nodes with incremental changes, such as newly added rendering nodes, rendering nodes with changed attribute information or drawing operations, and rendering nodes with changed animation step information, but do not include deleted rendering nodes. Therefore, the above process of traversing marked rendering nodes can also be called an incremental traversal process. Similarly, the above list that only includes dirty area drawing operations can also be called an incremental drawing instruction list.

[0188] It can be understood that the incremental traversal process can improve the efficiency of generating a drawing instruction list, reduce the occupation of CPU resources, and obtain a certain load benefit, such as a 5% load benefit.

[0189] (2) UniRender's rendering thread

[0190] In this embodiment, the rendering thread of UniRender is mainly used to draw image frames according to the drawing instruction list from the main thread and send the image frames to the display. Specifically, it includes the following contents (3) to (4).

[0191] (3) Draw the image frame according to the drawing instruction list from the main thread.

[0192] First, after the main thread generates the drawing instruction list of the unified rendering tree, it needs to synchronize the drawing instruction list to the rendering thread. For example, the main thread copies the drawing instruction list and sends it to the rendering thread; or, the main thread sends the address information of the drawing instruction list to the rendering thread; or, the main thread transfers the pointer of the address of the drawing instruction list to the rendering thread so that the rendering thread calls the drawing instruction list. Compared with copying the drawing instruction list or the address information of the drawing instruction list and sending it to the rendering thread, the data transfer method of pointer transfer only takes the time of pointer assignment, and the process is more convenient and efficient.

[0193] It should be noted that the data structures of drawing operations are different in UniRender's main thread and rendering thread. For example Fig.11 As shown in the figure, the data structure operated by the main thread is RenderNode, i.e., the rendering node. The data structure operated by the rendering thread is RenderNodeDrawble, i.e., the rendering node that can be drawn. In other words, RenderNodeDrawble holds RenderNode information. In other words, RenderNodeDrawble strongly references RenderNode.

[0194] See also Fig.12 As shown, in the RenderNode of the main thread, the data structure of the drawing operation is DrawOp List, which includes the drawing operations required to draw the view, such as operations of drawing circles, lines, and directions, but does not include attribute information such as the size, color, and transparency of the view. Exemplarily, the DrawOp List includes multiple types of drawing operations such as geometry DrawOp List, background DrawOp List, content DrawOp List, child node RenderNodeDrawOp List, foreground DrawOp List, and overlay DrawOp List.

[0195] See also Fig.12As shown, in the RenderNodeDrawble of the rendering thread, the data structure of the drawing operation is drawing DrawOp List. Compared with DrawOp List, drawing DrawOp List can be understood as adding view attribute information such as size, color, transparency, etc. on the basis of the view's drawing operation. Exemplarily, drawingDrawOp List includes multiple types such as geometry drawing DrawOp List, background drawing DrawOp List, content drawingDrawOp List, child node drawing RenderNodeDrawOp List, foreground drawing DrawOp List, and Overlay drawing DrawOp List.

[0196] Therefore, after receiving the drawing instruction list, the rendering thread needs to prepare drawing elements for the drawing operations DrawOp List of each rendering node (ie, prepare the attribute information of the rendering node) according to the node information of each rendering node in the drawing instruction list, and generate the corresponding drawing DrawOp List.

[0197] For example Fig.13 As shown, after the main thread sends the entire rendering node 2 as a drawing operation to the rendering thread, the rendering thread converts the background DrawOp List in the rendering node 2 into a background drawingDrawOp List in combination with the attribute information of the rendering node 2.

[0198] For example Fig.14 As shown, after the main thread sends the entire rendering node 7 as a drawing operation to the rendering thread, the rendering thread converts the sub-node RenderNodeDrawOp List in the rendering node 7 into the sub-node drawing RenderNodeDrawOp List according to the attribute information of the rendering node 7.

[0199] Subsequently, the rendering thread traverses each drawing operation in the drawing instruction list to generate a drawing result. For ease of description, this embodiment refers to the result obtained by drawing according to the incremental drawing instruction list as an incremental drawing result or a dirty area drawing result. For example, the rendering thread traverses the drawing operation of the rendering node 2 to change the background color of its view. Or, the rendering thread traverses the drawing operation of the rendering node 7 to regenerate the drawing result of its view.

[0200] Finally, the rendering thread generates the current image frame based on the drawing result of the previous image frame (called the previous frame drawing result) and the incremental drawing result obtained this time. Exemplarily, since the current drawing result is the drawing result of the dirty area, the rendering thread can use the current drawing result to replace the corresponding part of the previous frame drawing result to obtain the drawing result of the current image frame. For example, the drawing result of the current rendering node 2 is used to replace the drawing result of the rendering node 2 in the previous frame drawing result; and the drawing result of the current rendering node 7 is used to replace the drawing result of the rendering node 7 in the previous frame drawing result to obtain the drawing result of the current image frame.

[0201] (4) The rendering thread sends the image frame to the display.

[0202] The rendering thread stores the bitmap of the image frame into the frame buffer, and the DSS obtains the bitmap of the image frame from the frame buffer and displays the image frame on the screen according to the bitmap.

[0203] It should be noted that this embodiment does not limit the specific tasks performed by the main thread and rendering thread of UniRender. In other words, this embodiment does not limit the preprocessing results sent by the main thread of UniRender to the rendering thread. Therefore, the main thread and rendering thread of UniRender can also be configured to perform the following operations.

[0204] In some embodiments, the main thread of UniRender is used to perform the above process (1), that is, updating the unified rendering tree according to the application message, and sending the updated unified rendering tree to the rendering thread. The rendering thread is used to perform the above processes (2) to (4), that is, generating a drawing instruction list (such as a full drawing instruction list and an incremental drawing instruction list) according to the updated unified rendering tree, drawing image frames according to the drawing instruction list, and sending the image frames for display.

[0205] In some other embodiments, the main thread of UniRender is used to perform the above process (1), that is, updating the unified rendering tree according to the application message, and sending the updated unified rendering tree to the rendering thread. The rendering thread directly traverses the updated unified rendering tree to generate image frames, and sends the image frames for display.

[0206] After introducing the main thread and rendering thread of UniRender, the following is an exemplary description of the process in which an electronic device runs the main thread and the rendering thread in parallel to generate an image frame.

[0207] Fig.15 1 is a schematic flow chart of a method for generating an image frame provided by an embodiment of the present application. Taking UniRender generating a first image frame and a second image frame as an example, the method specifically includes the following steps S1501 to S1504.

[0208] S1501, after receiving a first VSync signal, the main thread generates a first preprocessing result of a first image frame.

[0209] In some embodiments, the first preprocessing result is a unified rendering tree of the first image frame.

[0210] In other embodiments, the first preprocessing result is a first drawing instruction list for the first image frame. The first drawing instruction list may be a full drawing instruction list or an incremental drawing instruction list, which is not limited in this embodiment. The incremental drawing instruction list includes drawing instructions for newly added rendering nodes in the unified rendering tree of the first image frame, and / or drawing instructions for rendering nodes whose information has been changed. The full drawing instruction list includes drawing instructions for all rendering nodes in the unified rendering tree of the first image frame, wherein the unified rendering tree is a rendering tree obtained by merging rendering trees of all applications performing unified rendering.

[0211] S1502: The main thread synchronizes the first preprocessing result to the rendering thread.

[0212] In some embodiments, the main thread sends the first pre-processing result to the rendering thread.

[0213] In some other embodiments, the main thread sends address information of the first preprocessing result to the rendering thread, and the address information is used to access the first preprocessing result.

[0214] In some other embodiments, the main thread transfers the pointer of the first preprocessing result from the main thread to the rendering thread. It can be understood that the time consumption of data transmission through pointer transfer is only the time consumption of pointer assignment, and the process is more convenient and efficient.

[0215] S1503: The rendering thread generates a first image frame according to the first preprocessing result.

[0216] It should be noted that for the same image frame (e.g., the first image frame), the main thread needs to generate the preprocessing result of the image frame first, and then the rendering thread can generate the image frame according to the preprocessing result. Therefore, the main thread and the rendering thread cannot run in parallel when generating the same image frame.

[0217] S1504, during the running of the rendering thread, after the main thread receives the second VSync signal, it generates a second preprocessing result of the second image frame, and the second preprocessing result is used to generate the second image frame.

[0218] In the above process, since the main thread will serially process the second preprocessing result of the second image frame after generating the first preprocessing result of the first image frame, the second preprocessing result may affect or cover the first preprocessing result. Therefore, in order to ensure the parallel operation of the main thread and the rendering thread, the main thread needs to send the first preprocessing result to the rendering thread to avoid the main thread changing the first preprocessing result during the processing of the second image frame.

[0219] For example, the main thread needs to send the first drawing instruction list of the first image frame to the rendering thread before it can start generating the second drawing instruction list of the second image frame, so as to avoid the first drawing instruction list being modified by the main thread while the rendering thread is calling the first drawing instruction list. Alternatively, the main thread needs to send the unified rendering tree of the first image frame to the rendering thread before it can start updating the unified rendering tree according to the second image frame, so as to avoid the unified rendering tree being modified by the main thread while the rendering thread is using the unified rendering tree.

[0220] In this embodiment, see Fig.16 As shown, the main thread can serially preprocess the first image frame, the second image frame, the third image frame, etc., which are waiting to be drawn in sequence, to obtain corresponding preprocessing results, such as the first preprocessing result, the second preprocessing result, the third preprocessing result, etc., and synchronize them to the rendering thread. The rendering thread can serially receive each preprocessing result and generate corresponding image frames according to the preprocessing results, such as generating a first image frame according to the first preprocessing result, generating a second image frame according to the second preprocessing result, and generating a third image frame according to the third preprocessing result, etc.

[0221] In the above process, it can be seen that the main thread and the rendering thread run in parallel when processing different image frames. For example, the process of the main thread processing the second image frame and the process of the rendering thread processing the first image frame are run in parallel. Therefore, the method provided in this embodiment can increase the generation rate of image frames, reduce the situation of frame drops or freezes during the display process of electronic devices, and improve user experience.

[0222] In the process of UniRender using the main thread and the rendering thread to generate image frames in parallel, the rendering thread can start running at different times. For example, the rendering thread can start running after receiving the VSync signal and synchronously obtaining the drawing instruction list from the main thread, or the rendering thread can start running immediately after synchronously obtaining the drawing instruction list from the main thread.

[0223] The following takes the example that the preprocessing result sent by the main thread to the rendering thread is a drawing instruction list to specifically explain the above two methods of generating image frames.

[0224] (1) The rendering thread starts running after receiving the VSync signal

[0225] In this embodiment, as shown in Table 1, the main thread of UniRender starts running after receiving the VSync signal, and the rendering thread starts working after synchronously obtaining the drawing instruction list from the main thread and receiving the VSync signal.

[0226] Table 1

[0227]

[0228] Fig.17 17 is a schematic flow chart of a method for generating an image frame provided by another embodiment of the present application. The method specifically includes the following steps S1701 to S1706.

[0229] S1701, the main thread receives a first VSync signal.

[0230] S1702: In response to a first VSync signal, the main thread generates a first drawing instruction list for a first image frame.

[0231] Taking the first drawing instruction list as an example, the main thread generates the first drawing instruction list including: the main thread updates the unified rendering tree, traverses the updated unified rendering tree to determine the dirty area, and generates an incremental drawing instruction list corresponding to the dirty area. The incremental drawing instruction list is the first drawing instruction list.

[0232] S1703: The main thread synchronizes the first drawing instruction list to the rendering thread.

[0233] Exemplarily, the main thread transfers the pointer of the first drawing instruction list to the rendering thread.

[0234] S1704, the rendering thread and the thread receive a second VSync signal.

[0235] It should be noted that the first VSync signal and the second VSync signal are VSync signals transmitted at a fixed frequency. In other words, the first VSync signal and the second VSync signal have the same content but are transmitted at different times.

[0236] S1705 , in response to the second VSync signal, the rendering thread generates a first image frame according to the first drawing instruction list.

[0237] In this embodiment, both the main thread and the rendering thread are started by the VSync signal, which can reduce the impact of the main thread on the rendering thread and improve the independence of the main thread and the rendering thread.

[0238] S1706 , in response to the second VSync signal, while the rendering thread generates the first image frame according to the first drawing instruction list, the main thread generates a second drawing instruction list for the second image frame, where the second drawing instruction list is used to generate the second image frame.

[0239] It should be noted that the first image frame and the second image frame are two image frames that the electronic device needs to draw continuously, and the drawing order of the first image frame is earlier than that of the second image frame. Fig.18 As shown, after the main thread hands over the first drawing instruction list of the first image frame to the rendering thread for processing, while the rendering thread draws the first image frame according to the first drawing instruction list, the main thread can generate a second drawing instruction list for the second image frame in parallel, thereby improving the image frame generation efficiency, reducing frame drops or freezes during the display process of the electronic device, and improving the user experience.

[0240] In this embodiment, see Fig.19 As shown in the figure, assuming that the time it takes UniRender to generate a single frame is constant at T (ms), then the maximum frame rate F supported by UniRender when generating image frames serially is max1 =1000 / T.

[0241] After UniRender is split into the main thread and the rendering thread, when the main thread and the rendering thread run in parallel, the maximum frame rate F supported by UniRender max2 The following formulas (1) and (2) are satisfied.

[0242] T UI +T render =T (1)

[0243]

[0244] Where T is the time it takes UniRender to generate a single frame, T UI is the time taken by UniRender’s main thread to process image frames, T render It is the time taken by UniRender's rendering thread to process the image frame. In addition, min(T UI , T render ) indicates T UI and T render The minimum value in man(T UI , T render ) indicates T UI and T render The maximum value in .

[0245] Solving formula (1) and formula (2) together yields:

[0246] When TUI =T render = T / 2, the parallelism between the main thread and the rendering thread is the highest. After calculation, the maximum frame rate F supported by UniRender after splitting max2 =2000 / T, which is the maximum frame rate F supported before splitting max1 twice, which can significantly increase the rate at which electronic devices generate image frames.

[0247] When T UI =T / 3, T render =2T / 3, the main thread and rendering thread can run in parallel. After calculation, the maximum frame rate F supported by UniRender after splitting max2 =1500 / T, which is the maximum frame rate F supported before splitting max1 It can also increase the rate at which electronic devices generate image frames by 1.5 times.

[0248] (2) The rendering thread starts running after synchronously obtaining the drawing instruction list from the main thread

[0249] In this embodiment, as shown in Table 2, the main thread of UniRender starts running after receiving the VSync signal, and the rendering thread works after synchronously obtaining the drawing instruction list from the main thread without waiting for the VSync signal.

[0250] Table 2

[0251] name Run time Main Thread After receiving the VSync signal Rendering Thread After synchronously obtaining the drawing instruction list from the main thread

[0252] Fig. 20 This is a schematic flow chart of a method for generating an image frame provided by another embodiment of the present application. The method specifically includes the following steps S2001 to S2006.

[0253] S2001, the main thread receives the first VSync signal.

[0254] S2002: In response to a first VSync signal, the main thread generates a first drawing instruction list for a first image frame.

[0255] S2003: The main thread synchronizes the first drawing instruction list to the rendering thread.

[0256] Exemplarily, the main thread transfers the pointer of the first drawing instruction list to the rendering thread.

[0257] S2004: The rendering thread generates a first image frame according to the first drawing instruction list.

[0258] S2005, the main thread receives a second VSync signal.

[0259] S2006 , in response to the second VSync signal, the main thread generates a second drawing instruction list for a second image frame, where the second drawing instruction list is used to generate the second image frame.

[0260] For S2005 to S2006, it should be noted that when the main thread finishes processing the first image frame, the next VSync signal after the first VSync signal (i.e., the second VSync signal) usually has not arrived yet, so the main thread needs to wait for the second VSync signal while the rendering thread is generating the first image frame. The main thread can start generating the second image frame only after the second VSync signal arrives.

[0261] In this embodiment, see Fig.21 As shown, the main thread can quickly start the rendering thread according to the first drawing instruction list, thereby improving the efficiency of generating image frames.

[0262] In this embodiment, see Fig. 22 As shown in the figure, assuming that the time it takes UniRender to generate a single frame is constant at T (ms), then the maximum frame rate F supported by UniRender when generating image frames serially is max1 =1000 / T. After UniRender is split into the main thread and the rendering thread, since the main thread and the rendering thread run in parallel, a rendering cycle of UniRender is greater than max(T UI , T render ) but less than T. And, T UI =T render =T / 2, the maximum frame rate F after UniRender split max2 Can also reach F max2 =2000 / T. Therefore, the method provided in this embodiment can significantly improve the efficiency of generating image frames.

[0263] In order to further improve the efficiency of image frame generation, the main thread can use multiple sub-threads to work in parallel to quickly generate a drawing instruction list. For example, the main thread can use multiple sub-threads to generate drawing instruction sub-lists of different foreground applications in parallel, and after merging these drawing instruction sub-lists, obtain the first drawing instruction list of the first image frame. The details are as follows.

[0264] Fig.23 This is a schematic flow chart of a main thread generating a first drawing instruction list provided by an embodiment of the present application. Taking the main thread including a first sub-thread and a second sub-thread running in parallel as an example, the process specifically includes the following steps S2301 to S2304.

[0265] S2301, the main thread receives a first VSync signal.

[0266] S2302: The main thread uses the first sub-thread to determine a first drawing instruction sub-list corresponding to the first part of the first image frame.

[0267] S2303: The main thread uses the second sub-thread to determine a second drawing instruction sub-list corresponding to the second part of the first image frame, and the first sub-thread and the second sub-thread run in parallel.

[0268] S2304: The main thread generates a first drawing instruction list for the first image frame according to the first drawing instruction sub-list and the second drawing instruction sub-list.

[0269] It should be noted that, in this embodiment, the first drawing instruction list of the first image frame may be a full drawing instruction list or an incremental drawing instruction list, and this embodiment does not limit this.

[0270] In one example, the process of the main thread generating an incremental drawing instruction list for the first image frame includes:

[0271] First, after receiving the first VSync signal, the main thread updates the unified rendering tree of the first image frame.

[0272] Then, the main thread determines the number M of foreground applications whose user interfaces have changed according to the updated unified rendering tree. Fig.24 As shown in the example, the rendering nodes 1 to 4 of the unified rendering tree belong to window 1, the rendering nodes 5 to 6 belong to window 2, and the rendering node 7 belongs to window 3. Among them, the main thread determines that the rendering nodes 2 and 7 are dirty areas after dirty area calculation, and the node change information of the rendering node 2 is: the background color is changed to color A. The child nodes of the rendering node 7 are deleted and it becomes a leaf node. Based on this, Fig.24 In the unified rendering tree shown, the main thread determines that there are two windows whose user interface has changed, namely window 1 and window 3. Since application 1 corresponds to window 1, application 2 corresponds to window 2, and application 3 corresponds to window 3, the main thread determines that the number of foreground applications whose user interface has changed is M=2.

[0273] Then, the main thread uses M sub-threads to respectively determine the drawing instruction sub-lists corresponding to the incremental rendering nodes in the M target objects. Fig.24Taking the unified rendering tree shown as an example, the main thread uses two sub-threads to process the incremental rendering nodes of window 1 and window 3 respectively. For example, sub-thread 1 is used to process each incremental rendering node of window 3. Specifically, sub-thread 1 reversely traverses the rendering nodes in window 3 to perform dirty area calculation, determines that the dirty area is rendering node 7, and copies the entire rendering node 7 to the drawing instruction sub-list 1 as a drawing operation. And, sub-thread 2 is used to process each incremental rendering node of window 1. Specifically, sub-thread 2 reversely traverses the rendering nodes in window 1 to perform dirty area calculation, and copies the background drawing operation of the entire rendering node 2 to the drawing operation list.

[0274] Finally, the main thread merges the drawing instruction sublists of the M target objects into the first drawing instruction list. Fig.24 As shown, the main thread merges the drawing instruction sub-list 1 corresponding to window 3 and the drawing instruction sub-list 2 corresponding to window 1 into the first drawing instruction list of the entire first image frame.

[0275] In summary, through the above processes S2301 to S2304, the main thread can use multiple sub-threads to work in parallel to generate a drawing instruction list, thereby improving the efficiency of generating image frames.

[0276] In addition, the rendering thread may also use multiple rendering sub-threads to work in parallel to quickly execute the drawing operations in the drawing instruction list to generate image frames. Taking the process of the rendering thread generating the first image frame according to the first drawing instruction list of the first image frame as an example, the method is specifically as follows.

[0277] Fig.25 This is a flowchart of a rendering thread generating a first image frame according to a first drawing instruction list provided by an embodiment of the present application. The flow specifically includes the following steps S2501 to S2503.

[0278] S2501: The rendering thread uses a first rendering sub-thread to obtain a first drawing result according to a first part of a first drawing instruction list.

[0279] S2502: The rendering thread uses a second rendering sub-thread to obtain a second rendering result according to a second part of the first rendering instruction list, and the first rendering sub-thread and the second rendering sub-thread run in parallel.

[0280] In some embodiments, the first part of the first drawing instruction list may be a first drawing instruction sub-list in the main thread, and the second part may be a second drawing instruction sub-list in the main thread.

[0281] In some other embodiments, different parts of the first drawing instruction list, such as the first part and the second part, may be re-divided by the rendering thread. For example, the rendering thread may split the first drawing instruction list into N parts according to the number N of FBOs.

[0282] S2503: The rendering thread generates a first image frame according to the first drawing result and the second drawing result.

[0283] In this embodiment, the rendering thread may split the first drawing instruction list into N sub-lists, such as the first part and the second part, according to the number of FBOs. Subsequently, the rendering thread uses N rendering sub-threads, such as the first rendering sub-thread and the second rendering sub-thread, to process different sub-lists of the first drawing instruction list respectively, and finally obtains the first image frame.

[0284] It should be noted that FBO is a mechanism for rendering to texture or multiple rendering targets, which can be understood as a container. It provides a flexible way to perform off-screen rendering, post-processing, and rendering to texture.

[0285] In one example, the rendering thread uses multiple rendering sub-threads to process the first drawing instruction list in parallel, and the process of generating the first image frame includes the following contents:

[0286] First, the rendering thread can create N FBOs and split the first drawing command list into N sublists for off-screen rendering. Taking N=3 as an example, the rendering thread can split the first drawing command list into sublists 1 to 3, where sublists 1 to 3 are respectively: FBO1 DrawCmdList, FBO2 DrawCmdList, and FBO3 DrawCmdList.

[0287] Then, the rendering thread uses N rendering sub-threads to process the N sub-lists in parallel to obtain N local rendering results. Fig.26 As shown, the rendering thread uses rendering sub-thread 1 to draw FBO1 DrawCmdList to obtain drawing result 1; uses rendering sub-thread 2 to draw FBO1 DrawCmdList to obtain drawing result 2; uses rendering sub-thread 3 to draw FBO1 DrawCmdList to obtain drawing result 3.

[0288] Finally, the rendering thread generates a corresponding first image frame according to the N local rendering results and the previous frame rendering result.

[0289] In some embodiments, the first drawing instruction list is a full drawing instruction list, and the rendering thread can generate a first image frame according to the sum of the N local drawing results.

[0290] In some other embodiments, the first drawing instruction list is an incremental drawing instruction list, and the rendering thread generates a first image frame according to the N local drawing results and the previous frame drawing result. The previous frame drawing result is the drawing result of the previous image frame. The rendering thread uses the N local drawing results to replace the corresponding parts of the previous frame drawing result to obtain the first image frame.

[0291] In summary, through the above processes S2501 to S2503, the rendering thread can use multiple rendering sub-threads to work in parallel to generate image frames, thereby improving the generation efficiency of the image frames.

[0292] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0293] An embodiment of the present application also provides an electronic device, which includes one or more processors and a memory; the memory is coupled to the one or more processors; the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the image frame generation method shown in the above-mentioned embodiments.

[0294] The present application also provides a chip system. Fig. 27 As shown, the chip includes a processor and a memory, in which a computer program is stored. When the computer program is executed by the processor, the method for generating an image frame in the above-mentioned embodiments is implemented.

[0295] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for generating an image frame provided in the above embodiments is implemented.

[0296] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by an electronic device, the electronic device implements the method for generating an image frame provided in the above embodiments.

[0297] It should be understood that the processor mentioned in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0298] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (doubledatarate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlink DRAM, SLDRAM), and direct rambus RAM (DR RAM).

[0299] In the embodiments provided in the present application, the division of each framework or module is only a logical function division. There may be other division methods in actual implementation. For example, multiple frameworks or modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0300] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0301] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0302] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0303] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for generating an image frame, characterized in that: Applied to an electronic device, the unified rendering process of the electronic device includes a main thread and a rendering thread, and the method includes: After receiving the first vertical synchronization signal, the main thread generates a first preprocessing result of the first image frame; The main thread synchronizes the first preprocessing result to the rendering thread; When the rendering thread generates the first image frame according to the first preprocessing result, after the main thread receives the second vertical synchronization signal, the main thread generates a second preprocessing result of the second image frame, and the second preprocessing result is used to generate the second image frame; The rendering thread generates the first image frame according to the first preprocessing result.

2. The method according to claim 1, characterized in that The first preprocessing result is a first drawing instruction list of the first image frame; wherein the first drawing instruction list is an incremental drawing instruction list or a full drawing instruction list, The incremental drawing instruction list includes drawing instructions of newly added rendering nodes in the unified rendering tree of the first image frame, and / or drawing instructions of rendering nodes whose information is changed; The full rendering instruction list includes rendering instructions of all rendering nodes in a unified rendering tree of the first image frame, wherein the unified rendering tree is a rendering tree obtained by merging rendering trees of all applications performing unified rendering.

3. The method according to claim 2, characterized in that The first image frame includes a first part from the first application and a second part from the second application. After receiving the first vertical synchronization signal, the main thread generates a first preprocessing result of the first image frame, including: After the main thread receives the first vertical synchronization signal, the first sub-thread is used to determine a first drawing instruction sub-list corresponding to the first part; After the main thread receives the first vertical synchronization signal, the second sub-thread is used to determine a second drawing instruction sub-list corresponding to the second part; wherein the first sub-thread and the second sub-thread are run in parallel; The main thread generates the first drawing instruction list of the first image frame according to the first drawing instruction sub-list and the second drawing instruction sub-list.

4. The method according to claim 2 or 3, characterized in that: The rendering thread generates the first image frame according to the first preprocessing result, including: The rendering thread uses a first rendering sub-thread to obtain a first drawing result according to a first part of the first drawing instruction list; The rendering thread uses a second rendering sub-thread to obtain a second rendering result according to the second part of the first rendering instruction list; wherein the first rendering sub-thread and the second rendering sub-thread run in parallel; The rendering thread generates the first image frame according to the first drawing result and the second drawing result.

5. The method according to claim 4, characterized in that The rendering thread generates the first image frame according to the first drawing result and the second drawing result, including: When the first drawing instruction list is an incremental drawing instruction list, the rendering thread generates the first image frame according to the first drawing result and the second drawing result, and the drawing result of the previous image frame.

6. The method according to any one of claims 2 to 5, characterized in that: The first drawing instruction list is an incremental drawing instruction list. After the main thread receives the first vertical synchronization signal, it generates a first preprocessing result of the first image frame, including: After receiving the first vertical synchronization signal, the main thread receives node change information of a rendering tree of at least one application; The main thread generates the incremental drawing instruction list of the first image frame according to the node change information.

7. The method according to any one of claims 2 to 5, characterized in that: The first drawing instruction list is a full drawing instruction list. After receiving the first vertical synchronization signal, the main thread generates a first preprocessing result of the first image frame, including: After receiving the first vertical synchronization signal, the main thread receives a rendering tree of at least one application; The main thread generates the full drawing instruction list of the first image frame according to the rendering tree of the at least one application.

8. The method according to claim 1, characterized in that The first preprocessing result is a unified rendering tree of the first image frame, wherein the unified rendering tree is a rendering tree obtained by merging rendering trees of all applications performing unified rendering.

9. The method according to any one of claims 1 to 8, characterized in that: The rendering thread generates the first image frame according to the first preprocessing result, including: After receiving the first preprocessing result, the rendering thread generates the first image frame according to the first preprocessing result.

10. The method according to any one of claims 1 to 8, characterized in that: The rendering thread generates the first image frame according to the first preprocessing result, including: After receiving the second vertical synchronization signal, the rendering thread generates the first image frame according to the first preprocessing result.

11. The method according to any one of claims 1 to 10, characterized in that: The main thread synchronizing the first preprocessing result to the rendering thread includes: The main thread sends the first preprocessing result to the rendering thread; or, The main thread sends address information of the first preprocessing result to the rendering thread, wherein the address information is used to access the first preprocessing result; or, The main thread transfers the pointer of the first preprocessing result to the rendering thread.

12. An electronic device, characterized in that: The electronic device comprises: one or more processors and a memory; The memory is coupled to the one or more processors; The memory is used to store computer program codes, where the computer program codes include computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 11.

13. A chip system, applied to an electronic device, the chip system comprising one or more processors, the processors being used to call computer instructions so that the electronic device executes the method as claimed in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images