Image frame generation method, electronic device, chip system and storage medium

Abstract

The present application discloses a method for generating an image frame, an electronic device, a chip system, and a storage medium, and relates to the field of electronic technology. The method is applied to an electronic device, and the unified rendering process of the electronic device includes a main thread and a rendering thread. The method includes: after the main thread receives a first vertical synchronization signal, it generates a first preprocessing result of a first image frame; the main thread synchronizes the first preprocessing result to the rendering thread; the rendering thread generates a first image frame according to the first preprocessing result; in the process of the rendering thread generating the first image frame according to the first preprocessing result, after the main thread receives a second vertical synchronization signal, it generates a second preprocessing result of a second image frame, and the second preprocessing result is used to generate a second image frame. The technical solution provided in this embodiment can increase the generation rate of image frames, reduce frame drops or freezes during the display process of the electronic device, and improve the user experience.

Description

[0001] This application is a divisional application. The application number of the original application is 202410697901.4, the original application date is May 30, 2024, and the name of the invention is "Image frame generation method, electronic device, chip system and storage medium". The entire content of the original application is incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of electronic technology, and in particular to a method for generating an image frame, an electronic device, a chip system, and a storage medium. Background Art

[0003] With the advancement of computer technology, electronic devices now support running one or more applications simultaneously in the foreground, known as foreground applications. Because each foreground application requires its user interface to be displayed on the screen, and the display area of ​​an electronic device's screen is limited, the image frame actually displayed by the electronic device is the result of combining the user interfaces of each foreground application.

[0004] Currently, electronic devices can use unified rendering to combine the user interfaces of various foreground applications into a single image frame for display. However, within a unified rendering cycle, the unified rendering process (UniRender) needs to perform numerous operations, such as updating render nodes, traversing render nodes to generate image frames, and sending image frames for display. Due to the relatively complex UniRender rendering process and the slow rate of image frame generation, electronic devices may experience frame drops or freezes during display, resulting in a poor user experience. Summary of the Invention

[0005] The present application provides a method for generating an image frame, an electronic device, a chip system and a storage medium, which are used to solve the problem in the prior art that the unified rendering process generates image frames at a slow rate, resulting in frame drops or freezes in the electronic device during the display process.

[0006] To achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a method for generating an image frame, which is applied to an electronic device, wherein the unified rendering process of the electronic device includes a main thread and a rendering thread, and the method includes: after the main thread receives a first vertical synchronization signal, generating a first preprocessing result of a first image frame; the main thread synchronizes the first preprocessing result to the rendering thread; the rendering thread generates the first image frame according to the first preprocessing result; in the process of the rendering thread generating the first image frame according to the first preprocessing result, after the main thread receives a second vertical synchronization signal, generating a second preprocessing result of a second image frame, and the second preprocessing result is used to generate the second image frame.

[0008] The first vertical synchronization signal and the second vertical synchronization signal are signals with the same content that are sent at a fixed frequency. The first image frame and the second image frame are two different image frames that are displayed sequentially on the screen.

[0009] In this embodiment, during the unified rendering process of the electronic device, the main thread processes the second image frame and the rendering thread processes the first image frame in parallel. Therefore, the method provided by this embodiment can increase the image frame generation rate, reduce frame drops or freezes during the electronic device display process, and improve the user experience.

[0010] In some embodiments, the first preprocessing result is a first drawing instruction list of a first image frame; wherein the first drawing instruction list is an incremental drawing instruction list or a full drawing instruction list, wherein the incremental drawing instruction list includes drawing instructions for newly added rendering nodes in a unified rendering tree of the first image frame, and / or drawing instructions for rendering nodes whose information has been changed; and the full drawing instruction list includes drawing instructions for all rendering nodes in a unified rendering tree of the first image frame, wherein the unified rendering tree is a rendering tree obtained by merging rendering trees of all applications performing unified rendering.

[0011] It's important to note that when the main thread generates a full drawing instruction list, it doesn't need to perform dirty region calculations. This means it doesn't need to determine the drawing instructions for newly added render nodes in the unified render tree and / or for render nodes with modified information, making the generation process relatively simple. Incremental drawing instruction lists, on the other hand, contain fewer drawing operations, reducing the workload of subsequent rendering threads and improving their efficiency.

[0012] In some embodiments, a first image frame includes a first portion from a first application and a second portion from a second application. After a main thread receives a first vertical synchronization signal, the main thread generates a first preprocessing result of the first image frame, including: after the main thread receives the first vertical synchronization signal, using a first sub-thread to determine a first drawing instruction sub-list corresponding to the first portion; after the main thread receives the first vertical synchronization signal, using a second sub-thread to determine a second drawing instruction sub-list corresponding to the second portion; wherein the first sub-thread and the second sub-thread run in parallel; and the main thread generates a first drawing instruction list for the first image frame based on the first drawing instruction sub-list and the second drawing instruction sub-list.

[0013] In this embodiment, the main thread uses multiple sub-threads to process different parts of the first image frame in parallel, which can improve the efficiency of generating the first drawing instruction list of the first image frame and further improve the efficiency of generating the image frame.

[0014] In some embodiments, a rendering thread generates a first image frame based on a first preprocessing result, including: the rendering thread uses a first rendering sub-thread to obtain a first drawing result based on a first part of a first drawing instruction list; the rendering thread uses a second rendering sub-thread to obtain a second drawing result based on a second part of the first drawing instruction list; wherein the first rendering sub-thread and the second rendering sub-thread run in parallel; and the rendering thread generates the first image frame based on the first drawing result and the second drawing result.

[0015] For example, the rendering thread may split the first drawing instruction list into N sublists, such as the first and second parts, based on the number of off-screen frame buffer objects (FBOs). Subsequently, N rendering sub-threads, such as the first and second rendering sub-threads, are determined to process the N sublists, respectively, ultimately obtaining the first image frame.

[0016] It should be noted that the first part and the second part of the first drawing instruction list may also correspond to the first drawing instruction sub-list and the second drawing instruction sub-list in the main thread respectively, and this embodiment does not limit this.

[0017] In this embodiment, the rendering thread uses multiple rendering sub-threads to process different parts of the first drawing instruction list in parallel, so that the drawing operations in the drawing instruction list can be quickly executed to generate the first image frame.

[0018] In some embodiments, the rendering thread generates a first image frame based on the first drawing result and the second drawing result, including: when the first drawing instruction list is an incremental drawing instruction list, the rendering thread generates the first image frame based on the first drawing result, the second drawing result, and the drawing result of the previous image frame. For example, the rendering thread replaces corresponding portions of the drawing result of the previous image frame with the first drawing result and the second drawing result to obtain the first image frame.

[0019] In this embodiment, the incremental rendering method is used to generate the first image frame, which can reduce the amount of data processed by the main thread and the rendering thread and increase the rate of generating image frames.

[0020] In some embodiments, the first drawing instruction list is an incremental drawing instruction list. After the main thread receives the first vertical synchronization signal, it generates a first preprocessing result of the first image frame, including: after the main thread receives the first vertical synchronization signal, it receives node change information of the rendering tree of at least one application; the main thread generates an incremental drawing instruction list for the first image frame according to the node change information.

[0021] For example, the main thread updates the unified rendering tree according to the node change information of the application, traverses the unified rendering tree to perform dirty area calculations, and copies the dirty area drawing instructions into the incremental drawing instruction list.

[0022] In this embodiment, during the process of generating the incremental drawing instruction list according to the node update information, the main thread has a smaller amount of computation, and the generated incremental drawing instruction list contains fewer drawing instructions, which can improve the efficiency of image frame generation.

[0023] In some embodiments, the first drawing instruction list is a full drawing instruction list, and after the main thread receives the first vertical synchronization signal, it generates a first preprocessing result of the first image frame, including: after the main thread receives the first vertical synchronization signal, it receives a rendering tree of at least one application; the main thread generates a full drawing instruction list for the first image frame based on the rendering tree of the at least one application.

[0024] For example, after receiving the render trees of each application, the main thread merges the render trees of each application to obtain a unified render tree. Subsequently, each render node in the unified render tree is recorded as a drawing instruction in a buffer area to form a full drawing instruction list.

[0025] In some embodiments, the first pre-processing result is a unified rendering tree of the first image frame, wherein the unified rendering tree is a rendering tree obtained by merging rendering trees of all applications performing unified rendering.

[0026] In some embodiments, the rendering thread generates the first image frame according to the first preprocessing result, including: after the rendering thread receives the first preprocessing result, generating the first image frame according to the first preprocessing result.

[0027] In this embodiment, the main thread can quickly start the rendering thread through the first pre-processing result (such as the first drawing instruction list), thereby increasing the generation rate of image frames.

[0028] In some embodiments, the rendering thread generates the first image frame according to the first preprocessing result, including: after the rendering thread receives the second vertical synchronization signal, generating the first image frame according to the first preprocessing result.

[0029] In this embodiment, both the main thread and the rendering thread are started by the VSync signal, which can reduce the impact of the main thread on the rendering thread and improve the independence of the main thread and the rendering thread.

[0030] In some embodiments, the main thread synchronizes the first preprocessing result to the rendering thread, including: the main thread sends the first preprocessing result to the rendering thread; or, the main thread sends address information of the first preprocessing result to the rendering thread, wherein the address information is used to access the first preprocessing result; or, the main thread transfers a pointer to the address of the first preprocessing result to the rendering thread.

[0031] It can be understood that compared with copying the first preprocessing result or the address information of the first preprocessing result and then sending it to the rendering thread, the time consumed by the pointer transfer data transmission method is only the time consumed by the pointer assignment, and the process is more convenient and efficient.

[0032] In a second aspect, this embodiment provides an electronic device, comprising: one or more processors and a memory. The memory is coupled to the one or more processors. The memory is configured to store computer program code, which includes computer instructions. The one or more processors invoke the computer instructions to cause the electronic device to perform the method described in the first aspect.

[0033] In a third aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device. The chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method shown in the first aspect above.

[0034] In a fourth aspect, this embodiment provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method shown in the first aspect above is implemented.

[0035] In a fifth aspect, this embodiment provides a computer program product, which includes a computer program. When the computer program is executed by an electronic device, the electronic device performs the method shown in the first aspect above.

[0036] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic diagram of a user interface provided in an embodiment of the present application;

[0038] Figure 2 is a structural diagram of an electronic device provided by this embodiment;

[0039] Figure 3 This is a schematic diagram of the software architecture of an electronic device provided by one embodiment of the present application;

[0040] Figure 4is a schematic diagram of a unified rendering process provided by an embodiment of the present application;

[0041] Figure 5 is a schematic diagram of a unified rendering exception scenario provided by an embodiment of the present application;

[0042] Figure 6 This is a schematic diagram of a main thread and a rendering thread included in a unified rendering process provided by an embodiment of the present application;

[0043] Figure 7 is a schematic diagram of a main thread and a rendering thread included in a unified rendering process provided by another embodiment of the present application;

[0044] Figure 8 This is a schematic diagram of the process of the main thread updating the unified rendering tree provided by an embodiment of the present application;

[0045] Figure 9 This is a schematic diagram of node relationships in a unified rendering tree provided by an embodiment of the present application;

[0046] Figure 10 is a schematic diagram of labeled rendering nodes provided by one embodiment of the present application;

[0047] Figure 11 This is a schematic diagram of the data structure of a drawing operation in a rendering thread provided by an embodiment of the present application;

[0048] Figure 12 This is a schematic diagram of instructions in the main thread and rendering thread provided by an embodiment of the present application;

[0049] Figure 13 This is a schematic diagram of a rendering thread converting a data format of a drawing operation provided by an embodiment of the present application;

[0050] Figure 14 This is a schematic diagram of a rendering thread converting a data format of a drawing operation provided by another embodiment of the present application;

[0051] Figure 15 is a schematic flow chart of a method for generating an image frame provided by an embodiment of the present application;

[0052] Figure 16 This is a schematic diagram of the operation flow of the main thread and the rendering thread provided by an embodiment of the present application;

[0053] Figure 17 is a schematic flow chart of a method for generating an image frame provided in another embodiment of the present application;

[0054] Figure 18 This is a schematic diagram of the operation flow of the main thread and the rendering thread provided by another embodiment of the present application;

[0055] Figure 19 is a schematic diagram of a frame rate for generating image frames provided by an embodiment of the present application;

[0056] Figure 20 is a schematic flow chart of a method for generating an image frame provided in yet another embodiment of the present application;

[0057] Figure 21 This is a schematic diagram of the operation flow of the main thread and the rendering thread provided by another embodiment of the present application;

[0058] Figure 22 is a schematic diagram of a frame rate for generating image frames provided by another embodiment of the present application;

[0059] Figure 23 This is a flowchart of a main thread generating a first drawing instruction list provided by an embodiment of the present application;

[0060] Figure 24 This is a schematic diagram of a process in which a main thread generates a first drawing instruction list, provided by an embodiment of the present application;

[0061] Figure 25 A flowchart of a rendering thread generating a first image frame according to a first drawing instruction list provided by an embodiment of the present application;

[0062] Figure 26 This is a schematic diagram of a process in which a rendering thread generates a first image frame according to a first drawing instruction list, provided by an embodiment of the present application;

[0063] Figure 27 It is a structural diagram of the chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] It should be understood that in the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a way to describe the association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0065] In this embodiment, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0066] To facilitate understanding, the following first introduces the relevant terms and concepts involved in the embodiments of this application.

[0067] (1) User interface

[0068] The user interface (UI) is the interface between applications and operating systems, allowing users to interact and exchange information. It is used to convert information from its internal form to a user-interpretable form. In other words, the UI is the interface source code written in a specific computer language, such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on the electronic device, ultimately presenting user-readable content. In short, the UI facilitates user interaction with electronic devices.

[0069] (2) Graphical User Interface

[0070] A graphical user interface (GUI) is a commonly used form of user interface, representing a user interface related to computer operations that uses graphics. A GUI can display visual interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets on electronic device screens.

[0071] (3) Display interface

[0072] In this embodiment, the display interface may also be referred to as an image frame or a screen interface, specifically referring to the content displayed on the screen of the electronic device. When the electronic device has multiple foreground applications, the display interface is the result of synthesizing the user interfaces of the multiple foreground applications.

[0073] (4) Vertical synchronization signal

[0074] The vertical synchronization (VSync) signal is a concept in computer graphics that controls the output and display of image frames. Its primary function is to eliminate screen tearing, a phenomenon that occurs when the refresh rate of computer graphics is out of sync with the display's refresh rate, resulting in the top and bottom portions of the image refreshing at different times. Specifically, when the display finishes displaying one image frame and is about to display the next, the display's controller sends a VSync signal. The frequency of the VSync signal is related to the refresh rate of the electronic device; for example, the frequency of the VSync signal can be the same as the refresh rate of the electronic device's screen.

[0075] VSync signals typically include software VSync signals and hardware VSync signals, and software VSync signals and hardware VSync signals typically maintain cycle synchronization. For example, software VSync signals include VSync-APP and VSync-RS. Among them, VSync-APP is used to trigger the rendering process of the application. VSync-RS is used to trigger the unified rendering process of the rendering service (RS). Hardware VSync signals include VSync-HW, which is used to trigger the display refresh process of the monitor.

[0076] (5) Rendering Tree

[0077] A render tree is a data structure used to generate an application's user interface. It records some or all of the information required to generate a single frame of the application's user interface. When an application has multiple windows or displays on different display areas (e.g., different screens), multiple render trees must be generated independently.

[0078] Each render tree can contain multiple render nodes, each of which includes render attributes and a draw operation struct (DrawOP) list. Render attributes include information about the view to be rendered by the render node, such as its position, size, and transparency on the surface. The draw operation list records the draw operations and their order. The draw operation determines the content of the view to be rendered by the render node, such as a line, rectangle, or circle.

[0079] The technical solutions provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0080] With the development of computer technology, electronic devices support running one or more applications (applications, Apps) in the foreground at the same time. For the sake of ease of description, this embodiment refers to the application running in the foreground as the foreground application. Since each foreground application needs to present its user interface on the screen, but the display area of ​​the electronic device screen is limited, the content actually displayed by the electronic device (i.e., the image frame) is the display effect of the user interfaces of each foreground application synthesized according to specific rules. It should be noted that the specific rules can be determined according to the system configuration of the electronic device or the user operation, and this embodiment does not limit this.

[0081] For example Figure 1As shown, the foreground applications of the electronic device include a status bar, an information application, and a video application. In the image frame of the electronic device, the user interface of the status bar is located at the top of the image frame; the user interfaces of the information application and the video application are displayed in split screen according to user instructions, and occupy the upper and lower halves of the remaining image frame respectively.

[0082] During the operation of the foreground application, its user interface usually needs to change continuously, including changing the displayed content and the display position of the displayed content according to the configuration of the application itself or according to user operations. Figure 1 For example, Figure 1 The status bar in an app needs to refresh the user interface based on information such as cellular signal type, wireless fidelity (Wi-Fi) connection status, battery level, and time. Messaging apps need to refresh the user interface based on user input and received messages. Video apps need to refresh the user interface based on the video frame being played or user actions.

[0083] Because the image frame of an electronic device is the composite display of the user interfaces of various foreground applications, any changes to the foreground application's user interface will also require the image frame of the electronic device to change accordingly. Therefore, before each refresh of the image frame, the electronic device must generate a new image frame based on the running status of all foreground applications.

[0084] The following is an exemplary description of the process of an electronic device generating an image frame.

[0085] In this embodiment, the electronic device includes a terminal device, which may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device may be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiments of the present application do not limit the specific technology and specific device form adopted by the terminal device.

[0086] Figure 2 2 is a schematic diagram of the structure of an electronic device provided in this embodiment. The electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display 294, and a subscriber identification module (SIM) card interface 295. The sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, an air pressure sensor 280C, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 280F, a proximity light sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, etc.

[0087] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0088] For example, when the electronic device 200 is a mobile phone or a tablet computer, it may include all the components shown in the figure, or may include only some of the components shown in the figure.

[0089] The processor 210 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0090] The controller may be the nerve center and command center of the electronic device 200. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0091] Processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 210 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 210. If processor 210 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 210 latency, and thus improves system efficiency.

[0092] The charging management module 240 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 240 can receive charging input from the wired charger via the USB interface 230. In some wireless charging embodiments, the charging management module 240 can receive wireless charging input via the wireless charging coil of the electronic device 200. While charging the battery 242, the charging management module 240 can also provide power to the electronic device via the power management module 241.

[0093] The power management module 241 is used to connect the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 and provides power to the processor 210, the internal memory 221, the external memory, the display 294, the camera 293, and the wireless communication module 260. The power management module 241 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance), etc.

[0094] In some other embodiments, the power management module 241 may also be provided in the processor 210. In some other embodiments, the power management module 241 and the charging management module 240 may also be provided in the same device.

[0095] The wireless communication function of the electronic device 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor and the baseband processor.

[0096] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0097] The mobile communication module 250 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for the electronic device 200. The mobile communication module 250 may include at least one filter, a switch, a power amplifier, a low-noise amplifier (LNA), and the like. The mobile communication module 250 can receive electromagnetic waves from the antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 250 can also amplify the signals modulated by the modem processor and convert them into electromagnetic waves for radiation via the antenna 1.

[0098] In some embodiments, at least some functional modules of the mobile communication module 250 may be provided in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 250 and at least some functional modules of the processor 210 may be provided in the same device.

[0099] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 270A, the receiver 270B, etc.) or displays an image or video through the display screen 294. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 210 and be set in the same device as the mobile communication module 250 or other functional modules.

[0100] The wireless communication module 260 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 200. The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 210. The wireless communication module 260 can also receive the signal to be sent from the processor 210, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0101] In some embodiments, antenna 1 of electronic device 200 is coupled to mobile communication module 250, and antenna 2 is coupled to wireless communication module 260, so that electronic device 200 can communicate with a network and other devices via wireless communication technology. Wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. GNSS may include the global positioning system (GPS), the global navigation satellite system (GLONASS), the Beidou navigation satellite system (BDS), the quasi-zenith satellite system (QZSS) and / or the satellite based augmentation system (SBAS).

[0102] Electronic device 200 implements display functionality through a GPU, display screen 294, and an application processor. A GPU is a microprocessor for image processing that connects display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 210 may include one or more GPUs that execute program instructions to generate or modify display information.

[0103] The display screen 294 is used to display images, videos, etc. For example, the image frames in the embodiments of the present application. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 200 may include one or N display screens 294, where N is a positive integer greater than one.

[0104] The electronic device 200 can implement a shooting function through an ISP, a camera 293, a video codec, a GPU, a display screen 294, and an application processor.

[0105] The ISP processes data fed back by camera 293. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization for image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 293.

[0106] The camera 293 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 200 may include 1 or N cameras 293, where N is a positive integer greater than 1.

[0107] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 200 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0108] Video codecs are used to compress or decompress digital video. Electronic device 200 may support one or more video codecs. This allows electronic device 200 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0109] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 200, such as image recognition, face recognition, speech recognition, and text comprehension.

[0110] In an embodiment of the present application, the NPU or other processors can be used to perform operations such as analyzing and processing images in the video stored in the electronic device 200.

[0111] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200. The external memory card communicates with the processor 210 via the external memory interface 220 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0112] The internal memory 221 can be used to store computer executable program code, which includes instructions. The processor 210 executes various functional applications and data processing of the electronic device 200 by running the instructions stored in the internal memory 221. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and an application required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the electronic device 200 (such as audio data, a phone book, etc.).

[0113] In addition, the internal memory 221 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0114] The electronic device 200 can implement audio functions through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone jack 270D, and the application processor.

[0115] The audio module 270 is used to convert digital audio signals into analog audio signals for output, and is also used to convert analog audio input into digital audio signals. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be provided in the processor 210, or some functional modules of the audio module 270 can be provided in the processor 210.

[0116] Speaker 270A, also called a "horn," is used to convert audio electrical signals into sound signals. Electronic device 200 can listen to music or listen to hands-free calls through speaker 270A. For example, the speaker can play the comparison and analysis results provided in the embodiments of the present application.

[0117] The receiver 270B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 200 receives a call or a voice message, the user can place the receiver 270B close to the ear to hear the voice.

[0118] The microphone 270C, also known as a "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 270C to input the sound signal into the microphone 270C. The electronic device 200 can be provided with at least one microphone 270C. In other embodiments, the electronic device 200 can be provided with two microphones 270C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 200 can also be provided with three, four or more microphones 270C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0119] The headphone jack 270D is used to connect a wired headphone and can be a USB interface 230 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0120] The buttons 290 include a power button, a volume button, and the like. The buttons 290 may be mechanical buttons or touch buttons. The electronic device 200 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 200.

[0121] Motor 291 can generate vibration prompts. Motor 291 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 294, motor 291 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0122] The indicator 292 may be an indicator light, which may be used to indicate the charging status, power level change, messages, missed calls, notifications, etc.

[0123] The SIM card interface 295 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 200 by inserting it into or removing it from the SIM card interface 295. The electronic device 200 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 295 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 295 can also be compatible with different types of SIM cards. The SIM card interface 295 can also be compatible with external memory cards. The electronic device 200 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 200 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 200 and cannot be separated from the electronic device 200.

[0124] Figure 3 Schematic diagram of the software architecture of an electronic device provided by an embodiment of the present application. Figure 3 As shown, a layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, the software architecture of an electronic device is divided into four layers: application layer, application framework layer, system library layer, and kernel layer, from top to bottom.

[0125] The application layer can include a series of application packages. Figure 3 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, music, desktop, gallery, call, navigation, Bluetooth and video, etc.

[0126] An application process includes at least a main thread, also known as a UI thread. This main thread generates a render tree for the application based on its execution. For example, if the application supports off-screen rendering, the application process also includes a render thread, which generates a user interface based on the render tree. This user interface typically exists as a bitmap within the computer.

[0127] In some embodiments, if a foreground application uses unified rendering (UniRender), then when each VSync signal (such as VSync-App) arrives, the main thread of the foreground application generates a rendering tree based on the running status of the foreground application and sends the rendering tree to the application framework layer for further processing.

[0128] In other embodiments, if a foreground application uses off-screen rendering, then when each VSync signal arrives, the main thread of the foreground application first generates a rendering tree, and then the rendering thread of the foreground application draws the user interface according to the rendering tree, and sends the generated user interface to the application framework layer for further processing.

[0129] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. For example Figure 3 As shown, the application framework layer may include window management services, display management services, content providers, view systems, phone managers, resource managers, notification managers, and the like.

[0130] The window management service is responsible for starting, adding, and deleting windows. It can determine the application displayed on the window and the creation, destruction, and property changes of the application's layer, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0131] The display management service can obtain the number and size of display areas, and is responsible for starting, adding, and deleting display areas.

[0132] Content providers are used to store and retrieve data and make it accessible to applications. This data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0133] The phone manager is used to provide communication functions for electronic devices, such as the management of call status (including answering, hanging up, etc.).

[0134] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, etc.

[0135] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically, without requiring user interaction. For example, the Notification Manager can be used to notify users of completed downloads, message reminders, and so on. Furthermore, the Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog interfaces on the screen. Examples include displaying text messages in the status bar, emitting alert sounds, vibrating electronic devices, or flashing indicator lights.

[0136] The view system includes visual controls, such as those that display text and images. The view system is used to build applications. An image frame can be composed of one or more views. For example, the view that contains a text notification icon can include a view that displays text and a view that displays an image.

[0137] In this implementation, the view system also includes a RenderService (RS) module for unified rendering. RenderService runs a unified rendering process (UniRender) to receive render trees from one or more foreground applications and merge them into a unified render tree. UniRender also synchronizes layer information, such as layer creation, destruction, and property changes, with the Window Management Service, and displays information, such as screen size, with the Display Management Service.

[0138] It should be noted that, in this embodiment, the unified rendering process (UniRender) and the application process are independent of each other. Among them, the application and the unified rendering process complete data interaction through inter-process communication (IPC). For the specific content of the unified rendering process, please refer to the patent application with application number 2021114105136 and the invention name “Interface Generation Method and Electronic Device” and the patent application with application number 202111410643X and the invention name “Interface Generation Method and Electronic Device”, which will not be repeated here.

[0139] Optionally, in some embodiments of the present application, the view system further includes a surface compositor (SurfaceFlinger). On an electronic device configured with a whitelist, if the foreground application is not on the whitelist, after the main thread of the foreground application generates a render tree, the rendering thread of the foreground application generates a user interface, which is then handed over to SurfaceFlinger for layer synthesis to generate an image frame for display.

[0140] Optionally, in some embodiments of the present application, when the foreground application includes both applications in the whitelist and applications not in the whitelist, UniRender is responsible for generating the user interface of the applications in the whitelist. After UniRender generates the user interface, it passes the user interface to SurfaceFlinger. SurfaceFlinger then performs layer synthesis of the user interface and the user interfaces of other applications not in the whitelist to generate an image frame for display.

[0141] The system library can include multiple functional modules, such as the surface manager, media libraries, and graphics processing libraries. The details are as follows.

[0142] The surface manager is used to manage the display subsystem and can fuse two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0143] The media library supports playback and recording of a variety of common audio and video formats, as well as still image files. The media library can support a variety of audio and video coding formats, such as Moving Picture Experts Group 4 (MPEG4), H.264, Moving Picture Experts Group Audio Layer III (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR), Joint Photographic Experts Group (JPEG), and Portable Network Graphics (PNG).

[0144] The graphics processing library includes a 3D graphics processing library and a 2D graphics engine. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, layer synthesis, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0145] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, sensor drivers, audio drivers, and virtual card drivers.

[0146] Based on the hardware and software architecture of the electronic device shown in the above embodiments, the electronic device can generate image frames using either off-screen rendering or unified rendering. However, compared to off-screen rendering, unified rendering can reduce the number of rendering times, reduce the power consumption of the electronic device, and improve the user experience. Therefore, unified rendering is currently commonly used to generate image frames.

[0147] In some embodiments, the process of the electronic device generating an image frame using a unified rendering method is as follows: Figure 4 shown.

[0148] See also Figure 4 As shown, taking the foreground application including application 1 to application 4 as an example, the process of the electronic device generating an image frame using a unified rendering method specifically includes the following processes (1) to (4).

[0149] (1) The foreground application sends the render tree to RenderService.

[0150] When a VSync signal arrives, each foreground application uses its main thread to generate a render tree based on its current state and sends the render tree to the RenderService across processes. For example, Applications 1 through 4 use their main threads to generate render trees 1 through 4 and send them to the RenderService.

[0151] Exemplarily, the process of generating a render tree by the main thread of the foreground application includes: invalidating the foreground application's view hierarchy. The view hierarchy represents the parent-child relationship between views in the user interface. The main thread then traverses the foreground application's views, re-layouts each view, determines each view's attribute information and drawing operation (DrawOP), and treats each view's attribute information and drawing operation (DrawOP) as a render node to form a render tree.

[0152] It's important to note that views are the fundamental elements of an application's user interface. A control on a user interface can correspond to one or more views. The user interface an application displays consists of multiple nested views, each with a parent-child relationship. Therefore, the parent-child relationship between render nodes in the render tree generated by the main thread traversing the views is the same as the parent-child relationship between views. In other words, the parent-child relationship between views determines the nesting relationship between render nodes, thereby ensuring that the rendering thread (such as the application's rendering thread or the rendering thread of a unified rendering process) correctly renders the user interface when generating the interface based on the render tree.

[0153] (2) RenderService merges the various render trees into a unified render tree.

[0154] RenderService uses UniRender to receive render trees from each foreground application, obtain window control information and display area information from the window management service and display management service, and then combines the render trees of foreground applications in the same display area into a unified render tree based on the window control information and display area information. The window control information includes the Z-order of the window, and the display area information includes the identification information of the display area.

[0155] In some embodiments, UniRender can generate a new root rendering node and use the original root rendering root node of each rendering tree as a child node of the same level as the new root rendering root node, thereby generating a unified rendering tree. Figure 4 As shown, UniRender generates a new root rendering node and uses the original root rendering nodes of rendering trees 1 to 4 as child nodes of the same level of the new rendering root node, thereby generating a unified rendering tree.

[0156] Optionally, during the generation of the unified render tree, UniRender can first determine the upper and lower obscuration relationships between the layers of the foreground application based on the Z-order of the layers corresponding to the foreground application, and then delete the rendering nodes corresponding to the completely obscured views during the generation of the unified render tree.

[0157] (3) RenderService generates image frames based on the unified rendering tree.

[0158] After generating a unified render tree, RenderService's UniRender calls the layer processing library, sequentially traversing each render node and executing the drawing operations for each render node in the unified render tree in the unified cache to generate an image frame. It should be noted that the display information for this image frame is stored in a bitmap, and the electronic device's display can display the image frame on the screen based on the bitmap.

[0159] (4) RenderService sends the image frame to the display subsystem.

[0160] UniRender of RenderService stores the bitmap of the image frame in a frame buffer. The display subsystem (DSS) obtains the bitmap from the frame buffer and displays the image frame on the screen based on the bitmap. For example, the frame buffer can be an on-screen buffer.

[0161] Through the above processes (1) to (4), the electronic device can combine the user interfaces of various foreground applications into image frames for display through unified rendering. However, in the above unified rendering process, UniRender needs to perform many operations within a rendering cycle, and the rendering process is relatively complex, resulting in low rendering efficiency.

[0162] For example Figure 5 As shown in the figure, when the refresh rate of an electronic device is 120fps (frames per second), the time interval between two adjacent VSync signals is only 8.3ms, that is, a rendering cycle is only 8.3ms. During a rendering cycle, UniRender needs to perform operations such as generating a unified render tree, generating image frames based on the unified render tree, and sending the image frames to the display to complete the rendering of each image frame.

[0163] It is understandable that due to the short unified rendering cycle and the complex rendering process of UniRender, UniRender may not be able to complete the rendering work within a unified rendering cycle, resulting in the inability to send the bitmap of the image frame to the display in time, which in turn causes the electronic device to drop frames or jam during the display process, resulting in a poor user experience.

[0164] To this end, the present application provides a method for generating image frames, which is used to increase the rate of generating image frames in a unified rendering mode, thereby improving the situation of frame drops or freezes in the display process of electronic devices and improving user experience.

[0165] In this embodiment, see Figure 6 As shown, the unified rendering process UniRender consists of a main thread and a rendering thread, which share the tasks required by UniRender. The main thread preprocesses information during the image frame generation process and synchronizes the preprocessing results to the rendering thread. The rendering thread then generates the image frame based on the main thread's preprocessing results.

[0166] It is worth noting that the main thread supports serial processing of different image frames, and the rendering thread also supports serial processing of different image frames. In addition, the main thread and rendering thread can also run in parallel when processing different image frames.

[0167] In some embodiments of this application, see Figure 7 As shown, the tasks performed by UniRender's main thread and rendering thread specifically include the following (1) and (2).

[0168] (1) UniRender's main thread

[0169] In this embodiment, the main thread of UniRender (hereinafter referred to as the main thread) is used to update the unified rendering tree according to application messages and generate a drawing instruction list for the image frame according to the updated unified rendering tree. Specifically, it includes the following contents (1) to (2).

[0170] (1) Update the unified rendering tree according to application messages.

[0171] An application message is a message sent by at least one foreground application. It can be the unified render tree of the foreground application or node change information for at least one render node in the unified render tree of the foreground application. Exemplarily, this node change information can include updated attribute information or drawing operations for the render node, or animation step information for the render node. Animation step information includes information about the render node's movement between multiple user interfaces, such as its starting position, step rate, and end position.

[0172] In some embodiments, upon receiving a VSync signal, the foreground application generates a render tree and sends the render tree to the main thread. Based on this, the main thread can update the unified render tree based on the application message by receiving and merging the render trees sent by each foreground application to obtain a new unified render tree.

[0173] In other embodiments, since in most cases, the transformation of the user interface of two adjacent frames of the foreground application is usually not large or even does not change, the foreground application can determine the changed rendering node (i.e., the changed rendering node) after receiving the VSync signal, and only send the node change information of the changed rendering node to the main thread to reduce the amount of data interacted across processes and save power consumption of electronic devices. Based on this, the process of the main thread updating the unified rendering tree according to the application message can be: receiving the node change information sent by each foreground application, and updating the unified rendering tree according to the node change information. For example, based on the unified rendering tree of the previous frame of the user interface, a new rendering node is added, the rendering properties and drawing operations of the rendering node are updated, or the rendering node is deleted, etc.

[0174] In an example, the unified rendering tree in the main thread is as follows Figure 8 As shown in (a) in the figure, the unified rendering tree includes a root rendering node 0, and the child nodes of the root rendering node 0 include rendering node 1, rendering node 5, and rendering node 7. The child nodes of rendering node 1 include rendering node 2 and rendering node 4. The child nodes of rendering node 2 include rendering node 3. The child nodes of rendering node 5 include rendering node 6. The child nodes of rendering node 7 include rendering node 8. The child nodes of rendering node 8 include rendering node 9. Assume that the main thread receives node change information about rendering node 2, rendering node 8, and rendering node 9 from the application, wherein the node change information of rendering node 2 is: the background color is changed to color A, and the node change information of rendering node 8 is: the node is deleted. The node change information of rendering node 9 is: the image scale is changed to scale B. Then, the unified rendering tree updated by the main thread is as follows: Figure 8 As shown in (b) in .

[0175] It's important to note that in a render tree, a parent node includes its child nodes, or in other words, the drawing operations of a parent node include the drawing operations of its child nodes. Therefore, when the main thread updates a target render node, it must traverse the target render node's parent nodes, starting from the root node of the unified render tree and traversing the tree in descending order, before finding the target render node and updating its node information.

[0176] by Figure 8 Take the unified rendering tree shown in (a) as an example, the node relationship between the unified rendering tree can be seen in Figure 9 As shown. Specifically, it includes the following contents: the drawing operation of root rendering node 0 (i.e., RenderNodeDrawOp No. 0) includes the drawing operations of rendering node 1, rendering node 5, and rendering node 7 (i.e., RenderNodeDrawOp No. 1, 5, and 7). The drawing operation of rendering node 1 (i.e., RenderNodeDrawOp No. 1) includes the drawing operations of rendering node 2 and rendering node 4 (i.e., RenderNodeDrawOp No. 2 and 4). The drawing operation of rendering node 2 (i.e., RenderNodeDrawOp No. 2) includes the drawing operation of rendering node 3 (i.e., RenderNodeDrawOp No. 3). The drawing operation of rendering node 5 (i.e., RenderNodeDrawOp No. 5) includes the drawing operation of rendering node 6 (i.e., RenderNodeDrawOp No. 6). The drawing operation of rendering node 7 (i.e., RenderNodeDrawOp No. 7) includes the drawing operation of rendering node 8 (i.e., RenderNodeDrawOp No. 8). The drawing operation of rendering node 8 (i.e., RenderNodeDrawOp No. 8) includes the drawing operation of rendering node 9 (i.e., RenderNodeDrawOp No. 9).

[0177] Based on this, when the main thread updates Render Node 2, it needs to search for Render Node 1 from Render Node 0. Only then can it find Render Node 2 in Render Node 1 and update its node information. Similarly, when the main thread deletes Render Node 8, it needs to search for Render Node 7 from Render Node 0 and then delete Render Node 8 from Render Node 7.

[0178] Optionally, when updating the unified render tree, the main thread can also mark the changed render node and all its upper-level render nodes for subsequent processing. Figure 8 Taking the updated unified render tree shown in (b) of Figure 1 as an example, the main thread can mark render node 2, its parent render nodes 1 and 0, and render node 7. The main thread does not mark deleted render nodes. For example, render nodes 8 and 9 are not marked.

[0179] It should be noted that in a render tree (including a unified render tree), the calling interface of a child node is in the parent node. Therefore, in order to find a changed render node while traversing the render nodes, the main thread needs to mark all higher-level render nodes of the changed render node.

[0180] (2) Generate a drawing instruction list based on the updated unified rendering tree.

[0181] In some embodiments, the main thread traverses each render node, treating each render node as a drawing operation and copying it into the unified render tree's draw command list (DrawCmdList). This means the main thread traverses the entire unified render tree to generate the full draw command list for the unified render tree. As will be appreciated, this full traversal process is time-consuming, and the large number of draw operations in the full draw command list can also result in a longer drawing process for subsequent image frames.

[0182] In other embodiments, the main thread traverses the updated unified rendering tree using a dirty region stack technique, and completes dirty region calculation and copying of dirty region drawing operations in one traversal to generate an incremental drawing instruction list.

[0183] The dirty area calculation performed by the main thread refers to the main thread determining the rendering nodes in the unified rendering tree that need to be redrawn due to changes. Figure 8 Taking the updated unified render tree shown in (b) as an example, the main thread determines through dirty area calculation that its dirty area includes render node 2 and render node 7. Render node 7 becomes dirty because it has lost its child nodes (i.e., render node 8 and render node 9), which requires redrawing.

[0184] In addition, copying the dirty area drawing operation includes copying the drawing operation corresponding to the dirty area to the drawing instruction list. Figure 7 Taking the update process shown as an example, since only the background of Render Node 2 needs to be redrawn, the main thread copies the updated background drawing operation list (background DrawOp List) of Render Node 2 into the main thread's drawing instruction list. Furthermore, since Render Node 7 has lost its child nodes and is exposed again on the image frame, the entire Render Node 7 is copied into the main thread's drawing instruction list as a single drawing operation.

[0185] It should be noted that, typically, the unified rendering process directly traverses the render nodes of the unified render tree to draw image frames. However, in this embodiment, to enable parallel execution of the main thread and the rendering thread, the render node to be redrawn needs to be copied, for example, by copying the render node as a whole to the drawing instruction list. It will be appreciated that after copying the render node as a whole to the drawing instruction list, the subsequent execution of the rendering thread no longer relies on the unified render tree. Even if individual render nodes in the unified render tree are modified, the rendering thread can still operate normally.

[0186] Optionally, in order to improve the efficiency of the main thread in generating the drawing instruction list, the main thread can only traverse the marked rendering nodes to generate the drawing instruction list. Figure 8 The updated unified rendering tree shown in (b) has marked rendering nodes such as Figure 10 As shown, it includes rendering node 0, rendering node 1, rendering node 2 and rendering node 7. Therefore, the main thread only needs to Figure 10 The rendering nodes in are traversed to generate a drawing instruction list.

[0187] It's important to note that in the above process, marked render nodes are used to identify render nodes that have undergone incremental changes, such as newly added render nodes, render nodes with changed attribute information or drawing operations, or render nodes with changed animation step information. Deleted render nodes are not included. Therefore, the above process of traversing marked render nodes can also be called an incremental traversal process. Similarly, the above list that only includes dirty area drawing operations can also be called an incremental drawing instruction list.

[0188] It can be understood that the incremental traversal process can improve the efficiency of generating the drawing instruction list, reduce the occupation of CPU resources, and obtain a certain load benefit, such as a load benefit of 5%.

[0189] (2) UniRender's rendering thread

[0190] In this embodiment, the rendering thread of UniRender is mainly used to draw image frames according to the drawing instruction list from the main thread and send the image frames to the display. Specifically, it includes the following contents (3) to (4).

[0191] (3) Draw the image frame according to the drawing instruction list from the main thread.

[0192] First, after the main thread generates the unified rendering tree's drawing instruction list, it needs to synchronize that drawing instruction list with the rendering thread. For example, the main thread can copy the drawing instruction list and send it to the rendering thread; or, it can send the address of the drawing instruction list to the rendering thread; or, it can transfer the pointer to the address of the drawing instruction list to the rendering thread, allowing the rendering thread to call the drawing instruction list. Compared to copying the drawing instruction list or its address and sending it to the rendering thread, the pointer transfer data transfer method only takes the time required to assign the pointer, making it more convenient and efficient.

[0193] It should be noted that the data structures of drawing operations are different in UniRender's main thread and rendering thread. For example Figure 11 As shown, the data structure operated by the main thread is RenderNode, or render node. The data structure operated by the rendering thread is RenderNodeDrawble, or render node to be drawn. In other words, RenderNodeDrawble holds RenderNode information. In other words, RenderNodeDrawble strongly references RenderNode.

[0194] See also Figure 12 As shown in the figure, in the RenderNode of the main thread, the data structure of the drawing operation is DrawOp List. The DrawOp List includes the drawing operations required to draw the view, such as operations to draw circles, lines, and directions, but does not include attribute information such as the view size, color, and transparency. Exemplarily, the DrawOp List includes multiple types of drawing operations, such as geometry DrawOp List, background DrawOp List, content DrawOp List, child node RenderNodeDrawOp List, foreground DrawOp List, and overlay DrawOp List.

[0195] See also Figure 12As shown, in the rendering thread's RenderNodeDrawble, the data structure of the drawing operation is a drawing DrawOp List. Compared to a DrawOp List, a drawing DrawOp List can be understood as adding view attribute information such as size, color, and transparency to the view's drawing operation. For example, a drawing DrawOp List includes multiple types, such as a geometry drawing DrawOp List, a background drawing DrawOp List, a content drawing DrawOp List, a child node drawing RenderNodeDrawOp List, a foreground drawing DrawOp List, and an overlay drawing DrawOp List.

[0196] Therefore, after receiving the drawing instruction list, the rendering thread needs to prepare the drawing elements (i.e. prepare the attribute information of the rendering node) for the drawing operation DrawOp List of each rendering node according to the node information of each rendering node in the drawing instruction list, and generate the corresponding drawing DrawOp List.

[0197] For example Figure 13 As shown, after the main thread sends the entire rendering node 2 as a drawing operation to the rendering thread, the rendering thread converts the background DrawOp List in the rendering node 2 into a background drawingDrawOp List based on the attribute information of the rendering node 2.

[0198] For example Figure 14 As shown, after the main thread sends the entire rendering node 7 as a drawing operation to the rendering thread, the rendering thread converts the sub-node RenderNodeDrawOp List in the rendering node 7 into the sub-node drawing RenderNodeDrawOp List according to the attribute information of the rendering node 7.

[0199] The rendering thread then iterates over each drawing operation in the drawing instruction list and generates a drawing result. For ease of description, this embodiment refers to the result obtained by drawing based on the incremental drawing instruction list as an incremental drawing result, or a dirty area drawing result. For example, the rendering thread iterates over the drawing operations of render node 2 to change the background color of its view. Alternatively, it iterates over the drawing operations of render node 7 to regenerate the drawing result of its view.

[0200] Finally, the rendering thread generates the current image frame based on the drawing result of the previous image frame (called the previous frame drawing result) and the incremental drawing result obtained this time. For example, since the current drawing result is the drawing result of the dirty area, the rendering thread can use the current drawing result to replace the corresponding part of the previous frame drawing result to obtain the drawing result of the current image frame. For example, the drawing result of the current rendering node 2 is used to replace the drawing result of the rendering node 2 in the previous frame drawing result; and the drawing result of the current rendering node 7 is used to replace the drawing result of the rendering node 7 in the previous frame drawing result to obtain the drawing result of the current image frame.

[0201] (4) The rendering thread sends the image frame to the display.

[0202] The rendering thread stores the bitmap of the image frame into the frame buffer. The DSS obtains the bitmap of the image frame from the frame buffer and displays the image frame on the screen according to the bitmap.

[0203] It should be noted that this embodiment does not restrict the specific tasks performed by UniRender's main thread and rendering thread. In other words, this embodiment does not restrict the preprocessing results that UniRender's main thread sends to the rendering thread. Therefore, UniRender's main thread and rendering thread can also be configured to perform the following operations.

[0204] In some embodiments, the UniRender main thread is used to perform the above process (1), i.e., updating the unified rendering tree according to the application message and sending the updated unified rendering tree to the rendering thread. The rendering thread is used to perform the above processes (2) to (4), i.e., generating a drawing instruction list (e.g., a full drawing instruction list, an incremental drawing instruction list) according to the updated unified rendering tree, drawing image frames according to the drawing instruction list, and sending the image frames for display.

[0205] In other embodiments, the main thread of UniRender is used to perform the above process (1), that is, to update the unified rendering tree according to the application message and send the updated unified rendering tree to the rendering thread. The rendering thread directly traverses the updated unified rendering tree to generate image frames and sends the image frames to the display.

[0206] After introducing the main thread and rendering thread of UniRender, the following is an exemplary description of the process of an electronic device running the main thread and rendering thread in parallel to generate an image frame.

[0207] Figure 15 FIG1 is a schematic flow chart of a method for generating an image frame provided by an embodiment of the present application. Taking UniRender generating a first image frame and a second image frame as an example, the method specifically includes the following steps S1501 to S1504.

[0208] S1501: After receiving a first VSync signal, the main thread generates a first preprocessing result of a first image frame.

[0209] In some embodiments, the first pre-processing result is a unified rendering tree of the first image frame.

[0210] In other embodiments, the first preprocessing result is a first drawing instruction list for the first image frame. The first drawing instruction list can be a full drawing instruction list or an incremental drawing instruction list, which is not limited in this embodiment. The incremental drawing instruction list includes drawing instructions for newly added rendering nodes in the unified rendering tree of the first image frame and / or drawing instructions for rendering nodes whose information has changed. The full drawing instruction list includes drawing instructions for all rendering nodes in the unified rendering tree of the first image frame, wherein the unified rendering tree is a rendering tree obtained by merging the rendering trees of all applications performing unified rendering.

[0211] S1502: The main thread synchronizes the first preprocessing result to the rendering thread.

[0212] In some embodiments, the main thread sends the first pre-processing result to the rendering thread.

[0213] In some other embodiments, the main thread sends address information of the first preprocessing result to the rendering thread, and the address information is used to access the first preprocessing result.

[0214] In some other embodiments, the main thread transfers the pointer of the first pre-processing result from the main thread to the rendering thread. It can be understood that the time consumed by the data transfer method through pointer transfer is only the time consumed by pointer assignment, and the process is more convenient and efficient.

[0215] S1503: The rendering thread generates a first image frame according to the first preprocessing result.

[0216] It should be noted that for the same image frame (e.g., the first image frame), the main thread must first generate the preprocessing results for the image frame before the rendering thread can generate the image frame based on the preprocessing results. Therefore, the main thread and the rendering thread cannot run in parallel when generating the same image frame.

[0217] S1504 , during the execution of the rendering thread, after the main thread receives the second VSync signal, it generates a second preprocessing result of the second image frame, where the second preprocessing result is used to generate the second image frame.

[0218] In the above process, since the main thread will serially process the second pre-processing result of the second image frame after generating the first pre-processing result of the first image frame, the second pre-processing result may affect or overwrite the first pre-processing result. Therefore, to ensure the parallel operation of the main thread and the rendering thread, the main thread needs to send the first pre-processing result to the rendering thread to prevent the main thread from changing the first pre-processing result while processing the second image frame.

[0219] For example, the main thread needs to send the first drawing instruction list of the first image frame to the rendering thread before it can start generating the second drawing instruction list of the second image frame, to prevent the first drawing instruction list from being modified by the main thread while the rendering thread is calling the first drawing instruction list. Alternatively, the main thread needs to send the unified render tree of the first image frame to the rendering thread before it can start updating the unified render tree based on the second image frame, to prevent the unified render tree from being modified by the main thread while the rendering thread is using it.

[0220] In this embodiment, see Figure 16 As shown, the main thread can serially preprocess the first image frame, the second image frame, the third image frame, and so on, which are waiting to be drawn, to obtain corresponding preprocessing results, such as the first preprocessing result, the second preprocessing result, the third preprocessing result, and so on, and synchronize them to the rendering thread. The rendering thread can serially receive each preprocessing result and generate corresponding image frames based on the preprocessing results, such as generating the first image frame based on the first preprocessing result, generating the second image frame based on the second preprocessing result, and generating the third image frame based on the third preprocessing result, etc.

[0221] In the above process, it can be seen that the main thread and the rendering thread run in parallel when processing different image frames. For example, the main thread processes the second image frame and the rendering thread processes the first image frame in parallel. Therefore, the method provided by this embodiment can increase the image frame generation rate, reduce frame drops or freezes during the electronic device display process, and improve the user experience.

[0222] When UniRender generates frames in parallel using the main thread and the render thread, the render thread can start running at different times. For example, the render thread can start running after receiving the VSync signal and synchronously obtaining the drawing instruction list from the main thread, or the render thread can start running immediately after synchronously obtaining the drawing instruction list from the main thread.

[0223] The following takes the example of the pre-processing result sent by the main thread to the rendering thread as a drawing instruction list to specifically explain the above two methods of generating image frames.

[0224] (1) The rendering thread starts running after receiving the VSync signal

[0225] In this embodiment, as shown in Table 1, the main thread of UniRender starts running after receiving the VSync signal, and the rendering thread starts working after synchronously obtaining the drawing instruction list from the main thread and receiving the VSync signal.

[0226] Table 1

[0227]

[0228] Figure 17 FIG1 is a schematic flow chart of a method for generating an image frame provided in another embodiment of the present application. The method specifically includes the following steps S1701 to S1706.

[0229] S1701: The main thread receives a first VSync signal.

[0230] S1702 : In response to a first VSync signal, the main thread generates a first drawing instruction list for a first image frame.

[0231] For example, if the first drawing command list is an incremental drawing command list, the main thread generates the first drawing command list by updating the unified render tree, traversing the updated unified render tree to determine dirty regions, and generating an incremental drawing command list corresponding to the dirty regions. This incremental drawing command list is the aforementioned first drawing command list.

[0232] S1703: The main thread synchronizes the first drawing instruction list to the rendering thread.

[0233] Exemplarily, the main thread transfers the pointer of the first drawing instruction list to the rendering thread.

[0234] S1704: The rendering thread and the thread receive a second VSync signal.

[0235] It should be noted that the first VSync signal and the second VSync signal are VSync signals transmitted at a fixed frequency. In other words, the first VSync signal and the second VSync signal have the same content but are transmitted at different times.

[0236] S1705 , in response to the second VSync signal, the rendering thread generates a first image frame according to the first drawing instruction list.

[0237] In this embodiment, both the main thread and the rendering thread are started by the VSync signal, which can reduce the impact of the main thread on the rendering thread and improve the independence of the main thread and the rendering thread.

[0238] S1706 , in response to the second VSync signal, while the rendering thread generates the first image frame according to the first drawing instruction list, the main thread generates a second drawing instruction list for the second image frame, where the second drawing instruction list is used to generate the second image frame.

[0239] It should be noted that the first image frame and the second image frame are two image frames that the electronic device needs to draw continuously, and the drawing order of the first image frame is earlier than that of the second image frame. Figure 18 As shown, after the main thread hands over the first drawing instruction list of the first image frame to the rendering thread for processing, while the rendering thread draws the first image frame according to the first drawing instruction list, the main thread can generate a second drawing instruction list for the second image frame in parallel, thereby improving the generation efficiency of the image frame, reducing frame drops or freezes during the display process of the electronic device, and improving the user experience.

[0240] In this embodiment, see Figure 19 As shown in the figure, assuming that the time it takes UniRender to generate a single frame is constant at T (ms), then the maximum frame rate F supported by UniRender when generating image frames serially is max1 =1000 / T.

[0241] After UniRender is split into the main thread and the rendering thread, when the main thread and the rendering thread run in parallel, the maximum frame rate supported by UniRender is F max2 The following formulas (1) and (2) are satisfied.

[0242] T UI +T render =T (1)

[0243]

[0244] Among them, T is the time it takes UniRender to generate a single frame of image, T UI The time it takes for UniRender's main thread to process image frames, T render It is the time taken by UniRender's rendering thread to process the image frame. In addition, min(T UI , T render ) indicates T UI and T render The minimum value in man(T UI , T render ) indicates T UI and T render The maximum value in .

[0245] Solving formula (1) and formula (2) together yields:

[0246] When TUI =T render = T / 2, the parallelism between the main thread and the rendering thread is the highest. After calculation, the maximum frame rate F supported by UniRender after splitting max2 =2000 / T, which is the maximum frame rate F supported before splitting max1 twice as fast as the original image, which can significantly increase the rate at which electronic devices generate image frames.

[0247] When T UI =T / 3, T render =2T / 3, the main thread and rendering thread can run in parallel. After calculation, the maximum frame rate F supported by UniRender after splitting max2 =1500 / T, which is the maximum frame rate F supported before splitting max1 It can also increase the rate at which electronic devices generate image frames by 1.5 times.

[0248] (2) The rendering thread starts running after synchronously obtaining the drawing instruction list from the main thread

[0249] In this embodiment, as shown in Table 2, the main thread of UniRender starts running after receiving the VSync signal, and the rendering thread starts working after synchronously obtaining the drawing instruction list from the main thread without waiting for the VSync signal.

[0250] Table 2

[0251] name Runtime Main thread After receiving the VSync signal Rendering Thread After synchronously obtaining the drawing instruction list from the main thread

[0252] Figure 20 FIG2 is a schematic flow chart of a method for generating an image frame provided in another embodiment of the present application. The method specifically includes the following steps S2001 to S2006.

[0253] S2001, the main thread receives the first VSync signal.

[0254] S2002 : In response to a first VSync signal, the main thread generates a first drawing instruction list for a first image frame.

[0255] S2003: The main thread synchronizes the first drawing instruction list to the rendering thread.

[0256] Exemplarily, the main thread transfers the pointer of the first drawing instruction list to the rendering thread.

[0257] S2004: The rendering thread generates a first image frame according to the first drawing instruction list.

[0258] S2005: The main thread receives a second VSync signal.

[0259] S2006 , in response to the second VSync signal, the main thread generates a second drawing instruction list for a second image frame, where the second drawing instruction list is used to generate the second image frame.

[0260] Regarding S2005-S2006, it should be noted that when the main thread finishes processing the first image frame, the next VSync signal after the first VSync signal (i.e., the second VSync signal) usually has not yet arrived. Therefore, the main thread needs to wait for the second VSync signal while the rendering thread is generating the first image frame. The main thread can only start generating the second image frame after the second VSync signal arrives.

[0261] In this embodiment, see Figure 21 As shown, the main thread can quickly start the rendering thread based on the first drawing instruction list, thereby improving the efficiency of generating image frames.

[0262] In this embodiment, see Figure 22 As shown in the figure, assuming that the time it takes UniRender to generate a single frame is constant at T (ms), then the maximum frame rate F supported by UniRender when generating image frames serially is max1 =1000 / T. After UniRender is split into the main thread and the rendering thread, since the main thread and the rendering thread run in parallel, a rendering cycle of UniRender is greater than max(T UI , T render ) but is less than T. And, T UI =T render =T / 2, the maximum frame rate F after UniRender split max2 Can also reach F max2 =2000 / T. Therefore, the method provided in this embodiment can significantly improve the efficiency of generating image frames.

[0263] To further improve the efficiency of image frame generation, the main thread can use multiple sub-threads to work in parallel to quickly generate drawing instruction lists. For example, the main thread can use multiple sub-threads to generate drawing instruction sub-lists for different foreground applications in parallel, and then merge these drawing instruction sub-lists to generate the first drawing instruction list for the first image frame. The details are as follows.

[0264] Figure 23 This is a schematic flow chart of a main thread generating a first drawing instruction list provided by an embodiment of the present application. Taking the main thread including a first sub-thread and a second sub-thread running in parallel as an example, the flow specifically includes the following steps S2301 to S2304.

[0265] S2301: The main thread receives a first VSync signal.

[0266] S2302: The main thread uses the first sub-thread to determine a first drawing instruction sub-list corresponding to the first part of the first image frame.

[0267] S2303: The main thread uses the second sub-thread to determine a second drawing instruction sub-list corresponding to the second part of the first image frame, and the first sub-thread and the second sub-thread run in parallel.

[0268] S2304: The main thread generates a first drawing instruction list for the first image frame according to the first drawing instruction sub-list and the second drawing instruction sub-list.

[0269] It should be noted that, in this embodiment, the first drawing instruction list of the first image frame may be a full drawing instruction list or an incremental drawing instruction list, and this embodiment does not limit this.

[0270] In one example, the process of the main thread generating an incremental drawing instruction list for the first image frame includes:

[0271] First, the main thread updates the unified rendering tree of the first image frame after receiving the first VSync signal.

[0272] Then, the main thread determines the number M of foreground applications whose user interfaces have changed based on the updated unified rendering tree. For example, the unified rendering tree updated by the main thread is as follows: Figure 24 As shown in the example, the unified rendering tree's rendering nodes 1 to 4 belong to window 1, rendering nodes 5 to 6 belong to window 2, and rendering node 7 belongs to window 3. The main thread, after dirty area calculation, determines that rendering nodes 2 and 7 are dirty areas, and the node change information for rendering node 2 is: the background color is changed to color A. Rendering node 7 has its child nodes deleted and becomes a leaf node. Based on this, Figure 24 In the unified rendering tree shown, the main thread determines that there are two windows with user interface changes: window 1 and window 3. Since application 1 corresponds to window 1, application 2 corresponds to window 2, and application 3 corresponds to window 3, the main thread determines that the number of foreground applications with user interface changes is M = 2.

[0273] Then, the main thread uses M sub-threads to determine the drawing instruction sub-lists corresponding to the incremental rendering nodes in the M target objects. Figure 24Taking the unified rendering tree shown as an example, the main thread uses two child threads to process the incremental rendering nodes of window 1 and window 3, respectively. For example, child thread 1 processes each incremental rendering node of window 3. Specifically, child thread 1 reversely traverses the rendering nodes in window 3 to perform dirty region calculation, determines that the dirty region is rendering node 7, and copies the entire rendering node 7 as a single drawing operation to drawing instruction sublist 1. Furthermore, child thread 2 processes each incremental rendering node of window 1. Specifically, child thread 2 reversely traverses the rendering nodes in window 1 to perform dirty region calculation, and copies the background drawing operation of the entire rendering node 2 to the drawing operation list.

[0274] Finally, the main thread merges the drawing instruction sublists of the M target objects into the first drawing instruction list. Figure 24 As shown, the main thread merges the drawing instruction sub-list 1 corresponding to window 3 and the drawing instruction sub-list 2 corresponding to window 1 into the first drawing instruction list of the entire first image frame.

[0275] In summary, through the above processes S2301 to S2304 , the main thread can use multiple sub-threads to work in parallel to generate a drawing instruction list, thereby improving the efficiency of generating image frames.

[0276] Alternatively, the rendering thread may employ multiple rendering sub-threads to work in parallel to quickly execute the drawing operations in the drawing instruction list and generate an image frame. Taking the process of a rendering thread generating a first image frame according to a first drawing instruction list of the first image frame as an example, the method is specifically described as follows.

[0277] Figure 25 This is a flowchart of a rendering thread generating a first image frame according to a first drawing instruction list provided by an embodiment of the present application. The flowchart specifically includes the following steps S2501 to S2503.

[0278] S2501: The rendering thread uses a first rendering sub-thread to obtain a first rendering result according to a first part of a first drawing instruction list.

[0279] S2502: The rendering thread uses a second rendering sub-thread to obtain a second rendering result according to the second part of the first rendering instruction list, and the first rendering sub-thread and the second rendering sub-thread run in parallel.

[0280] In some embodiments, the first portion of the first drawing instruction list may be a first drawing instruction sub-list in the main thread, and the second portion may be a second drawing instruction sub-list in the main thread.

[0281] In other embodiments, different parts of the first drawing instruction list, such as the first part and the second part, may be re-divided by the rendering thread. For example, the rendering thread may split the first drawing instruction list into N parts according to the number N of FBOs.

[0282] S2503: The rendering thread generates a first image frame according to the first drawing result and the second drawing result.

[0283] In this embodiment, the rendering thread can split the first drawing instruction list into N sublists, such as a first portion and a second portion, based on the number of FBOs. Subsequently, the rendering thread uses N rendering sub-threads, such as the first rendering sub-thread and the second rendering sub-thread, to process the different sublists of the first drawing instruction list, ultimately generating a first image frame.

[0284] It should be noted that FBO is a mechanism for rendering to textures or multiple render targets, which can be understood as a container. It provides a flexible way to perform off-screen rendering, post-processing, and rendering to textures.

[0285] In one example, a rendering thread uses multiple rendering sub-threads to process the first drawing instruction list in parallel, and a process of generating the first image frame includes the following:

[0286] First, the rendering thread can create N FBOs and split the first draw command list into N sublists for off-screen rendering. Taking N = 3 as an example, the rendering thread can split the first draw command list into sublists 1 to 3, where sublists 1 to 3 are: FBO1 DrawCmdList, FBO2 DrawCmdList, and FBO3 DrawCmdList.

[0287] Then, the rendering thread uses N rendering sub-threads to process the N sub-lists in parallel to obtain N local rendering results. Figure 26 As shown, the rendering thread uses rendering sub-thread 1 to draw FBO1 DrawCmdList, and obtains drawing result 1; uses rendering sub-thread 2 to draw FBO1 DrawCmdList, and obtains drawing result 2; uses rendering sub-thread 3 to draw FBO1 DrawCmdList, and obtains drawing result 3.

[0288] Finally, the rendering thread generates a corresponding first image frame according to the N local rendering results and the previous frame rendering result.

[0289] In some embodiments, the first drawing instruction list is a full drawing instruction list, and the rendering thread can generate a first image frame according to the sum of the N local drawing results.

[0290] In other embodiments, the first drawing instruction list is an incremental drawing instruction list. The rendering thread generates a first image frame based on the N partial drawing results and the previous frame drawing result. The previous frame drawing result is the drawing result of the previous image frame. The rendering thread replaces the corresponding portion of the previous frame drawing result with the N partial drawing results to generate the first image frame.

[0291] In summary, through the above processes S2501 to S2503 , the rendering thread can use multiple rendering sub-threads to work in parallel to generate image frames, thereby improving the efficiency of image frame generation.

[0292] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0293] An embodiment of the present application also provides an electronic device, which includes one or more processors and a memory; the memory is coupled to the one or more processors; the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the image frame generation method shown in the above-mentioned embodiments.

[0294] The present application also provides a chip system. Figure 27 As shown, the chip includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method for generating an image frame in the above-mentioned embodiments is implemented.

[0295] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for generating an image frame provided in the above embodiments is implemented.

[0296] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by an electronic device, the electronic device implements the image frame generation method provided in the above embodiments.

[0297] It should be understood that the processor mentioned in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0298] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0299] In the embodiments provided in this application, the division of each framework or module is merely a logical function division. In actual implementation, there may be other division methods, for example, multiple frameworks or modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0300] In addition, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or software functional modules.

[0301] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0302] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0303] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for generating an image frame, characterized in that: Applied to an electronic device, wherein a unified rendering process of the electronic device includes a main thread and a rendering thread, the method includes: After receiving the first vertical synchronization signal, the main thread receives an application message sent by at least one foreground application; the application message is a rendering tree of the foreground application, or node change information of at least one rendering node in the rendering tree of the foreground application; The main thread updates a unified rendering tree according to the application message sent by the at least one foreground application, and generates a first preprocessing result of a first image frame according to the unified rendering tree, where the first preprocessing result is a first drawing instruction list of the first image frame; The main thread synchronizes the first preprocessing result to the rendering thread; When the rendering thread generates the first image frame according to the first preprocessing result, after the main thread receives the second vertical synchronization signal, the main thread generates a second preprocessing result for the second image frame, and the second preprocessing result is used to generate the second image frame; The rendering thread generates the first image frame according to the first preprocessing result.

2. The method according to claim 1, characterized in that The first drawing instruction list is an incremental drawing instruction list or a full drawing instruction list, The incremental drawing instruction list includes drawing instructions for newly added rendering nodes in the unified rendering tree of the first image frame and / or drawing instructions for rendering nodes whose information has been changed; the full drawing instruction list includes drawing instructions for all rendering nodes in the unified rendering tree of the first image frame, wherein the unified rendering tree is a rendering tree obtained by merging the rendering trees of all applications performing unified rendering.

3. The method according to claim 2, characterized in that The first image frame includes a first portion from a first application and a second portion from a second application, the main thread updates a unified rendering tree according to the application message sent by the at least one foreground application, and generates a first preprocessing result of the first image frame according to the unified rendering tree, including: The main thread updates the unified rendering tree according to the application message sent by the at least one foreground application; The main thread uses a first sub-thread to determine a first drawing instruction sub-list corresponding to the first part according to the unified rendering tree; The main thread uses a second sub-thread to determine a second drawing instruction sub-list corresponding to the second part according to the unified rendering tree; wherein the first sub-thread and the second sub-thread run in parallel; The main thread generates the first drawing instruction list of the first image frame according to the first drawing instruction sub-list and the second drawing instruction sub-list.

4. The method according to claim 2 or 3, characterized in that The rendering thread generates the first image frame according to the first preprocessing result, including: The rendering thread uses a first rendering sub-thread to obtain a first drawing result according to the first part of the first drawing instruction list; The rendering thread uses a second rendering sub-thread to obtain a second rendering result according to the second part of the first rendering instruction list; wherein the first rendering sub-thread and the second rendering sub-thread run in parallel; The rendering thread generates the first image frame according to the first drawing result and the second drawing result.

5. The method according to claim 4, characterized in that The rendering thread generates the first image frame according to the first drawing result and the second drawing result, including: When the first drawing instruction list is an incremental drawing instruction list, the rendering thread generates the first image frame according to the first drawing result, the second drawing result, and the drawing result of the previous image frame.

6. The method according to claim 1, characterized in that The first preprocessing result is a unified rendering tree of the first image frame, wherein the unified rendering tree is a rendering tree obtained by merging rendering trees of all applications that perform unified rendering.

7. The method according to any one of claims 1 to 3 or 5 to 6, characterized in that The rendering thread generates the first image frame according to the first preprocessing result, including: After receiving the first preprocessing result, the rendering thread generates the first image frame according to the first preprocessing result.

8. The method according to any one of claims 1 to 3 or 5 to 6, characterized in that The rendering thread generates the first image frame according to the first preprocessing result, including: After receiving the second vertical synchronization signal, the rendering thread generates the first image frame according to the first preprocessing result.

9. The method according to any one of claims 1 to 3 or 5 to 6, characterized in that The main thread synchronizing the first preprocessing result to the rendering thread includes: The main thread sends the first preprocessing result to the rendering thread; or, The main thread sends address information of the first preprocessing result to the rendering thread, wherein the address information is used to access the first preprocessing result; or, The main thread transfers the pointer of the first pre-processing result to the rendering thread.

10. An electronic device, characterized in that: The electronic device includes: one or more processors and memory; the memory being coupled to the one or more processors; The memory is used to store computer program code, where the computer program code includes computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 9.

11. A chip system, applied to an electronic device, comprising one or more processors, wherein the processors are configured to call computer instructions so that the electronic device executes the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images