Method for generating image frame, and electronic device, chip system and storage medium

Abstract

The present application relates to the technical field of electronics. Provided are a method for generating an image frame, and an electronic device, a chip system and a storage medium. The method is applied to an electronic device, wherein a unified render process of the electronic device comprises a main thread and a render thread. The method comprises: a main thread generating a first pre-processing result of a first image frame after receiving a first vertical synchronization signal; the main thread synchronizing the first pre-processing result to a render thread; the render thread generating the first image frame on the basis of the first pre-processing result; and during the process of the render thread generating the first image frame on the basis of the first pre-processing result, the main thread generating a second pre-processing result of a second image frame after receiving a second vertical synchronization signal, wherein the second pre-processing result is used for generating the second image frame. By means of the technical solution provided in the embodiments, the generation rate of an image frame can be increased, and frame drops or jank during display by an electronic device can be reduced, thereby improving the user experience.

Description

Image frame generation methods, electronic devices, chip systems, and storage media

[0001] This application claims priority to Chinese Patent Application No. 202410697901.4, filed on May 30, 2024, entitled "Method for Generating Image Frames, Electronic Device, Chip System and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of electronic technology, and in particular to a method for generating image frames, an electronic device, a chip system, and a storage medium. Background Technology

[0003] With the development of computer technology, electronic devices can support running one or more applications simultaneously in the foreground, i.e., foreground applications. Since each foreground application needs to present its user interface on the screen, but the display area of ​​an electronic device screen is limited, the image frame actually displayed by the electronic device is the combined effect of the user interfaces of each foreground application.

[0004] Currently, electronic devices can use unified rendering to merge the user interfaces of various foreground applications into a single image frame for display. However, within a unified rendering cycle, the UniRender process needs to perform numerous operations, such as updating rendering nodes, traversing rendering nodes to generate image frames, and sending the image frames for display. Because the UniRender rendering process is relatively complex and the rate of generating image frames is slow, it causes frame drops or stuttering during electronic device display, resulting in a poor user experience. Summary of the Invention

[0005] This application provides a method for generating image frames, an electronic device, a chip system, and a storage medium to solve the problem that the rate at which the unified rendering process generates image frames in the prior art is slow, causing frame drops or stuttering during the display of electronic devices.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] In a first aspect, this application provides a method for generating an image frame, applied to an electronic device. The unified rendering process of the electronic device includes a main thread and a rendering thread. The method includes: after the main thread receives a first vertical synchronization signal, it generates a first preprocessing result of a first image frame; the main thread synchronizes the first preprocessing result to the rendering thread; the rendering thread generates the first image frame according to the first preprocessing result; during the process of the rendering thread generating the first image frame according to the first preprocessing result, after the main thread receives a second vertical synchronization signal, it generates a second preprocessing result of a second image frame, and the second preprocessing result is used to generate the second image frame.

[0008] The first vertical synchronization signal and the second vertical synchronization signal are identical signals transmitted at a fixed frequency. The first image frame and the second image frame are two different image frames displayed sequentially on the screen.

[0009] In this embodiment, during the unified rendering process of the electronic device, the main thread's processing of the second image frame and the rendering thread's processing of the first image frame run in parallel. Therefore, the method provided in this embodiment can improve the image frame generation rate, reduce frame drops or stuttering during electronic device display, and improve the user experience.

[0010] In some embodiments, the first preprocessing result is a first drawing instruction list for the first image frame; wherein the first drawing instruction list is an incremental drawing instruction list or a full drawing instruction list, wherein the incremental drawing instruction list includes drawing instructions for newly added rendering nodes in the unified rendering tree of the first image frame, and / or drawing instructions for rendering nodes with changed information; the full drawing instruction list includes drawing instructions for all rendering nodes in the unified rendering tree of the first image frame, wherein the unified rendering tree is a rendering tree obtained by merging the rendering trees of all applications performing unified rendering.

[0011] It should be noted that when the main thread generates the full list of drawing instructions, there is no need to perform dirty region calculations, meaning there is no need to determine the drawing instructions for newly added rendering nodes and the drawing instructions for rendering nodes with changed information in the unified rendering tree. The generation process is relatively simple. Furthermore, the incremental drawing instruction list contains fewer drawing operations, which reduces the workload of subsequent rendering threads and improves their efficiency.

[0012] In some embodiments, the first image frame includes a first part from a first application and a second part from a second application. After receiving the first vertical synchronization signal, the main thread generates a first preprocessing result of the first image frame, including: after receiving the first vertical synchronization signal, the main thread uses a first sub-thread to determine a first drawing instruction sub-list corresponding to the first part; after receiving the first vertical synchronization signal, the main thread uses a second sub-thread to determine a second drawing instruction sub-list corresponding to the second part; wherein the first sub-thread and the second sub-thread run in parallel; the main thread generates a first drawing instruction list for the first image frame based on the first drawing instruction sub-list and the second drawing instruction sub-list.

[0013] In this embodiment, the main thread uses multiple sub-threads to process different parts of the first image frame in parallel, which can improve the efficiency of generating the first drawing instruction list of the first image frame, thereby improving the generation efficiency of the first image frame.

[0014] In some embodiments, the rendering thread generates a first image frame based on a first preprocessing result, including: the rendering thread using a first rendering sub-thread to obtain a first rendering result based on a first part of a first drawing instruction list; the rendering thread using a second rendering sub-thread to obtain a second rendering result based on a second part of the first drawing instruction list; wherein the first rendering sub-thread and the second rendering sub-thread run in parallel; and the rendering thread generates the first image frame based on the first rendering result and the second rendering result.

[0015] For example, the rendering thread can divide the first drawing instruction list into N sub-lists based on the number of off-screen frame buffer objects (FBOs), such as the first part and the second part mentioned above. Subsequently, N rendering sub-threads, such as the first rendering sub-thread and the second rendering sub-thread mentioned above, are used to process the N sub-lists respectively, and finally obtain the first image frame.

[0016] It should be noted that the first part and the second part of the first drawing instruction list can also correspond to the first drawing instruction sublist and the second drawing instruction sublist in the main thread, respectively. This embodiment does not impose any restrictions on this.

[0017] In this embodiment, the rendering thread uses multiple rendering sub-threads to process different parts of the first drawing instruction list in parallel, which can quickly complete the drawing operations in the drawing instruction list and generate the first image frame.

[0018] In some embodiments, the rendering thread generates a first image frame based on a first drawing result and a second drawing result, including: when the first drawing instruction list is an incremental drawing instruction list, the rendering thread generates the first image frame based on the first drawing result, the second drawing result, and the drawing result of the previous image frame. For example, the rendering thread can obtain the first image frame by replacing the corresponding part of the drawing result of the previous image frame with the first drawing result and the second drawing result.

[0019] In this embodiment, the first image frame is generated by incremental rendering, which can reduce the amount of data processed by the main thread and the rendering thread and improve the rate of generating the first image frame.

[0020] In some embodiments, the first drawing instruction list is an incremental drawing instruction list. After the main thread receives the first vertical synchronization signal, it generates a first preprocessing result of the first image frame, including: after the main thread receives the first vertical synchronization signal, it receives node change information of the rendering tree of at least one application; the main thread generates an incremental drawing instruction list of the first image frame based on the node change information.

[0021] For example, the main thread updates the unified rendering tree based on the application's node change information, traverses the unified rendering tree to perform dirty region calculations, and copies the dirty region drawing instructions to the incremental drawing instruction list.

[0022] In this embodiment, during the process of generating an incremental drawing instruction list based on node update information, the main thread has a smaller computational load, and the number of drawing instructions in the incremental drawing instruction list it generates is smaller, which can improve the efficiency of image frame generation.

[0023] In some embodiments, the first drawing instruction list is a full drawing instruction list. After the main thread receives the first vertical synchronization signal, it generates a first preprocessing result of the first image frame, including: after the main thread receives the first vertical synchronization signal, it receives the rendering tree of at least one application; the main thread generates a full drawing instruction list of the first image frame based on the rendering tree of at least one application.

[0024] For example, after the main thread receives the rendering trees from various applications, it merges the rendering trees of the various applications to obtain a unified rendering tree. Subsequently, each rendering node in the unified rendering tree is recorded as a drawing instruction into a buffer, forming a full list of drawing instructions.

[0025] In some embodiments, the first preprocessing result is a unified rendering tree of the first image frame, wherein the unified rendering tree is a rendering tree obtained by merging the rendering trees of all applications performing unified rendering.

[0026] In some embodiments, the rendering thread generates a first image frame based on a first preprocessing result, including: after receiving the first preprocessing result, the rendering thread generates a first image frame based on the first preprocessing result.

[0027] In this embodiment, the main thread can quickly start the rendering thread to run using the first preprocessing result (e.g., the first drawing instruction list), thereby improving the generation rate of the first image frame.

[0028] In some embodiments, the rendering thread generates a first image frame based on a first preprocessing result, including: after receiving a second vertical synchronization signal, the rendering thread generates a first image frame based on the first preprocessing result.

[0029] In this embodiment, both the main thread and the rendering thread are started by the VSync signal, which reduces the impact of the main thread on the rendering thread and improves the independence of the main thread and the rendering thread.

[0030] In some embodiments, the main thread synchronizes the first preprocessing result to the rendering thread, including: the main thread sending the first preprocessing result to the rendering thread; or, the main thread sending the address information of the first preprocessing result to the rendering thread, wherein the address information is used to access the first preprocessing result; or, the main thread transferring the pointer to the address of the first preprocessing result to the rendering thread.

[0031] It is understandable that, compared to copying the first preprocessing result or its address information and sending it to the rendering thread, the time consumed by the pointer transfer data transfer method is only the time consumed by pointer assignment, and the process is more convenient and efficient.

[0032] Secondly, this embodiment provides an electronic device comprising one or more processors and a memory. The memory is coupled to the one or more processors. The memory stores computer program code, including computer instructions, which the one or more processors invoke to cause the electronic device to perform the methods shown in the first aspect above.

[0033] Thirdly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the method shown in the first aspect above.

[0034] Fourthly, this embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method shown in the first aspect above.

[0035] Fifthly, this embodiment provides a computer program product including a computer program that, when run by an electronic device, causes the electronic device to perform the method shown in the first aspect above.

[0036] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0037] Figure 1 is a schematic diagram of a user interface provided in an embodiment of this application;

[0038] Figure 2 is a schematic diagram of the structure of an electronic device provided in this embodiment;

[0039] Figure 3 is a schematic diagram of the software architecture of an electronic device provided in an embodiment of this application;

[0040] Figure 4 is a schematic diagram of a unified rendering process provided in an embodiment of this application;

[0041] Figure 5 is a schematic diagram of a unified rendering of anomaly scenarios provided in an embodiment of this application;

[0042] Figure 6 is a schematic diagram of the main thread and rendering thread included in the unified rendering process provided in an embodiment of this application;

[0043] Figure 7 is a schematic diagram of the main thread and rendering thread included in the unified rendering process provided in another embodiment of this application;

[0044] Figure 8 is a schematic diagram of the process of updating the unified rendering tree by the main thread according to an embodiment of this application;

[0045] Figure 9 is a schematic diagram of node relationships in a unified rendering tree provided in an embodiment of this application;

[0046] Figure 10 is a schematic diagram of a marked rendering node provided in an embodiment of this application;

[0047] Figure 11 is a schematic diagram of the data structure for drawing operations in a rendering thread provided in an embodiment of this application;

[0048] Figure 12 is a schematic diagram of instructions in the main thread and rendering thread provided in an embodiment of this application;

[0049] Figure 13 is a schematic diagram of the rendering thread converting the data format of the drawing operation according to an embodiment of this application;

[0050] Figure 14 is a schematic diagram of the rendering thread converting the data format of the drawing operation according to another embodiment of this application;

[0051] Figure 15 is a schematic flowchart of an image frame generation method provided in an embodiment of this application;

[0052] Figure 16 is a schematic diagram of the operation flow of the main thread and the rendering thread provided in an embodiment of this application;

[0053] Figure 17 is a schematic flowchart of an image frame generation method provided in another embodiment of this application;

[0054] Figure 18 is a schematic diagram of the operation flow of the main thread and the rendering thread provided in another embodiment of this application;

[0055] Figure 19 is a schematic diagram of the frame rate of the generated image frame provided in an embodiment of this application;

[0056] Figure 20 is a schematic flowchart of an image frame generation method provided in another embodiment of this application;

[0057] Figure 21 is a schematic diagram of the operation flow of the main thread and the rendering thread provided in another embodiment of this application;

[0058] Figure 22 is a schematic diagram of the frame rate of the generated image frame provided in another embodiment of this application;

[0059] Figure 23 is a flowchart of the main thread generating a first drawing instruction list according to an embodiment of this application;

[0060] Figure 24 is a schematic diagram of the process of the main thread generating a first drawing instruction list according to an embodiment of this application;

[0061] Figure 25 is a flowchart of a rendering thread generating a first image frame according to a first drawing instruction list provided in an embodiment of this application;

[0062] Figure 26 is a schematic diagram of the process by which a rendering thread generates a first image frame according to a first drawing instruction list, provided in an embodiment of this application.

[0063] Figure 27 is a schematic diagram of the chip system provided in an embodiment of this application. Detailed Implementation

[0064] It should be understood that in the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0065] In this embodiment, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0066] To facilitate understanding, the relevant terms and concepts involved in the embodiments of this application will be introduced below.

[0067] (1) User interface

[0068] A user interface (UI) is the interface through which applications and operating systems interact and exchange information with users. It facilitates the conversion between the internal form of information and a user-readable format. In other words, a user interface is the source code written in a specific computer language such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on an electronic device, ultimately presenting content that the user can recognize. In short, the existence of a user interface makes interaction with electronic devices more convenient.

[0069] (2) Graphical User Interface

[0070] A graphical user interface (GUI) is a common form of user interface, which is a user interface related to computer operation displayed graphically. GUIs can display visual interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets on the screen of electronic devices.

[0071] (3) Display Interface

[0072] In this embodiment, the display interface can also be referred to as an image frame or screen interface, specifically referring to the content presented on the screen by the electronic device. When the electronic device has multiple foreground applications, the display interface is the result of combining the user interfaces of the multiple foreground applications.

[0073] (4) Vertical synchronization signal

[0074] Vertical synchronization (VSync) is a concept in computer graphics used to control the output and display of image frames. Its main function is to eliminate screen tearing, which occurs when the computer graphics and the monitor's refresh rate are out of sync, resulting in two parts of the image refreshing at different times. Specifically, before displaying the next image frame, the monitor's controller sends a VSync signal. The frequency of the VSync signal is related to the refresh rate of the electronic device; for example, the frequency of the VSync signal may be the same as the refresh rate of the electronic device's screen.

[0075] VSync signals typically include software VSync signals and hardware VSync signals, which are usually synchronized periodically. For example, software VSync signals include VSync-APP and VSync-RS. VSync-APP is used to trigger the application's rendering process, while VSync-RS is used to trigger the unified rendering process of the render service (RS). Hardware VSync signals include VSync-HW, which is used to trigger the display's refresh process.

[0076] (5) Rendering Tree

[0077] A render tree is a data structure used to generate the user interface of an application. It records some or all of the information needed to generate a frame of the application's user interface. When an application has multiple windows or is displayed on different display areas (such as different screens), multiple render trees need to be generated independently.

[0078] Each render tree can contain multiple render nodes, each including render attributes and a list of draw operation structs (DrawOPs). Render attributes include information such as the position, size, and opacity of the view to be rendered by the render node within the surface. The draw operation list records the draw operations and their drawing order. Draw operations determine the content of the view to be rendered by the render node, such as lines, rectangles, and circles.

[0079] The technical solutions provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0080] With the development of computer technology, electronic devices support running one or more applications (Apps) simultaneously in the foreground. For ease of description, this embodiment refers to the application running in the foreground as a foreground application. Since each foreground application needs to display its user interface on the screen, but the display area of ​​an electronic device's screen is limited, the content actually displayed by the electronic device (i.e., image frames) is the composite display effect of the user interfaces of each foreground application according to specific rules. It should be noted that these specific rules can be determined based on the system configuration of the electronic device or user operation; this embodiment does not impose any limitations on them.

[0081] For example, as shown in Figure 1, the foreground applications of an electronic device include a status bar, a messaging application, and a video application. In the image frame of the electronic device, the user interface of the status bar is located at the top of the image frame; the user interfaces of the messaging application and the video application are displayed in a split screen according to the user's instructions, and occupy the upper and lower halves of the remaining image frame, respectively.

[0082] During operation, the user interface of a foreground application typically needs to constantly change. This includes adjusting the displayed content and its position based on the application's configuration or user actions. For example, as shown in Figure 1, the status bar needs to refresh its user interface based on information such as cellular signal type, Wi-Fi connection status, battery level, and time. Messaging applications need to refresh their user interface based on user input and received messages. Video applications need to refresh their user interface based on the video frames to be played or user actions.

[0083] Since the image frames of an electronic device are the composite display effect of the user interfaces of various foreground applications, the image frames of the electronic device also need to change as the user interfaces of the foreground applications change. Therefore, before refreshing the image frames each time, the electronic device needs to generate new image frames based on the running status of all foreground applications.

[0084] The process of an electronic device generating an image frame is illustrated below.

[0085] In this embodiment, the electronic device includes a terminal device, which can also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device can be a mobile phone, smart TV, wearable device, tablet computer, computer with wireless transceiver capabilities, virtual reality (VR) terminal device, augmented reality (AR) terminal device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, etc. The embodiments of this application do not limit the specific technology or device form used in the terminal device.

[0086] Figure 2 is a schematic diagram of the structure of an electronic device provided in this embodiment. The electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, buttons 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, a barometric pressure sensor 280C, a magnetic sensor 280D, an accelerometer sensor 280E, a distance sensor 280F, a proximity sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, etc.

[0087] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0088] For example, when the electronic device 200 is a mobile phone or a tablet computer, it may include all the components shown in the figure, or it may include only some of the components shown in the figure.

[0089] Processor 210 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0090] The controller can be the nerve center and command center of the electronic device 200. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0091] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0092] The charging management module 240 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 240 receives charging input from the wired charger via a USB interface 230. In some wireless charging embodiments, the charging management module 240 receives wireless charging input via the wireless charging coil of the electronic device 200. While charging the battery 242, the charging management module 240 can also supply power to the electronic device via the power management module 241.

[0093] The power management module 241 connects the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, providing power to the processor 210, internal memory 221, external memory, display 294, camera 293, and wireless communication module 260, etc. The power management module 241 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance).

[0094] In some other embodiments, the power management module 241 may also be located within the processor 210. In other embodiments, the power management module 241 and the charging management module 240 may also be located in the same device.

[0095] The wireless communication function of electronic device 200 can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor, and baseband processor.

[0096] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 200 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0097] The mobile communication module 250 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G on the electronic device 200. The mobile communication module 250 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1.

[0098] In some embodiments, at least some functional modules of the mobile communication module 250 may be disposed in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 250 and at least some modules of the processor 210 may be disposed in the same device.

[0099] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 270A, receiver 270B, etc.) or displays images or videos through the display screen 294. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 210 and may be housed in the same device as the mobile communication module 250 or other functional modules.

[0100] The wireless communication module 260 can provide solutions for wireless communication applications on the electronic device 200, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 210. The wireless communication module 260 can also receive signals to be transmitted from processor 210, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0101] In some embodiments, antenna 1 of electronic device 200 is coupled to mobile communication module 250, and antenna 2 is coupled to wireless communication module 260, enabling electronic device 200 to communicate with networks and other devices via wireless communication technology. Wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. GNSS can include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0102] Electronic device 200 implements display functions through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0103] Display screen 294 is used to display images, videos, etc., such as image frames in the embodiments of this application. Display screen 294 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 200 may include one or N displays screens 294, where N is a positive integer greater than 1.

[0104] Electronic device 200 can perform shooting functions through ISP, camera 293, video codec, GPU, display screen 294 and application processor.

[0105] The ISP (Image Signal Processor) is used to process data fed back from the camera 293. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 293.

[0106] Camera 293 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 200 may include one or N cameras 293, where N is a positive integer greater than 1.

[0107] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 200 selects a frequency, the DSP is used to perform Fourier transforms on the frequency energy.

[0108] Video codecs are used to compress or decompress digital video. Electronic device 200 may support one or more video codecs. Thus, electronic device 200 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0109] An NPU (Neural Processing Unit) is a neural network (NN) computing processor that, by borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, rapidly processes input information and can continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0110] In this embodiment of the application, the NPU or other processor can be used to perform operations such as analysis and processing of images in the video stored in the electronic device 200.

[0111] The external storage interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200. The external memory card communicates with the processor 210 through the external storage interface 220 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0112] Internal memory 221 can be used to store computer executable program code, which includes instructions. Processor 210 executes various functional applications and data processing of electronic device 200 by running the instructions stored in internal memory 221. Internal memory 221 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function (such as sound playback, image playback, etc.). The data storage area may store data created during the use of electronic device 200 (such as audio data, phonebook, etc.).

[0113] In addition, the internal memory 221 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0114] Electronic device 200 can implement audio functions through audio module 270, speaker 270A, receiver 270B, microphone 270C, headphone jack 270D, and application processor.

[0115] The audio module 270 is used to convert digital audio signals into analog audio signals for output, and also to convert analog audio inputs into digital audio signals. The audio module 270 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 270 may be located in the processor 210, or some functional modules of the audio module 270 may be located in the processor 210.

[0116] The speaker 270A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 200 can listen to music or hands-free calls through the speaker 270A. For example, the speaker can play the comparison analysis results provided in the embodiments of this application.

[0117] The receiver 270B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 200 answers a telephone call or voice message, the receiver 270B can be brought close to the ear to listen to the voice.

[0118] Microphone 270C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 270C, inputting the sound signal into microphone 270C. Electronic device 200 may have at least one microphone 270C. In some embodiments, electronic device 200 may have two microphones 270C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 200 may also have three, four, or more microphones 270C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0119] The headphone jack 270D is used to connect wired headphones. The headphone jack 270D can be a USB 230 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0120] Buttons 290 include a power button, volume buttons, etc. Buttons 290 can be mechanical buttons or touch-sensitive buttons. Electronic device 200 can receive button input and generate key signal inputs related to user settings and function control of electronic device 200.

[0121] Motor 291 can generate vibration alerts. Motor 291 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can be corresponding to touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 291 can also correspond to different vibration feedback effects for touch operations applied to different areas of the display screen 294. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0122] Indicator 292 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0123] The SIM card interface 295 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 295 to make contact with and separate from the electronic device 200. The electronic device 200 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 295 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 295 is also compatible with different types of SIM cards. The SIM card interface 295 is also compatible with external memory cards. The electronic device 200 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 200 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 200 and cannot be separated from the electronic device 200.

[0124] Figure 3 is a schematic diagram of the software architecture of an electronic device according to an embodiment of this application. Referring to Figure 3, the layered architecture divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the software architecture of the electronic device is divided into four layers, from top to bottom: the application layer, the application framework layer, the system library, and the kernel layer.

[0125] The application layer can include a series of application packages. As shown in Figure 3, application packages can include camera, gallery, calendar, call, map, navigation, WLAN, music, desktop, gallery, call, navigation, Bluetooth, and video, etc.

[0126] An application's process includes at least a main thread, also known as the UI thread, which is used to generate a render tree based on the application's runtime behavior. For example, if the application supports off-screen rendering, the process also includes a render thread, which generates the user interface from the render tree. This user interface is typically represented internally as a bitmap.

[0127] In some embodiments, if a foreground application uses UniRender, then when each VSync signal (such as VSync-App) arrives, the main thread of the foreground application generates a render tree based on the running status of the foreground application and sends the render tree to the application framework layer for further processing.

[0128] In other embodiments, if a foreground application uses off-screen rendering, then when each VSync signal arrives, the main thread of the foreground application first generates a render tree, and then the rendering thread of the foreground application draws the user interface based on the render tree and sends the generated user interface to the application framework layer for further processing.

[0129] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example, as shown in Figure 3, the application framework layer may include window management services, display management services, content providers, view systems, phone managers, resource managers, notification managers, etc.

[0130] The window management service is responsible for starting, adding, and deleting windows. It can determine the application displayed on the window, as well as the creation, destruction, and attribute changes of the application's layers. It can also determine whether there is a status bar, lock the screen, and capture the screen.

[0131] The display management service can obtain the number and size of display areas, and is responsible for starting, adding, and deleting display areas.

[0132] Content providers store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, phone calls made and received, browsing history and bookmarks, phone books, etc.

[0133] A phone manager is used to provide communication functions for electronic devices. For example, it manages call status (including connection and disconnection).

[0134] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, etc.

[0135] The notification manager allows applications to display notifications in the status bar. These can be used to convey informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. Additionally, the notification manager can also display notifications as icons or scrolling text in the system's top status bar, such as notifications from background applications, or as dialog-style notifications on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0136] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. An image frame can consist of one or more views. For example, a view including a text message notification icon can include views for displaying text and views for displaying images.

[0137] In this implementation, the view system also includes a functional module for unified rendering: RenderService (RS). RenderService can receive rendering trees from one or more foreground applications by running a unified rendering process (UniRender), and merge these one or more rendering trees into a unified rendering tree. Additionally, UniRender can synchronize layer information through the window management service, such as layer creation, destruction, and attribute changes, and can also synchronize display area information, such as screen size, from the display management service.

[0138] It should be noted that in this embodiment, the UniRender process and the application process are independent processes. The application and the UniRender process interact with each other through inter-process communication (IPC). For details regarding the UniRender process, please refer to patent applications No. 2021114105136 and No. 202111410643X, both entitled "Interface Generation Method and Electronic Device," which will not be repeated here.

[0139] Optionally, in some embodiments of this application, the view system further includes a SurfaceFlinger. On an electronic device configured with a whitelist, if the foreground application is not on the whitelist, after the main thread of the foreground application generates the render tree, the rendering thread of the foreground application generates the user interface, which is then handed over to SurfaceFlinger for layer composition to generate image frames for display.

[0140] Optionally, in some embodiments of this application, when the foreground application includes both whitelisted and non-whitelisted applications, the user interface generation of the whitelisted applications is handled by UniRender. After UniRender generates the user interface, it passes the user interface to SurfaceFlinger. SurfaceFlinger then composites the user interface with the user interfaces of other non-whitelisted applications to generate an image frame for display.

[0141] System libraries can include multiple functional modules. Examples include a surface manager, media libraries, and graphics processing libraries. See below for details.

[0142] The Surface Manager is used to manage the display subsystem and can merge two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0143] The media library supports playback and recording of various commonly used audio and video formats, as well as still image files. It supports multiple audio and video coding formats, such as Moving Picture Experts Group 4 (MPEG4), H.264, Moving Picture Experts Group Audio Layer III (MP3), Advanced Audio Coding (AAC), Adaptive Multi-rate (AMR), Joint Photographic Experts Group (JPEG), and lossless compressed bitmap graphics format (PNG).

[0144] The graphics processing library includes a 3D graphics processing library and a 2D graphics engine. The 3D graphics processing library is used for 3D graphics drawing, image rendering, layer compositing, and layer processing. The 2D graphics engine is a drawing engine for 2D graphics.

[0145] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, sensor drivers, audio drivers, and virtual card drivers.

[0146] Based on the hardware and software architecture of the electronic device illustrated in the above embodiments, the electronic device can generate image frames using either off-screen rendering or unified rendering. However, compared to off-screen rendering, unified rendering reduces the number of rendering passes, lowers the power consumption of the electronic device, and improves the user experience. Therefore, unified rendering is currently the more common method for generating image frames.

[0147] In some embodiments, the process of generating image frames using a unified rendering method by an electronic device is shown in Figure 4.

[0148] Referring to Figure 4, taking the foreground application as an example that includes applications 1 to 4, the process of generating image frames using a unified rendering method on electronic devices specifically includes the following processes (1) to (4).

[0149] (1) The front-end application sends the rendering tree to the RenderService.

[0150] Upon each VSync signal, each foreground application generates a render tree using its main thread based on its runtime status and sends the render tree across processes to the RenderService. For example, applications 1 through 4 each generate render trees 1 through 4 using their respective main threads and send them to the RenderService.

[0151] For example, the process of the main thread of the foreground application generating the render tree includes: disabling the view hierarchy of the foreground application. The view hierarchy represents the parent-child relationships between views in the user interface. Then, the main thread traverses the views of the foreground application, re-layouts each view, determines the attribute information and drawing operations (DrawOPs) of each view, and treats each view's attribute information and drawing operations (DrawOPs) as a render node, forming the render tree.

[0152] It's important to note that a view is the fundamental element that constitutes the user interface of an application. A single control on the user interface can correspond to one or more views. The user interface displayed by an application consists of multiple nested views, with parent-child relationships between different views. Therefore, the parent-child relationships between rendering nodes in the rendering tree generated by the main thread traversing the views are the same as the parent-child relationships between the views themselves. In other words, the parent-child relationships between views determine the nesting relationships between different rendering nodes, thus ensuring that the rendering thread (such as the application's rendering thread or the rendering thread of a unified rendering process) can correctly render the user interface when generating the interface based on the rendering tree.

[0153] (2) RenderService merges the various rendering trees into a unified rendering tree.

[0154] RenderService uses UniRender to receive the rendering trees sent by each foreground application, and obtains window control information and display area information from the window management service and display management service. Then, it combines the rendering trees of foreground applications located in the same display area into a unified rendering tree based on the window control information and display area information. Among them, the window control information includes the Z-order of the window, and the display area information includes the identification information of the display area.

[0155] In some embodiments, UniRender can generate a new root rendering node and use the original root rendering nodes of each rendering tree as child nodes at the same level as the new root rendering node, thereby generating a unified rendering tree. For example, as shown in Figure 4, UniRender generates a new root rendering node and uses the original root rendering nodes of rendering trees 1 to 4 as child nodes at the same level as the new root rendering node, thereby generating a unified rendering tree.

[0156] Optionally, during the generation of the unified render tree, UniRender can first determine the top and bottom occlusion relationships between the layers of the foreground application based on the Z-order of the corresponding layers of the foreground application, and then delete the rendering nodes corresponding to the completely occluded views during the generation of the unified render tree.

[0157] (3) RenderService generates image frames based on the unified rendering tree.

[0158] After generating the unified render tree, RenderService's UniRender calls the layer processing library to traverse each render node sequentially. It then performs drawing operations on each render node in the unified render tree within the unified cache, generating an image frame. It's important to note that the display information for this image frame is carried on a bitmap, and the electronic device's display can then render the image frame on the screen based on the bitmap.

[0159] (4) RenderService sends the image frame to the display subsystem.

[0160] The RenderService's UniRender class stores the bitmap of the image frame into a frame buffer. The display subsystem (DSS) retrieves the bitmap from the frame buffer and displays the image frame on the screen based on the bitmap. For example, this frame buffer can be an on-screen buffer.

[0161] Through the above processes (1) to (4), the electronic device can merge the user interfaces of various foreground applications into image frames for display through unified rendering. However, in the above unified rendering process, UniRender needs to perform a lot of operations within a rendering cycle, and the rendering process is relatively complex, resulting in low rendering efficiency.

[0162] For example, as shown in Figure 5, when the refresh rate of an electronic device is 120fps (frames per second), the time interval between two adjacent VSync signals is only 8.3ms, meaning that one rendering cycle is only 8.3ms. Within one rendering cycle, UniRender needs to perform operations such as generating a unified rendering tree, generating image frames based on the unified rendering tree, and sending the image frames to the display in order to complete the rendering of one image frame.

[0163] It is understandable that, due to the short unified rendering cycle and the complexity of UniRender's rendering process, UniRender may not be able to complete the rendering work within a unified rendering cycle, resulting in the inability to send the bitmap of the image frame to the display in a timely manner. This can lead to frame drops or lag on electronic devices during display, resulting in a poor user experience.

[0164] Therefore, this application provides a method for generating image frames to improve the rate of generating image frames in a unified rendering mode, thereby improving the situation of frame drops or stuttering during the display of electronic devices and enhancing the user experience.

[0165] In this embodiment, referring to Figure 6, the UniRender unified rendering process includes a main thread and a rendering thread, which share the tasks required by UniRender. The main thread performs preprocessing of information during image frame generation and synchronizes the preprocessing results to the rendering thread. The rendering thread then generates image frames based on the preprocessing results from the main thread.

[0166] It's worth noting that both the main thread and the rendering thread support serial processing of different image frames. Furthermore, the main thread and the rendering thread can also run in parallel while processing different image frames.

[0167] In some embodiments of this application, as shown in Figure 7, the tasks performed by the main thread and rendering thread of UniRender specifically include the following (i) and (ii).

[0168] (I) UniRender's main thread

[0169] In this embodiment, the main thread of UniRender (hereinafter referred to as the main thread) is used to update the unified rendering tree according to the application message, and to generate a list of drawing instructions for image frames according to the updated unified rendering tree. Specifically, it includes the following contents (1) to (2).

[0170] (1) Update the unified rendering tree based on the application message.

[0171] Application messages refer to messages sent by at least one foreground application. These messages can be the foreground application's rendering tree or node change information for at least one rendering node within that tree. For example, the node change information could be updated attribute information of the rendering node, drawing operations, or animation step information. The animation step information includes the movement information of the rendering node across multiple user interfaces, such as its starting position, step rate, and ending position.

[0172] In some embodiments, the foreground application generates a render tree upon receiving the VSync signal and sends the render tree to the main thread. Based on this, the process by which the main thread updates the unified render tree according to application messages can be: receiving and merging the render trees sent by various foreground applications to obtain a new unified render tree.

[0173] In other embodiments, since the changes in the user interface between two adjacent frames of the foreground application are usually minor or even nonexistent in most cases, the foreground application, upon receiving the VSync signal, can identify the changed rendering nodes (i.e., the changed rendering nodes) and only send the node change information of these changed rendering nodes to the main thread. This reduces the amount of data exchanged across processes and saves power consumption of the electronic device. Based on this, the process by which the main thread updates the unified rendering tree according to application messages can be: receiving node change information sent by each foreground application and updating the unified rendering tree based on the node change information. For example, based on the unified rendering tree of the previous frame's user interface, adding rendering nodes, updating the rendering attributes and drawing operations of rendering nodes, or deleting rendering nodes, etc.

[0174] In one example, the unified rendering tree in the main thread is shown in Figure 8(a). This unified rendering tree includes a root rendering node 0. The child nodes of root rendering node 0 include rendering nodes 1, 5, and 7. The child nodes of rendering node 1 include rendering nodes 2 and 4. The child nodes of rendering node 2 include rendering node 3. The child nodes of rendering node 5 include rendering node 6. The child nodes of rendering node 7 include rendering node 8. The child nodes of rendering node 8 include rendering node 9. Suppose the main thread receives node change information from the application regarding rendering nodes 2, 8, and 9. The node change information for rendering node 2 is: background color changed to color A; the node change information for rendering node 8 is: node deleted; and the node change information for rendering node 9 is: image scale changed to scale B. Then, the unified rendering tree updated by the main thread is shown in Figure 8(b).

[0175] It's important to note that in the rendering tree, a parent node includes its child nodes, or in other words, the drawing operation of a parent node includes the drawing operation of its child nodes. Therefore, when the main thread updates a target rendering node, it needs to start from the root node of the unified rendering tree and traverse the upper-level nodes of the target rendering node in descending order of hierarchy before it can find the target rendering node and update its node information.

[0176] Taking the unified rendering tree shown in Figure 8(a) as an example, the node relationships between the unified rendering trees can be seen in Figure 9. Specifically, it includes the following: The drawing operation of the root rendering node 0 (i.e., RenderNodeDrawOp 0) includes the drawing operations of rendering nodes 1, 5, and 7 (i.e., RenderNodeDrawOp 1, 5, and 7). The drawing operation of rendering node 1 (i.e., RenderNodeDrawOp 1) includes the drawing operations of rendering nodes 2 and 4 (i.e., RenderNodeDrawOp 2 and 4). The drawing operation of rendering node 2 (i.e., RenderNodeDrawOp 2) includes the drawing operation of rendering node 3 (i.e., RenderNodeDrawOp 3). The drawing operation of rendering node 5 (i.e., RenderNodeDrawOp 5) includes the drawing operation of rendering node 6 (i.e., RenderNodeDrawOp 6). The drawing operation of rendering node 7 (i.e., RenderNodeDrawOp 7) includes the drawing operation of rendering node 8 (i.e., RenderNodeDrawOp 8). The drawing operation of rendering node 8 (i.e., RenderNodeDrawOp No. 8) includes the drawing operation of rendering node 9 (i.e., RenderNodeDrawOp No. 9).

[0177] Therefore, when the main thread updates render node 2, it needs to find render node 1 in render node 0 before it can find render node 2 in render node 1 and update its node information. Similarly, when the main thread deletes render node 8, it needs to find render node 7 in render node 0 before it can delete render node 8 from render node 7.

[0178] Optionally, during the process of updating the unified rendering tree, the main thread can also mark changed rendering nodes and all their parent rendering nodes for subsequent processing. Taking the updated unified rendering tree shown in Figure 8(b) as an example, the main thread can mark rendering node 2, its parent rendering nodes 1 and 0, and rendering node 7. The main thread does not mark deleted rendering nodes. For example, rendering nodes 8 and 9 are not marked.

[0179] It should be noted that in the rendering tree (including the unified rendering tree), the calling interface of the child node is in the parent node. Therefore, in order to find the changed rendering node during the traversal of the rendering nodes, the main thread needs to mark all the higher-level rendering nodes of the changed rendering node.

[0180] (2) Generate a list of drawing instructions based on the updated unified rendering tree.

[0181] In some embodiments, the main thread iterates through each rendering node, treating each node as a drawing operation and copying it to the unified rendering tree's drawing instruction list (DrawCmdList). That is, the main thread performs a full traversal of the unified rendering tree, generating a complete drawing instruction list for the entire tree. It's understandable that this full traversal process is time-consuming, and the large number of drawing operations in the complete drawing instruction list leads to a longer drawing time for subsequent image frames.

[0182] In other embodiments, the main thread traverses the updated unified render tree using the dirty region stack technique, and completes the dirty region calculation and copying of dirty region drawing operations in one traversal to generate an incremental drawing instruction list.

[0183] The dirty region calculation performed by the main thread refers to the main thread identifying the rendering nodes in the unified rendering tree that need to be redrawn due to changes. Taking the updated unified rendering tree shown in Figure 8(b) as an example, the main thread determines that the dirty regions include rendering node 2 and rendering node 7 after the dirty region calculation. Rendering node 7 becomes dirty because it lost its child nodes (i.e., rendering nodes 8 and 9), which requires redrawing.

[0184] Additionally, copying the dirty area drawing operation involves copying the drawing operation corresponding to the dirty area to the drawing instruction list. Taking the update process shown in Figure 7 as an example, since only the background of rendering node 2 needs to be redrawn, the main thread copies the updated background drawing instruction list (background DrawOp List) of rendering node 2 to the main thread's drawing instruction list. Furthermore, since rendering node 7 has lost its child nodes and is now exposed on the image frame again, rendering node 7 as a whole is copied to the main thread's drawing instruction list as a drawing operation.

[0185] It should be noted that, normally, the unified rendering process directly traverses the rendering nodes of the unified rendering tree to draw image frames. However, in this embodiment, to achieve parallel operation of the main thread and the rendering thread, it is necessary to copy the rendering nodes to be redrawn, for example, by copying the rendering nodes as a whole to the drawing instruction list. It can be understood that after copying the rendering nodes as a whole to the drawing instruction list, the subsequent execution of the rendering thread will no longer depend on the unified rendering tree; even if individual rendering nodes in the unified rendering tree are modified, the rendering thread can still run normally.

[0186] Optionally, to improve the efficiency of the main thread in generating the drawing instruction list, the main thread can traverse only the marked rendering nodes to generate the drawing instruction list. For example, for the updated unified rendering tree shown in Figure 8(b), its marked rendering nodes are shown in Figure 10, including rendering node 0, rendering node 1, rendering node 2, and rendering node 7. Therefore, the main thread only needs to traverse the rendering nodes in Figure 10 to generate the drawing instruction list.

[0187] It should be noted that in the above process, the marked rendering nodes are used to identify rendering nodes that have undergone incremental changes, such as newly added rendering nodes, rendering nodes whose attribute information or drawing operations have changed, and rendering nodes whose animation step information has changed, but do not include deleted rendering nodes. Therefore, the process of traversing the marked rendering nodes described above can also be called an incremental traversal process. Similarly, the list that only includes dirty area drawing operations can also be called an incremental drawing instruction list.

[0188] It is understandable that the incremental traversal process can improve the efficiency of generating the drawing instruction list, reduce the CPU resource consumption, and obtain a certain load gain, such as a 5% load gain.

[0189] (ii) UniRender's rendering thread

[0190] In this embodiment, the UniRender rendering thread is mainly used to draw image frames according to the drawing instruction list from the main thread and send the image frames to the display. Specifically, it includes the following contents (3) to (4).

[0191] (3) Draw image frames according to the list of drawing instructions from the main thread.

[0192] First, after generating the drawing instruction list for the unified rendering tree, the main thread needs to synchronize this list with the rendering thread. For example, the main thread can copy the drawing instruction list and send it to the rendering thread; or, the main thread can send the address information of the drawing instruction list to the rendering thread; or, the main thread can transfer a pointer to the address of the drawing instruction list to the rendering thread so that the rendering thread can call the drawing instruction list. Compared to copying the drawing instruction list or its address information and sending it to the rendering thread, the pointer transfer method only takes the time of pointer assignment, making it more convenient and efficient.

[0193] It's important to note that the data structures used for drawing operations differ between the main thread and the rendering thread of UniRender. For example, as shown in Figure 11, the main thread operates on the RenderNode data structure, which is the rendering node. The rendering thread, on the other hand, operates on the RenderNodeDrawble data structure, which is the rendering node that can be drawn. In other words, RenderNodeDrawble holds RenderNode information. Or, to put it another way, RenderNodeDrawble strongly references RenderNode.

[0194] Referring to Figure 12, in the RenderNode of the main thread, the data structure for drawing operations is a DrawOp List. The DrawOp List includes the drawing operations required to draw the view, such as drawing circles, lines, and directions, but does not include attribute information such as the view's size, color, or transparency. For example, the DrawOp List includes various types of drawing operations such as geometry DrawOp List, background DrawOp List, content DrawOp List, child node RenderNode DrawOp List, foreground DrawOp List, and overlay DrawOp List.

[0195] Referring to Figure 12, in the RenderNodeDrawable of the rendering thread, the data structure for drawing operations is a drawing DrawOp List. Compared to a regular DrawOp List, a drawing DrawOp List can be understood as adding view attribute information such as size, color, and transparency to the view's drawing operations. For example, the drawing DrawOp List includes various types such as geometry drawing DrawOp List, background drawing DrawOp List, content drawing DrawOp List, child node drawing RenderNodeDrawOp List, foreground drawing DrawOp List, and overlay drawing DrawOp List.

[0196] Therefore, after receiving the drawing instruction list, the rendering thread needs to prepare the drawing elements (i.e., prepare the attribute information of the rendering node) for the drawing operation DrawOp List of each rendering node according to the node information of each rendering node in the drawing instruction list, and generate the corresponding drawing DrawOp List.

[0197] For example, as shown in Figure 13, after the main thread sends the entire rendering node 2 as a drawing operation to the rendering thread, the rendering thread combines the attribute information of rendering node 2 to convert the background DrawOp List in rendering node 2 into a background drawing DrawOp List.

[0198] For example, as shown in Figure 14, after the main thread sends the entire rendering node 7 as a drawing operation to the rendering thread, the rendering thread combines the attribute information of the rendering node 7 and converts the child node RenderNodeDrawOp List in the rendering node 7 into the child node drawing RenderNodeDrawOp List.

[0199] Subsequently, the rendering thread iterates through each drawing operation in the drawing instruction list and generates a drawing result. For ease of description, this embodiment refers to the result obtained from drawing according to the incremental drawing instruction list as the incremental drawing result, or the dirty area drawing result. For example, the rendering thread iterates through the drawing operations of rendering node 2 to change the background color of its view. Or, it iterates through the drawing operations of rendering node 7 to regenerate the drawing result of its view.

[0200] Finally, the rendering thread generates the current image frame based on the rendering result of the previous image frame (referred to as the previous frame rendering result) and the incremental rendering result obtained this time. For example, since the current rendering result is the result of rendering the dirty region, the rendering thread can use the current rendering result to replace the corresponding part in the previous frame rendering result to obtain the rendering result of the current image frame. For instance, the rendering result of rendering node 2 in this frame is used to replace the rendering result of rendering node 2 in the previous frame rendering result; and the rendering result of rendering node 7 in this frame is used to replace the rendering result of rendering node 7 in the previous frame rendering result to obtain the rendering result of the current image frame.

[0201] (4) The rendering thread sends the image frame to the display.

[0202] The rendering thread stores the bitmap of the image frame into the frame buffer. The DSS can then retrieve the bitmap of the image frame from the frame buffer and display the image frame on the screen based on the bitmap.

[0203] It should be noted that this embodiment does not limit the specific tasks executed by the UniRender main thread and rendering thread. In other words, this embodiment does not limit the preprocessing results sent by the UniRender main thread to the rendering thread. Therefore, the UniRender main thread and rendering thread can also be configured to perform the following operations.

[0204] In some embodiments, the main thread of UniRender is used to execute the above process (1), that is, to update the unified rendering tree according to the application message and to send the updated unified rendering tree to the rendering thread. The rendering thread is used to execute the above processes (2) to (4), that is, to generate a drawing instruction list (e.g., a full drawing instruction list and an incremental drawing instruction list) according to the updated unified rendering tree, to draw image frames according to the drawing instruction list, and to send the image frames to the display.

[0205] In other embodiments, the main thread of UniRender is used to perform the above process (1), namely, updating the unified rendering tree according to the application message and sending the updated unified rendering tree to the rendering thread. The rendering thread then directly traverses the updated unified rendering tree to generate image frames and displays the image frames.

[0206] After introducing UniRender's main thread and rendering thread, the following is an exemplary description of how an electronic device runs its main thread and rendering thread in parallel to generate image frames.

[0207] Figure 15 is a schematic flowchart of an image frame generation method provided in an embodiment of this application. Taking UniRender generating a first image frame and a second image frame as an example, the method specifically includes the following steps S1501 to S1504.

[0208] S1501, after the main thread receives the first VSync signal, it generates the first preprocessing result of the first image frame.

[0209] In some embodiments, the first preprocessing result is a unified rendering tree of the first image frame.

[0210] In other embodiments, the first preprocessing result is a first drawing instruction list for the first image frame. This first drawing instruction list can be a full drawing instruction list or an incremental drawing instruction list; this embodiment does not impose any limitations on this. The incremental drawing instruction list includes drawing instructions for newly added rendering nodes in the unified rendering tree of the first image frame, and / or drawing instructions for rendering nodes with changed information. The full drawing instruction list includes drawing instructions for all rendering nodes in the unified rendering tree of the first image frame, where the unified rendering tree is the rendering tree obtained by merging the rendering trees of all applications performing unified rendering.

[0211] S1502, the main thread synchronizes the first preprocessing result to the rendering thread.

[0212] In some embodiments, the main thread sends the first preprocessing result to the rendering thread.

[0213] In other embodiments, the main thread sends the address information of the first preprocessed result to the rendering thread, which is used to access the first preprocessed result.

[0214] In some other embodiments, the main thread transfers the pointer to the first preprocessing result from the main thread to the rendering thread. It can be understood that the time consumed by data transfer via pointer transfer is only the time consumed by pointer assignment, making the process more convenient and efficient.

[0215] S1503, the rendering thread generates the first image frame based on the first preprocessing result.

[0216] It should be noted that for the same image frame (e.g., the first image frame), the main thread needs to generate the preprocessing result of that image frame first, and then the rendering thread can generate the image frame based on that preprocessing result. Therefore, the main thread and the rendering thread cannot run in parallel when generating the same image frame.

[0217] S1504, during the execution of the rendering thread, after the main thread receives the second VSync signal, it generates the second preprocessing result of the second image frame, which is used to generate the second image frame.

[0218] In the above process, after the main thread generates the first preprocessing result of the first image frame, it will sequentially process the second preprocessing result of the second image frame. This may result in the second preprocessing result affecting or overwriting the first preprocessing result. Therefore, in order to ensure the parallel operation of the main thread and the rendering thread, the main thread needs to send the first preprocessing result to the rendering thread to prevent the main thread from changing the first preprocessing result while processing the second image frame.

[0219] For example, the main thread needs to send the first drawing instruction list of the first image frame to the rendering thread before it can start generating the second drawing instruction list of the second image frame. This is to prevent the first drawing instruction list from being modified by the main thread while the rendering thread is calling the first drawing instruction list. Alternatively, the main thread needs to send the unified rendering tree of the first image frame to the rendering thread before it can start updating the unified rendering tree based on the second image frame. This is to prevent the unified rendering tree from being modified by the main thread while the rendering thread is using the unified rendering tree.

[0220] In this embodiment, referring to Figure 16, the main thread can sequentially preprocess the first image frame, the second image frame, the third image frame, and so on, waiting to be drawn, to obtain corresponding preprocessing results, such as the first preprocessing result, the second preprocessing result, the third preprocessing result, and so on, and synchronize them to the rendering thread. The rendering thread can receive each preprocessing result sequentially and generate the corresponding image frame based on the preprocessing result, such as generating the first image frame based on the first preprocessing result, generating the second image frame based on the second preprocessing result, and generating the third image frame based on the third preprocessing result, etc.

[0221] As can be seen from the above process, the main thread and the rendering thread run in parallel when processing different image frames. For example, the main thread's processing of the second image frame and the rendering thread's processing of the first image frame run in parallel. Therefore, the method provided in this embodiment can improve the image frame generation rate, reduce frame drops or stuttering during electronic device display, and improve the user experience.

[0222] In UniRender's process of generating image frames in parallel using the main thread and the rendering thread, the rendering thread can start running at different times. For example, the rendering thread can start running after receiving the VSync signal and synchronously obtaining the list of drawing instructions from the main thread, or it can start running immediately after synchronously obtaining the list of drawing instructions from the main thread.

[0223] The following section uses the example of the preprocessing result sent by the main thread to the rendering thread as a list of drawing instructions to explain the two methods for generating image frames.

[0224] (1) The rendering thread starts running after receiving the VSync signal.

[0225] In this embodiment, as shown in Table 1, the main thread of UniRender starts running after receiving the VSync signal, and the rendering thread starts working after synchronously obtaining the drawing instruction list from the main thread and receiving the VSync signal.

[0226] Table 1

[0227] Figure 17 is a schematic flowchart of an image frame generation method provided in another embodiment of this application. The method specifically includes the following steps S1701 to S1706.

[0228] S1701, the main thread receives the first VSync signal.

[0229] S1702, in response to the first VSync signal, the main thread generates the first drawing instruction list for the first image frame.

[0230] Taking an incremental drawing instruction list as an example, the main thread generates the first drawing instruction list by: updating the unified rendering tree, traversing the updated unified rendering tree to determine dirty regions, and generating an incremental drawing instruction list corresponding to the dirty regions. This incremental drawing instruction list is the aforementioned first drawing instruction list.

[0231] S1703, the main thread synchronizes the first drawing instruction list to the rendering thread.

[0232] For example, the main thread transfers the pointer to the first drawing instruction list to the rendering thread.

[0233] S1704, the rendering thread and the thread receive the second VSync signal.

[0234] It should be noted that both the first and second VSync signals are VSync signals transmitted at a fixed frequency. In other words, the first and second VSync signals have the same content, but are transmitted at different times.

[0235] S1705, in response to the second VSync signal, the rendering thread generates the first image frame according to the first drawing instruction list.

[0236] In this embodiment, both the main thread and the rendering thread are started by the VSync signal, which reduces the impact of the main thread on the rendering thread and improves the independence of the main thread and the rendering thread.

[0237] S1706, in response to the second VSync signal, while the rendering thread generates the first image frame according to the first drawing instruction list, the main thread generates a second drawing instruction list for the second image frame, which is used to generate the second image frame.

[0238] It should be noted that the first image frame and the second image frame are two image frames that the electronic device needs to draw consecutively, and the drawing order of the first image frame is before that of the second image frame. Therefore, as shown in Figure 18, after the main thread hands over the first drawing instruction list of the first image frame to the rendering thread for processing, while the rendering thread is drawing the first image frame according to the first drawing instruction list, the main thread can generate the second drawing instruction list of the second image frame in parallel, thereby improving the efficiency of image frame generation, reducing frame drops or stuttering during electronic device display, and improving the user experience.

[0239] In this embodiment, referring to Figure 19, assuming that the time taken for UniRender to generate a single image frame is constant at T (ms), then the maximum frame rate F supported by UniRender when generating image frames serially is... max1 =1000 / T.

[0240] By splitting UniRender into a main thread and a rendering thread, and with the main thread and rendering thread running in parallel, UniRender supports a maximum frame rate F. max2 It satisfies the following formulas (1) and (2). T UI +T render =T (1)

[0241] Where T is the time taken for UniRender to generate a single image frame, T UI The time T takes to process image frames for the main thread of UniRender render This refers to the time taken for the UniRender rendering thread to process image frames. Additionally, min(T) UI T render ) represents T UI and T render The minimum value in man(T). UI T render ) represents T UI and T render The maximum value in.

[0242] Solving equations (1) and (2) together, we can find that:

[0243] When T UI =T renderAt time T / 2, the main thread and rendering thread achieve the highest degree of parallelism. Calculations show that the maximum frame rate F supported by UniRender after splitting is... max2 =2000 / T, which is the maximum frame rate F supported before the split. max1 Twice that of electronic devices, significantly increasing the rate at which electronic devices generate image frames.

[0244] When T UI When T = T / 3, T render When the frame rate is 2T / 3, the main thread and rendering thread can run in partial parallelism. Calculations show that the maximum frame rate F supported by UniRender after splitting is... max2 =1500 / T, which is the maximum frame rate F supported before the split. max1 It is 1.5 times faster and can also increase the rate at which electronic devices generate image frames.

[0245] (2) The rendering thread starts running after synchronously obtaining the drawing instruction list from the main thread.

[0246] In this embodiment, as shown in Table 2, the main thread of UniRender starts running after receiving the VSync signal, and the rendering thread works after synchronously obtaining the drawing instruction list from the main thread, without waiting for the VSync signal.

[0247] Table 2

[0248] Figure 20 is a schematic flowchart of an image frame generation method provided in another embodiment of this application. The method specifically includes the following steps S2001 to S2006.

[0249] S2001, the main thread receives the first VSync signal.

[0250] S2002, in response to the first VSync signal, the main thread generates the first drawing instruction list for the first image frame.

[0251] In S2003, the main thread synchronizes the first drawing instruction list to the rendering thread.

[0252] For example, the main thread transfers the pointer to the first drawing instruction list to the rendering thread.

[0253] S2004, the rendering thread generates the first image frame according to the first drawing instruction list.

[0254] S2005, the main thread receives the second VSync signal.

[0255] S2006, in response to the second VSync signal, the main thread generates a second drawing instruction list for the second image frame, which is used to generate the second image frame.

[0256] For S2005 and S2006, it should be noted that when the main thread finishes processing the first image frame, the next VSync signal (i.e., the second VSync signal) after the first VSync signal has usually not yet arrived. Therefore, the main thread needs to wait for the second VSync signal while the rendering thread is generating the first image frame. Only after the second VSync signal arrives can the main thread begin generating the second image frame.

[0257] In this embodiment, as shown in Figure 21, the main thread can quickly start the rendering thread to run using the first drawing instruction list, thereby improving the efficiency of image frame generation.

[0258] In this embodiment, referring to Figure 22, assuming that the time taken for UniRender to generate a single image frame is constant at T (ms), then the maximum frame rate F supported by UniRender when generating image frames serially is... max1 =1000 / T. However, after splitting UniRender into a main thread and a rendering thread, since the main thread and the rendering thread run in parallel, one rendering cycle of UniRender is greater than max(T). UI T render But less than T. And, T UI =T render =T / 2, the maximum frame rate F after UniRender splitting max2 It can also reach F max2 =2000 / T. Therefore, the method provided in this embodiment can significantly improve the efficiency of image frame generation.

[0259] To further improve the efficiency of image frame generation, the main thread can use multiple sub-threads to work in parallel to quickly generate a drawing instruction list. For example, the main thread can use multiple sub-threads to generate drawing instruction sub-lists for different foreground applications in parallel, and then merge these drawing instruction sub-lists to obtain the first drawing instruction list for the first image frame. See the diagram below for details.

[0260] Figure 23 is a schematic flowchart of the main thread generating a first drawing instruction list according to an embodiment of this application. Taking the main thread as including a first sub-thread and a second sub-thread running in parallel as an example, the process specifically includes the following steps S2301 to S2304.

[0261] S2301, the main thread receives the first VSync signal.

[0262] S2302, the main thread uses the first sub-thread to determine the first drawing instruction sub-list corresponding to the first part of the first image frame.

[0263] S2303, the main thread uses the second sub-thread to determine the second drawing instruction sub-list corresponding to the second part of the first image frame, and the first sub-thread and the second sub-thread run in parallel.

[0264] S2304, the main thread generates a first drawing instruction list for the first image frame based on the first drawing instruction sublist and the second drawing instruction sublist. For example, the main thread can merge the first drawing instruction sublist and the second drawing instruction sublist to generate the first drawing instruction list for the first image frame.

[0265] It should be noted that, in this embodiment, the first drawing instruction list of the first image frame can be either a full drawing instruction list or an incremental drawing instruction list, and this embodiment does not impose any restrictions on this.

[0266] In one example, the process by which the main thread generates the list of incremental rendering instructions for the first image frame includes:

[0267] First, after receiving the first VSync signal, the main thread updates the unified rendering tree of the first image frame.

[0268] Then, the main thread determines the number M of foreground applications whose user interface has changed based on the updated unified rendering tree. For example, taking the unified rendering tree updated by the main thread as shown in Figure 24, rendering nodes 1-4 belong to window 1, rendering nodes 5-6 belong to window 2, and rendering node 7 belongs to window 3. The main thread, after calculating the dirty regions, determines that rendering nodes 2 and 7 are dirty regions, and the node change information for rendering node 2 is: the background color changes to color A. Rendering node 7 has had its child nodes deleted, becoming a leaf node. Based on this, in the unified rendering tree shown in Figure 24, the main thread determines that there are two windows with changed user interfaces: window 1 and window 3. Since application 1 corresponds to window 1, application 2 corresponds to window 2, and application 3 corresponds to window 3, the main thread determines that the number M of foreground applications with changed user interfaces is 2.

[0269] Subsequently, the main thread uses M sub-threads to determine the drawing instruction sub-lists corresponding to the incremental rendering nodes in the M foreground applications. Taking the unified rendering tree shown in Figure 24 as an example, the main thread uses two sub-threads to process the incremental rendering nodes of window 1 and window 3 respectively. For example, sub-thread 1 processes each incremental rendering node of window 3. Specifically, sub-thread 1 traverses the rendering nodes in window 3 in reverse to perform dirty region calculation, determines the dirty region as rendering node 7, and performs a drawing operation on the entire rendering node 7, copying it to the drawing instruction sub-list 1. And, sub-thread 2 processes each incremental rendering node of window 1. Specifically, sub-thread 2 traverses the rendering nodes in window 1 in reverse to perform dirty region calculation, and copies the background drawing operation of the entire rendering node 2 to the drawing instruction list.

[0270] Finally, the main thread merges the drawing instruction sublists of the M foreground applications into a first drawing instruction list. For example, as shown in Figure 24, the main thread merges the drawing instruction sublist 1 corresponding to window 3 and the drawing instruction sublist 2 corresponding to window 1 into the first drawing instruction list for the entire first image frame.

[0271] In summary, through the processes S2301 to S2304 described above, the main thread can use multiple sub-threads to work in parallel to generate a drawing instruction list, thereby improving the efficiency of image frame generation.

[0272] Alternatively, the rendering thread can employ multiple rendering sub-threads working in parallel to quickly execute the drawing operations in the drawing instruction list and generate image frames. Taking the process of the rendering thread generating the first image frame based on the first drawing instruction list of the first image frame as an example, the specific method is as follows.

[0273] Figure 25 is a flowchart of a rendering thread generating a first image frame according to a first drawing instruction list provided in an embodiment of this application. The process specifically includes the following steps S2501 to S2503.

[0274] S2501, the rendering thread uses the first rendering sub-thread to obtain the first rendering result based on the first part of the first drawing instruction list.

[0275] S2502, the rendering thread uses the second rendering sub-thread to obtain the second rendering result according to the second part of the first rendering instruction list, and the first rendering sub-thread and the second rendering sub-thread run in parallel.

[0276] In some embodiments, the first part of the first drawing instruction list may be a first drawing instruction sublist in the main thread, and the second part may be a second drawing instruction sublist in the main thread.

[0277] In other embodiments, different portions of the first drawing instruction list, such as the first portion and the second portion, may be re-divided by the rendering thread. For example, the rendering thread may split the first drawing instruction list into N portions based on the number of FBOs N.

[0278] S2503, the rendering thread generates the first image frame based on the first drawing result and the second drawing result.

[0279] In this embodiment, the rendering thread can divide the first drawing instruction list into N sub-lists, such as a first part and a second part, based on the number of FBOs. Subsequently, the rendering thread uses N rendering sub-threads, such as the first rendering sub-thread and the second rendering sub-thread mentioned above, to process the different sub-lists of the first drawing instruction list respectively, and finally obtain the first image frame.

[0280] It's important to note that an FBO (Frame Optimizer) is a mechanism for rendering to textures or multiple render targets; it can be understood as a container. It provides a flexible way to perform off-screen rendering, post-processing, and rendering to textures.

[0281] In one example, the rendering thread uses multiple rendering sub-threads to process the first drawing instruction list in parallel. The process of generating the first image frame includes the following:

[0282] First, the rendering thread can create N FBOs and split the first drawing instruction list into N sublists for off-screen rendering. Taking N=3 as an example, the rendering thread can split the first drawing instruction list into sublist 1 to sublist 3, which are respectively: FBO1 DrawCmdList, FBO2 DrawCmdList, and FBO3 DrawCmdList.

[0283] Subsequently, the rendering thread uses N rendering sub-threads to process the N sub-lists in parallel, obtaining N partial drawing results. For example, as shown in Figure 26, the rendering thread uses rendering sub-thread 1 to draw FBO1 DrawCmdList, obtaining drawing result 1; uses rendering sub-thread 2 to draw FBO1 DrawCmdList, obtaining drawing result 2; and uses rendering sub-thread 3 to draw FBO1 DrawCmdList, obtaining drawing result 3.

[0284] Finally, the rendering thread generates the corresponding first image frame based on the N local drawing results and the previous frame's drawing results. For example, the first image frame is generated based on drawing result 1, drawing result 2, and drawing result 3.

[0285] In some embodiments, the first drawing instruction list is a full drawing instruction list, and the rendering thread can generate the first image frame based on the N local drawing results.

[0286] In other embodiments, the first drawing instruction list is an incremental drawing instruction list. In this case, the rendering thread needs to generate the first image frame based on these N local drawing results and the previous frame drawing result. The previous frame drawing result refers to the drawing result of the previous image frame. The rendering thread uses these N local drawing results to replace the corresponding parts in the previous frame drawing result to obtain the first image frame.

[0287] In summary, through the processes S2501 to S2503 described above, the rendering thread can employ multiple rendering sub-threads to work in parallel to generate image frames, thereby improving the efficiency of image frame generation.

[0288] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0289] This application also provides an electronic device, which includes one or more processors and a memory; the memory is coupled to one or more processors; the memory is used to store computer program code, which includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to execute the image frame generation method shown in the above embodiments.

[0290] This application also provides a chip system, as shown in FIG27. The chip includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the image frame generation method in the above embodiments.

[0291] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image frame generation method provided in the above embodiments.

[0292] This application also provides a computer program product, which includes a computer program that, when run by an electronic device, enables the electronic device to implement the image frame generation method provided in the above embodiments.

[0293] It should be understood that the processor mentioned in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0294] It should also be understood that the memory mentioned in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0295] In the embodiments provided in this application, the division of each framework or module is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple frameworks or modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0296] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0297] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0298] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0299] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method of generating an image frame, characterized by, The method is applied to an electronic device, a unified rendering process of the electronic device includes a main thread and a rendering thread, and the method includes the following steps: After the main thread receives a first vertical synchronization signal, a first preprocessing result of a first image frame is generated; The main thread synchronizes the first preprocessing result to the rendering thread; The rendering thread generates the first image frame according to the first preprocessing result; During the process that the rendering thread generates the first image frame according to the first preprocessing result, after the main thread receives a second vertical synchronization signal, a second preprocessing result of a second image frame is generated, and the second preprocessing result is used to generate the second image frame.

2. The method of claim 1, wherein, The first preprocessing result is a first drawing instruction list of the first image frame; wherein the first drawing instruction list is an incremental drawing instruction list or a full-amount drawing instruction list, The incremental drawing instruction list includes drawing instructions of a newly added rendering node in a unified rendering tree of the first image frame and / or drawing instructions of a rendering node with changed information; The full-amount drawing instruction list includes drawing instructions of all rendering nodes in the unified rendering tree of the first image frame, wherein the unified rendering tree is a rendering tree obtained by merging rendering trees of all application programs that are subjected to unified rendering.

3. The method of claim 2, wherein, The first image frame includes a first part from a first application program and a second part from a second application program, after the main thread receives the first vertical synchronization signal, the first preprocessing result of the first image frame is generated, including the following steps: After the main thread receives the first vertical synchronization signal, a first drawing instruction sub-list corresponding to the first part is determined by using a first sub-thread; After the main thread receives the first vertical synchronization signal, a second drawing instruction sub-list corresponding to the second part is determined by using a second sub-thread; wherein the first sub-thread and the second sub-thread run in parallel; The main thread generates the first drawing instruction list of the first image frame according to the first drawing instruction sub-list and the second drawing instruction sub-list.

4. The method according to claim 2 or 3, characterized in that, The rendering thread generates the first image frame according to the first preprocessing result, including the following steps: The rendering thread obtains a first drawing result according to a first part of the first drawing instruction list by using a first rendering sub-thread; The rendering thread obtains a second drawing result according to a second part of the first drawing instruction list by using a second rendering sub-thread; wherein the first rendering sub-thread and the second rendering sub-thread run in parallel; The rendering thread generates the first image frame according to the first drawing result and the second drawing result.

5. The method of claim 4, wherein, The rendering thread generates the first image frame according to the first drawing result and the second drawing result, including the following steps: When the first drawing instruction list is an incremental drawing instruction list, the rendering thread generates the first image frame according to the first drawing result and the second drawing result, and a drawing result of a previous image frame.

6. The method according to any one of claims 2 to 5, characterized in that, The first drawing instruction list is an incremental drawing instruction list, and the main thread generates the first preprocessing result of the first image frame after receiving the first vertical synchronization signal, including the following steps: The main thread receives node change information of a rendering tree of at least one application after receiving the first vertical synchronization signal; The main thread generates the incremental drawing instruction list of the first image frame according to the node change information.

7. The method according to any one of claims 2 to 5, characterized in that, The first drawing instruction list is a full drawing instruction list, and the main thread generates a first preprocessing result of the first image frame after receiving the first vertical synchronization signal, including: The main thread receives a rendering tree of at least one application after receiving the first vertical synchronization signal; The main thread generates the full drawing instruction list of the first image frame according to the rendering tree of the at least one application.

8. The method of claim 1, wherein, The first preprocessing result is a unified rendering tree of the first image frame, wherein the unified rendering tree is a rendering tree obtained by merging rendering trees of all applications that perform unified rendering.

9. The method according to any one of claims 1 to 8, characterized in that, The rendering thread generates the first image frame according to the first preprocessing result, including: The rendering thread generates the first image frame according to the first preprocessing result after receiving the first preprocessing result.

10. The method according to any one of claims 1 to 8, characterized in that, The rendering thread generates the first image frame according to the first preprocessing result, including: The rendering thread generates the first image frame according to the first preprocessing result after receiving the second vertical synchronization signal.

11. The method according to any one of claims 1 to 10, characterized in that, The main thread synchronizes the first preprocessing result to the rendering thread, including: The main thread sends the first preprocessing result to the rendering thread; or The main thread sends address information of the first preprocessing result to the rendering thread, wherein the address information is used to access the first preprocessing result; or The main thread transfers a pointer of the first preprocessing result to the rendering thread.

12. An electronic device, comprising: The electronic device includes one or more processors and a memory; The memory is coupled to the one or more processors; The memory is configured to store computer program code including computer instructions, and the one or more processors are configured to invoke the computer instructions to cause the electronic device to perform the method of any one of claims 1-11.

13. A chip system applied to an electronic device, the chip system including one or more processors configured to invoke computer instructions to cause the electronic device to perform the method of any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method of any one of claims 1-11. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method of any one of claims 1-11.

Images