Graph synthesis method, electronic equipment, storage medium and program product

Abstract

The invention provides a graph synthesis method, electronic equipment, a storage medium and a program product, and relates to the technical field of image processing. The method comprises the steps that if a graphic synthesis task is detected during execution of a compiling task, the electronic equipment determines whether the graphic synthesis task is a first type of graphic synthesis task or not, resources occupied when the first type of graphic synthesis task is executed are the same as resources occupied when the compiling task is executed, and the first type of graphic synthesis task is a first type of graphic synthesis task; and if the task is the graph synthesis task of the first type, interrupting the compiling task until the graph synthesis task of the first type is executed. In the scheme, when the first type of graphic synthesis task appears, execution of the compiling task is interrupted, and when the first type of graphic synthesis task is executed completely, the interrupted compiling task continues to be executed, so that it is guaranteed that the first type of graphic synthesis task is executed preferentially all the time; and the conditions of abnormal display and even blockage of the foreground application process are avoided.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a graphic synthesis method, electronic device, storage medium, and program product. Background Technology

[0002] In image processing technology, multiple graphical elements are typically pre-defined, and each graphical element has its own independent program. These programs for graphical elements can run on a Graphics Processing Unit (GPU) to control the rendering effects of graphics or perform general computing tasks. Generally, the programs for the graphical elements are first compiled into GPU-executable binary files, and then the compiled binary files are called for graphics processing.

[0003] During the compilation of binary files, there may be situations where it is necessary to display graphic content on the screen. When the resources required for graphic composition are occupied by the compilation task of the binary file, graphic composition and foreground display cannot be performed in a timely manner. This will affect the foreground display business, causing display abnormalities or even freezing of the foreground application process. Summary of the Invention

[0004] This application provides a graphics compositing method, electronic device, storage medium, and program product, aiming to solve the problem of compilation tasks blocking graphics compositing tasks, causing lag in front-end display services.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, a graphics compositing method is provided for use in an electronic device. The method includes: during the execution of a compilation task, if a graphics compositing task is detected, determining whether the graphics compositing task is a first type of graphics compositing task, wherein the compilation task is a task that compiles the program corresponding to the graphics elements used for graphics rendering into a GPU-executable binary file; if so, interrupting the compilation task until the first type of graphics compositing task is completed, wherein the resources occupied when executing the first type of graphics compositing task are the same as the resources occupied when executing the compilation task.

[0007] Alternatively, the method includes: if a first type of graphics compositing task is detected during the execution of a compilation task, the compilation task is interrupted until the first type of graphics compositing task is completed, wherein the compilation task is a task that compiles the program corresponding to the graphics elements used for graphics rendering into a GPU-executable binary file; the resources occupied when executing the first type of graphics compositing task are the same as the resources occupied when executing the compilation task.

[0008] Based on the above technical solution, the compilation task is executed dynamically. That is, when a first type of graphics compositing task appears, the execution of the compilation task is interrupted. When the first type of graphics compositing task is completed, the previously interrupted compilation task continues to be executed. This ensures that the first type of graphics compositing task is always executed first, thereby avoiding the situation where the foreground display business is affected, display abnormalities occur, or even the foreground application process freezes when the compilation task blocks the graphics compositing task.

[0009] In one embodiment of the first aspect, after determining whether the graphics compositing task is a first type of graphics compositing task, the method further includes: if the graphics compositing task is a second type of graphics compositing task, then executing the second type of graphics compositing task and the compilation task in parallel, wherein the resources occupied when executing the second type of graphics compositing task are different from the resources occupied when executing the compilation task.

[0010] Alternatively, if a second type of graphics compositing task is detected, the second type of graphics compositing task and the compilation task are executed in parallel, wherein the resources used when executing the second type of graphics compositing task are different from the resources used when executing the compilation task.

[0011] In this method, if a graphics synthesis task occurs during the execution of a compilation task, and the graphics synthesis task is of the second type, since the resources occupied when executing the second type of graphics synthesis task are different from those occupied when executing the compilation task, that is, the compilation task will not block the graphics synthesis task. In this case, the graphics synthesis task and the compilation task can be executed in parallel without interfering with each other.

[0012] In one embodiment of the first aspect, the software architecture of the electronic device includes a system service layer, on which a first process runs. The first process includes a main thread and a rendering thread. The method further includes: during the execution of a compilation task by the rendering thread, if the main thread detects a graphics compositing task, the main thread determines whether the graphics compositing task is a first type of graphics compositing task; if so, the main thread generates an interrupt instruction and sends it to the rendering thread, the interrupt instruction instructing the rendering thread to interrupt the compilation task; the rendering thread responds to the interrupt instruction and interrupts the compilation task; after the rendering thread interrupts the compilation task, the rendering thread executes the first type of graphics compositing task.

[0013] Alternatively, the software architecture of the electronic device includes a system service layer, in which a first process runs. The first process includes a main thread and a rendering thread. The method further includes: during the rendering thread's execution of a compilation task, if the main thread detects a first type of graphics compositing task; the main thread generates an interrupt instruction and sends it to the rendering thread, the interrupt instruction instructing the rendering thread to interrupt the compilation task; the rendering thread responds to the interrupt instruction and interrupts the compilation task; after the rendering thread interrupts the compilation task, the rendering thread executes the first type of graphics compositing task.

[0014] In this embodiment, if a graphics compositing task occurs during the rendering thread's compilation task, and the graphics compositing task is of the first type, the main thread will generate an interrupt command and send it to the rendering thread. The rendering thread will respond to the interrupt command, interrupt the compilation task, and then execute the graphics compositing task. This interrupt mechanism allows the already occupied rendering thread to prioritize the execution of the graphics compositing task, preventing the compilation task from blocking the graphics compositing task.

[0015] In one embodiment of the first aspect, the method further includes: after the rendering thread has completed the first type of graphics compositing task, the rendering thread continues to execute the interrupted compilation task.

[0016] In this application, during the compilation task, each frame of image composition corresponds to a graphics compositing task. In the current frame, if the main thread determines that the graphics compositing task is a type 1 task and uses GPU compositing, the rendering thread will interrupt the compilation task and switch to executing the graphics compositing task. If the next frame determines that the graphics compositing task is a type 2 task and uses HWC compositing, the rendering thread will enable interrupts and continue executing the compilation task. If the next frame's graphics compositing task is still a type 1 task and still uses GPU compositing, the rendering thread will not enable interrupts and will continue executing the graphics compositing task for the next frame.

[0017] In one embodiment of the first aspect, the software architecture of the electronic device further includes a hardware abstraction layer, on which a second process runs, and the method further includes: if the main thread determines that the graphics compositing task is a second type of graphics compositing task, then sending the second type of graphics compositing task to the second process; the second process executes the second type of graphics compositing task.

[0018] Alternatively, the software architecture of the electronic device may also include a hardware abstraction layer, in which a second process runs. The method may also include: if the main thread detects a second type of graphics compositing task, it sends the second type of graphics compositing task to the second process; the second process executes the second type of graphics compositing task.

[0019] In this embodiment, if the main thread determines that the graphics compositing task is a second type of graphics compositing task and uses HWC compositing, then the main thread will send the graphics compositing task to the second process, which will then execute the graphics compositing task. During the execution of the graphics compositing task by the second process, the rendering thread continuously executes the compilation task; these two processes do not interfere with each other.

[0020] In one embodiment of the first aspect, the compilation task includes multiple atomic tasks, and the rendering thread interrupts the compilation task in response to an interrupt instruction, including: the rendering thread responding to the interrupt instruction and stopping the execution of the next atomic task when the currently executing atomic task has finished executing.

[0021] Although compilation is a time-consuming task, it is composed of a series of atomic tasks, which provides the basis for dynamically executing compilation tasks. When the rendering thread receives an interrupt command, it will stop execution when the current atomic task has finished.

[0022] In one embodiment of the first aspect, the method further includes: after the compilation task is completed, if a first type of graphics synthesis task is detected, then the first type of graphics synthesis task is executed.

[0023] In one embodiment of the first aspect, the method further includes: after the compilation task is completed, if the main thread detects a first type of graphics compositing task, the main thread calls the rendering thread to execute the first type of graphics compositing task.

[0024] In this embodiment, after the compilation task is completed, the rendering thread can generate a compilation task completion signal and send it back to the main thread. Upon receiving the compilation task completion signal, if the main thread subsequently detects a first-type graphics compositing task, it does not need to send an interrupt command to the rendering thread; instead, it directly sends the first-type graphics compositing task to the rendering thread for execution.

[0025] In one embodiment of the first aspect, the first type of graphics compositing task is a GPU compositing task, and the second type of graphics compositing task is an HWC compositing task.

[0026] In a second aspect, an electronic device is provided, including a display screen for displaying a composited graphic, and the electronic device for performing the graphic compositing method of the first aspect or any possible implementation thereof.

[0027] Thirdly, a computer-readable storage medium is provided, which stores a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the graphics composition method of the first aspect or any possible implementation thereof.

[0028] Fourthly, a computer program product, characterized in that the computer program product includes: computer program code, which, when executed by an electronic device, causes the electronic device to perform the graphics synthesis method in the first aspect or any possible implementation thereof.

[0029] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. The technical effects obtained by the second, third, and fourth aspects are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0030] Figure 1 A schematic diagram of the electronic device is shown.

[0031] Figure 2 A software structure block diagram of an electronic device according to an embodiment of this application is shown.

[0032] Figure 3 A schematic flowchart of a graphic synthesis method provided in an embodiment of this application is shown.

[0033] Figure 4 A flowchart of a graphic synthesis method is shown.

[0034] Figure 5 A signaling diagram of a graphic synthesis method is shown.

[0035] Figure 6 A schematic diagram of a graphics synthesis task being blocked is shown.

[0036] Figure 7 This diagram illustrates the main thread SF sending an interrupt command to the rendering thread.

[0037] Figure 8 This illustration shows an optional structural schematic diagram of the graphic synthesis apparatus provided in an embodiment of this application.

[0038] Figure 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0039] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0040] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" do not necessarily imply that they are different.

[0041] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0042] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0043] When an application that requires graphics rendering is opened, the application initializes the graphics rendering environment, including loading and compiling the programs corresponding to the required graphics elements. These programs are, for example, shader programs. Compiling these programs can refer to converting the shader program code into a GPU-executable binary file, called a cache file, which can be executed on the GPU.

[0044] Shader compilation typically occurs during the first application runtime to support the application's UI display requirements. When the application needs to display graphical content, the application's startup screen must wait for the shader program to finish compiling before it can perform graphics rendering based on the compiled cache file and finally display it on the screen. This approach may impact the application's startup efficiency.

[0045] To improve application startup efficiency, some mobile operating systems may compile and cache shader programs during the phone's boot process. This allows applications to directly use the cached files upon startup without needing to recompile.

[0046] When a mobile phone is powered on, the bootloader controls the startup process. The bootloader is responsible for initializing the hardware and loading the operating system kernel. Once the bootloader has completed initialization and loaded the operating system kernel (usually the Linux kernel), the operating system begins to boot. During the Linux kernel startup process, critical system services and hardware drivers are initialized.

[0047] After the Android system boots up, it starts a series of system services, including the graphics composition manager (SurfaceFlinger). SurfaceFlinger is Android's display system service, responsible for managing the graphical windows of applications and system services and compositing them onto the display screen.

[0048] Once the Android system is ready, SurfaceFlinger starts the compilation task, compiling all shader programs into GPU-executable binaries (or instruction sets) and caching them as cache files. This way, when the application starts, it can directly use the cached files without recompiling, thus improving application startup efficiency.

[0049] However, during the boot process, phones typically display a custom startup screen or logo from the device manufacturer. Specifically, the Android system loads pre-defined boot animation resources, which usually include animation frames, background images, or video files. SurfaceFlinger receives these boot animation resources, composites them into a screen frame buffer, and finally sends the image data to the display controller for display on the device's screen.

[0050] In other words, during the boot process, it may be necessary to both compile cache files and compose the boot animation. If the resources used for composing the boot animation are already occupied by the compilation task of compiling cache files, the graphics compositing task for composing the boot animation cannot be processed in a timely manner, which will affect the display of the boot animation.

[0051] In addition, since compiling all the shader programs is a large task, it takes longer to consume resources. This will make the graphics compositing task for the boot animation wait longer, thus increasing the probability of the boot animation stuttering.

[0052] To address this issue, this application provides a graphics compositing method that proposes dynamically executing compilation tasks based on their type during the compilation process. Specifically, if a graphics compositing task is acquired during compilation, and this task is of the first type, the electronic device interrupts the compilation task. This interruption releases resources, allowing the electronic device to use those resources to execute the graphics compositing task until it is completed, at which point the interrupted compilation task resumes. This prioritizes the execution of first-type graphics compositing tasks, resolving the issue of compilation tasks blocking graphics compositing tasks and effectively preventing boot animation stuttering.

[0053] The graphic composition method provided in this application can be applied to electronic devices equipped with displays. Electronic devices include terminal devices, which can also be called terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. Electronic devices can be mobile phones, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) electronic devices, augmented reality (AR) electronic devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, and so on. The embodiments of this application do not limit the specific technologies or device forms used in the electronic devices.

[0054] To better understand the embodiments of this application, the structure of the electronic device of the embodiments of this application is described below.

[0055] Figure 1A schematic diagram of the electronic device is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a touch screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0056] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0057] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0058] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0059] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0060] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0061] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0062] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0063] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, touchscreen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0064] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0065] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 150 can provide solutions for wireless communication applications including 2G / 3G / 4G / 1G on the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.

[0066] In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the same device as at least some modules of the processor 110.

[0067] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0068] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with networks and other devices through wireless communication technology.

[0069] Electronic device 100 implements display functions through a GPU, a touch screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the touch screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0070] The touchscreen 194 is used to display images, display videos, and receive swipe operations, etc. The touchscreen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N touchscreens 194, where N is a positive integer greater than 1.

[0071] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, touch screen 194 and application processor.

[0072] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0073] Camera 193 is used to capture still images or videos. In some embodiments, electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0074] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0075] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0076] Internal memory 121 can be used to store executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located within the processor.

[0077] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0078] The sensor module 180 includes, for example, a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer, a distance sensor, a proximity sensor, a fingerprint sensor, a temperature sensor, a touch sensor, and an ambient light sensor.

[0079] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0080] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can be corresponding to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the touchscreen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0081] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0082] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0083] In practical applications, the operating system of electronic device 100 may include, but is not limited to, Symbian, Android, Windows, macOS, iOS, Blackberry, HarmonyOS, and other operating systems. This application does not impose any limitations on the operating system of the electronic device.

[0084] Figure 2 A software structure block diagram of an electronic device according to an embodiment of this application is shown.

[0085] For example, such as Figure 2As shown, the layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into five layers, from top to bottom: the application (APP) layer, the application framework (FWK) layer, the system runtime library layer (Native), the hardware abstraction layer (HAL), and the kernel layer.

[0086] The application layer includes one or more applications that require graphics rendering, such as cameras, games, and calendars. When an application needs to display an image, it can generate a graphics compositing task and send it to the graphics compositing manager.

[0087] The system runtime layer includes display system services, which include the graphics compositing manager SurfaceFlinger. SurfaceFlinger is implemented in a separate process (the first process), which includes the main thread (SF) and the rendering thread (renderengine). On one hand, during the boot process, when a boot animation needs to be displayed, SurfaceFlinger can generate graphics compositing tasks. On the other hand, SurfaceFlinger can receive graphics compositing tasks sent by upper-layer applications. SurfaceFlinger can invoke software and / or hardware to execute graphics compositing tasks.

[0088] The Hardware Abstraction Layer (HAWL) comprises the Hardware Composer (HWC) module and the GPU driver module. The HWC module handles hardware compositing and is implemented in a separate process (a second process). Its primary responsibility is to efficiently composite and display multiple layers on the screen using the GPU and other hardware resources. The GPU driver is the GPU driver interface, performing GPU-related tasks and then handing them over to the GPU hardware for display.

[0089] The kernel layer is the layer between hardware and software. The hardware can be a DPU, GPU, etc.

[0090] It should be noted that although the embodiments of this application are described using the Android system, the principle of the image synthesis method is also applicable to electronic devices with operating systems such as iOS or Windows.

[0091] The following example illustrates the workflow of the software and hardware of an electronic device.

[0092] When a higher-level application needs to display graphical content, taking the "Calendar" application as an example, such as... Figure 2 As shown, the upper-layer application "Calendar" generates graphics compositing tasks and sends them to the SurfaceFlinger runtime layer. The main thread SF in SurfaceFlinger selects a compositing strategy. Specifically, the main thread SF determines whether the graphics compositing task is a first-type task. If so, the main thread SF decides to use GPU compositing, sending the task to the rendering thread. The rendering thread executes the task and sends the result to the GPU for display via the GPU driver module. If the main thread SF determines that the task is not a first-type task, it decides to use HWC compositing, sending the task to the HWC module. The HWC module executes the task using a second process and sends the result to the DPU for display.

[0093] GPU compositing refers to a layer compositing method that uses the GPU to composite layers. GPU compositing can be software-based or client-side. It supports arbitrary compositing transformations. In the compositing strategies of some electronic devices (such as mobile phones), GPU compositing is used when HWC compositing cannot meet the compositing conditions, or HWC compositing and GPU compositing are used in combination to achieve the desired compositing effect. Specifically, layers that do not meet the conditions can be composited using GPU, and the resulting data can be used as input data for HWC compositing for secondary compositing.

[0094] HWC compositing refers to a layer compositing method that uses HWC to composite layers. During HWC compositing, each layer requires one transmission channel from the HWC, and since the number of transmission channels in HWC is limited, the number of layers that HWC can support for compositing is limited. HWC compositing is a hardware-based compositing method, meaning that a separate hardware component in the electronic device processes the compositing and overlay of layers. The composited result is directly sent to the display for display (i.e., sent for display). Its advantages include highly customized processes and results that can produce extremely high compositing efficiency, and relatively low power consumption. Because HWC supports a limited number of layers, when the number of layers to be composited exceeds the number of layers that HWC can support, the GPU needs to first merge some of the layers in the layers to be composited. Then, the GPU's compositing result is sent to the HWC along with the remaining layers in the layers to be composited. The HWC then composites the GPU's compositing result and the remaining layers to obtain the layer to be displayed. Finally, this layer is sent for display, and thus, a frame of image composed of the layers to be composited can be displayed on the screen.

[0095] The technical solution of this application and how it solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be implemented independently or in combination with each other. Similar or identical concepts or processes may not be described again in some embodiments.

[0096] Please refer to Figure 3 , Figure 3 A schematic flowchart of a graphic synthesis method provided in an embodiment of this application is shown. This method can be applied to electronic devices, or to chips or circuits within corresponding electronic devices. The method includes steps 301 to 306.

[0097] Step 301: Obtain the graphic composition task.

[0098] In this embodiment, when an upper-layer application or system service needs to display graphics on the screen, the upper-layer application or system service generates a graphics compositing task. This task is then obtained by the functional module in the phone responsible for managing the graphics windows of applications and system services. The graphics compositing task includes graphics data, which is provided by the system service or the upper-layer application.

[0099] Each frame of image is synthesized into a corresponding image synthesis task.

[0100] In one implementation, when the phone boots up, it loads pre-set boot animation resources, which typically include animation frames, background images, or video files. This allows the phone to generate and acquire a graphics compositing task, which is the task of compositing the boot animation.

[0101] In another implementation, when an application requiring graphics rendering starts, it provides the graphics data to be displayed on the screen, such as the application's interface data. In this way, the phone can generate and acquire graphics compositing tasks, which are tasks that compose the graphical content that the application needs to display.

[0102] Step 302: Determine whether the graphics compositing task is a first type of graphics compositing task. If yes, proceed to step 303; otherwise, proceed to step 306.

[0103] In this embodiment, after obtaining the graphics compositing task, a compositing strategy is selected. The compositing strategy is determined based on the type of the graphics compositing task. When the graphics compositing task is of the first type, the compositing strategy is GPU compositing; if the graphics compositing task is not of the first type, the compositing strategy is non-GPU compositing. For example, when the graphics compositing task is not of the first type, it is generally determined to be of the second type. In this case, non-GPU compositing can be understood as HWC compositing.

[0104] The mobile phone can select a compositing strategy based on the characteristics and requirements of the graphics compositing task. These characteristics and requirements include, for example, at least one of the following: whether layer blending is required, whether transparency processing is required, and whether transparency transformation is required. Other characteristics and requirements of the graphics compositing task may also be included, but are not exhaustively listed in this application.

[0105] Some characteristics and requirements of graphics compositing tasks may limit the use of HWC. For example, there may be a need to render specific graphic effects, display complex layers, or have special requirements for transparency, blending modes, and layer relationships. In such cases, the phone can determine that the graphics compositing task is of type one and choose to use GPU compositing. For details on the process of performing the graphics compositing task based on GPU compositing, please refer to the description of step 303.

[0106] For graphic compositing tasks with fewer layers and simpler requirements, the phone can determine that the task is of type two and select HWC compositing. For details on performing this graphic compositing task based on HWC compositing, please refer to step 306.

[0107] Step 303: Check if a compilation task is being executed. If yes, proceed to steps 304 and 305. If no, proceed to step 305.

[0108] Step 304: Abort the compilation task.

[0109] Step 305: Perform the first type of graphics composition task.

[0110] The compilation task is the task of compiling the programs corresponding to the graphics elements used for graphics rendering into GPU-executable binary files. The first type of graphics compositing task is the GPU compositing task.

[0111] The programs corresponding to graphical elements are, for example, shader programs. In graphics rendering, a separate shader program is usually set up for each graphical element. Shader programs are typically written in specific programming languages, such as OpenGL's GLSL (OpenGL Shading Language) or DirectX's HLSL (High Level Shading Language). These specific programming languages ​​cannot be directly recognized by the GPU. Therefore, the shader program needs to be compiled into GPU-executable binary code.

[0112] The process of a mobile phone performing a compilation task includes: the mobile phone calling the rendering thread renderengine to translate high-level shader code into low-level GPU instruction sets.

[0113] It's understandable that the rendering thread is occupied when a compilation task is being executed, and unoccupied when no compilation task is being executed. Therefore, the state of the rendering thread can be used to detect whether a compilation task is being executed.

[0114] Optionally, in this embodiment, when the compilation task ends, the mobile phone can obtain a compilation task completion signal. After obtaining the compilation task completion signal, a first status bit can be marked, indicating that the compilation task has not been executed. If the compilation task completion signal is not obtained, a second status bit can be marked, indicating that the compilation task is being executed. The mobile phone can determine whether it is executing a compilation task by whether it has obtained a compilation task completion signal. Alternatively, the mobile phone can determine whether it is executing a compilation task based on the marked status bits.

[0115] Optionally, the rendering thread is occupied during the execution of a compilation task, but not when the compilation task is not being executed. Therefore, it can be determined whether a compilation task is being executed based on whether the rendering thread is occupied.

[0116] It's important to note that the resources used for executing compilation tasks are the same as those used for executing the first type of graphics compositing task. Specifically, both the compilation and compositing tasks require the use of the renderengine thread. Therefore, if a compilation task is detected in progress, it means that the renderengine thread is occupied, which will hinder the graphics compositing task.

[0117] To address the problem of graphics compositing tasks being hindered, in this embodiment of the application, if a first type of graphics compositing task occurs during the execution of a compilation task, the compilation task needs to be interrupted to release the rendering thread, and then the rendering thread can be used to execute the first type of graphics compositing task.

[0118] The process of interrupting the compilation task can include: the rendering thread stopping the execution of the compilation task and entering a callable state.

[0119] In one implementation, the compilation task comprises multiple atomic tasks, and executing the compilation task actually involves executing each atomic task one by one. Interrupting the compilation task means that the phone completes the currently executing atomic task and then stops executing the next atomic task.

[0120] It should be noted that the phone also records the sequence number of the currently interrupted atomic task, so that the compilation task can continue from the interrupted position after the graphics synthesis task is completed.

[0121] In this embodiment, the completion of all atomic tasks signifies the completion of the compilation task. In this case, the rendering thread can generate a compilation task completion signal, so that when a graphics compositing task is subsequently acquired, steps 303 and 304 do not need to be executed; step 305 can be executed directly. Furthermore, once the compilation task is completed, the rendering thread is released and becomes available for further execution.

[0122] The process of executing the first type of graphics compositing task may include: the mobile phone acquiring the graphics compositing task, calling the rendering thread to perform graphics rendering, and then sending the rendered graphics to the GPU, which then composites the layers and displays them.

[0123] In this embodiment of the application, the rendering thread is automatically released after the graphics compositing task is completed. In this case, such as Figure 3 As shown, after step 305, the phone can call the rendering thread again to continue executing the compilation task.

[0124] Specifically, after the graphics compositing task is completed, the mobile phone may receive a new frame of the first type of graphics compositing task. In this case, the compilation task remains interrupted, and the rendering thread will continue to execute the new frame of graphics compositing task.

[0125] If the rendering thread does not receive a new graphics compositing task within a preset time after the graphics compositing task is completed, the rendering thread will be automatically released. In this case, the phone can call the rendering thread to restart the compilation task until the compilation task is completed or until the rendering thread receives a graphics compositing task again.

[0126] In the embodiments of this application, such as Figure 3 As shown in step 303, when it is detected that a compilation task is not being executed, it means that the rendering thread renderengine is not being used. In this case, as shown in step 305, the rendering thread can be directly used to execute the first type of graphics compositing task.

[0127] It should be noted that, in this embodiment of the application, if the compilation task has not been completed after the graphics compositing task is completed, the rendering thread will continue to execute the compilation task until the compilation task is completed, or until a new frame of graphics compositing task is received. If the compilation task is completed and no new graphics compositing task arrives, the process will end.

[0128] Step 306: Perform the graphics compositing task based on HWC compositing.

[0129] The second type of graphics compositing task is the HWC compositing task.

[0130] In this embodiment, when a graphics compositing task is not a first type of graphics compositing task, it is generally considered to be a second type of graphics compositing task. In this embodiment, when a second type of graphics compositing task is detected, the task can be executed based on the resources required to execute it.

[0131] In this embodiment of the application, the resources occupied by executing the second type of graphics synthesis task are different from the resources occupied by executing the compilation task.

[0132] Executing the second type of graphics compositing task consumes HWC resources, while executing the compilation task consumes the rendering thread. Therefore, it can be understood that executing the second type of graphics compositing task and executing the compilation task do not interfere with each other. In this case, the phone can directly call HWC resources to execute the graphics compositing task. Simultaneously, if a compilation task exists, the rendering thread does not need to interrupt the compilation task and continues its original execution. If no compilation task exists, the phone does not need to call the rendering thread.

[0133] In this embodiment of the application, the process of performing the second type of graphics compositing task may include: the mobile phone acquiring the graphics compositing task, sending the graphics compositing task to the HWC module, and the HWC module performing the graphics data in the graphics compositing task based on the second process to compose the final display frame. After the HWC compositing is completed, the final display frame is sent to the data processing unit (DPU), and the DPU sends the display frame to the display device (such as the mobile phone screen, external display, etc.) in an appropriate timing sequence to complete the actual display of the image.

[0134] In the graphics compositing method provided in this application embodiment, if a graphics compositing task is detected during the execution of a compilation task, the mobile phone will determine whether the graphics compositing task is a first type of graphics compositing task. The resources occupied when executing a first type of graphics compositing task are the same as those occupied when executing a compilation task. If it is a first type of graphics compositing task, the compilation task is interrupted until the first type of graphics compositing task is completed. This solution dynamically executes the compilation task; that is, when a first type of graphics compositing task appears, the execution of the compilation task is interrupted; when the first type of graphics compositing task is completed, the previously interrupted compilation task continues to be executed. This ensures that the first type of graphics compositing task is always executed with priority, thereby avoiding the situation where the foreground display service is affected, display abnormalities occur, or even the foreground application process freezes when the compilation task blocks the graphics compositing task.

[0135] It should be noted that, in this embodiment of the application, when the compilation task is completed and the mobile phone obtains a new graphics compositing task, and determines that the new graphics compositing task is a first-type graphics compositing task, the mobile phone can directly call the rendering thread to execute the new graphics compositing task. No interruption operation is required during this process.

[0136] Based on the above embodiments, this application also provides a graphic synthesis method, please refer to... Figure 4 and Figure 5 , Figure 4 A flowchart of a graphic synthesis method is shown. Figure 5 A signaling diagram of a graphics compositing method is shown. This method is applied to an electronic device whose software architecture includes a system service layer, on which a first process (SurfaceFlinger) runs. The first process includes a main thread (SF) and a rendering thread (renderengine). The electronic device's software architecture also includes a hardware abstraction layer, on which a second process runs. The method includes:

[0137] Step 401: The rendering thread executes the compilation task.

[0138] In this embodiment of the application, during the power-on phase, such as Figure 5 As shown, SurfaceFlinger will perform an initialization (Init) and wake up the rendering thread to execute the compilation task.

[0139] The compilation task refers to the task of compiling the shader program into a binary file that can be executed by the GPU. The compiled binary file is called a cache file, which is cached for use during graphics processing.

[0140] Step 402: During the compilation task, the main thread obtains the graphics compositing task and determines whether it is a first-type graphics compositing task. If yes, proceed to steps 403 to 404. If no, proceed to steps 405 to 406.

[0141] Step 403: The main thread generates an interrupt command and sends it to the rendering thread.

[0142] Step 404: The rendering thread responds to the interrupt command to interrupt the compilation task, and after the compilation task is interrupted, the rendering thread executes the first type of graphics compositing task.

[0143] During the boot process, a boot animation needs to be displayed. In this case, the system service generates a graphics compositing task and submits it to SurfaceFlinger. The system service submits the graphics data corresponding to each image frame to SurfaceFlinger frame by frame. After SurfaceFlinger obtains the graphics compositing task frame by frame, it then passes the task to the main thread SF. The main thread SF determines the type of the graphics compositing task or selects a compositing strategy for it.

[0144] In one implementation, the main thread SF can determine the type of graphics compositing task based on the characteristics and requirements of the graphics compositing task, or the main thread SF can select the compositing strategy of the graphics compositing task based on the characteristics and requirements of the graphics compositing task.

[0145] For example, when the number of layers in a graphics compositing task is large, the main thread SF can determine that the graphics compositing task is a first-type graphics compositing task. Alternatively, when the number of layers in a graphics compositing task is large, the HWC compositing method may be unable to meet the compositing conditions due to the number of channels. In this case, the main thread SF can choose GPU compositing as the compositing strategy for the graphics compositing task.

[0146] In this embodiment of the application, when the graphics compositing task is a first type of graphics compositing task, the resources occupied by executing the graphics compositing task are the same as those occupied by executing the compilation task. Therefore, it can be understood that when the graphics compositing task is a first type of graphics compositing task, executing the graphics compositing task conflicts with executing the compilation task.

[0147] For example, please refer to Figure 6 , Figure 6 This diagram illustrates a scenario where a graphics synthesis task is blocked. (For example...) Figure 6As shown, during the compilation of the cache file in the rendering thread, SurfaceFlinger sequentially receives three image frames corresponding to graphics compositing tasks, denoted as frame1, frame2, and frame3. Upon receiving frame1, the main thread SF determines that frame1 will undergo HWC compositing. HWC compositing does not conflict with compiling the cache file, so frame1 compositing succeeds. Upon receiving frame2, the main thread SF determines that frame2 will also undergo HWC compositing. HWC compositing does not conflict with compiling the cache file, so frame2 compositing succeeds. When the main thread SF receives frame3, it determines that frame3 will undergo GPU compositing. However, the rendering thread required for GPU compositing is already occupied by the compilation task. In this situation, frame3 compositing fails.

[0148] In this case, please refer to Figure 4 and Figure 7 As shown, Figure 7 The diagram illustrates how the main thread (SF) sends an interrupt command to the rendering thread. Specifically, for frame 3, the main thread (SF) can generate an interrupt command and send it to the rendering thread. Upon receiving the interrupt command, the rendering thread will interrupt the compilation task and then execute frame 3.

[0149] For example, the main thread SF can issue commands to the GPU via a graphics API (such as OpenGL ES or Vulkan) to request the execution of compositing operations. Based on the instructions and graphics data (included in the graphics compositing task) provided by the main thread SF, the GPU uses its parallel computing capabilities and dedicated compositing hardware units to composite multiple graphics buffers. The GPU stores the final composited image in one or more frame buffers and outputs it to a display device (such as a mobile phone screen, external monitor, etc.) to complete the actual display of the image.

[0150] In this embodiment, after frame3 completes execution, if the compilation task has not yet finished, the rendering thread will continue executing the compilation task until it is completed, or until a new frame's graphics compositing task is received. If the compilation task is completed and no new graphics compositing task arrives, the process will terminate.

[0151] Optionally, in this embodiment of the application, the rendering thread can generate an end signal after the compilation task is completed and send the end signal to the main thread SF.

[0152] This application interrupts the execution of the compilation task by using an interrupt instruction, thereby ensuring that the graphics synthesis task is executed first and avoiding the graphics synthesis task being blocked by the compilation task.

[0153] Optionally, in this embodiment of the application, the compilation task includes multiple atomic tasks. Interrupting the compilation task means that the rendering thread responds to the interrupt instruction, completes the currently executing atomic task, and then stops executing the next atomic task. Figure 5 As shown by the dashed line, the rendering thread interrupts the compilation task and switches to executing the graphics compositing task. The rendering thread records the sequence number of the currently interrupted atomic task so that subsequent atomic tasks can resume execution from the point of interruption, until all atomic tasks are completed, which means the compilation task is finished.

[0154] In this embodiment, after the main thread SF obtains the graphics compositing task, it selects a compositing strategy. When the graphics compositing task performs GPU compositing, it sends an interrupt command to the rendering thread to control the rendering thread to interrupt the compilation task and instead execute the graphics compositing task. This avoids the compilation task from hindering the graphics compositing task, which could cause the foreground display to lag.

[0155] Step 405: The main thread sends the graphics compositing task to the second process.

[0156] Step 406: The second process executes the graphics synthesis task.

[0157] In this embodiment, SurfaceFlinger can acquire the graphics compositing task, including the content, position, and transparency of each layer, and then send the graphics compositing task to the main thread SF, where SF selects the compositing strategy. If the main thread SF determines that the graphics compositing task is a second type of graphics compositing task, that is, if the compositing strategy is HWC compositing, then the main thread SF will send the graphics compositing task to a second process. In the second process, the HWC module uses hardware acceleration to directly composite these layers on the screen, without needing software rendering via the CPU or GPU, and finally sends the composited result to the DPU for display.

[0158] In this embodiment of the application, when the image compositing task is a second type of graphics compositing task, the process of executing the graphics compositing task and the process of the rendering thread executing the compilation task do not interfere with each other.

[0159] When a graphics compositing task performs GPU compositing, the rendering thread needs to interrupt the compilation task. However, when a graphics compositing task performs HWC compositing, the compilation task being executed by the rendering thread is unaffected.

[0160] It should be noted that, as Figure 5As shown, during the compilation task, each frame of image composition corresponds to a graphics compositing task. In the current frame, if the main thread determines that the graphics compositing task is a type 1 task and uses GPU compositing, the rendering thread will interrupt the compilation task and switch to executing the graphics compositing task. If the next frame determines that the graphics compositing task is a type 2 task and uses HWC compositing, the rendering thread will enable interrupts and continue executing the compilation task. If the next frame's graphics compositing task is still a type 1 task and still uses GPU compositing, the rendering thread will not enable interrupts and will continue executing the graphics compositing task for the next frame.

[0161] It should be noted that in the embodiments of this application, when the compilation task is completed and the main thread SF receives the graphics compositing task again, and determines that the graphics compositing task is a first type of graphics compositing task, the main thread SF does not need to send an interrupt instruction to the rendering thread, but can directly send the graphics compositing task to the rendering thread and call the rendering thread to execute the graphics compositing task.

[0162] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0163] The above combination Figures 1 to 7 The graphic synthesis method provided in the embodiments of this application has been described. The apparatus for performing the above method provided in the embodiments of this application is described below. Figure 8 As shown, Figure 8This illustration shows an optional structural diagram of a graphics compositing apparatus provided in an embodiment of this application. The graphics compositing apparatus may be the execution device in the embodiments of this application, or a chip or chip system within the execution device. The graphics compositing apparatus 800 includes a processing unit 801, wherein the processing unit 801 is configured to, during the execution of a compilation task, if a graphics compositing task is detected, determine whether the graphics compositing task is a first type of graphics compositing task, where the compilation task is the task of compiling the program corresponding to the graphics elements used for graphics rendering into a GPU-executable binary file; if so, interrupt the compilation task until the first type of graphics compositing task is completed, wherein the resources occupied when executing the first type of graphics compositing task are the same as the resources occupied when executing the compilation task.

[0164] Alternatively, processing unit 801 is configured to interrupt the compilation task if a first type of graphics compositing task is detected during the execution of the compilation task, until the first type of graphics compositing task is completed. The compilation task is the task of compiling the program corresponding to the graphics elements used for graphics rendering into a GPU-executable binary file. The resources occupied when executing the first type of graphics compositing task are the same as those occupied when executing the compilation task.

[0165] Optionally, the processing unit 801 is further configured to execute the second type of graphics compositing task and the compilation task in parallel if the graphics compositing task is a second type of graphics compositing task, wherein the resources occupied when executing the second type of graphics compositing task are different from the resources occupied when executing the compilation task.

[0166] Alternatively, the processing unit 801 is further configured to, if a second type of graphics compositing task is detected, execute the second type of graphics compositing task and the compilation task in parallel, wherein the resources occupied when executing the second type of graphics compositing task are different from the resources occupied when executing the compilation task.

[0167] Optionally, the processing unit 801 also runs a first process, which includes a main thread and a rendering thread. During the execution of a compilation task by the rendering thread, if the main thread detects a graphics compositing task, the main thread determines whether the graphics compositing task is a first type of graphics compositing task. If so, the main thread generates an interrupt instruction and sends it to the rendering thread, which instructs the rendering thread to interrupt the compilation task. The rendering thread responds to the interrupt instruction and interrupts the compilation task. After the rendering thread interrupts the compilation task, the rendering thread executes the first type of graphics compositing task.

[0168] Alternatively, the processing unit 801 may also run a first process, which includes a main thread and a rendering thread. If the main thread detects a first type of graphics compositing task while the rendering thread is executing a compilation task, the main thread generates an interrupt instruction and sends it to the rendering thread. The interrupt instruction instructs the rendering thread to interrupt the compilation task. The rendering thread responds to the interrupt instruction and interrupts the compilation task. After the rendering thread interrupts the compilation task, the rendering thread executes the first type of graphics compositing task.

[0169] Optionally, the processing unit 801 is also configured to allow the rendering thread to continue executing the interrupted compilation task after the rendering thread has completed the first type of graphics compositing task.

[0170] Optionally, the processing unit 801 also runs a second process. If the main thread determines that the graphics compositing task is a second type of graphics compositing task, it sends the second type of graphics compositing task to the second process; the second process executes the second type of graphics compositing task.

[0171] Alternatively, the processing unit 801 may also run a second process. If the main thread detects a second type of graphics compositing task, it sends the second type of graphics compositing task to the second process; the second process then executes the second type of graphics compositing task.

[0172] Optionally, the compilation task includes multiple atomic tasks, and the processing unit 801 is also used to respond to interrupt instructions in the rendering thread, and stop executing the next atomic task when the currently executing atomic task finishes.

[0173] Optionally, the processing unit 801 is further configured to execute the first type of graphics synthesis task if a first type of graphics synthesis task is detected after the compilation task has finished executing.

[0174] Optionally, the processing unit 801 is also configured to, after the compilation task has finished executing, if the main thread detects a first type of graphics compositing task, call the rendering thread to execute the first type of graphics compositing task.

[0175] Optionally, the first type of graphics compositing task is a GPU compositing task, and the second type of graphics compositing task is an HWC compositing task.

[0176] It should be noted that the aforementioned graphics synthesis device 800 is embodied in the form of a functional unit. The term "unit" here can be implemented in software and / or hardware, without specific limitation. For example, a "unit" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include application-specific integrated circuits (ASICs), electronic circuits, a processor (e.g., a shared processor, a proprietary processor, or a group processor, etc.) and memory for executing one or more software or firmware programs, merging logic circuits, and / or other suitable components supporting the described functions.

[0177] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0178] Figure 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application is shown. Figure 9 The electronic device 900 shown (which may specifically be a server) includes one or more processors 910, communication lines 940 and communication interfaces 930. Optionally, the electronic device 900 also includes a memory 920.

[0179] In some implementations, memory 920 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof.

[0180] The methods described in the embodiments of this application can be applied to or implemented by the processor 910. The processor 910 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor 910 or by instructions in the form of software. The processor 910 may be a general-purpose processor (e.g., a microprocessor or conventional processor), a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components. The processor 910 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0181] The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in mature storage media in the art, such as random access memory, read-only memory, programmable read-only memory, or electrically erasable programmable read-only memory (EEPROM). This storage medium is located in memory 920, and processor 910 reads information from memory 920 and, in conjunction with its hardware, completes the steps of the above method.

[0182] The processor 910, memory 920 and communication interface 930 can communicate with each other via communication line 940.

[0183] In the above embodiments, the instructions stored in the memory for execution by the processor can be implemented in the form of a computer program product. This computer program product can be pre-written into the memory, or it can be downloaded and installed into the memory as software.

[0184] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. For example, available media may include magnetic media (e.g., floppy disk, hard disk, or magnetic tape), optical media (e.g., digital versatile disc (DVD)), or semiconductor media (e.g., solid-state disk (SSD)).

[0185] This application provides a chip including a processor. The processor is used to call a computer program stored in memory to execute the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar to the related embodiments described above, and will not be repeated here.

[0186] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program or instructions. When the computer program or instructions are executed by a processor, they implement the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0187] As one possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disc storage; computer-readable media may include disk storage or other disk storage devices. Furthermore, any connecting cable may also be appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. As used herein, disks and optical discs include optical discs (CD), laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0188] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0189] The above specific embodiments further illustrate the purpose, technical solution and beneficial effects of this application. It should be understood that the above are only specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this application.

Claims

1. A method for graphic synthesis, characterized in that, Applied to electronic devices, the method includes: If a first type of graphics compositing task is detected during the execution of a compilation task, the compilation task is interrupted until the first type of graphics compositing task is completed. The compilation task is a task that compiles the program corresponding to the graphics elements used for graphics rendering into a GPU-executable binary file. The resources occupied when executing the first type of graphics compositing task are the same as those occupied when executing the compilation task.

2. The method according to claim 1, characterized in that, The method further includes: If a second type of graphics compositing task is detected, the second type of graphics compositing task and the compilation task are executed in parallel, wherein the resources used when executing the second type of graphics compositing task are different from the resources used when executing the compilation task.

3. The method according to claim 1 or 2, characterized in that, The software architecture of the electronic device includes a system service layer, on which a first process runs. The first process includes a main thread and a rendering thread. The method further includes: If the main thread detects the first type of graphics compositing task during the execution of the compilation task by the rendering thread, the main thread generates an interrupt instruction and sends it to the rendering thread, the interrupt instruction instructing the rendering thread to interrupt the compilation task. The rendering thread interrupts the compilation task in response to the interrupt command; After the rendering thread interrupts the compilation task, the rendering thread executes the first type of graphics compositing task.

4. The method according to claim 3, characterized in that, The method further includes: After the rendering thread completes the first type of graphics compositing task, the rendering thread continues to execute the interrupted compilation task.

5. The method according to claim 3 or 4, characterized in that, The software architecture of the electronic device further includes a hardware abstraction layer, on which a second process runs; the method further includes: If the main thread detects a second type of graphics compositing task, it sends the second type of graphics compositing task to the second process. The second process executes the second type of graphics synthesis task.

6. The method according to any one of claims 3-5, characterized in that, The compilation task includes multiple atomic tasks, and the rendering thread interrupts the compilation task in response to the interrupt instruction, including: In response to the interrupt command, the rendering thread stops executing the next atomic task when the currently executing atomic task finishes.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: After the compilation task is completed, if the first type of graphics compositing task is detected, then the first type of graphics compositing task is executed.

8. The method according to claim 7, characterized in that, The method further includes: After the compilation task is completed, if the main thread detects the first type of graphics compositing task, the main thread calls the rendering thread to execute the first type of graphics compositing task.

9. The method according to any one of claims 2-8, characterized in that, The first type of graphics compositing task is a GPU compositing task, and the second type of graphics compositing task is an HWC compositing task.

10. An electronic device, characterized in that, The device includes a display screen for displaying the synthesized graphics, and the electronic device is used to perform the method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1 to 9.

12. A computer program product, characterized in that, The computer program product includes: computer program code, which, when executed by an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 9.

Images