Layer composition method, electronic device and storage medium

Abstract

The present application relates to the technical field of terminals, and provides a layer composition method, an electronic device and a storage medium, capable of improving the layer composition efficiency, reducing frame loss problems, and improving the use experience of users. The method is applied to the electronic device, and comprises: acquiring layer information of layers to be composited of a current frame; on the basis of the layer information of said layers, determining a first detection result; if the first detection result does not meet a first condition, and a first target layer meeting a preset condition exists among said layers, controlling a GPU to composite the first target layer so as to obtain a first intermediate layer, and controlling an HWC to query a composition mode of said layers; and if the composition mode of a second target layer is HWC composition, controlling the HWC to composite the first intermediate layer and the second target layer, so as to obtain an image of the current frame, wherein the second target layer is a layer to be composited except the first target layer among said layers.

Description

Layer synthesis method, electronic device and storage medium Technical Field

[0001] The application relates to the field of terminal technology, and in particular to a layer synthesis method, electronic device and storage medium. Background Art

[0002] At present, in order to improve the efficiency of layer synthesis and reduce the synthesis pressure of the graphics processing unit (GPU), most electronic devices such as mobile phones are equipped with a hardware composer (HWC). HWC is mainly used to help the display synthesis system SurfaceFlinger (also known as SF) synthesize layers, thereby reducing the amount of layer synthesis performed by SF through the GPU. However, due to the limited performance of the HWC itself, after receiving the synthesis request of the layer, the HWC will first determine whether the layer should be synthesized by the GPU or the HWC based on its own capabilities and layer parameters, and then after determining who will synthesize the result, the corresponding layer parameters will be handed over to the corresponding components for synthesis. Among them, the process of determining who synthesizes the layer can be called a query synthesis strategy.

[0003] However, the existing layer synthesis process may cause frame loss, affecting the user experience.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a layer synthesis method, electronic device and storage medium, which can improve the efficiency of layer synthesis, reduce frame loss problems, and improve the user experience.

[0006] In order to achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a layer synthesis method, which is applied to an electronic device. The method includes: obtaining layer information of a layer to be synthesized in a current frame; determining a first detection result based on the layer information of the layer to be synthesized; if the first detection result does not meet a first condition, and there is a first target layer that meets a preset condition in the layer to be synthesized, controlling the GPU to synthesize the first target layer to obtain a first intermediate layer, and at the same time controlling the HWC to query the synthesis method of the layer to be synthesized; wherein the synthesis method of the first target layer that meets the preset condition is GPU synthesis; the preset condition includes any one or more of the following: the layer attribute parameters or display parameters of the layer have changed; the layer is a newly added layer, and the layer attribute parameters include a first flag, the first flag is used to indicate that the GPU is used for synthesis; the layer is a newly added layer, and the layer buffer size is greater than a preset size; the layer whose synthesis method in the previous frame was GPU synthesis, and whose position has changed or the visible area has increased; the position of the layer has rotated 180° compared with the previous frame. When the synthesis method of the second target layer is HWC synthesis, the HWC is controlled to synthesize the first intermediate layer and the second target layer to obtain the image of the current frame; wherein the second target layer is the layer to be synthesized other than the first target layer in the layers to be synthesized.

[0008] Based on the technical solution provided by this application, when all the layers to be synthesized in the current frame do not meet the parallel conditions (the hash value of the current frame is different from the hash value of the previous frame), when there are special layers that meet the preset conditions, GPU synthesis and query synthesis strategies can also be executed in parallel, reducing the time required for layer synthesis and the frame loss problem caused by excessive layer synthesis time, making the display effect of dynamic pictures more smooth and improving the user experience.

[0009] In a possible design method of the first aspect, based on the layer information of the layer to be synthesized, determining the first detection result includes: the electronic device calculates the hash value of the current frame based on the layer information of the layer to be synthesized; when the hash value of the current frame is the same as the hash value of the previous frame, determining that the first detection result meets the first condition; when the hash value of the current frame is different from the hash value of the previous frame, determining that the first detection result does not meet the first condition.

[0010] Based on the above design, it is possible to determine whether the first detection result meets the first condition by comparing the hashes, and further determine whether the GPU synthesis and query synthesis strategies can be executed in parallel.

[0011] In a possible design of the first aspect, after controlling the GPU to synthesize the first target layer to obtain a first intermediate layer and controlling the HWC to query the synthesis method of the layer to be synthesized, the method further includes: when there is a third target layer in the second target layer whose synthesis method is GPU synthesis, controlling the GPU to synthesize the third target layer into the first intermediate layer to obtain an updated first intermediate layer; the third target layer and the first target layer are different layers to be synthesized; controlling the HWC to synthesize the updated first intermediate layer and the layers to be synthesized in the second target layer except the third target layer to obtain an image of the current frame.

[0012] Based on this design, after executing the GPU synthesis and query synthesis strategies in parallel, the layer that requires GPU synthesis but has not yet been synthesized by the GPU (i.e., the third target layer) can be further synthesized into the intermediate layer (i.e., the first intermediate layer) obtained after executing the GPU synthesis and query synthesis strategies in parallel, thereby obtaining an updated intermediate layer. In this way, the layers to be synthesized that the HWC cannot synthesize can be smoothly handed over to the GPU for synthesis, while improving the efficiency of layer synthesis and ensuring that the final synthesis and display of the HWC can be completed smoothly.

[0013] In a possible design of the first aspect, the method further includes: when the first detection result does not meet the first condition and the first target layer does not exist in the layer to be synthesized, if there is a fourth target layer in the layer to be synthesized, controlling the GPU to synthesize the fourth target layer to obtain a second intermediate layer, and at the same time controlling the HWC to query the synthesis method of the layer to be synthesized; wherein the synthesis method of the fourth target layer is GPU synthesis; the synthesis method of the fourth target layer in the previous frame is GPU synthesis; when the synthesis method of the fifth target layer is HWC synthesis, controlling the HWC to synthesize the second intermediate layer and the fifth target layer to obtain an image of the current frame; wherein the fifth target layer is the layer to be synthesized other than the fourth target layer in the layer to be synthesized.

[0014] Based on the above design method, the electronic device can execute GPU synthesis and query synthesis strategies in parallel when the layer to be synthesized in the current frame does not meet the first condition (the hash value of the current frame is different from the hash value of the previous frame) and there is no special layer that meets the preset conditions. If there is a special layer that was GPU synthesized in the previous frame, it can reduce the time required for layer synthesis and the frame loss problem caused by too long layer synthesis time, making the display effect of dynamic pictures smoother and improving the user experience.

[0015] In a possible design of the first aspect, after controlling the GPU to synthesize the fourth target layer to obtain a second intermediate layer and controlling the HWC to query the synthesis method of the layer to be synthesized, the method further includes: when there is a sixth target layer in the fifth target layer whose synthesis method is GPU synthesis, controlling the GPU to synthesize the sixth target layer into the second intermediate layer to obtain an updated second intermediate layer; the sixth target layer and the fourth target layer are different layers to be synthesized; controlling the HWC to synthesize the updated second intermediate layer and the layers to be synthesized in the fifth target layer except the sixth target layer to obtain an image of the current frame.

[0016] Based on this design, after executing the GPU synthesis and query synthesis strategies in parallel, the layer that requires GPU synthesis but has not yet been synthesized by the GPU (i.e., the third target layer) can be further synthesized into the intermediate layer (i.e., the first intermediate layer) obtained after executing the GPU synthesis and query synthesis strategies in parallel, thereby obtaining an updated intermediate layer. In this way, the layers to be synthesized that the HWC cannot synthesize can be smoothly handed over to the GPU for synthesis, while improving the efficiency of layer synthesis and ensuring that the final synthesis and display of the HWC can be completed smoothly.

[0017] In a possible design method of the first aspect, the layer information includes: the identifier of the layer's buffer, layer attribute parameters and display parameters; the layer attribute parameters include any one or more of the following: blur parameter, transparency, position, size, name, type, visibility; the display parameters include any one or more of the following: color, transparency, blending mode, depth sorting, cropping parameters, occlusion parameters.

[0018] Based on the above design method, the electronic device can use rich layer information to determine whether the layer to be synthesized can be synthesized by the GPU or by the HWC. At the same time, it can accurately determine whether the first detection result meets the first condition, and then determine whether the GPU synthesis and query synthesis strategies can be executed in parallel, thereby ensuring the smooth progress of layer synthesis.

[0019] In a possible design manner of the first aspect, the newly added layer is a layer that does not exist in the previous frame.

[0020] Since the newly added layer itself increases the number of layers to be synthesized, it is more likely to exceed the synthesis capability of HWC. Therefore, based on the above design method, whether the layer is a newly added layer is used as a factor to determine whether the layer to be synthesized can be directly synthesized by the GPU, which may obtain more accurate judgment results.

[0021] In a possible design of the first aspect, before obtaining the layer information of the layer to be synthesized of the current frame, the method also includes: receiving a first operation from the user to start the first application; and obtaining the layer information of the layer to be synthesized of the current frame during the startup of the first application.

[0022] Because there can be significant changes between adjacent frames during application startup, existing layer synthesis logic can lead to excessively long synthesis times and frame drops. However, based on the aforementioned design approach, using the layer synthesis method provided in this application in application scenarios like the application startup process can improve layer synthesis efficiency, reduce frame drops, and make the application startup process smoother, enhancing the user experience.

[0023] In a second aspect, embodiments of the present application further provide a layer synthesis device that can be applied to electronic devices. The functions of the device can be implemented through hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as an acquisition module, a determination module, and a control module.

[0024] Among them, the acquisition module is used to obtain the layer information of the layer to be synthesized in the current frame. The determination module is used to determine the first detection result based on the layer information of the layer to be synthesized obtained by the acquisition module. The control module is used to control the GPU to synthesize the first target layer to obtain the first intermediate layer when the first detection result determined by the determination module does not meet the first condition and there is a first target layer that meets the preset conditions in the layer to be synthesized obtained by the acquisition module, and at the same time control the HWC to query the synthesis method of the layer to be synthesized; wherein the synthesis method of the first target layer that meets the preset conditions is GPU synthesis; the preset conditions include any one or more of the following: the layer attribute parameters or display parameters of the layer have changed; the layer is a newly added layer, and the layer attribute parameters include a first mark, the first mark is used to indicate that the GPU is used for synthesis; the layer is a newly added layer, and the layer buffer size is larger than the preset size; the layer whose synthesis method in the previous frame was GPU synthesis, and whose position has changed or the visible area has increased; the position of the layer has rotated 180° compared with the previous frame. The control module is also used to control the HWC to synthesize the first intermediate layer and the second target layer to obtain the image of the current frame when the synthesis method of the second target layer is HWC synthesis; wherein the second target layer is the layer to be synthesized other than the first target layer.

[0025] In a third aspect, the present application provides an electronic device comprising a display screen, a memory, and one or more processors; the display screen, the memory, and the processor are coupled; wherein the memory stores computer program code, and the computer program code comprises computer instructions, which, when executed by the processor, enable the electronic device to execute the layer synthesis method provided in the first aspect and any possible design method thereof.

[0026] In a fourth aspect, the present application provides a computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the layer synthesis method provided in the first aspect and any possible design method thereof.

[0027] In a fifth aspect, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the layer synthesis method provided in the first aspect and any possible design thereof.

[0028] It can be understood that the beneficial effects that can be achieved by the technical solutions provided in the second to fifth aspects mentioned above can be referred to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] FIG1 is a schematic diagram of a desktop provided in an embodiment of the present application;

[0030] FIG2 is a schematic diagram of the disassembly of the layers of the desktop provided in an embodiment of the present application;

[0031] FIG3 is a schematic diagram of a layer synthesis process provided by related technology;

[0032] FIG4 is a schematic diagram of a display process based on a Vsync signal provided by a related art;

[0033] FIG5 is a schematic diagram of a layer synthesis process provided by related art;

[0034] FIG6 is a schematic diagram showing the principle of a layer synthesis method provided in an embodiment of the present application;

[0035] FIG7 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0036] FIG8 is a schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application;

[0037] FIG9 is a first flow chart of a layer synthesis method according to an embodiment of the present application;

[0038] FIG10 is a schematic diagram of a user operation scenario provided in an embodiment of the present application;

[0039] FIG11 is a second flow chart of a layer synthesis method provided in an embodiment of the present application;

[0040] FIG12 is a third flow chart of a layer synthesis method provided in an embodiment of the present application;

[0041] FIG13 is a fourth flow chart of a layer synthesis method provided in an embodiment of the present application;

[0042] FIG14 is a schematic structural diagram of a layer synthesis device provided in an embodiment of the present application;

[0043] FIG15 is a schematic structural diagram of a chip system provided in an embodiment of the present application;

[0044] FIG16 is a schematic diagram of the structure of a computer program product provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application. As used in the specification and appended claims of the present application, the singular expressions "a", "an", "said", "above", "the" and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that " / " means or, for example, A / B can mean A or B; "and / or" in the text is merely a description of an association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0046] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0047] The terms "first" and "second" in the following embodiments of this application are used for descriptive purposes only and should not be understood as implying or suggesting relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0048] First, the nouns involved in the embodiments of this application are explained as follows:

[0049] (1) Buffer: A buffer is a memory area used to temporarily store data. The buffer is an important component of a computer system for temporarily storing data. The buffer is a reserved storage space in the memory that is used to buffer input or output data. Specifically, the buffer can coordinate the speed inconsistency between IO (input and output) devices and the central processing unit (CPU). By setting up a buffer between low-speed devices and high-speed devices, the constraints between the two can be removed, the efficiency of data transmission and processing can be improved, and thus the performance of the system can be improved. The buffer can be used to store data, and when the amount of data reaches a certain level, the actual read and write operations can be performed, which can reduce the number of read and write operations and improve efficiency. For example, in image processing, the image data can be read into the buffer, then processed, and finally the processed results can be written to the disk or display device.

[0050] (2) User Interface Thread (UI Thread): The UI thread, also known as the main thread, is the core thread responsible for handling the user interface (UI). It is responsible for rendering UI elements, including various controls, such as drawing events, and handling user input events, such as touch screens. All of these operations are scheduled and controlled by the main thread. It is stipulated that the controls in the activity are refreshed by the main thread, and other threads cannot refresh them directly.

[0051] (3) Render Thread: It is used to receive instructions from the UI thread and draw the UI components of each layer indicated by the UI thread into the frame buffer.

[0052] (4) Frame: A frame refers to a single image, the smallest unit, when an electronic device displays an image. A frame can be understood as a still image. Displaying multiple connected frames in rapid succession can create the illusion of motion. A frame can be synthesized from multiple layers. Before an electronic device displays a frame, it must first complete the drawing and rendering of each layer in the frame. After that, all the drawn and rendered layers are synthesized to obtain the final frame for display by the electronic device.

[0053] (5) Layer: Each application may correspond to one or more graphical interfaces, each of which is called a layer. Each frame of the display interface (or each frame of the screen) of an electronic device is usually composed of multiple layers.

[0054] For example, taking a mobile phone as an electronic device, as shown in FIG1 , a desktop frame displayed on the mobile phone may include a status bar 101, an application icon tray 102, a fixed icon bar 103, and a desktop background 104. The application icon tray 102 displays multiple application icons, which can be switched in response to a swipe operation within the application icon tray 102. The desktop shown in FIG1 can be divided into multiple layers, as shown in FIG2 . FIG2 exemplifies the layer structure of the desktop. As shown in FIG2 , the desktop shown in FIG1 may include four layers: an interactive layer 201, a fixed icon layer 202, a status bar layer 203, and a background layer 204. The icon layer 201 is the layer where user-operated and switchable controls (i.e., the application icon tray 102) reside. The fixed icon layer 202 is the layer where the fixed icon bar 103, whose entire content cannot be switched in response to a user swipe, resides. The status bar layer 203 is the layer where the status bar 101 resides. Background layer 204 is the layer where desktop background 104 (e.g., theme, wallpaper, etc.) resides. In some scenarios, if a user swipes application icon tray 102 to switch the application icons displayed in the tray, the desktop application will display a switching animation during the switching process. During this switching animation, only the interactive layer 201 will change.

[0055] Changes to a layer in the display interface can include changes in layer attribute parameters, layer content, and display parameters. Layer content refers to the content displayed in the layer, such as status icons such as network status, battery status, device connection status, phone settings status, or time in the status bar layer. Layer attribute parameters refer to parameters that define the characteristics of the layer itself, including the layer's type, name, visibility, position, size, transparency, blur parameters, etc. These parameters determine the basic properties and behavior of the layer and can be accessed and modified during the drawing and rendering process. Display parameters refer to parameters that control the appearance and effects of the layer in the final display, including color, transparency, blending mode (such as premultiplied per-pixel alpha blending and per-plane alpha blending), depth sorting (i.e., the front-to-back order of layers on the Z axis), clipping, and occlusion. These parameters are used to further control and adjust the layer during the rendering process to achieve the desired display effect.

[0056] (6) Display synthesis system (surfaceflinger, SF): surfaceflinger is a system service that is mainly used to implement functions such as layer creation, control, and management. In the display system of an electronic device, layer synthesis can be achieved through surfaceflinger. For example, after the application is started, surfaceflinger can create a layer for the application. During the operation of the electronic device, surfaceflinger can obtain the layers to be displayed by various applications running on the electronic device, and synthesize the obtained layers through the GPU and / or HWC. As an example, after the application is started, a layer creation request can be sent to surfaceflinger. After surfaceflinger receives the layer creation request from the application (specifically, it can be the UI thread of the application), it creates a corresponding layer for the application and returns the layer control surfacecontrol pointer of the created layer to the application so that the application can draw and render the layer. Among them, the created layer has corresponding layer attribute parameters and display parameters.

[0057] (7) Hardware composer (HWC): HWC (hwcomposer) is The hardware abstraction layer (HAL) layer module for layer synthesis and display in the SurfaceFlinger is implemented by a device-specific method and is usually completed by electronic device manufacturers (OEMs) to provide hardware support for SurfaceFlinger. The role of HWC is to help the display synthesis system SurfaceFlinger (which may be called SF synthesizer) synthesize layers, thereby reducing the amount of synthesis required by the SF synthesizer to perform layer synthesis through the GPU. It can abstract objects such as overlays and 2D bit block transmitters to synthesize images, and communicate with specialized layer synthesis hardware to synthesize layers. For example, the layer synthesis hardware used by HWC for layer synthesis may include hardware such as a digital signal processor (DSP), a vision processing unit (VPU), and a display processing unit (DPU).

[0058] Typically, HWC supports the following capabilities: at least four overlays: a status bar (displaying network status, etc.), a system bar (displaying home button, etc.), an application interface (displaying the content of the foreground application), and a wallpaper or background. The overlay can be larger than the display screen; for example, the wallpaper size can be larger than the display screen size. Premultiplied per-pixel alpha blending (premultiplied per-pixel alpha blending means that the color value of each pixel in the layer has been premultiplied by the corresponding alpha value before alpha blending) and per-plane alpha blending (per-plane alpha blending is used for transparency blending between multiple layers, so that the transparency of one layer can affect the transparency of all layers below it. In per-plane alpha blending, the transparency of each pixel is calculated based on the plane where the pixel is located, rather than based on a single pixel). To support protected content, a hardware path for protected video playback must be provided.

[0059] In electronic devices, SurfaceFlinger can use Open Graphics Library Embedded Systems (OpenGL ES) to synthesize layers, which requires and consumes GPU resources. However, most GPUs are not optimized for layer synthesis. When SurfaceFlinger receives a layer synthesis request from an application and synthesizes the layer through the GPU, the GPU resources are occupied, and the application or other applications cannot obtain enough GPU resources to perform other tasks such as layer rendering. This situation may cause the performance of the application to degrade, especially graphics-intensive applications. To solve this problem, current electronic devices can hand over the synthesis of some layers to the HWC using a dedicated hardware module for layer synthesis, thereby reducing the synthesis pressure on the GPU. In this way, applications in electronic devices can better utilize the GPU for self-rendering (that is, rendering the layers in the display image required by the application itself), improving graphics performance and user experience.

[0060] Specifically, the capabilities of electronic devices vary greatly, making it difficult to directly use application programming interface (API) parameters to indicate the types and number of layers supported by the display hardware, whether layers can be rotated and blended during the compositing process, and restrictions on layer positioning and hardware compositing. Therefore, the process of using HWC for layer compositing in electronic devices is as follows:

[0061] First, after receiving a drawing request for a frame of image, SurfaceFlinger provides HWC with information about all layers of the frame of image, allowing HWC to decide how to process these layers based on its hardware capabilities.

[0062] Specifically, after receiving information about all layers of a frame, the HWC determines whether each layer should be synthesized by the GPU or the HWC based on its own capabilities. That is, after receiving information about multiple layers of a frame, the HWC queries the synthesis strategy and sends the query to SurfaceFlinger. The synthesis strategy includes the synthesis method of each layer, that is, whether each layer is synthesized by the GPU or the HWC.

[0063] After that, SurfaceFlinger uses the GPU to composite all layers that need GPU synthesis into an output buffer according to the synthesis strategy. Then, SurfaceFlinger can hand over this output buffer and other layers that need HWC synthesis to HWC for final overlay synthesis and display.

[0064] FIG3 exemplifies a schematic diagram of a layer processing process. Taking a mobile phone as an example, as shown in FIG3 , assuming that the layers that need to be synthesized for a frame of image that the mobile phone currently needs to display may include: layer 11, layer 12, layer 13, and layer 14. After these layers are drawn and rendered by the application, they can be passed to surfaceflinger for layer synthesis. After surfaceflinger collects these layers, it can first be handed over to HWC to query the synthesis strategy to determine the synthesis method of each layer to be synthesized. In some embodiments, the HWC can determine the synthesis method of the layer with rounded corners in the layer to be synthesized to be synthesized by the GPU, and the synthesis method of the remaining layers to be synthesized to be synthesized by the HWC. Of course, how to obtain the specific synthesis method of each layer to be synthesized, that is, the synthesis strategy, the HWC also needs to combine its own capabilities and the content, size and other parameters of each layer to be synthesized, and use any feasible rules to determine, and this application does not impose specific restrictions on this.

[0065] Based on the above synthesis strategy, for example, the synthesis mode of layer 12 and layer 14 can be set to GPU synthesis mode, and the synthesis mode of layer 11 and layer 13 can be set to HWC synthesis mode.

[0066] After determining the composition method for each layer to be composited, SurfaceFlinger can hand off compositing of layers 12 and 14 (with GPU compositing) to the GPU, and hand off compositing of layers 11 and 13 (with HWC compositing) to the HWC. After compositing layers 12 and 14, the GPU submits the resulting intermediate layer to SurfaceFlinger in the form of an output buffer.

[0067] After that, surfaceflinger will hand over the intermediate layer synthesized by the GPU to HWC, which will synthesize the final target interface and transmit it to the display for display.

[0068] (8) Vertical synchronization signal (Vsync): When a display screen displays a row of pixels, it is called a row scan. When a display screen displays all rows of pixels, it is called a field scan. The physical signal that the display screen starts to scan a field is called a vertical synchronization signal. After scanning a field, the display screen can display a frame of image data, that is, the display screen can display a frame of image data within a Vsync signal cycle. How many fields can the display screen refresh per second, that is, how many frames of image data can the display screen display per second, the electronic device needs to generate that many Vsync signals per second. In order to avoid display freezes and improve the visual performance of graphics, the display system in the electronic device synchronizes the various processes of image display (including drawing, rendering, synthesis and display transmission) through the Vsync signal. For a certain frame of image, when the electronic device displays the frame of image based on the Vsync signal, it needs to execute the application rendering process in the Nth Vsync cycle, the layer synthesis process in the N+1th Vsync cycle, and the hardware display transmission process in the N+2th Vsync cycle. In other words, it takes at least 2 Vsync cycles for a frame of image to go from application rendering to hardware display transmission.

[0069] Typically, the Vsync signal can simultaneously trigger the application drawing and rendering process, the layer synthesis process, and the hardware display process.

[0070] For example, Figure 4 is a schematic diagram of a Vsync signal-based display process provided by related art. As shown in Figure 4, for an application in an electronic device that needs to display three frames: Frame 1, Frame 2, and Frame 3, the application can begin drawing and rendering Frame 1 upon receiving the Nth Vsync signal (VsyncN); begin drawing and rendering Frame 2 upon receiving the N+1th Vsync signal (VsyncN+1); and begin drawing and rendering Frame 3 upon receiving the N+2th Vsync signal (VsyncN+2). The drawing and rendering process does not exceed one Vsync cycle, that is, the time interval between two adjacent Vsync signals.

[0071] The surfaceflinger display synthesis system can start layer synthesis of frame 1 when receiving the N+1th Vsync signal VsyncN+1; start layer synthesis of frame 2 when receiving the N+2th Vsync signal VsyncN+2; and start layer synthesis of frame 3 when receiving the N+2th Vsync signal VsyncN+2. The layer synthesis process does not exceed one Vsync cycle, that is, it does not exceed the time interval between two adjacent Vsync signals.

[0072] The surfaceflinger display synthesis system can send frame 1 for display when receiving the N+2th Vsync signal VsyncN+2, so that the display starts displaying; when receiving the N+3th Vsync signal VsyncN+3, it can send frame 5 for display, so that the display starts displaying; when receiving the N+4th Vsync signal VsyncN+4, it can send frame 3 for display, so that the display starts displaying. The display process of each frame lasts for one Vsync cycle, that is, one frame of image is displayed in the time interval between each two adjacent Vsync signals after the display starts displaying the image.

[0073] Currently, electronic devices with display functions have a need to improve the efficiency of layer synthesis and reduce the pressure on the GPU for layer synthesis. For example, taking a mobile phone as an example, as shown in FIG5 , the process of performing layer synthesis in an existing electronic device may include the following steps:

[0074] Step 1: After receiving the layer drawing instruction, the user interface (UI) thread of the application sends a layer creation request to the surfaceflinger main thread (SF main thread) to request the creation of a target layer. The target layer can be one layer or multiple layers.

[0075] Step 2: After receiving the layer creation request from the UI thread, the SF main thread returns the target layer's surface control pointer to the UI thread. If the target layer requested by the layer creation request has not been created before, the SF main thread will create the layer after receiving the layer creation request and then return the target layer's surface control pointer to the UI thread.

[0076] Step 3. After the UI thread receives the surfacecontrol pointer, it can use the surfacecontrol pointer to draw the target layer content in the corresponding memory area and set the layer attribute parameters and display parameters of the target layer.

[0077] Step 4: The UI thread calls the rendering thread to render the target layer. Specifically, the UI thread may send the address pointer of the memory area storing the layer content, layer attribute parameters, and display parameters of the target layer to the rendering thread.

[0078] Step 5: The rendering thread draws the layer content of the target layer into the target buffer, and sends the target buffer, the layer attribute parameters of the target layer, and the display parameters to the SF binder thread.

[0079] Step 6: After the SF binder thread receives the target buffer (specifically, it can be a pointer to the target buffer), the layer attribute parameters and display parameters of the target layer, it will store them in SF to wait for the Vsync signal (also called Vsync-SF signal) to wake up the SF main thread for layer synthesis.

[0080] Step 7: The timer starts timing after receiving the vsync signal, and wakes up the timer thread after the timing ends.

[0081] Step 8: After the timer thread is started, it sends a start instruction to the SF main thread to instruct the SF main thread to start layer synthesis.

[0082] Step 9. After receiving the start instruction, the SF main thread can determine the layer information of the corresponding layer in all layers of the current frame based on the target buffer, the layer attribute parameters of the target layer, and the display parameters, and determine all visible layers in the current frame.

[0083] The layer information includes layer attribute parameters, display parameters, and the handle of the buffer to which it belongs.

[0084] Specifically, a visible layer refers to a layer that is ultimately to be presented on the display screen, and the visibility in the layer attribute parameters of the visible layer is "visible". Of course, in practice, "visible" can be a specific value.

[0085] Step 10: The SF main thread uses the layer information of each visible layer in the current frame to generate the hash value of the current frame.

[0086] Step 11: The SF main thread determines whether the hash value of the current frame is the same as the hash value of the previous frame.

[0087] If the hash value of the current frame differs from the previous frame's hash value, it can be assumed that the layers to be displayed in the current frame have changed from those in the previous frame. In this case, the query synthesis strategy and GPU synthesis are executed sequentially. In this case, step 11 is followed by step 12. After that, steps 13, 14, 15, and 16 are executed in sequence.

[0088] If the hash value of the current frame is the same as the hash value of the previous frame, it can be considered that the layer to be displayed in the current frame and the layer to be displayed in the previous frame have not changed. At this time, the query synthesis strategy and GPU synthesis can be executed in parallel, that is, step 17 (including step 17A and step 17B) and step 18 are started at the same time.

[0089] Step 12: The SF main thread sends the layer information of all visible layers to the HWC thread and requests to query the synthesis strategy.

[0090] Step 13: The HWC thread determines the synthesis method of all visible layers based on its own capabilities and the layer information of all visible layers, that is, the first synthesis strategy, and sends the first synthesis strategy to the SF main thread.

[0091] Step 14: The SF main thread stores the hash value of the current frame and the first synthesis strategy.

[0092] This makes it easier for the SF main thread to compare the hashes of the two frames when synthesizing the layers of the next frame, thereby determining the subsequent layer synthesis process.

[0093] Step 15: The SF main thread sends a first composition instruction to the SF composition thread.

[0094] The first GPU synthesis instruction carries layer information of the first layer whose synthesis method is GPU synthesis in the first synthesis strategy. The first synthesis instruction is used to instruct to place the intermediate layer synthesized by the first layer in the intermediate buffer.

[0095] The intermediate buffer can be selected by the SF main thread from the bufferqueue.

[0096] Based on this, the SF synthesis thread will use the GPU to synthesize the first layer, and the synthesized intermediate layer exists in the intermediate buffer.

[0097] Step 16: The SF synthesis thread uses the GPU to synthesize the first layer to obtain an intermediate layer. Specifically, the intermediate buffer containing the layer content of the intermediate layer is obtained.

[0098] After step 16, proceed to step 26.

[0099] Step 17A: The SF main thread adds a first flag to the layer information of the second layer. The first flag is used to indicate that GPU compositing is used.

[0100] Step 17B: The SF main thread sends the layer information of all visible layers to the HWC thread, requesting to query the synthesis strategy.

[0101] After step 17, proceed to step 19.

[0102] Step 18: The SF main thread sends a second synthesis instruction to the SF synthesis thread.

[0103] The second GPU synthesis instruction carries the layer information of the second layer in all visible layers whose synthesis method is GPU synthesis in the synthesis strategy of the previous frame. The second synthesis instruction is used to indicate that the middle layer synthesized by the second layer is placed in the middle buffer.

[0104] After step 18, proceed to step 23.

[0105] Step 19: Based on its own capabilities and the layer information of all visible layers, the HWC thread determines the synthesis method of other layers marked as GPU synthesis when the synthesis method of some visible layers is marked as GPU synthesis, and then obtains the second synthesis strategy and sends the second synthesis strategy to the SF main thread.

[0106] After step 19, proceed to step 20.

[0107] Step 20: The SF main thread stores the hash value of the current frame and the second synthesis strategy.

[0108] After step 20, proceed to step 21.

[0109] Step 21: The SF main thread determines whether the second synthesis strategy is the same as the synthesis strategy of the previous frame.

[0110] If it is determined that the second synthesis strategy is the same as the synthesis strategy of the previous frame, it means that all layers in the current frame that need to be synthesized by the GPU have been sent to the SF synthesis thread for synthesis. Here, the SF main thread can determine that all layers that need GPU synthesis have been synthesized, and can subsequently instruct the HWC to perform the final synthesis, that is, execute step 26.

[0111] If it is determined that the second synthesis strategy is different from the synthesis strategy of the previous frame image, it means that in addition to the second layer, there may be a third layer among all the layers of the current frame that requires GPU synthesis. At this time, the SF main thread can send the layer information of the third layer to the SF synthesis thread for synthesis, that is, execute step 22.

[0112] Step 22: The SF main thread sends a third composition instruction to the SF composition thread.

[0113] Among them, the third synthesis indication carries the layer information of the third layer other than the second layer in the layer whose synthesis method is GPU synthesis in the second synthesis strategy. The third synthesis indication is used to instruct to synthesize the third layer with the intermediate layer in the intermediate buffer.

[0114] After step 22, execute step 25.

[0115] Step 23: The SF synthesis thread uses the GPU to synthesize the second layer to obtain an intermediate layer, which may be a buffer of the intermediate layer.

[0116] After step 23, execute step 23A.

[0117] Step 23A: The SF composition thread sends a composition completion indication to the SF main thread.

[0118] The composition is complete indicating that the intermediate layer composition is complete.

[0119] After step 23A, execute step 21.

[0120] In practice, through specific synchronization methods, the time it takes for the HWC to return the composition strategy to the SF main thread and for the SF composition thread to use the GPU to composite the intermediate layer and return it to the SF main thread can be the same. That is, step 23A and step 19 (or step 20) will be executed simultaneously, or after either step 19 or step 23A is completed, the subsequent process will wait until the other is completed. This allows the execution to proceed to step 21 after steps 19 and 23A, ensuring the smooth implementation of the entire layer composition process.

[0121] Step 24: The SF synthesis thread uses the GPU to update the middle layer using the third layer.

[0122] Specifically, it may be to update the intermediate layer in the intermediate buffer.

[0123] The intermediate layer updated in step 24 may specifically be the intermediate layer obtained in step 23 .

[0124] After step 24, proceed to step 25.

[0125] Step 25: The SF composition thread sends a composition completion indication to the SF main thread.

[0126] Step 26: The SF main thread sends the identifier of the intermediate layer to the HWC thread and instructs the HWC to synthesize all layers of the current frame and send them for display.

[0127] There are three situations in which the synthesis completion indication obtained by the SF main thread in step 25: the first is that when step 25 is executed after step 16 is completed, the SF synthesis thread sends the synthesis completion indication to the SF main thread; the second is that when step 23A is executed, the SF synthesis thread sends the synthesis completion indication to the SF main thread; the third is that when step 25 is executed after step 24 is completed, the SF synthesis thread sends the synthesis completion indication to the SF main thread.

[0128] The identifier of the middle layer may specifically be the identifier of the middle buffer.

[0129] Step 27: The HWC thread synthesizes the intermediate layer and the layer whose synthesis method is HWC synthesis in the target synthesis strategy to obtain the image of the current frame and sends it for display.

[0130] The target synthesis strategy can be the first synthesis strategy or the second synthesis strategy.

[0131] Step 28: After the HWC thread completes the display transmission, it returns a display transmission completion indication to the SF main thread, so that SF can clearly proceed with the synthesis of the next frame layer.

[0132] Based on the entire layer synthesis process described above, it can be seen that in some dynamic image display scenarios, such as application startup or exit, volume bar or notification message pop-up, the electronic device is likely to experience significant changes in the layers between the previous and next frames during the display process (for example, changes in parameters such as rounded corners, transparency, and blur). However, based on the existing layer synthesis logic shown in Figure 5, during the layer synthesis process of these frames, the SF judgment needs to serially execute the query synthesis strategy and GPU synthesis. In other words, in this case, the entire layer synthesis process must first query the synthesis strategy through the HWC to obtain the synthesis method for each layer to be synthesized; secondly, the GPU synthesizes some of the layers to be synthesized; and finally, the HWC synthesizes the GPU synthesis results with the remaining layers to be synthesized. During the entire layer synthesis process, the HWC needs to wait for the GPU synthesis to complete before synthesizing the GPU synthesis results with the remaining layers. This reduces the efficiency and flexibility of layer synthesis, which may result in the inability to complete layer synthesis and display within the display cycle of a single frame, causing frame loss and affecting the user experience.

[0133] In response to the above-mentioned problems, an embodiment of the present application provides a method for determining the cause of a frame loss failure, which is applied to an electronic device. In this technical solution, as shown in FIG6 , after the electronic device has determined all the layers to be synthesized in the current frame, it will first determine whether all the layers to be synthesized in the current frame meet the parallel conditions. Specifically, the electronic device can determine the hash value of the current frame based on the layer information of all the layers to be synthesized. The layer information may include layer attribute parameters, display parameters, and an identifier of the buffer corresponding to the layer. When the hash value of the current frame is different from the hash value of the previous frame, it is determined that all the layers to be synthesized in the current frame do not meet the parallel conditions; when the hash value of the current frame is the same as the hash value of the previous frame, it is determined that all the layers to be synthesized in the current frame meet the parallel conditions.

[0134] When all the layers to be synthesized in the current frame do not meet the parallel conditions, in order to improve the efficiency of layer synthesis, the electronic device will hand over the first layer that meets the preset conditions among all the layers to be synthesized to the GPU for synthesis, and at the same time request the HWC to query the synthesis strategy for other layers to obtain the synthesis strategy of the current frame. Among them, the preset conditions include any one or more of the following: the layer attribute parameters or display parameters of the layer have changed; the layer is a newly added layer, and the layer attribute parameters include a first mark, the first mark is used to indicate that the GPU is forced to be used for synthesis; the layer is a newly added layer, and the buffer size of the layer is larger than the preset size (for example, 2160px*2160px); the synthesis method of the previous frame is GPU synthesis, and the position parameters have changed or the visible area has become larger; the position parameters of the layer have rotated 180° compared to the previous frame. Of course, in practice, the preset conditions can also be any other feasible conditions, which can be determined based on the experience of the mobile phone manufacturer's designers. This application does not impose specific restrictions on this.

[0135] The composition strategy of the current frame includes the composition methods of all layers to be composited, where the composition method of the first layer is GPU composition.

[0136] Afterwards, after the first layer is synthesized using the GPU, the synthesized intermediate layer can be provided to the HWC.

[0137] After obtaining the synthesis strategy of the current frame through HWC, if there is a second layer in addition to the first layer in the layer synthesized by GPU, the electronic device will also hand over the second layer to the GPU for synthesis to update the intermediate layer, and provide the updated intermediate layer to HWC.

[0138] Finally, the electronic device can use HWC to synthesize the intermediate layer and the layer to be synthesized, whose synthesis strategy indicates that the synthesis method is HWC synthesis, to obtain the image of the current frame and send it for display.

[0139] In the technical solution provided in this application, when the hash value of the current frame is different from the hash value of the previous frame, GPU synthesis and query synthesis strategies can be executed in parallel, reducing the time required for layer synthesis and the frame loss problem caused by excessive layer synthesis time, making the display effect of dynamic images more smooth and improving the user experience.

[0140] The technical solutions provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0141] The technical solution provided in this application can be applied to electronic devices with dynamic effect display functions. In some embodiments, the electronic device can be a mobile phone, a tablet computer, a handheld computer, a personal computer (PC), an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, an in-vehicle device, a smart home device, and / or a smart city device, etc. The embodiments of this application do not impose any special restrictions on the specific type of the electronic device.

[0142] For example, taking the electronic device as a mobile phone as an example, FIG7 shows a schematic structural diagram of an electronic device provided in an embodiment of the present application.

[0143] 7 , the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a display 193, a subscriber identification module (SIM) card interface 194, and a camera 195. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0144] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), a display processing unit (DPU), a vision processing unit (VPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0145] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0146] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0147] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0148] The charging management module 140 is used to receive charging input from a power supply device (e.g., a charger, laptop charger, etc.). The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input through the wireless charging coil of the electronic device.

[0149] While charging the battery 142, the charging management module 140 can also power the electronic device through the power management module 141. Specifically, the battery 142 can be composed of multiple batteries connected in series. The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110.

[0150] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 and provides power to the processor 110, the internal memory 121, the display 193, the camera 195, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery voltage, current, battery cycle count, and battery health status (leakage, impedance). In other embodiments, the power management module 141 can also be provided in the processor 110.

[0151] The external memory interface 120 can be used to connect to an external non-volatile memory device to expand the storage capacity of the electronic device. The external non-volatile memory device communicates with the processor 110 via the external memory interface 120 to implement data storage. For example, files such as music and videos can be stored in the external non-volatile memory device.

[0152] The internal memory 121 may include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs). The RAM can be directly read and written by the processor 110 and can be used to store executable programs (e.g., machine instructions) of the operating system or other running programs, as well as user and application data. The NVM can also store executable programs and user and application data, and can be pre-loaded into the RAM for direct reading and writing by the processor 110.

[0153] A touch sensor, also known as a "touch control device," can be provided on the display screen 193. The touch sensor and the display screen 193 form a touch screen, also known as a "touch screen." The touch sensor is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided via the display screen 193. In other embodiments, the touch sensor can also be provided on the surface of the electronic device, at a location different from that of the display screen 193.

[0154] The ambient light sensor is used to sense ambient light brightness. For example, the ambient light sensor can measure the light intensity of four channels of ambient light. The ambient light sensor outputs the measured light intensity of the four channels of ambient light to processor 110. Processor 110 can process the light intensity of the four channels of ambient light output by the ambient light sensor to obtain the ambient light intensity. In the bright screen state, the electronic device can adaptively adjust the display brightness based on the obtained ambient light intensity.

[0155] The pressure sensor is used to sense pressure signals and can convert pressure signals into electrical signals. In some embodiments, the pressure sensor can be set on the display screen 193. There are many types of pressure sensors, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. When a touch operation is applied to the display screen 193, the electronic device monitors the touch operation intensity based on the pressure sensor. The electronic device can also calculate the position of the touch based on the monitoring signal of the pressure sensor. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, the instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

[0156] In some embodiments, the electronic device may include 1 or N cameras 195, where N is a positive integer greater than 1. In an embodiment of the present application, the type of camera 195 can be distinguished based on the hardware configuration and physical location. For example, the camera provided on the side of the display screen 193 of the electronic device can be called a front camera, and the camera provided on the side of the back cover of the electronic device can be called a rear camera; for another example, a camera with a short focal length and a larger viewing angle can be called a wide-angle camera, and a camera with a long focal length and a small viewing angle can be called a normal camera. Among them, the length of the focal length and the size of the viewing angle are relative concepts, and there are no specific parameters to limit them. Therefore, wide-angle cameras and normal cameras are also relative concepts, and can be specifically distinguished based on physical parameters such as focal length and viewing angle.

[0157] The electronic device implements display functionality through a GPU, display screen 193, and an application processor. The GPU is a microprocessor for image processing that connects display screen 193 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0158] The electronic device can implement a shooting function through an ISP, a camera 195, a video codec, a GPU, a display screen 193, and an application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information. In the embodiment of the present application, the GPU function is used during the frame drawing process of each image frame to achieve better display effects and performance of the final displayed image.

[0159] The ISP processes data fed back by camera 195. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise and brightness. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be incorporated into camera 195. Camera 195 is used to capture still images or video.

[0160] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when an electronic device selects a frequency, the DSP performs a Fourier transform on the frequency energy.

[0161] Display screen 193 is used to display images, videos, and the like. Display screen 193 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, the electronic device can include one or N display screens 193, where N is a positive integer greater than one.

[0162] In an embodiment of the present application, the display screen 193 can be used to display the page that the electronic device needs to display. The display screen can also display a picture display interface and display images taken from any one or more cameras 195 in the picture display interface.

[0163] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem and baseband processor.

[0164] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in an electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0165] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to electronic devices. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter, amplify, and perform other processing on the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0166] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 193. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0167] The wireless communication module 160 can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0168] SIM card interface 194 is used to connect a SIM card. A SIM card can be connected to and disconnected from the electronic device by inserting or removing it from the SIM card interface 194. An electronic device may support one or more SIM card interfaces. SIM card interface 194 can support Nano SIM cards, Micro SIM cards, and SIM cards. Multiple cards can be inserted into the same SIM card interface 194 simultaneously. SIM card interface 194 is also compatible with external memory cards. Electronic devices interact with the network through SIM cards to implement functions such as call and data communications. Each SIM card corresponds to one user number.

[0169] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present invention is only a schematic illustration and does not constitute a structural limitation of the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0170] Of course, it is understood that FIG7 is merely an example of an electronic device in the form of a mobile phone. If the electronic device is a tablet computer, handheld computer, PC, PDA, wearable device (such as a smart watch, smart bracelet), or other device form factors, the structure of the electronic device may include fewer or more structures than shown in FIG7, and this is not limited here.

[0171] It is understandable that, in general, the realization of electronic device functions requires not only hardware support but also software cooperation. The software system of the electronic device can adopt a layered architecture, event-driven architecture, micro-core architecture, micro-service architecture, or cloud architecture. Taking the system as an example, the software structure of the electronic device is illustrated.

[0172] Figure 8 is a schematic diagram of the layered architecture of the software system of the electronic device provided in an embodiment of the present application. The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces (e.g., APIs).

[0173] In some examples, as shown in FIG8 , in an embodiment of the present application, the software of an electronic device is divided into five layers, namely, from top to bottom, the application layer, the framework layer (or application framework layer), the system library and Android runtime (Android runtime), the HAL layer (hardware abstraction layer), and the driver layer (or kernel layer). Among them, the system library and Android runtime can also be called the local framework layer or native layer.

[0174] The application layer may include a series of applications. As shown in FIG7 , the application layer may include applications (APPs) such as camera, gallery, calendar, map, WLAN, Bluetooth, news, music, video, short message, call, navigation, and instant messaging.

[0175] When an application is working, it can start the corresponding UI thread to begin drawing and rendering the display interface. If the UI thread needs to adjust the target layer in the display interface or create a target layer, it can request the corresponding surfacecontrol pointer from the SF main thread, so that the UI thread can draw the layer content of the target layer and set the layer attribute parameters and display parameters of the target layer. After that, the UI thread can send the layer content, layer attribute parameters and display parameters of the target layer to the drawing thread, so that the rendering thread can draw and render the target layer.

[0176] The framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions or services. For example, the application framework layer may include an activity manager, a window manager, a content provider, an audio service, a view system, a telephony manager, a resource manager, a notification manager, a package manager, etc., but the embodiments of this application do not impose any restrictions on this.

[0177] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0178] Content providers are used to store and retrieve data and make it accessible to applications. This data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0179] The view system includes visual controls, such as controls for displaying text and images. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface containing a text notification icon can include a view for displaying text and a view for displaying images. In some embodiments, the view system can also include or start a rendering thread to perform operations such as drawing and rendering.

[0180] After the rendering thread receives the layer content, layer attribute parameters, and display parameters of the target layer from the UI thread, the rendering thread can draw the content of the target layer into the target buffer, and send the target buffer, the layer attribute parameters, and display parameters of the target layer to the display composition system (SurfaceFlinger, SF) for storage.

[0181] The phone manager is used to provide communication functions for electronic devices. For example, the phone manager can manage the call status of the call application (including initiation, connection, and hang up).

[0182] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0183] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0184] Package Manager in The system is used to manage application packages. It allows applications to obtain detailed information about installed applications and their services, permissions, etc. The package manager is also used to manage events such as application installation, uninstallation, and upgrades.

[0185] The system library can include multiple functional modules. For example: surface manager, media library, OpenGL ES, SGL, etc. The surface manager is used to manage the display subsystem and provides 2D and 3D layer fusion for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. OpenGL ES is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing. SGL is the drawing engine for 2D drawing.

[0186] The Android runtime consists of core libraries and the ART virtual machine. The Android runtime is responsible for scheduling and management of the Android system. The core libraries consist of two parts: one for Java-based functions and the other for the Android core library. The application layer and application framework layer run in the ART virtual machine. The ART virtual machine executes Java files from the application layer and application framework layer as binary files. The ART virtual machine is responsible for performing functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0187] In an embodiment of the present application, a compositing system (SurfaceFlinger, SF) may also be displayed in the system library. SF may include an SF binder thread, an SF timer thread, an SF main thread, and an SF compositing thread (SF RenderEngine rendering thread). The SF binder thread is used to communicate with other processes or threads, the SF timer thread can instruct the SF main thread to start layer compositing after startup, and the SF compositing thread can use the GPU for layer compositing.

[0188] In some embodiments, the SF binder thread can receive the target buffer, layer attribute parameters of the target layer, and display parameters from the rendering thread and store them in the SF. The timer corresponding to the SF timer thread can start counting after receiving the Vsync signal, and trigger the SF timer thread to start after the count ends. Once started, the SF timer thread can instruct the SF main thread to start layer compositing.

[0189] After receiving the instruction, the SF main thread can work with the SF synthesis thread to synthesize the layers. For specific implementation, please refer to the relevant descriptions in the subsequent embodiments, which will not be described in detail here.

[0190] It should be noted that, compared to existing layer synthesis processes, the SF main thread in the layer synthesis process provided by this application uses specific methods to enable GPU synthesis and query synthesis strategies to be executed in parallel when the hash value of the previous frame differs from the hash value of the previous frame, thereby improving layer synthesis efficiency and reducing layer synthesis time. The specific methods used will be described in detail in subsequent embodiments and will not be elaborated on here.

[0191] The HAL layer is an interface layer located between the operating system kernel and the hardware circuit. Its purpose is to abstract the hardware. It hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system, making it hardware-independent and portable across multiple platforms. The HAL layer provides a standard interface to display device hardware capabilities to the higher-level Java API framework (i.e., the framework layer). The HAL layer contains multiple library modules, each of which implements an interface for a specific type of hardware component, such as the audio HAL audio module, the bluetooth HAL Bluetooth module, the camera HAL camera module (also known as the camera HAL or camera hardware abstraction module), and the sensors HAL sensor module (or sensor service).

[0192] In the embodiment of the present application, the HAL layer also includes a hardware composer (HWC). The relevant introduction of the hardware composer can refer to the relevant content in the explanation of terms in the above embodiment, and will not be repeated here.

[0193] In this embodiment of the present application, upon receiving a request from the SF main thread to query the composition strategy, the HWC is configured to determine the composition method (i.e., composition strategy) for each layer based on the layer information of all layers from the SF main thread, and return the composition strategy to the SF main thread. The HWC is also configured to, upon receiving information about intermediate layers (e.g., buffer identifiers) and an instruction to composite and display them from the SF main thread, composite the intermediate layers with all layers whose composition method is HWC composition, to obtain the final current frame image and display it.

[0194] The kernel layer is the layer between hardware and software. It includes at least a display driver, a camera driver, an audio driver, a sensor driver, a battery driver, and other components, though this application does not limit these. Specifically, the sensor driver may include drivers for each sensor in the electronic device, such as the ambient light sensor. Upon receiving a display instruction from the HWC, the display driver controls the display to display the current frame image from the HWC.

[0195] The technical solutions provided in the embodiments of this application can be implemented in electronic devices having the above-mentioned hardware architecture or software architecture.

[0196] Based on the software architecture shown in FIG8 above, the layer synthesis method provided by the embodiment of the present application is introduced below in conjunction with FIG9. FIG9 is a flow chart of a layer synthesis method provided by the embodiment of the present application. Referring to FIG9, the layer synthesis method may include S901-S938:

[0197] S901. After responding to a layer drawing instruction, the UI thread sends a first request to the SF main thread.

[0198] The first request is used to request the surface control pointer of the target layer. The first request may include the identifier of the target layer, such as the name. The target layer may be one layer or multiple layers.

[0199] In an embodiment of the present application, a layer drawing instruction may be generated after a user performs any possible operation on an electronic device, for example, the operation may be opening an application, exiting an application, switching application pages, or sliding an application page.

[0200] For example, consider a mobile phone as an electronic device, where a layer drawing instruction is triggered by a user opening an application. As shown in FIG10(a), the mobile phone's desktop may display multiple application icons. To view images in the gallery, the user may click on the gallery application icon 1002 on the desktop. In response to this click, the phone may first display an application launch animation, such as a blurred image centered on the gallery application icon that gradually zooms in until it reaches full screen. After the launch animation is complete, the image display interface shown in FIG10(b) may be displayed. During the process of the gallery application being triggered to open and displaying the image display page, the phone may display multiple frames, including the multiple frames included in the application launch animation and the image display interface frame. Each frame may include multiple layers, each of which can be a target layer. Layer drawing instructions may be automatically generated by the phone or its application after each frame is displayed after the user clicks to open the gallery application.

[0201] In this application, the gallery application may be referred to as the first application, and the click operation performed by the user on the gallery application icon 1002 may be referred to as the first operation. The layer composition method provided in this application may be implemented during the startup of the first application after the user performs the first operation. The startup process of the first application may refer to the period from the time the user performs the first operation until the first application displays the main interface.

[0202] Of course, the layer synthesis method provided in this application can also be implemented in the display process of other dynamic images.

[0203] Of course, in practice, when the layer drawing instruction is generated and by which module it is generated can be determined according to actual needs, and this application does not impose specific restrictions on this.

[0204] In some embodiments, the first request may be a layer creation request requesting the creation of a new layer, and the new layer may be the target layer. The first request may also be a layer adjustment request requesting the adjustment of an already created layer, and the already created layer may be the target layer. Of course, the SF main thread may also determine whether the target layer has been created based on the target layer identifier carried in the first request, and further determine whether the first request is a layer creation request or a layer adjustment request. This application does not impose any specific restrictions on this.

[0205] exist In the system, the surfacecontrol pointer is used to control and manage the display of the graphical interface (i.e., layer). It is part of the Android graphics system, used to handle windows, surfaces, and operations related to screen display. Among them, windows and surfaces can be considered as layers. Specifically, the surfacecontrol pointer can be an address pointer pointing to a storage area in the memory allocated by the SF main thread for the target layer. Exemplarily, the memory can specifically refer to random access memory (RAM). A storage area in the memory allocated to the target layer can be called an off-screen buffer.

[0206] S902. The SF main thread sends the surfacecontrol pointer of the target layer to the UI thread.

[0207] If the target layer is a newly created layer, the SF main thread can allocate a storage area for the target layer after receiving the first request, and return the corresponding surfacecontrol pointer to the UI thread.

[0208] If the target layer is not a newly created layer, after receiving the first request, the SF main thread can determine the storage area corresponding to the target layer based on the identifier of the target layer carried in the first request, and then return the corresponding surfacecontrol pointer to the UI thread.

[0209] S903. The UI thread draws the layer content of the target layer based on the surfacecontrol pointer of the target layer from the SF main thread, and sets the layer attribute parameters and display parameters of the target layer.

[0210] Once the UI thread obtains the surfacecontrol pointer of the target layer, it can draw the layer content of the target layer in the memory area corresponding to the pointer and set the corresponding layer attribute parameters and display parameters.

[0211] For example, layer attribute parameters may include blur parameter, transparency, position, size, name, type, visibility, etc. Display parameters may include color, transparency, blending mode, depth sorting (or Z-axis sorting), clipping parameters, occlusion parameters, etc.

[0212] In some embodiments, in combination with FIG9 and as shown in FIG11 , S903 may specifically include S9031 and S9032:

[0213] S9031. The UI thread performs measurement and layout based on the surfacecontrol pointer of the target layer from the SF main thread.

[0214] Based on measurement and layout measures, the UI thread can determine the size and position of each element in the layer content of the target layer, thereby facilitating the subsequent drawing of the layer content of the target layer.

[0215] S9032. The UI thread draws the layer content of the target layer according to the measurement and layout results, and sets the layer attribute parameters and display parameters of the target layer.

[0216] In one embodiment, the UI thread can call an OpenGL ES3.1 interface or a Vulkan library function interface to draw the layer content of the target layer. Of course, in practice, the UI thread can use any feasible method to draw the target layer, and the embodiment of the application does not specifically limit the drawing method of the UI thread.

[0217] After the UI thread completes drawing the target layer, it can call the rendering thread to perform a step of drawing and rendering on the target layer, that is, execute S904.

[0218] S904: The UI thread calls the rendering thread to render the target layer.

[0219] In one possible implementation, the UI thread can use a binder-based cross-process call mechanism to call the rendering thread, wake up the rendering thread, and instruct the rendering thread to render the target layer. Specifically, the UI thread can send a drawing instruction to the rendering thread through the cross-process call mechanism to instruct the rendering thread to render the target layer. The drawing instruction can carry a pointer or identifier to the memory area where the UI thread draws the target layer.

[0220] S905: The rendering thread in the mobile phone draws the layer content of the target layer into the target buffer, and sends the target buffer, the layer attribute parameters and display parameters of the target layer to the SF binder thread.

[0221] After the rendering thread receives a call from the UI thread or receives a drawing instruction from the UI thread, the rendering thread can draw the layer content of the target layer into the target buffer, and send the target buffer, the layer attribute parameters and display parameters of the target layer to the SF binder thread so that SF can perform subsequent layer synthesis.

[0222] In some embodiments, in combination with FIG9 and as shown in FIG11 , S905 may specifically include S9051-S9053:

[0223] S9051. The rendering thread selects an empty buffer from the buffer queue bufferqueue as the target buffer.

[0224] In one possible implementation, the rendering thread may select a blank buffer from the bufferqueue as the target buffer according to a first-in, first-out principle. Specifically, the rendering thread may select the blank buffer that enters the bufferqueue earliest as the target buffer.

[0225] In another possible implementation, the rendering thread may select a blank buffer from the bufferqueue as the target buffer based on priority. Specifically, the rendering thread may select the blank buffer that best meets the specific conditions (i.e., the highest priority) based on specific conditions (e.g., buffer size, availability, or special requirements) as the target buffer.

[0226] Of course, in practice, how the rendering thread selects an empty buffer from the bufferqueue as the target buffer can be determined according to actual needs, and this application does not impose any specific restrictions on this.

[0227] S9052: The rendering thread draws the layer content in the target layer into the target buffer.

[0228] Specifically, the rendering thread can use OpenGL (Open Graphics Library) instructions to draw each element of the layer content of the target layer into the target buffer. During this process, the rendering thread can also use some rendering techniques to optimize drawing performance (such as batching and clipping).

[0229] After the rendering thread completes drawing the target layer, it needs to instruct the SF to start layer synthesis. At this time, the rendering thread needs to inform the SF main thread of the data related to the target layer through the SF binder thread, that is, execute S9052.

[0230] S9053: The rendering thread sends the target buffer, the layer attribute parameters of the target layer, and the display parameters to the SF binder thread.

[0231] Specifically, because the target buffer stores a large amount of data, it's inconvenient to directly transmit all of it. Therefore, the rendering thread can send the target buffer's address or pointer, the target layer's layer attribute parameters, and display parameters to the SF binder thread. Alternatively, the target layer's layer attribute parameters and display parameters can also be stored in the target buffer. In this case, the rendering thread can send only the target buffer's address or pointer to the SF binder thread.

[0232] In some embodiments, the rendering thread may send first instruction information to the SF binder thread while sending the target buffer, layer attribute parameters of the target layer, and display parameters. The first instruction information is used to instruct the SF to perform layer synthesis.

[0233] After the SF binder thread receives information about the target layer from the rendering thread (target buffer, layer attribute parameters and display parameters of the target layer), this information can be stored in SF for use by the SF main thread when performing layer synthesis. At the same time, the SF binder thread can request a Vsync signal (specifically, a Vsync-SF signal) to wake up the SF main thread for layer synthesis. Of course, in practice, the Vsync signal is not requested, but is periodically generated by the hardware in the electronic device according to the screen refresh rate. Therefore, after the SF binder thread receives information about the target layer from the rendering thread, it will wait for the Vsync signal to wake up the SF main thread for layer synthesis. That is, execute S906.

[0234] S906 , the SF binder thread stores the target buffer, the layer attribute parameters and the display parameters of the target layer in the SF, and waits for the Vsync signal to wake up the SF main thread for layer synthesis.

[0235] Exemplarily, the target buffer stored by the SFbinder thread in the SF may specifically be the address or pointer of the target buffer. If the layer attribute parameters and display parameters of the target layer are both present in the target buffer, S906 may specifically be that the SF binder thread stores the address or pointer of the target buffer in the SF.

[0236] S907 , the timer in the SF starts timing after receiving the vsync signal, and wakes up the SF timer thread to start after the timing ends.

[0237] Once the timer thread is started, it instructs the SF main thread to begin layer compositing. Generally, the timer is set to a short duration to ensure that the interval between compositing two frames of image layers matches the system interval corresponding to the refresh rate. For example, at a refresh rate of 120Hz, the system interval can be 1000 / 120 = 8.3ms, or a multiple thereof.

[0238] In a possible implementation, the timer in the SF may register a callback function for the Vsync signal in the hardware abstraction layer, so that when a new Vsync signal appears, the hardware abstraction layer notifies the timer through the callback function to start timing.

[0239] The purpose of using the Vsync signal to wake up the SF main thread through the timer and timer thread to start layer synthesis can be the following two aspects:

[0240] (1) During the entire process of layer synthesis and display, the SF main thread is usually responsible for handling a large number of tasks, and these tasks usually need to be executed at a high frequency. The SF main thread may face a large workload, especially when dealing with complex interfaces and a large amount of dynamic content. The timer thread can be used to decouple the SF main thread from the direct response to the Vsync signal, reducing the workload of the SF main thread. Moreover, if the execution timing or frequency of the SF main thread needs to be changed, it is only necessary to adjust the timer parameters instead of modifying the code of the SF main thread. This decoupling method improves the flexibility and maintainability of the program.

[0241] (2) Due to hardware limitations (SF cannot accurately respond to Vsync signals), race conditions between threads in a multi-threaded environment, and excessive system load, the SF main thread may experience delays or jitter when responding directly to Vsync signals. However, the timer thread can trigger the execution of the SF main thread according to a predetermined time interval, avoiding the above-mentioned problems that may be encountered when responding directly to Vsync signals. This makes the stability and accuracy of the Vsync signal waking up the SF main thread for layer compositing more improved.

[0242] S908. The SF timer thread sends second indication information to the SF main thread.

[0243] The second indication information is used to instruct the SF main thread to perform layer synthesis.

[0244] S909. The SF main thread responds to the second indication information and determines the layer information of the optional layer corresponding to the target layer in all layers of the current frame based on the target buffer, the layer attribute parameters of the target layer, and the display parameters.

[0245] The layer information may include: an identifier or pointer of the buffer corresponding to the layer, layer attribute parameters, and display parameters. Specifically, the identifier or pointer of the buffer corresponding to the layer may be a handle of the buffer corresponding to the layer.

[0246] After receiving the instruction from the timer thread (such as the second instruction information), the SF main thread will first obtain all the layers currently registered in SF and the corresponding layer information, and use these layers as the layers of the current frame. Afterwards, since the layer information of these layers at this time is obtained during the display process of the previous frame, the SF main thread needs to update the layer information of the optional layers corresponding to the target layer in all layers of the current frame based on the information of the target layer. Among them, the SF main thread can determine the optional layers corresponding to the target layer in all layers of the current frame by the name of the layer. For example, the names of all layers of the current frame may include: a, A, Ab, ab; the name of the target layer may be A. Then, the layer named A in all layers of the current frame can be determined as the optional layer corresponding to the target layer.

[0247] It should be noted that a single image frame may contain multiple layers, and the layer information of each layer may change in the next frame, and the corresponding application of each layer may also be different. Therefore, the above S901-S909 can be executed multiple times within a single Vsync cycle, or multiple similar processes can be executed in parallel. Each time the SF main thread obtains the latest layer information for a layer, it will update the layer information of the corresponding layer before layer synthesis to ensure that the final displayed image meets the expectations.

[0248] To improve compositing efficiency, the SF main thread can omit any processing on invisible layers that do not need to be displayed on the display screen and only composite the visible layers that need to be displayed on the display screen. Based on this, after the SF main thread completes updating the layer information of the layers in the current frame, it needs to select the visible layer as the layer to be composited in the current frame, that is, execute S910.

[0249] S910. The SF main thread uses the visible layers among all layers of the current frame as the layers to be synthesized of the current frame.

[0250] Among them, the visible layer refers to the layer that needs to be presented on the display screen in the end. The visibility in the layer attribute parameters of the visible layer is "visible". Of course, in practice, "visible" can be a specific value. For example, the value of visibility in the layer attribute parameters can be "true" or "false". When the value of visibility is "true", the visibility is "visible", that is, the corresponding layer is a visible layer; when the value of visibility is "false", the visibility is "invisible", that is, the corresponding layer is an invisible layer.

[0251] After the electronic device obtains the layer information of the layer to be synthesized in the current frame, it can further determine whether the query synthesis strategy and GPU synthesis processes can be performed in parallel. Specifically, it can determine whether the first detection result meets the parallel condition. In this application, the parallel condition can be referred to as the first condition.

[0252] In some embodiments, the first detection result may be whether the hash value of the current frame is the same as the hash value of the previous frame. That is, S911 and S912 are executed after S910. It should be noted that the electronic device may obtain the layer information of all layers to be synthesized or the layer information of some layers to be synthesized.

[0253] S911. The SF main thread determines the hash value of the current frame based on the layer information of the layer to be synthesized in the current frame.

[0254] In practice, SF can determine the hash value of the current frame based on the layer information of all layers to be synthesized in the current frame.

[0255] Specifically, the SF main thread can use a specific hash algorithm to process the layer information of the layer to be synthesized to obtain the hash value of the current frame. For example, the code of the hash algorithm can be as follows:

[0256] S912. The SF main thread determines whether the hash value of the current frame is the same as the hash value of the previous frame.

[0257] If the hash value of the current frame is different from the hash value of the previous frame, it can be considered that the layer to be displayed in the current frame (i.e., the layer to be synthesized) and the layer to be displayed in the previous frame have changed. According to the existing logic, it is determined that the first detection result does not meet the parallel conditions at this time, and the query synthesis strategy and GPU synthesis need to be executed serially. This will cause the layer synthesis to take a long time, which may cause frame loss problems. Therefore, in order to avoid the serial execution of the query synthesis strategy and GPU synthesis, the SF main thread can first determine whether there is a first layer that meets the preset conditions among all the layers to be synthesized in the current frame, that is, execute S913. If so, the first layer can be controlled to perform GPU synthesis, and the query synthesis strategy can be executed at the same time, that is, the query synthesis strategy and GPU synthesis are executed in parallel to improve the efficiency of layer synthesis.

[0258] If the hash value of the current frame is the same as the hash value of the previous frame, it can be considered that the layer to be displayed in the current frame and the layer to be displayed in the previous frame have not changed. At this time, it is determined that the first detection result meets the parallel condition, and then the query synthesis strategy (control HWC query synthesis strategy) and GPU synthesis (control GPU to synthesize the second layer whose synthesis method in the previous frame synthesis strategy is GPU synthesis) can be executed in parallel, that is, S914 and S928 are started at the same time.

[0259] S913. The SF main thread determines whether there is a first layer that meets the preset conditions among all the layers to be synthesized in the current frame.

[0260] The preset conditions may be determined by the electronic device designer based on the functional limitations of the HWC itself, statistical data on which layers need to be synthesized by the GPU in practice, and work experience. In this application, the first layer may be referred to as the first target layer.

[0261] Exemplarily, the preset conditions may include any one or more of the following: the layer attribute parameters or display parameters of the layer have changed; the layer is a newly added layer, and the layer attribute parameters include a first mark, the first mark is used to indicate the use of GPU for synthesis; the layer is a newly added layer, and the layer buffer size is larger than the preset size (for example, 2160px*2160px); the synthesis method of the previous frame is GPU synthesis, and the position of the layer has changed or the visible area has become larger; the position of the layer has rotated 180° compared to the previous frame. The buffer size of the layer can be obtained from the memory based on the buffer pointer of the layer. Among them, the newly added layer refers to the layer that did not appear in the previous frame, that is, the layer that did not exist in the previous frame.

[0262] The first mark in the layer attribute parameters can be that after the SF main thread obtains all the layers to be synthesized, it determines the layers to be synthesized that need to be synthesized by the GPU according to any feasible rules or methods, and then adds the first mark to the layer attribute parameters of such layers to be synthesized.

[0263] Of course, in practice, the preset conditions may also be any other feasible conditions, which may be determined according to actual needs, and this application does not impose any specific restrictions on this.

[0264] When it is determined that there is no first layer that meets the preset conditions among all the layers to be synthesized in the current frame, the SF main thread needs to execute the query synthesis strategy and GPU synthesis in series, that is, execute S915-S919 in sequence.

[0265] If it is determined that a first layer that meets the preset conditions exists among all layers to be synthesized in the current frame, the electronic device can concurrently perform GPU synthesis (synthesizing the first layer) and control the HWC to query the synthesis strategy. Specifically, the SF main thread can simultaneously hand over the first layer to the GPU for synthesis while using the HWC to query the synthesis strategy based on the layer information of all layers to be synthesized. In other words, S920 and S921 are executed simultaneously.

[0266] S914. The SF main thread sends a first query request to the HWC thread.

[0267] The first query request carries the layer information of all layers to be composited and indicates that the composite method for the second layer is GPU composite. The second layer is the layer whose composite method is GPU composite in the composite strategy of the previous frame among all layers to be composited. For example, if the composite strategy of the previous frame indicates that the composite method for layer A (A can be the name of the layer) is GPU composite, then layer A is the second layer.

[0268] Specifically, indicating that the second layer's composition method is GPU composition can be implemented by adding a first flag to the layer information of the second layer. The first query request is used to request a composition strategy, which includes requesting a composition method for the layer to be composed. Subsequent similar query requests are similar and will not be further described.

[0269] In some embodiments, adding the first mark to the layer information of the second layer may be performed by the SF main thread before S914 and after S912 is performed when it is determined that the hash value of the current frame is the same as the hash value of the previous frame.

[0270] After the HWC thread receives the first query request from the SF main thread, it can obtain the first query strategy based on the request and return it to the SF main thread. That is, S929 is executed after S914.

[0271] S915. The SF main thread sends a second query request to the HWC thread.

[0272] The second query request carries the layer information of all layers to be synthesized, and is used to query the synthesis strategy.

[0273] In some embodiments, if the layer information of the layers to be synthesized exists in the buffer of the layers to be synthesized, the second query request may carry the identifiers of the buffers of all the layers to be synthesized, rather than the layer information of all the layers to be synthesized. Similar contents in subsequent embodiments are similar.

[0274] S916 . The HWC thread determines a second compositing strategy based on the layer information of all layers to be composited in response to the second query request, and sends the first compositing strategy to the SF main thread.

[0275] After receiving the second query request from the SF main thread, the HWC thread can determine the composition method for each layer to be composited based on its own capabilities and the layer information of all layers to be composited, that is, obtain the second composition strategy. After determining the second composition strategy, it can send the second composition strategy to the SF main thread.

[0276] S917. The SF main thread receives the second synthesis strategy from the HWC thread and stores the hash value of the current frame and the second synthesis strategy.

[0277] After SF stores the hash value and the second synthesis strategy of the current frame, the hash value and the second synthesis strategy of the current frame can be used as the hash value and the synthesis strategy of the previous frame in the layer synthesis process of the next frame.

[0278] In one possible implementation, there may be a specific storage space in SF, and the specific storage control is used to store the previous frame variable. The previous frame variable is used to store the hash value of the previous frame and the synthesis strategy of the previous frame. Each time the SF main thread stores the hash value of the current frame and the synthesis strategy from the HWC thread (such as the second synthesis strategy / third synthesis strategy / first synthesis strategy / fifth synthesis strategy, etc.), it can use the hash value of the current frame and the synthesis strategy from the HWC thread to update the content in the previous frame variable. In this way, in the layer synthesis process of the next frame, the corresponding data can be obtained from the previous frame variable for use.

[0279] S918. The SF main thread sends a first composition instruction to the SF composition thread.

[0280] The first composition instruction carries the layer information of the third layer, whose composition method is GPU composition in the second composition strategy, and the identifier of the first intermediate buffer. The first composition instruction is used to instruct the intermediate layer of the third layer to be placed in the first intermediate buffer. This first intermediate buffer can be a blank buffer selected from the bufferqueue by the SF main thread. The specific implementation of the SF main thread selecting a blank buffer from the bufferqueue can be referred to the relevant description after S9051 in the above embodiment and will not be repeated here.

[0281] In some embodiments, all layer information of a layer may be stored in the layer buffer. In this case, the layer information of a layer sent between different threads may be an identifier or pointer to the layer buffer. The same applies to subsequent embodiments.

[0282] It should be noted that there is no necessary order between S917 and S918. S917 can be executed first, S918 can be executed first, or S917 and S918 can be executed at the same time.

[0283] S919. The SF synthesis thread controls the GPU to synthesize the third layer in response to the first synthesis instruction to obtain an intermediate layer.

[0284] Specifically, the SF compositing thread controls the GPU to composite the third layer. Specifically, it can control the GPU to fuse all the layer contents of the third layer into the first intermediate buffer based on the layer information of the third layer. At this time, the layer in the first intermediate buffer is the intermediate layer (specifically, the layer content in the first intermediate buffer is the layer content of the intermediate layer).

[0285] After the SF synthesis thread obtains the intermediate layer, it can inform the SF main thread of the situation so that it can proceed with the subsequent synthesis process. Specifically, the SF synthesis thread can send a synthesis completion indication indicating that the synthesis of the intermediate layer is completed to the SF main thread, that is, execute S935.

[0286] In some embodiments, in combination with FIG9 , as shown in FIG11 , S918 may include S918A, and S919 may include S9191 and S9192 :

[0287] S918A: The SF main thread selects a blank buffer from the bufferqueue as a first intermediate buffer, and sends a first composition instruction to the SF composition thread.

[0288] S9192. The SF synthesis thread responds to the first synthesis instruction, uses the first preset function to call the GPU to synthesize the third layer, and places the synthesized layer content in the first intermediate buffer.

[0289] The first preset function is used to instruct the GPU to work asynchronously, that is, to instruct the GPU to perform the task of synthesizing the third layer in parallel while performing other tasks, and to place the layer content of the synthesized layer in the first intermediate bufferr.

[0290] Exemplarily, the first healing function may be a glFlush function.

[0291] S9192. The SF synthesis thread calls a second preset function to obtain a predetermined mark.

[0292] The predetermined mark is used to indicate whether the GPU synthesis is completed or not. For example, the second preset function may be a gl function, and the predetermined mark may be a Fence.

[0293] In Science Fiction, calling the gl function to obtain a fence to track the completion of GPU compositing is a mechanism used to ensure that layer compositing operations have completed. A fence is a synchronization object used to transmit signals between multiple threads or processes. In Science Fiction, a fence indicates whether layer compositing operations on the GPU are complete. When the Science Fiction compositing thread instructs the GPU to perform layer compositing, it creates a fence object and passes it to the GPU. After layer compositing is complete, the GPU marks the fence as complete, indicating that the GPU compositing is complete.

[0294] Therefore, in order to promptly know whether the GPU has completed compositing, the SF compositing thread can obtain the Fence in real time or periodically through the gl function and return it to the SF main thread. When the Fence indicates that the GPU compositing is complete, it is equivalent to the SF compositing thread returning a compositing completion indication to the SF main thread.

[0295] In the subsequent embodiments, the specific implementation of S918 and S919 is similar and will not be described in detail.

[0296] S920. The SF main thread sends a third query request to the HWC thread.

[0297] The third query request carries the layer information of all layers to be composited and indicates that the composite method for the first layer is GPU composite. Specifically, indicating that the composite method for the first layer is GPU composite can be implemented by adding a first flag to the layer information of the first layer. The third query request is used to query the composite strategy.

[0298] In some embodiments, adding the first mark to the layer information of the second layer may be performed by the SF main thread before S920 and after S913 is performed when it is determined that the first layer exists.

[0299] After S920, execute S922.

[0300] S921. The SF main thread sends a second composition instruction to the SF composition thread.

[0301] The second composition instruction carries the layer information of all first layers, and is used to instruct to composite the second layers into the first intermediate buffer.

[0302] After receiving the layer information of all first layers from the SF main thread, the SF synthesis thread may control the GPU to synthesize all first layers to obtain an intermediate layer, that is, execute S923.

[0303] S922: The HWC thread determines a third synthesis strategy in response to the third query request, and sends the third synthesis strategy to the SF main thread.

[0304] Specifically, after receiving the third query request, the HWC thread can determine the compositing method for the layers other than the first layer based on its own capabilities and the layer information of the layers other than the first layer. This can then determine the compositing method for all layers to be composited, i.e., the third compositing strategy. In the third compositing strategy, the first layer is composited using GPU compositing.

[0305] After S922, S924 and S925 are executed.

[0306] S923. The SF synthesis thread controls the GPU to synthesize the first layer in response to the second synthesis instruction to obtain an intermediate layer.

[0307] Specifically, the SF compositing thread utilizes the GPU to composite the first layer. Specifically, it controls the GPU to fuse all of the first layer's layer content into a first intermediate buffer based on the first layer's layer information. This first intermediate buffer includes the layer content of the intermediate layer. The intermediate layer obtained by compositing the first layer in step S923 can be referred to as the first intermediate layer.

[0308] After the SF synthesis thread obtains the intermediate layer, it can inform the SF main thread of the situation to facilitate the main thread to perform subsequent synthesis processes. Specifically, after synthesizing the intermediate layer, the SF synthesis thread can send a synthesis completion indication to the SF main thread, that is, execute S923A.

[0309] For the specific implementation of S921 and S923, reference may be made to the relevant descriptions of S918 and S919 in the aforementioned embodiments, which will not be repeated here.

[0310] S923A: The SF composition thread sends a composition completion indication to the SF main thread.

[0311] The synthesis completion indication is used to indicate that the synthesis of the intermediate layer is completed.

[0312] In practice, through specific synchronization means, when GPU synthesis and query synthesis strategies are executed in parallel, the time taken by HWC to return the synthesis strategy to the SF main thread and the time taken by the SF synthesis thread to obtain the intermediate layer through GPU synthesis and inform the SF main thread can be the same, that is, S923A and step S922 will be executed at the same time, or after either step S923A or S922 is completed, the subsequent process will be executed only after the other is completed.

[0313] Since the intermediate layer being synthesized by the SF compositing thread is created by compositing the first layer using the GPU, it's uncertain whether there are any other layers in the layers to be synthesized that require GPU synthesis. Therefore, after obtaining the third compositing strategy and the identifier of the intermediate layer, the SF main thread must determine whether there are any other layers in the layers to be synthesized that require GPU synthesis, other than the first layer. This can be determined by determining whether there are any other layers in the layers to be synthesized that require GPU synthesis, other than the first layer, in the layers to be synthesized that the third compositing strategy indicates are synthesized using GPU synthesis. That is, after both S923A and S922, step S925 must be executed.

[0314] S924. The SF main thread receives the third synthesis strategy from the HWC thread and stores the hash value of the current frame and the third synthesis strategy.

[0315] After SF stores the hash value and the third synthesis strategy of the current frame, the hash value and the third synthesis strategy of the current frame can be used as the hash value and the synthesis strategy of the previous frame in the layer synthesis process of the next frame image.

[0316] S925. The SF main thread determines whether there is a fourth layer among all the layers to be synthesized based on the third synthesis strategy.

[0317] The fourth layer is a layer to be synthesized other than the first layer among the layers to be synthesized, whose synthesis method is GPU synthesis, indicated by the third synthesis strategy. That is, the fourth layer and the first layer are different layers to be synthesized. In this application, the fourth layer can be referred to as the third target layer.

[0318] If, based on the third compositing strategy, it is determined that the fourth layer does not exist among the layers to be synthesized, this indicates that the only layers to be synthesized by the GPU in the current frame include the first layer, and all layers have already been sent to the SF synthesis thread for synthesis. The SF main thread does not need to perform any other actions here, only needing to control the HWC to perform the final layer synthesis and send it for display. This means that S936 is then executed. In this application, the absence of a fourth layer among the layers to be synthesized can be referred to as the synthesis method for the second target layer being HWC synthesis. The second target layer is the layer to be synthesized other than the first target layer among the layers to be synthesized.

[0319] If the third compositing strategy determines that a fourth layer exists among the layers to be composited, this indicates that, in addition to the first layer, a fourth layer requiring GPU compositing exists among all the layers to be composited in the current frame. The SF main thread can then send the layer information of the fourth layer to the SF compositing thread, enabling it to use the GPU to update the intermediate layer composited from the first layer, thereby executing step S926. In this application, the presence of a fourth layer among the layers to be composited can be referred to as the presence of a third target layer in the second target layer whose compositing method is GPU compositing.

[0320] It should be noted that there is no necessary order between S924 and S925. S924 may be executed first, or S925 may be executed first, or S924 and S925 may be executed simultaneously.

[0321] S926. The SF main thread sends a third composition instruction to the SF composition thread.

[0322] The third composition instruction carries the layer information of all fourth layers, and the third composition instruction is used to instruct to composite the fourth layer with the intermediate layer in the first intermediate buffer.

[0323] After receiving the third composition instruction from the SF main thread, the SF composition thread can control the GPU to composite the fourth layer into the intermediate layer composited from the first layer for updating, thereby obtaining an updated intermediate layer.

[0324] S927 . The SF synthesis thread controls the GPU to synthesize the fourth layer into the intermediate layer synthesized from the first layer in response to the third synthesis instruction.

[0325] Specifically, the SF synthesis thread can control the GPU to merge the layer content of the fourth layer with the layer content of the intermediate layer in the first intermediate buffer based on the layer information of the fourth layer, thereby updating the layer content of the intermediate layer synthesized from the first layer in the first intermediate buffer.

[0326] The specific implementation of S926 and S927 can refer to the relevant description of S918 and S919 in the above embodiment, which will not be repeated here. In this way, all layers that cannot be synthesized by HWC can be smoothly synthesized by GPU, ensuring that the final synthesis and display of HWC can be completed smoothly.

[0327] In some embodiments, when updating an intermediate layer, the SF may select a new empty buffer as the second intermediate buffer and place the updated intermediate layer in the second intermediate buffer. In this case, the third synthesis instruction in S926 is used to indicate that the new intermediate layer (specifically, the layer content of the new intermediate layer) obtained after synthesizing the fourth layer with the intermediate layer in the intermediate buffer is placed in the second intermediate buffer. S927 may specifically be that the SF synthesis thread uses the GPU to fuse the layer content of the fourth layer with the layer content of the intermediate layer in the first intermediate buffer based on the layer information of the fourth layer, and then places the layer content of the fused new intermediate layer in the second intermediate buffer.

[0328] In this embodiment, S926 may specifically include: the SF main thread selecting a blank buffer from the bufferqueue as the second intermediate buffer and sending a third composition instruction to the SF composition thread. S927, when implemented, may include: the SF composition thread, in response to the third composition instruction, calling a first preset function to instruct the GPU to composite the fourth layer with the intermediate layer in the first intermediate buffer, and placing the layer content of the composited layer in the second intermediate buffer; and the SF composition thread calling a second preset function to obtain a predetermined marker.

[0329] After the SF synthesis thread updates the intermediate layer, it can inform the SF main thread of the completion of the intermediate layer update (or the completion of the intermediate layer synthesis) so that it can perform the subsequent synthesis process.

[0330] S928. The SF main thread sends a fourth composition instruction to the SF composition thread.

[0331] The fourth synthesis instruction carries the layer information of all second layers, and is used to instruct to place the intermediate layer synthesized from the second layers in the first intermediate buffer.

[0332] After receiving the fourth synthesis instruction from the SF main thread, the SF synthesis thread may control the GPU to synthesize all first layers to obtain an intermediate layer, that is, execute S930 .

[0333] S929 . The HWC thread determines a first synthesis strategy in response to the first query request, and sends the first synthesis strategy to the SF main thread.

[0334] Specifically, after receiving the first query request, the HWC thread can determine the compositing method for the layers other than the second layer in the layers to be composited based on its own capabilities and the layer information of the layers other than the second layer. This can then determine the compositing method for all layers to be composited, i.e., the first compositing strategy. In the first compositing strategy, the second layer is composited using GPU compositing.

[0335] After S929, S931 and S932 are executed.

[0336] S930 : The SF synthesis thread controls the GPU to synthesize the second layer in response to the fourth synthesis instruction to obtain an intermediate layer.

[0337] Specifically, the SF compositing thread controls the GPU to composite the second layer. Specifically, the GPU can be controlled to fuse the layer contents in all buffers of the second layer into the first intermediate buffer based on the layer information of the second layer. At this time, the layer in the first intermediate buffer is the intermediate layer.

[0338] After the SF synthesis thread obtains the intermediate layer, it can inform the SF main thread of the situation to facilitate the main thread to perform subsequent synthesis processes. Specifically, after synthesizing the intermediate layer, the SF synthesis thread can send a synthesis completion indication to the SF main thread, that is, execute S930A.

[0339] For the specific implementation of S928 and S930, reference may be made to the relevant descriptions of S918 and S919 in the aforementioned embodiments, which will not be repeated here.

[0340] S930A: The SF composition thread sends a composition completion indication to the SF main thread.

[0341] In practice, through specific synchronization means, when GPU synthesis and query synthesis strategies are executed in parallel, the time taken by HWC to return the synthesis strategy to the SF main thread and the time taken by the SF synthesis thread to obtain the intermediate layer through GPU synthesis and inform the SF main thread can be made to be the same. That is, S930A and step S929 will be executed and completed at the same time, or after either step S930A or S929 is completed, the subsequent process will be executed only after the other is completed.

[0342] Since the intermediate layer being synthesized by the SF compositing thread is created by compositing the second layer using the GPU, it's uncertain whether there are any other layers in the to-be-composited layers that require GPU compositing. Therefore, after obtaining the first compositing strategy and the identifier of the intermediate layer, the SF main thread must also determine whether there are any other layers in the to-be-composited layers that require GPU compositing, other than the second layer. This can be determined by determining whether the first compositing strategy is the same as the compositing strategy of the previous frame. That is, after both steps S930A and S929, step S932 must be executed.

[0343] S931. The SF main thread receives the first synthesis strategy from the HWC thread and stores the hash value of the current frame and the first synthesis strategy.

[0344] After SF stores the hash value and the first synthesis strategy of the current frame, the hash value and the first synthesis strategy of the current frame can be used as the hash value and the synthesis strategy of the previous frame in the layer synthesis process of the next frame image.

[0345] S932. The SF main thread determines whether the first synthesis strategy is the same as the synthesis strategy of the previous frame.

[0346] If the first compositing strategy is determined to be the same as the compositing strategy of the previous frame, then all layers to be composited by the GPU in the current frame have been sent to the SF compositing thread for compositing. The SF main thread does not need to take any other actions here, but only needs to instruct the HWC to perform the final layer composition and send it for display. Then, S936 is executed.

[0347] If it is determined that the first synthesis strategy is different from the synthesis strategy of the previous frame image, it means that in addition to the second layer, there may be a fifth layer among all the layers to be synthesized in the current frame that requires GPU synthesis. At this time, the SF main thread can send the layer information of the fifth layer to the SF synthesis thread, so that it can use the GPU to update the intermediate layer synthesized by the second layer, that is, execute step S933.

[0348] It should be noted that there is no necessary order between S931 and S932. S931 can be executed first, S932 can be executed first, or S931 and S932 can be executed at the same time.

[0349] S933. The SF main thread sends a fifth composition instruction to the SF composition thread.

[0350] The fifth composition instruction carries the layer information of all fifth layers, and is used to instruct to composite the fifth layer with the intermediate layers in the first intermediate buffer.

[0351] The fifth layer is a layer other than the second layer among the layers whose synthesis method is GPU synthesis in the first synthesis strategy.

[0352] After receiving the layer information of all fifth layers from the SF main thread, the SF composition thread can control the GPU to use the fifth layer to update the intermediate layer composed of the second layer to obtain the latest intermediate layer, that is, execute S934.

[0353] S934: In response to the fifth synthesis instruction, the SF synthesis thread controls the GPU to use the fifth layer to update the intermediate layer synthesized from the second layer.

[0354] Specifically, the SF synthesis thread controls the GPU to use the fifth layer to update the intermediate layer synthesized by the second layer. Specifically, it can control the GPU to merge the layer content of the fifth layer with the layer content of the intermediate layer in the first intermediate buffer according to the layer information of the fifth layer, thereby updating the layer content of the intermediate layer synthesized by the second layer in the first intermediate buffer.

[0355] In some embodiments, when updating an intermediate layer, the SF may select a new empty buffer as the second intermediate buffer and place the updated intermediate layer in the second intermediate buffer. In this case, the fifth synthesis instruction in S933 is used to indicate that the new intermediate layer (specifically, the layer content of the new intermediate layer) obtained after synthesizing the fifth layer with the intermediate layer in the first intermediate buffer is placed in the second intermediate buffer. S934 may specifically be that the SF synthesis thread controls the GPU to fuse the layer content of the fourth layer with the layer content of the intermediate layer in the first intermediate buffer based on the layer information of the fifth layer, and then places the layer content of the fused new intermediate layer in the second intermediate buffer.

[0356] The specific implementation of S933 and S934 can refer to the relevant descriptions of S926 and S928 in the aforementioned embodiment, and will not be repeated here.

[0357] After the SF synthesis thread updates the intermediate layer, it can inform the SF main thread of the completion of the intermediate layer update so that it can perform the subsequent synthesis process, that is, execute S935.

[0358] S935. The SF composition thread sends a composition completion indication to the SF main thread.

[0359] After S935, execute S936.

[0360] S936. The SF main thread sends a synthesis instruction to the HWC thread.

[0361] The composition instruction carries the identifier of the intermediate layer and instructs the HWC to combine the intermediate layer with the layer whose composition method is HWC composition in the final composition strategy and then send it for display. The final composition strategy can be the first composition strategy, the second composition strategy, or the third composition strategy.

[0362] If S935 is executed before S936, and S919 is executed before S935, the final synthesis strategy is the second synthesis strategy; if S932 is executed before S936, or S934 and S935 are executed in sequence before S936, the final synthesis strategy is the first synthesis strategy; if S925 is executed before S936, or S925 and S935 are executed at once before S936, the final synthesis strategy is the third synthesis strategy.

[0363] In some embodiments, the synthesis instruction may also carry layer information of all layers whose synthesis method in the final synthesis strategy is HWC synthesis.

[0364] There are five situations for the synthesis completion indication when S935 is executed:

[0365] The first case is when S935 is executed after S919 is executed, and the SF synthesis thread sends a synthesis completion instruction to the SF main thread; the second case is when S923A is executed, and the SF synthesis thread sends a synthesis completion instruction to the SF main thread; the third case is when S935 is executed after S927 is executed, and the SF synthesis thread sends a synthesis completion instruction to the SF main thread; the fourth case is when S930A is executed, and the SF synthesis thread sends a synthesis completion instruction to the SF main thread; the fifth case is when S935 is executed after S934 is executed, and the SF synthesis thread sends a synthesis completion instruction to the SF main thread.

[0366] In the third and fifth cases, if the SF compositing thread places the updated intermediate layer in the second intermediate buffer when updating it, the intermediate layer identifier can be the identifier of the second intermediate buffer. In all other cases, the intermediate layer identifier is the identifier of the first intermediate buffer.

[0367] S937. The HWC thread responds to the synthesis instruction, synthesizes the intermediate layer and the layer to be synthesized whose synthesis method is indicated by the final synthesis strategy as HWC synthesis, obtains the image of the current frame, and sends it for display.

[0368] After receiving the synthesis instruction, the HWC thread can fuse the layer content of the sixth layer with the layer content of the middle layer according to the layer information of the sixth layer, so as to obtain the image of the current frame.

[0369] In some embodiments, the HWC thread's synthesis of the intermediate layer and the sixth layer can be completed by the DPU. Specifically, the HWC thread can send the layer information of the sixth layer and the identifier of the intermediate layer to the display driver. The display driver can control the DPU to synthesize the intermediate layer and the sixth layer when the latest tearing effect (TE) signal is generated, thereby obtaining the image of the current frame. The display driver can then send the image of the current frame to the integrated circuit (IC) for display.

[0370] After the HWC thread completes layer synthesis and sends it for display, in order to ensure smooth layer synthesis of the next frame, S938 can be executed after S937.

[0371] S938. After the HWC thread completes the display transmission, it returns a display transmission completion indication to the SF main thread.

[0372] In this way, the SF can clearly know that it can synthesize the layer of the next frame. For example, the display completion indication can include releaseFence.

[0373] In some embodiments, after the HWC thread completes compositing of all layers, it may also return a composition completion indication to the SF main thread, where the composition completion indication is used to indicate that all layers have been composited. For example, the composition completion indication may include a retireFence.

[0374] In other embodiments, when the SF main thread receives the display completion indication and synthesis end indication from the HWC, the SF main thread will also perform corresponding post-processing. Specifically, the SF main thread will execute the callback callback of the application monitoring the display completion, that is, return the display completion indication message to the applications corresponding to all the layers to be synthesized in the current frame. At the same time, the SF main thread can also update the timestamp parameters of all the layers to be synthesized to clarify the time of the latest synthesis and display of all the layers to be synthesized. Finally, if there is a buffer of a layer to be synthesized that has not been consumed due to various possible reasons, that is, the layer to be synthesized has not been synthesized, it can be instructed to be processed when the next Vsync-SF signal arrives.

[0375] Based on the technical solution provided in the embodiments of the present application, when all the layers to be synthesized in the current frame do not meet the parallel conditions (the hash value of the current frame is different from the hash value of the previous frame), when there are special layers that meet the preset conditions, GPU synthesis and query synthesis strategies can also be executed in parallel, reducing the time required for layer synthesis and the frame loss problem caused by excessive layer synthesis time, making the display effect of dynamic pictures more smooth and improving the user experience.

[0376] In some embodiments, when the SF main line determines that the hash value of the current frame is different from the hash value of the previous frame, if it is determined that there is no first layer that meets the preset conditions among all the layers to be synthesized in the current frame, then in order to improve the layer synthesis efficiency and reduce the layer synthesis time. At this time, the electronic device can force all the layers to be synthesized in the current frame that are synthesized by GPU synthesis in the synthesis strategy of the previous frame to be directly GPU synthesized, and query the synthesis strategy at the same time, thereby achieving the purpose of parallel execution of GPU synthesis and query synthesis strategy. Based on this, in combination with Figure 9, as shown in Figure 12, after S913 is executed, when it is determined that there is no first layer that meets the preset conditions among all the layers to be synthesized in the current frame, the method also includes S1201-S1209:

[0377] S1201. The SF main thread determines whether there is a sixth layer among all layers to be synthesized in the current frame.

[0378] The sixth layer is the layer whose compositing method in the previous frame's compositing strategy is GPU compositing. For example, if the compositing method of layer B (B can be the name of the layer) in the previous frame's compositing strategy is GPU compositing, then layer B is the sixth layer. In this application, the sixth layer can be referred to as the fourth target layer.

[0379] If it is determined that the sixth layer does not exist among all layers to be synthesized in the current frame, the SF main thread needs to execute the query synthesis strategy and GPU synthesis in series, that is, execute S915-S919 in sequence. The specific content of S915-S919 is not shown in Figure 12. The specific content and implementation of S915-S919 can be referred to Figure 9 and the related description of Figure 9, and will not be repeated here.

[0380] If it is determined that a sixth layer exists among all layers to be synthesized in the current frame, the electronic device can execute GPU synthesis and query synthesis strategies in parallel. Specifically, the SF main thread can hand over the sixth layer to the GPU for synthesis (specifically, it can control the GPU to synthesize the sixth layer to obtain the second intermediate layer) while simultaneously controlling the HWC to query the synthesis strategy (specifically, it can control the HWC to query the synthesis method of the layers to be synthesized). At this time, S1202 and S1203 are executed simultaneously.

[0381] S1202. The SF main thread sends a fourth query request to the HWC thread.

[0382] The fourth query request carries the layer information of all layers to be composited and indicates that the composite method for the sixth layer is GPU composite. Specifically, indicating that the composite method for the sixth layer is GPU composite can be implemented by adding a first flag to the layer information of the sixth layer. The fourth query request is used to query the composite strategy.

[0383] In some embodiments, adding the first mark to the layer information of the sixth layer may be performed by the SF main thread before S1202 and after S1201 is performed when it is determined that the sixth layer exists.

[0384] After the HWC thread receives the fourth query request from the SF main thread, it can obtain the fourth query strategy based on the request and return it to the SF main thread. That is, S1204 is executed after S1202.

[0385] S1203: The SF main thread sends a sixth composition instruction to the SF composition thread.

[0386] The sixth composition instruction carries the layer information of all sixth layers, and is used to instruct to composite the sixth layer into the first intermediate buffer.

[0387] After the SF synthesis thread receives the layer information of all the sixth layers from the SF main thread, it can control the GPU to synthesize all the sixth layers to obtain the intermediate layer, that is, execute S1205.

[0388] S1204: The HWC thread determines a fourth synthesis strategy in response to the fourth query request, and sends the fourth synthesis strategy to the SF main thread.

[0389] Specifically, after receiving the fourth query request, the HWC thread can determine the compositing method for the layers other than the sixth layer based on its own capabilities and the layer information of the layers other than the sixth layer. This can then determine the compositing method for all layers to be composited, i.e., the fourth compositing strategy. In the fourth compositing strategy, the compositing method for the sixth layer is GPU compositing.

[0390] After S1204, S1206 and S1207 are executed.

[0391] S1205 . The SF synthesis thread controls the GPU to synthesize the sixth layer in response to the sixth synthesis instruction to obtain an intermediate layer.

[0392] The specific implementation of S1205 can refer to the relevant description of S923 in the above embodiment, which will not be repeated here. In this application, the intermediate layer synthesized from the sixth layer obtained in S1205 can be called the second intermediate layer.

[0393] After the SF synthesis thread obtains the intermediate layer, it can inform the SF main thread of the situation to facilitate the main thread to perform subsequent synthesis processes. Specifically, after synthesizing the intermediate layer, the SF synthesis thread can send a synthesis completion indication to the SF main thread, that is, execute S1205A.

[0394] S1205A: The SF composition thread sends a composition completion indication to the SF main thread.

[0395] The synthesis completion indication is used to indicate that the synthesis of the intermediate layer is completed.

[0396] In practice, through specific synchronization means, when GPU synthesis and query synthesis strategies are executed in parallel, the time taken by HWC to return the synthesis strategy to the SF main thread and the time taken by the SF synthesis thread to obtain the intermediate layer through GPU synthesis and inform the SF main thread can be the same, that is, S1205A and step S1204 will be executed and completed at the same time, or after either step S1205A or S1204 is completed, the subsequent process will be executed only after the other is completed.

[0397] Since the intermediate layer being synthesized by the SF compositing thread is created by compositing the sixth layer using the GPU, it's uncertain whether there are any other layers in the layers to be synthesized that require GPU synthesis. Therefore, after obtaining the fourth compositing strategy and the identifier of the intermediate layer, the SF main thread must determine whether there are any other layers in the layers to be synthesized that require GPU synthesis, other than the sixth layer. This can be determined by determining whether there are any other layers in the layers for which the fourth compositing strategy indicates GPU synthesis. Therefore, step S1207 must be executed after both S1205A and S1204.

[0398] S1206. The SF main thread receives the fourth composition strategy from the HWC thread and stores the hash value of the current frame and the fourth composition strategy.

[0399] After SF stores the hash value and the fourth synthesis strategy of the current frame, the hash value and the fourth synthesis strategy of the current frame can be used as the hash value and the synthesis strategy of the previous frame in the layer synthesis process of the next frame image.

[0400] S1207. The SF main thread determines whether there is a seventh layer among all the layers to be synthesized based on the fourth synthesis strategy.

[0401] The seventh layer is a layer to be synthesized other than the sixth layer among the layers to be synthesized whose synthesis method is GPU synthesis according to the fourth synthesis strategy, i.e., the sixth layer and the seventh layer are different layers to be synthesized. In this application, the seventh layer can be referred to as the sixth target layer.

[0402] If, based on the fourth compositing strategy, it is determined that the seventh layer does not exist among all the layers to be composited, then the only layer to be composited by the GPU in the current frame is the sixth layer, and all layers have already been sent to the SF compositing thread for synthesis. The SF main thread does not need to take any other action here, only needing to instruct the HWC to perform the final layer synthesis and send it for display. That is, S936 is then executed. In this application, the absence of the seventh layer among the layers to be composited can be referred to as the HWC synthesis method for the fifth target layer. The fifth target layer is the layer to be composited other than the fourth target layer among the layers to be composited.

[0403] If the fourth compositing strategy determines that a seventh layer exists among all layers to be composited, this indicates that, in addition to the sixth layer, a seventh layer requiring GPU synthesis exists among all layers to be composited in the current frame. The SF main thread can then send the layer information of the seventh layer to the SF compositing thread, enabling it to use the GPU to update the intermediate layer composited from the sixth layer using the seventh layer, thereby executing step S1208. In this application, the presence of a seventh layer among the layers to be composited can be referred to as the presence of a sixth target layer in the fifth target layer, whose synthesis method is GPU synthesis.

[0404] It should be noted that there is no necessary order between S1206 and S1207. S1206 can be executed first, S1207 can be executed first, or S1206 and S1207 can be executed at the same time.

[0405] S1208. The SF main thread sends a seventh composition instruction to the SF composition thread.

[0406] The seventh composition instruction carries all layer information of the seventh layer, and is used to instruct to composite the seventh layer with the intermediate layer in the first intermediate buffer.

[0407] The seventh layer is a layer other than the sixth layer among the layers whose synthesis method is GPU synthesis in the fourth synthesis strategy.

[0408] After receiving the seventh composition instruction from the SF main thread, the SF composition thread can control the GPU to composite the seventh layer into the intermediate layer composed of the sixth layer for updating, thereby obtaining an updated intermediate layer.

[0409] S1209 , the SF synthesis thread controls the GPU to synthesize the seventh layer into the intermediate layer synthesized from the sixth layer in response to the seventh synthesis instruction.

[0410] In this application, the intermediate layer obtained after synthesizing the seventh layer into the intermediate layer obtained by synthesizing the sixth layer can be referred to as the updated second intermediate layer.

[0411] The specific implementation of S1208 and S1209 can be referenced to the related descriptions of S926 and S927 in the aforementioned embodiment and will not be repeated here. Based on the technical solution corresponding to S1209, the layers to be synthesized that the HWC cannot synthesize can be smoothly handed over to the GPU for synthesis, which not only improves the efficiency of layer synthesis, but also ensures that the final synthesis and display of the HWC can be completed smoothly.

[0412] In the above technical solution, the final synthesis strategy in S937 may also be the fourth synthesis strategy. If S1207 is executed before S936, or S1209 and S935 are executed in sequence before S936, the final synthesis strategy is the fourth synthesis strategy.

[0413] Based on the technical solutions corresponding to the above S1201-S1209, the electronic device can execute GPU synthesis and query synthesis strategies in parallel when the layer to be synthesized in the current frame does not meet the parallel conditions (the hash value of the current frame is different from the hash value of the previous frame) and there is no special layer that meets the preset conditions. If there is a special layer that was GPU synthesized in the previous frame, it can also reduce the time required for layer synthesis and the frame loss problem caused by too long layer synthesis time, making the display effect of dynamic pictures more smooth and improving the user experience.

[0414] Typically, to avoid complex updates to a layer's history buffer (the buffer generated for that layer during previous layer synthesis) and improve layer synthesis efficiency, electronic devices typically generate a new buffer for the layer during the synthesis process rather than modifying the layer's history buffer. However, as the number of layer synthesizers increases, the number of history buffers for the same layer increases, placing increasing pressure on memory. Therefore, in some embodiments, to reduce memory pressure, when querying a synthesis strategy, the HWC also determines whether the history buffer of the layer to be synthesized needs to be cleared based on the layer's layer information. In other words, all synthesis strategies (the first, second, third, and fourth synthesis strategies) can also include a clear instruction for each layer to be synthesized. This clear instruction indicates whether to clear the layer's history buffer or not. After the SF main thread receives the synthesis strategy, it clears the contents of the history buffer of the layer to be synthesized that is the target of the clear instruction, leaving its history buffer as a blank buffer in the bufferqueue.

[0415] In order to more clearly illustrate the process of layer synthesis by the electronic device, the process of layer synthesis by the electronic device is described below with reference to FIG13. As shown in FIG13, the process includes S1301-S1314:

[0416] S1301: Update the layer attribute parameters and display parameters of all layers and determine the layer to be synthesized in the current frame.

[0417] Among them, all layers refer to all layers currently registered in the SF of the electronic device.

[0418] The specific implementation of S1301 can refer to the relevant descriptions of S901-S910 in the aforementioned embodiment, and will not be repeated here.

[0419] S1302: Calculate the hash value of the current frame based on the layer information of each layer to be synthesized.

[0420] The layer information may include: an identifier or pointer to the layer's corresponding buffer, layer attribute parameters, and display parameters. Specifically, the identifier or pointer to the layer's corresponding buffer may be a handle to the layer's corresponding buffer. The specific contents of the layer attribute parameters and display parameters can be found in the related descriptions in the previous embodiments and will not be repeated here.

[0421] The specific implementation of S1302 can refer to the relevant description of S911 in the above embodiment, and will not be repeated here.

[0422] S1303: Determine whether the hash value of the current frame is the same as the hash value of the previous frame.

[0423] When the hash value of the current frame is the same as the hash value of the previous frame, it can be considered that the layer to be displayed in the current frame and the layer to be displayed in the previous frame have not changed. At this time, it is determined that the first detection result meets the parallel condition, and the query synthesis strategy and GPU synthesis can be executed in parallel, that is, S1304 is executed.

[0424] When the hash value of the current frame is different from the hash value of the previous frame, in order to execute the query synthesis strategy and GPU synthesis in parallel, it is possible to first determine whether there is a special layer that meets the preset conditions in the layers to be synthesized, that is, execute S1305.

[0425] S1304 , executing query synthesis strategy and GPU synthesis in parallel.

[0426] S1304 may specifically be controlling the GPU to synthesize the layers to be synthesized whose synthesis mode in the previous frame is GPU synthesis while controlling the HWC to query the synthesis mode of the layers to be synthesized.

[0427] The specific implementation of S1304 may refer to the relevant descriptions of S914, S928, S929, S930 and S930A in the aforementioned embodiments, which will not be repeated here.

[0428] S1305: Determine whether there is a layer to be synthesized that meets preset conditions among the layers to be synthesized.

[0429] In the case that there is a layer to be synthesized that meets the preset conditions among the layers to be synthesized, it is determined that the query synthesis strategy and GPU synthesis can be executed in parallel, that is, S1309 is executed.

[0430] When there is a layer to be synthesized that meets the preset conditions in the layers to be synthesized, in order to execute the query synthesis strategy and GPU synthesis in parallel, it can be further determined whether there is a special layer in the layers to be synthesized whose synthesis method in the previous frame is GPU synthesis, that is, execute S1310.

[0431] The specific implementation of S1305 can refer to the relevant description of S913 in the above embodiment, which will not be repeated here.

[0432] S1306: Record the hash and synthesis strategy of the current frame into the "previous frame" variable.

[0433] The specific meaning of the "previous frame" variable can refer to the relevant statements between S917 and S918 in the aforementioned embodiment, and will not be repeated here.

[0434] The specific implementation of S1306 can refer to the relevant description of S931 in the above embodiment, which will not be repeated here.

[0435] S1307: Determine whether the synthesis strategy of the current frame is the same as the synthesis strategy of the previous frame.

[0436] When the synthesis strategy of the current frame is the same as that of the previous frame, it is determined that all the layers to be synthesized in the current frame that need to be synthesized by the GPU have been synthesized by the GPU. At this time, no other actions are required. It is only necessary to control the HWC to perform the final layer synthesis and send it for display, that is, execute S1315.

[0437] When the synthesis strategy of the current frame is different from that of the previous frame, it means that there are still layers to be synthesized in all the layers to be synthesized in the current frame that need to be synthesized by the GPU. At this time, these layers to be synthesized can be handed over to the GPU for synthesis to update the synthesis result of the GPU after S1304 is executed, that is, execute step S1308.

[0438] S1308. Update the GPU synthesis result.

[0439] The specific implementation of S1308 can refer to the relevant descriptions of S934 and S935 in the above embodiments, and will not be repeated here.

[0440] After S1308, execute S1315.

[0441] S1309 , executing the query synthesis strategy and GPU synthesis in parallel.

[0442] When S1309 is executed after S1305, the specific implementation of S1309 can refer to the relevant descriptions of S920, S922, S921, S923 and S923A in the aforementioned embodiments, and will not be repeated here.

[0443] When S1309 is executed after S1310, the specific implementation of S1309 can refer to the relevant descriptions of S1202, S1204, S1203, S1205 and S1205A in the aforementioned embodiments, and will not be repeated here.

[0444] S1310: Determine whether there is a layer to be synthesized whose previous frame was synthesized by GPU among the layers to be synthesized.

[0445] If there is a layer to be synthesized whose previous frame is GPU synthesized in the layers to be synthesized, it is determined that the query synthesis strategy and GPU synthesis can be executed in parallel, that is, S1309 is executed.

[0446] When there is a layer to be synthesized that meets the preset conditions in the layers to be synthesized, in order to execute the query synthesis strategy and GPU synthesis in parallel, it can be further determined whether there is a special layer in the layers to be synthesized whose synthesis method in the previous frame is GPU synthesis, that is, execute S1310.

[0447] The specific implementation of S1310 can refer to the relevant description of S913 in the above embodiment, which will not be repeated here.

[0448] S1311. Record the hash and synthesis strategy of the current frame into the "previous frame" variable.

[0449] The specific meaning of the "previous frame" variable can refer to the relevant statements between S917 and S918 in the aforementioned embodiment, and will not be repeated here.

[0450] When S1309 is executed after S1305, the specific implementation of S1311 can refer to the relevant description of S924 in the above embodiment, which will not be repeated here.

[0451] When S1309 is executed after S1310, the specific implementation of S1311 can refer to the relevant description of S1206 in the above embodiment, which will not be repeated here.

[0452] S1312: Serially execute query synthesis strategy and GPU synthesis.

[0453] Specifically, S1312 first obtains the synthesis mode of the layer to be synthesized, and then controls the GPU to synthesize the layer to be synthesized whose synthesis mode is GPU synthesis.

[0454] The specific implementation of S1312 can refer to the relevant descriptions of S915-S919 and S935 in the aforementioned embodiment, and will not be repeated here.

[0455] Execute S1315 after S1312.

[0456] S1313: Determine whether there is a layer to be synthesized that requires GPU synthesis among the layers to be synthesized.

[0457] If there is no layer to be synthesized that requires GPU synthesis in the layers to be synthesized, it is determined that all the layers to be synthesized in the current frame that need to be synthesized by the GPU have been synthesized by the GPU. At this time, no other actions are required. It is only necessary to control the HWC to perform the final layer synthesis and send it for display, that is, execute S1315.

[0458] If there are layers to be synthesized that require GPU synthesis among the layers to be synthesized, it means that there are layers to be synthesized that require GPU synthesis among all the layers to be synthesized in the current frame. At this time, these layers to be synthesized can be handed over to the GPU for synthesis to update the synthesis result of the GPU after S1309 is executed, that is, execute step S1314.

[0459] S1314: Update the GPU synthesis result.

[0460] When S1309 is executed after S1305, the specific implementation of S1314 can refer to the relevant descriptions of S927 and S935 in the above embodiment, which will not be repeated here.

[0461] When S1309 is executed after S1310, the specific implementation of S1314 can refer to the relevant description of S1209 in the above embodiment, which will not be repeated here.

[0462] S1315: Control HWC to perform final layer synthesis and display.

[0463] The specific implementation of S1315 can refer to the relevant descriptions of S936 and S937 in the above-mentioned embodiment, which will not be repeated here.

[0464] The beneficial effects of the embodiment corresponding to FIG13 above can refer to the beneficial effects of the layer synthesis method provided in the aforementioned embodiment, which will not be repeated here.

[0465] It is understandable that, in order to realize the above functions, the above electronic device includes hardware structures and / or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0466] The embodiment of the present application can divide the functional modules of the above-mentioned electronic device according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0467] In the case of dividing each functional module into corresponding functional modules, as shown in FIG14 , the embodiment of the present application further provides a layer synthesis device, which can be applied to an electronic device. The electronic device can include: an acquisition module 1401 , a determination module 1402 , and a control module 1403 .

[0468] The acquisition module 1401 is configured to acquire layer information of the layer to be synthesized in the current frame. The determination module 1402 is configured to determine a first detection result based on the layer information of the layer to be synthesized acquired by the acquisition module 1401. The control module 1403 is configured to, if the first detection result determined by the determination module 1402 does not meet the first condition and a first target layer that meets a preset condition exists among the layers to be synthesized acquired by the acquisition module 1401, control the GPU to synthesize the first target layer to obtain a first intermediate layer, and simultaneously control the HWC to query the synthesis mode of the layer to be synthesized. The synthesis mode of the first target layer that meets the preset condition is GPU synthesis. The preset condition includes any one or more of the following: a change in the layer attribute parameters or display parameters of the layer; the layer is a newly added layer and the layer attribute parameters include a first flag that indicates GPU synthesis; the layer is a newly added layer and the layer buffer size is larger than a preset size; the layer was synthesized by GPU in the previous frame and its position has changed or its visible area has increased; the position of the layer has rotated 180 degrees compared to the previous frame. The control module 1403 is also used to control the HWC to synthesize the first intermediate layer and the second target layer to obtain the image of the current frame when the synthesis method of the second target layer is HWC synthesis; wherein the second target layer is the layer to be synthesized other than the first target layer in the layers to be synthesized.

[0469] In addition, the cooperation among the acquisition module 1401, the determination module 1402 and the control module 1403 can also enable the layer synthesis apparatus to complete all the steps of the layer synthesis method provided in the aforementioned embodiment.

[0470] Regarding the layer synthesis device in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the layer synthesis method in the above-mentioned embodiment, and will not be further elaborated here. The related beneficial effects can also be referred to the related beneficial effects of the above-mentioned layer synthesis method, and will not be repeated here.

[0471] The present application also provides an electronic device comprising: a display screen, a memory, and one or more processors; the display screen, the memory, and the processors are coupled; wherein the memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device executes the layer composition method provided in the aforementioned embodiment. The specific structure of the electronic device can be referenced with the structure of the electronic device shown in FIG7 .

[0472] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the layer synthesis method provided in the aforementioned embodiment.

[0473] The present application also provides a chip system, as shown in FIG15 . The chip system 1500 includes at least one processor 1501, a memory, and at least one interface circuit 1502. The processor 1501 and the interface circuit 1502 can be interconnected via a line. For example, the interface circuit 1502 can be used to receive signals from other devices (e.g., network-side devices). In another example, the interface circuit 1502 can be used to send signals to other devices (e.g., network-side devices).

[0474] For example, interface circuit 1502 can read instructions or computer programs stored in memory and send them to processor 1501. When the instructions or computer programs are executed by processor 1501, the various steps of the layer control method provided in the above embodiment can be implemented. Of course, the chip system can also include other discrete components, which are not specifically limited in this embodiment of the application.

[0475] An embodiment of the present application also provides a computer program product, which includes executable instructions. When the computer program product is run on an electronic device, the electronic device executes the layer synthesis method provided in the aforementioned embodiment.

[0476] FIG16 schematically illustrates a conceptual partial view of a computer program product provided by an embodiment of the present invention. In one embodiment, the computer program product is provided using a signal-bearing medium 1600. Signal-bearing medium 1600 may include one or more program instructions that, when executed by one or more processors, may provide the functionality or portions of the functionality described above with respect to FIG4 . Thus, for example, one or more features or steps of the layer composition method provided in the aforementioned embodiments may be performed by one or more instructions associated with signal-bearing medium 1600. Furthermore, the program instructions in FIG16 also depict example instructions.

[0477] In some examples, the signal-bearing medium 1600 may include a computer-readable medium 1601, such as, but not limited to, a hard drive, a compact disk (CD), a digital video disk (DVD), a digital tape, a memory, a read-only memory (ROM), or a random access memory (RAM), and the like.

[0478] In some implementations, the signal bearing medium 1600 may include a computer recordable medium 1602 such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, or the like.

[0479] In some embodiments, signal bearing medium 1600 may include communication medium 1603 such as, but not limited to, digital and / or analog communication media (eg, fiber optic cables, waveguides, wired communication links, wireless communication links, etc.).

[0480] Signal bearing medium 1600 may be communicated by a wireless form of communication medium 1603 (eg, a wireless communication medium conforming to the IEEE 802.150 standard or other transmission protocols). The one or more program instructions may be, for example, computer executable instructions or logic implemented instructions.

[0481] In some examples, a data writing device for receiving and storing external data may also be included, which may be configured to provide various operations, functions, or actions in response to one or more program instructions in the computer-readable medium 1601, the computer-recordable medium 1602, and / or the communication medium 1603.

[0482] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0483] In the several embodiments provided in this application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0484] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0485] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0486] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0487] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for layer synthesis, characterized in that, Applied to an electronic device, the electronic device includes a Graphics Processing Unit (GPU) and a Hardware Compositor (HWC), and the method includes: Obtain the layer information of the layer to be composed in the current frame; Determine a first detection result based on the layer information of the layer to be composed; When the first detection result does not meet the first condition and there is a first target layer that meets the preset conditions in the layer to be composed, control the GPU to compose the first target layer to obtain a first intermediate layer, and at the same time control the HWC to query the composition method of the layer to be composed; wherein, the composition method of the first target layer that meets the preset conditions is GPU composition; the preset conditions include any one or more of the following: the layer attribute parameters or display parameters of the layer have changed; the layer is a newly added layer, and the layer attribute parameters include a first flag for indicating composition using the GPU; the layer is a newly added layer, and the buffer size of the layer is greater than the preset size; in the previous frame, the composition method was GPU composition, and the layer with a position change or a larger visible area; the position of the layer has rotated 180° compared to the previous frame; When the composition methods of the second target layers are all HWC composition, control the HWC to compose the first intermediate layer and the second target layers to obtain the image of the current frame; Wherein, the second target layer is the layer to be composed in the layer to be composed except the first target layer.

2. The method according to claim 1, characterized in that, The determining the first detection result based on the layer information of the layer to be composed includes: The electronic device calculates the hash value of the current frame based on the layer information of the layer to be composed; When the hash value of the current frame is the same as the hash value of the previous frame, determine that the first detection result meets the first condition; When the hash value of the current frame is different from the hash value of the previous frame, determine that the first detection result does not meet the first condition.

3. The method according to claim 1 or 2, characterized in that, After controlling the GPU to compose the first target layer to obtain a first intermediate layer and at the same time controlling the HWC to query the composition method of the layer to be composed, the method further includes: When there is a third target layer with a composition method of GPU composition in the second target layers, control the GPU to compose the third target layer into the first intermediate layer to obtain the updated first intermediate layer; the third target layer and the first target layer are different layers to be composed; Control the HWC to compose the updated first intermediate layer and the layers to be composed in the second target layers except the third target layer to obtain the image of the current frame.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: When the first detection result does not meet the first condition and the first target layer does not exist in the layer to be synthesized, if a fourth target layer exists in the layer to be synthesized, control the GPU to synthesize the fourth target layer to obtain a second intermediate layer, and at the same time control the HWC to query the synthesis method of the layer to be synthesized; wherein, the synthesis method of the fourth target layer is GPU synthesis; the synthesis method of the fourth target layer in the previous frame is GPU synthesis; When the synthesis methods of the fifth target layers are all HWC synthesis, control the HWC to synthesize the second intermediate layer and the fifth target layer to obtain the image of the current frame; wherein, the fifth target layer is the layer to be synthesized in the layer to be synthesized except the fourth target layer.

5. The method according to claim 4, wherein After controlling the GPU to synthesize the fourth target layer to obtain a second intermediate layer and at the same time controlling the HWC to query the synthesis method of the layer to be synthesized, the method further includes: When a sixth target layer with a synthesis method of GPU synthesis exists in the fifth target layer, control the GPU to synthesize the sixth target layer into the second intermediate layer to obtain the updated second intermediate layer; the sixth target layer and the fourth target layer are different layers to be synthesized; Control the HWC to synthesize the updated second intermediate layer and the layers to be synthesized in the fifth target layer except the sixth target layer to obtain the image of the current frame.

6. The method according to any one of claims 1-5, characterized in that The layer information includes: the identifier of the buffer of the layer, layer attribute parameters, and display parameters; The layer attribute parameters include any one or more of the following: blur parameter, transparency, position, size, name, type, visibility; The display parameters include any one or more of the following: color, transparency, blending mode, depth sorting, clipping parameters, occlusion parameters.

7. The method according to any one of claims 1-6, characterized in that, The newly added layer is a layer that did not exist in the previous frame.

8. The method according to any one of claims 1-7, characterized in that, Before obtaining the layer information of the layer to be synthesized for the current frame, the method further includes: Receive a first operation from the user and start the first application; Obtain the layer information of the layer to be synthesized for the current frame during the startup process of the first application.

9. An electronic device, characterized in that, Includes: A display screen, a memory, and one or more processors; the display screen and the memory are coupled to the processor; wherein, the memory stores computer program code, and the computer program code includes computer instructions. When the computer instructions are executed by the processor, the electronic device executes the layer synthesis method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, Includes computer instructions. When the computer instructions run on an electronic device, the electronic device executes the layer synthesis method according to any one of claims 1-8.

Images