Frame synthesis method, electronic device, and computer-readable storage medium

Abstract

This invention relates to a frame composition method, an electronic device, and a computer-readable storage medium, and pertains to the field of terminals. In this method, during the switching of an electronic device from a first user interface to a second user interface, while composing M first layers of a first frame, if the attributes of the first layer satisfy at least one of a plurality of preset attribute items, then composition prediction is performed on the first frame, where M is a positive integer. This method can increase the probability of performing composition prediction on the frame, saving the composition time of the first frame, thereby avoiding screen stuttering on the electronic device and improving the user experience.

Description

Frame synthesis method, electronic device and computer-readable storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on January 10, 2024, with application number 202410042526.X and application name “Frame synthesis method, electronic device and computer-readable storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminals, and in particular to a frame synthesis method, an electronic device, and a computer-readable storage medium. Background Art

[0003] Electronic devices display images on screens, enabling user interaction. In some scenarios, the need to rotate, scale, and blur layers requires frame synthesis to be processed on the GPU, resulting in longer synthesis times and potentially causing image freezes, impacting the user experience. Summary of the Invention

[0004] The present application provides a frame synthesis method, electronic device, and computer-readable storage medium. When the M first layers in a first frame meet a first preset condition, the electronic device directly performs synthesis prediction on the first frame. This can increase the probability of performing synthesis prediction on the frame, save synthesis time for the first frame, thereby avoiding screen freezes in the electronic device and improving the user experience.

[0005] In a first aspect, the present application provides a frame synthesis method, which is applied to an electronic device, the method comprising: displaying a first user interface on a display screen of the electronic device. During a period when the electronic device switches from the first user interface to display a second user interface, in a process of synthesizing M first layers of a first frame, if an attribute of the first layer satisfies at least one of a plurality of preset attribute items, where M is a positive integer; the electronic device performs synthesis prediction on the first frame; wherein the plurality of preset attribute items include scaling.

[0006] In the frame synthesis method provided in the present application, the display screen of the electronic device displays a first user interface, and during the period when the electronic device switches from the first user interface to display a second user interface, a first frame is also synthesized. During the process of the electronic device synthesizing the M first layers of the first frame, if the attributes of the first layer meet at least one of multiple preset attribute items (including a zoom effect), a synthesis prediction is performed on the first frame. This can increase the probability of performing a synthesis prediction on the first frame, save the synthesis time of the first frame, thereby avoiding screen freezes of the electronic device and improving the user experience.

[0007] It can be understood that synthesis prediction refers to a method of asynchronous execution of the GPU synthesis of M first layers and the hardware synthesizer determining the synthesis method of layer requests other than the M first layers in the first frame. This method allows the electronic device to determine the synthesis method of other layer requests in the first frame without affecting the electronic device's execution of GPU synthesis, thereby fully utilizing the computing power of the electronic device for GPU synthesis.

[0008] Optionally, when the synthesis mode requested by a layer other than the M first layers in the first frame is GPU synthesis, the electronic device performs GPU synthesis on the layer.

[0009] Optionally, the first user interface and the second user interface may be in a screen-off state.

[0010] Optionally, the rendering attribute is zoom, which is used to indicate whether to render the target layer with a zoom-in effect or a zoom-out effect.

[0011] In a possible implementation, the first preset condition further includes that a plurality of preset attribute items include: rounded corners, blur effect, special angle rotation, high dynamic range (HDR) and shadow effect.

[0012] In the implementation manner provided by the present application, the plurality of preset attribute items also include rounded corners, blur effects, special angle rotations, high dynamic range lighting synthesis HDR, and shadow effects. As can be seen from the above, when a layer satisfies at least one of the plurality of preset attribute items, the electronic device can directly perform GPU synthesis on the layer. In the implementation manner of the present application, more preset attribute items are shown, thereby increasing the probability that the layer in the first frame satisfies at least one of the plurality of preset attribute items, thereby enabling the electronic device to directly perform GPU synthesis on more layers.

[0013] In a possible implementation, the electronic device performs synthesis prediction on the first frame, including: when the second frame includes a layer whose requested synthesis method is GPU synthesis, the electronic device performs synthesis prediction on the first frame; the second frame is the previous frame of the first frame.

[0014] In the embodiment of the present application, the second frame includes a layer whose requested synthesis method is GPU synthesis. Combined with the fact that the second frame is the previous frame of the first frame, it can be known that the first frame is likely to also include a layer whose requested synthesis method is GPU synthesis (because the two frames do not change much, if the previous frame includes a layer whose requested synthesis method is GPU synthesis, the next frame is likely to also include a layer whose requested synthesis method is GPU synthesis). Therefore, in this case, it can be determined to perform synthesis prediction on the first frame, which can increase the probability of performing synthesis prediction on the first frame, save synthesis time for the first frame, thereby avoiding screen freezes on electronic devices and improving the user experience.

[0015] In a possible implementation, the electronic device performs synthesis prediction on the first frame, including: when the first frame is an animation frame, the electronic device performs synthesis prediction on the first frame.

[0016] In the implementation of this application, the first frame is a motion effect frame. It is understandable that because motion effects require displaying complex and diverse dynamic images, the composition of the motion effect frames is more complex than that of ordinary frames during synthesis, and their layer request synthesis method generally includes GPU synthesis. Therefore, in this case, it can be determined that synthesis prediction should be performed on the first frame, which can increase the probability of performing synthesis prediction on the first frame, save synthesis time for the first frame, thereby avoiding screen freezes on electronic devices and improving the user experience.

[0017] In one possible embodiment, the animation frame includes one or more of the following: an image displayed when the electronic device starts the target application; or, an image displayed when the electronic device exits the target application; or, an image displayed when the electronic device enters the multitasking interface; or, an image displayed when the electronic device enters the target application from the multitasking interface.

[0018] In an embodiment of the present application, the animation frame includes one or more of the following: an image displayed when the electronic device starts the target application, or an image displayed when the electronic device exits the target application, or an image displayed when the electronic device enters the multitasking interface, or an image displayed when the electronic device enters the target application from the multitasking interface.

[0019] Optionally, an image displayed during a certain process of the electronic device can be set as a motion effect frame in a preset manner. For example, an image displayed during the process of unlocking the screen of the electronic device can be set as a motion effect frame.

[0020] Optionally, the animation frame also includes an image displayed during the process of switching the screen refresh rate.

[0021] In a possible implementation, the electronic device displays the motion effect frame on the display screen while switching from the first user interface to display the second user interface.

[0022] In the implementation manner of the present application, the electronic device switches from the first user interface to display the second user interface device, and a motion effect frame is displayed on the display screen of the electronic device.

[0023] Optionally, the motion effect frame can also be displayed in the background of the electronic device. For example, if the target application is started in the background of the electronic device, the screen during the target application startup process is displayed in the background.

[0024] In a possible implementation, the electronic device performs synthesis prediction on the first frame, further comprising: when the second frame does not include a layer whose requested synthesis method is GPU synthesis, the electronic device performs synthesis prediction on the first frame; and the second frame is the previous frame of the first frame.

[0025] In the implementation manner of the present application, when the second frame does not include a layer whose requested synthesis method is GPU synthesis, and the first frame is a motion effect frame, the electronic device performs synthesis prediction on the first frame. It can be understood that even if the previous frame (the second frame) of the first frame does not include a layer whose requested synthesis method is GPU synthesis, since the first frame is a motion effect frame, the first frame may include multiple layers whose requested synthesis method is GPU synthesis. Therefore, in this case, it can be determined that synthesis prediction is performed on the first frame, which can increase the probability of performing synthesis prediction on the first frame, save synthesis time for the first frame, thereby avoiding screen freezes of the electronic device and improving the user experience.

[0026] In a possible implementation, the layer data of the first frame is the same as the layer data of the second frame; or, the layer data of the first frame is different from the layer data of the second frame; and the second frame is a previous frame of the first frame.

[0027] In the embodiment of the present application, the layer data of the first frame and the layer data of the second frame may be the same or different. When the layer data of the first frame and the layer data of the second frame are the same, it indicates that the first frame is the same as the second frame. Combined with the fact that the second frame includes a layer whose requested synthesis method is GPU synthesis, it indicates that the first frame also includes a layer whose requested synthesis method is GPU synthesis. Therefore, in this case, it can be determined that synthesis prediction is performed on the first frame, which can increase the probability of performing synthesis prediction on the first frame, save synthesis time for the first frame, thereby avoiding screen freezes of electronic devices and improving user experience.

[0028] The layer data of the first frame is different from the layer data of the second frame, indicating that the second frame has changed compared to the first frame, combined with the fact that the second frame includes a layer whose requested synthesis method is GPU synthesis. It can be understood that when the previous frame of the first frame includes a layer whose requested synthesis method is GPU synthesis, even if the previous frame of the first frame is different from the first frame, synthesis prediction is performed on the first frame. This implementation method can increase the probability of the first frame entering the synthesis prediction, thereby reducing the synthesis time of the first frame, avoiding screen freezes, and thus improving the user experience.

[0029] In a possible implementation, the synthesis method of the first frame is the same as that of the second frame; or, the synthesis method of the first frame is different from that of the second frame; and the second frame is a previous frame of the first frame.

[0030] In the embodiment of the present application, the synthesis method of the first frame and the synthesis method of the second frame can be the same or different. In the case where the synthesis method of the first frame is the same as the synthesis method of the second frame, combined with the fact that the second frame includes a layer whose synthesis method is requested to be GPU synthesized, it indicates that the first frame also includes a layer whose synthesis method is requested to be GPU synthesized. Therefore, in this case, it can be determined that synthesis prediction is performed on the first frame, which can increase the probability of performing synthesis prediction on the first frame, save synthesis time for the first frame, thereby avoiding screen freezes of the electronic device and improving the user experience.

[0031] The synthesis method of the first frame is different from that of the second frame, indicating that the second frame has changed compared to the first frame, combined with the fact that the second frame includes a layer whose synthesis method is GPU synthesis. It can be understood that when the previous frame of the first frame includes a layer whose synthesis method is GPU synthesis, even if the previous frame of the first frame is different from the first frame, synthesis prediction is performed on the first frame. This implementation method can increase the probability of the first frame entering the synthesis prediction, thereby reducing the synthesis time of the first frame, avoiding screen freezes, and thus improving the user experience.

[0032] In a second aspect, the present application provides an electronic device comprising a display screen, a memory, and a processor coupled to the memory; the display screen is used to display an interface, the memory stores a computer program, and when the processor executes the above computer program, the electronic device implements any one of the methods described in the above first aspect.

[0033] In a third aspect, the present application provides a computer-readable storage medium, which stores a computer program or computer instructions, and the aforementioned computer program or computer instructions are executed by a processor to implement any method described in the first aspect.

[0034] In a fourth aspect, an embodiment of the present application provides a computer program product. When the computer program product is executed by a processor, the method described in any one of the above-mentioned first aspects will be implemented.

[0035] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a memory, wherein the memory is used to store computer programs or computer instructions, and the processor is used to execute the computer programs or computer instructions stored in the memory, so that the chip executes any method described in the first aspect above.

[0036] The solutions provided in the second to fifth aspects are used to implement or cooperate with the corresponding methods provided in the first aspect, and therefore can achieve the same or corresponding beneficial effects as the corresponding methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG1 is a schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of the present application;

[0038] FIG2 is a software structure block diagram of a terminal device provided in an embodiment of the present application;

[0039] FIG3 is a flow chart of a frame synthesis method provided in an embodiment of the present application;

[0040] FIG4a is a first schematic diagram of an interface switching method provided by an embodiment of the present application;

[0041] FIG4 b is a schematic diagram of a motion effect playback provided by an embodiment of the present application;

[0042] FIG5 is a second schematic diagram of an interface switching provided by an embodiment of the present application;

[0043] FIG6 is a third schematic diagram of an interface switching provided in an embodiment of the present application;

[0044] FIG7 is a fourth schematic diagram of an interface switching provided in an embodiment of the present application;

[0045] FIG8 is a schematic diagram of a method of not performing synthesis prediction provided by an embodiment of the present application;

[0046] FIG9 is a schematic diagram of performing synthetic prediction according to an embodiment of the present application;

[0047] FIG10 is a schematic diagram of a workflow of a user interface switching process software provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The terms used in the following examples of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular expressions "a," "an," "said," "above," "the," and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and encompasses any or all possible combinations of one or more of the listed items.

[0049] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0050] The frame synthesis processing method provided in the embodiment of the present application can be applied to electronic devices with display functions.

[0051] Electronic devices include terminal devices, which can also be called terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. Terminal devices can be mobile phones, smart TVs, wearable devices, tablet computers (Pads), computers with wireless transceiver functions, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. The embodiments of the present application do not limit the specific technology and specific device form adopted by the terminal devices.

[0052] In order to better understand the embodiments of the present application, the structure of the terminal device in the embodiments of the present application is introduced below:

[0053] FIG1 is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.

[0054] The following embodiments are described in detail using electronic device 100 as an example. It should be understood that electronic device 100 may have more or fewer components than those shown in FIG1 , may combine two or more components, or may have a different component configuration. The various components shown in FIG1 may be implemented in hardware, including one or more signal processing and / or application-specific integrated circuits, software, or a combination of hardware and software.

[0055] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a camera 193, a display screen 194, etc. The sensor module 180 may include a pressure sensor 180A, a touch sensor 180B, etc.

[0056] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0057] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0058] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0059] The charging management module 140 is configured to receive charging input from a charger, which may be a wireless charger or a wired charger.

[0060] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, etc.

[0061] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for image synthesis. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0062] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLED, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0063] The camera 193 is used to capture still images or videos. The electronic device 100 may include one or N cameras 193, where N is a positive integer greater than one.

[0064] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0065] Video codecs are used to compress or decompress digital videos. Electronic device 100 may support one or more video codecs. In this way, electronic device 100 can play or record videos in multiple encoding formats.

[0066] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0067] The external memory interface 120 can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0068] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, applications required for at least one function (such as face recognition function, fingerprint recognition function, mobile payment function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as face information template data, fingerprint information template, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0069] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0070] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0071] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0072] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0073] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals.

[0074] The pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a device comprising at least two parallel plates with conductive material. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to the display screen 194, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0075] Touch sensor 180B, also known as a "touch panel," can be disposed on display screen 194. The touch sensor 180B and display screen 194 form a touch screen, also known as a "touch screen." Touch sensor 180B is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via display screen 194. In other embodiments, touch sensor 180B can also be disposed on the surface of electronic device 100, at a location different from that of display screen 194.

[0076] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture, etc. In the embodiment of the present application, the Android system with a layered architecture is used as an example to exemplify the software structure of the electronic device 100 .

[0077] Figure 2 is a block diagram of the software structure of a terminal device in an embodiment of the present application. The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers: the application layer, the application framework layer, the Android runtime and system libraries, the hardware abstraction layer, and the kernel layer.

[0078] The application layer can include a series of application packages. As shown in Figure 2, the application package can include applications such as phone, email, calendar, camera, etc.

[0079] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0080] As shown in FIG2 , the application framework layer may include an image composition system, a window manager, a frame rate control system, a view system, a package manager, an input manager, and the like.

[0081] The image synthesis system is used to control image synthesis and generate vertical synchronization (Vsync) signals. The image synthesis system includes a synthesis thread, a Vsync thread, and a queue buffer thread. The synthesis thread is awakened by the Vsync signal to perform synthesis. The Vsync thread is used to generate the next Vsync signal based on the Vsync signal request. The queue buffer thread is used to store cache, generate Vsync signal requests, and wake up the synthesis thread. The queue buffer thread contains one or more cache queues, each used to store cache corresponding to different applications.

[0082] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0083] The frame rate control system is used to adjust the screen refresh rate.

[0084] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0085] The package manager is used for program management within the system, such as application installation, uninstallation, and upgrades.

[0086] The input manager is used to manage input devices. For example, the input system can determine input operations such as mouse clicks, keyboard inputs, and touch swipes.

[0087] The application framework layer may also include an activity manager and a resource manager.

[0088] The Activity Manager is used to manage the lifecycle of each application and the navigation back function. It is responsible for creating the Android main thread and maintaining the lifecycle of each application.

[0089] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0090] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for scheduling and management of the Android system.

[0091] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0092] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0093] The system library can include multiple functional modules, such as image rendering library, image synthesis library, open graphics library, function library, and media library.

[0094] The image rendering library (RenderEngine) is used to render 2D or 3D images. The image synthesis library (surface flinger) is used to synthesize 2D or 3D images.

[0095] In a possible implementation, the application renders the image using the image rendering library, and then sends the rendered image to the cache queue of the image synthesis system. Whenever a Vsync signal arrives, the image synthesis system sequentially retrieves a frame of image to be synthesized from the cache queue, and then performs image synthesis using the image synthesis library.

[0096] The Open Graphics Library (OpenGL) is a cross-platform, cross-language graphics programming interface. It is widely used for 2D and 3D graphics rendering and is a core component of many applications, games, and web browsers. For example, OpenGL is used to pass data drawn by an app to the GPU for processing.

[0097] The function library provides macros, type definitions, string operation functions, mathematical calculation functions, and input and output functions used in the C language.

[0098] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0099] The hardware abstraction layer (HAL) can contain multiple library modules, such as hardware composers (HWCs), graphics processing units (GPUs), audio drivers, and sensor drivers. The Android system can load the corresponding library modules for device hardware, enabling the application framework layer to access the device hardware. Device hardware can include, for example, LCD displays and cameras in electronic devices.

[0100] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, audio drivers, and sensor drivers.

[0101] Hardware can be audio devices, Bluetooth devices, camera devices, sensor devices, etc.

[0102] The following describes the workflow of the software and hardware of the electronic device 100 by way of example, in conjunction with the scenario of application startup or interface switching within an application.

[0103] When the touch sensor 180K in the touch panel receives a touch operation, the kernel layer processes the touch operation into an original input event (including touch coordinates, touch force, timestamp of the touch operation and other information). The original input event is stored in the kernel layer. The kernel layer reports the original input event to the input manager of the application framework layer through the input processing library. The input manager of the application framework layer parses the information of the original input event (including: operation type and reported point position, etc.) and determines the focus application based on the current focus, and sends the parsed information to the focus application. The focus can be the touch point in a touch operation or the click position in a mouse click operation. The focus application is the application running in the foreground of the terminal device or the application corresponding to the touch position in the touch operation. The focus application determines the control corresponding to the original input event based on the parsed information of the original input event (for example, the reported point position).

[0104] Taking the touch operation as a touch sliding operation, and the control corresponding to the touch sliding operation as the list control of the WeChat application as an example, the WeChat application calls the image rendering library in the system library through the view system of the application framework layer to draw and render the image. The WeChat application sends the drawn and rendered image to the cache queue of the image synthesis system. The image synthesis library in the system library synthesizes the drawn and rendered image in the image synthesis system into the WeChat interface. The image synthesis system uses the display driver of the kernel layer to make the screen (display) display the corresponding interface of the WeChat application.

[0105] To facilitate understanding, some of the examples given are provided for reference to the description of concepts related to the embodiments of the present application.

[0106] 1. Frame: This refers to the smallest unit of a single image in an interface display. A frame can be thought of as a still image. Displaying multiple frames in rapid succession can create the illusion of motion. Frame rate refers to the number of frames refreshed per second, or the number of times per second the graphics processor in a terminal device refreshes the image. A higher frame rate results in smoother and more realistic animation. The more frames per second, the smoother the displayed motion.

[0107] It should be noted that before the interface displays a frame, it usually needs to go through processes such as layer drawing, layer rendering, and layer synthesis.

[0108] 2. Layer drawing: This refers to the image drawing of the display interface. The display interface can be composed of one or more layers (views). Each layer can be drawn by the visual controls of the view system. Each layer is composed of sublayers. A sublayer corresponds to a small widget in the layer. For example, a sublayer corresponds to a symbol in the image layer.

[0109] 3. Layer rendering: This involves applying coloring to the drawn layer or adding 3D effects. For example, 3D effects can include lighting, shadows, and textures. Another example is layer rendering, which can scale the layer, round corners, blur it, rotate it at a specific angle, perform HDR synthesis, and perform shadow effects.

[0110] 4. Layer synthesis: It is the process of synthesizing multiple or more rendered layers into a display interface.

[0111] 5. Frame synthesis: including layer drawing, layer rendering and layer synthesis operations.

[0112] In some scenarios, in order to alleviate the problem of motion effect stuttering caused by long GPU synthesis time, a synthesis prediction mechanism is proposed to reduce the CPU synthesis time. However, the existing synthesis prediction mechanism is prone to synthesis prediction failure and cannot effectively alleviate the problem of motion effect stuttering caused by long GPU synthesis time. In view of this, the present application provides a frame synthesis method, which directly performs synthesis prediction on the first frame when the M first layers in the first frame meet the first preset condition. This can increase the probability of performing synthesis prediction on the frame, save the synthesis time of the first frame, thereby avoiding screen stuttering of electronic devices and improving the user experience.

[0113] Please refer to Figure 3, which is a flowchart of a frame synthesis method provided by an embodiment of the present application. As shown in Figure 3, the workflow of the electronic device 100 software includes but is not limited to steps S301-S308. The execution entities (units) in Figure 3 include: applications, image rendering libraries, image synthesis libraries, asynchronous synthesizers, and hardware synthesizers. Among them, applications are, for example, application packages of the application layer in Figure 2 above, such as telephones, email, calendars, or cameras. The image rendering library is, for example, the image rendering library shown in Figure 2 above. The image synthesis library is, for example, the image synthesis library shown in Figure 2 above. The hardware synthesizer is, for example, the hardware synthesizer shown in Figure 2 above. The asynchronous synthesizer is used to perform synthesis prediction on the frame. For example, the electronic device 100 creates a first thread and a second thread (the first thread and the second thread execute in parallel and do not affect each other). The first thread is used to process the layers that need to be synthesized by the GPU (as shown in step S305), and the second thread is used to determine the synthesis method requested by other layers in the frame based on the information fed back by the hardware synthesizer. The second thread determines the layer that needs to be GPU synthesized, sends it to the first thread, and the first thread performs the synthesis process (as shown in step S307).

[0114] S301: The application responds to a first operation (optional).

[0115] The application responds to the first operation, and the display screen of the electronic device switches from displaying the first user interface to displaying the second user interface. The operation type of the first operation is, for example, pressing, moving, releasing, sliding, etc. The first operation is, for example, operating the electronic device through one or more of the following, including: display screen, physical button, image sensor, microphone, etc. It can be understood that this application does not limit the first operation. The solution provided in this application is applied to the period when the display screen of the electronic device switches from the first user interface to the second user interface, and the display screen also displays at least one frame of the scene. Of course, in some scenarios, the switching of the display screen from the first user interface to the second user interface will run in the background of the electronic device and will not be directly presented on the display screen of the electronic device. This scenario still falls within the scope of protection of this application.

[0116] Next, a possible implementation of the first operation is provided. For example, as shown in FIG4a(a), in response to a user operating the "Communication" application icon 101 on the phone's home screen (e.g., the user clicks icon 101), the phone displays interface 102 as shown in FIG4a(b). Interface 102 is the interface after the "Communication" application is launched. In some embodiments, as shown in FIG4a(b), interface 102 may be an interface showing the user's contact list.

[0117] The interface shown in (a) of 4a may be referred to as the first user interface, and interface 102 may be referred to as the second user interface. During the transition from the first interface to the second interface, the electronic device may further display multiple transitional images (referred to as frames). For example, when a user clicks icon 101, the image displayed in interface 102 increases in size until the image displayed in interface 102 fills the entire screen (or display screen).

[0118] For example, the mobile phone plays six frames of image frames continuously during the process of switching from the first interface to the second interface. When the user clicks the "Communication" application icon 101, the mobile phone continuously plays the image frames of frame 1, frame 2, frame 3, frame 4, frame 5, and frame 6 as shown in Figure 4b.

[0119] Optionally, the first user interface and / or the second user interface may be in a screen-off state. For example, the first user interface is in a screen-off state, and the application responds to a first operation (e.g., an unlock operation) to cause the display screen to light up and display the second user interface, as shown in (a) of Figure 4a. For another example, the display screen displays the first user interface, as shown in (a) of Figure 4a, and the application responds to a first operation (e.g., a lock operation) to cause the display screen to be in a screen-off state.

[0120] In one possible implementation, when the application responds to the first operation, the electronic device sends the layer data to be drawn to the layer rendering library. For example, when the application responds to the first operation, the electronic device may initiate a layer drawing task through the schedule traversals function of the ViewRoot class in the application framework layer, and then pass the data to be drawn to the GPU for processing through the open graphics library in the system library. The GPU completes the drawing of the data, for example, the image rendering library in the system library performs drawing processing on the data through the GPU. The specific processing process can be referred to in step S302.

[0121] S302: The image rendering library renders data of the first frame.

[0122] The first frame may be any frame in the user interface switching process. Taking FIG. 4 b as an example, the first frame may be any frame among frame 1, frame 2, frame 3, frame 4, frame 5 and frame 6.

[0123] For example, the image rendering library can be understood as a rendering thread. The rendering thread can apply for cache space to store the texture data of the application. When the rendering thread detects that the data in the cache space is updated, the rendering thread will read the texture data of the application in the cache space and control the GPU to render the texture data of the application. For example, the updated data in the cache space maintained by the rendering thread can be called the texture data of the application of the first frame. The rendering thread can read the texture data of the application of the first frame in the cache space and control the GPU to render the texture data of the application of the first frame.

[0124] It should be noted that the embodiments of this application do not limit how the GPU renders the texture data of the application. It is understood that the data obtained by GPU rendering exists in layers, so multiple layers can be obtained through GPU rendering. The layers obtained by GPU rendering include various data, such as scaling, rounded corners, blur effects, special angle rotation, high dynamic range (HDR) synthesis, and shadow effects.

[0125] In one possible implementation, the data obtained by rendering / drawing is filled into a cache space and passed to the image synthesis library, allowing the layer synthesis library to align and perform synthesis operations. It should be noted that the cache space for caching the texture data of the application and the cache space for caching the first frame after rendering can be the same cache space or a different cache space, which is not limited in this embodiment of the application.

[0126] S303: The image synthesis library determines the synthesis method of the layer request in the first frame.

[0127] In one possible implementation, the image synthesis library can read the rendered first frame (the first frame includes multiple layers rendered by the GPU) from the above-mentioned cache space, and then perform synthesis operations on the multiple layers in the first frame. It is understandable that the electronic device can synthesize the layers that need to be synthesized through the GPU and the hardware synthesizer. Among them, the synthesis method of GPU synthesis can cope with complex synthesis requirements. For example, the layers that the hardware synthesizer cannot synthesize can be synthesized by the GPU. For example, when the rendering effect of the layer includes one or more of scaling, rounded corners, blur effects, special angle rotation, high dynamic lighting synthesis HDR and shadow effects, the hardware synthesizer cannot synthesize it. In this case, the GPU can be allowed to synthesize the layers including the above-mentioned rendering effects. It should be noted that with the development of technology, the hardware synthesizer may have more powerful functions, so the function of the hardware synthesizer should not be used as a limitation of this application.

[0128] In one possible implementation, after reading the first rendered frame, the image compositing library determines the properties of each layer in the first frame. Then, based on the properties of each layer, it determines whether the compositing method requested for that layer is GPU compositing.

[0129] Exemplarily, the synthesis mode of the layer request may be determined by judging whether the properties of the layer in the first frame satisfy at least one property among a plurality of preset property items.

[0130] The aforementioned multiple preset property items include scaling. If the properties of a layer in the first frame include scaling, the requested composition type (requestedCompositionType) for that layer is determined to be GPU composition. For ease of description, the layer with the requested composition type set to GPU composition can be referred to as the first layer. Using the above determination method, M first layers can be determined from the multiple layers included in the first frame, where M is a positive integer. For example, M is 1, 2, 3, or 5. Scaling refers to the situation where the layer size obtained by the image rendering library differs from the actual displayed size of the layer. If the layer size obtained by the image rendering library is smaller than the actual displayed size of the layer, the layer property is reduced. If the layer size obtained by the image rendering library is larger than the actual displayed size of the layer, the layer property is method. Referring to Frames 1 and 2 shown in Figure 4b, both Frames 1 and 2 include four layers: a signal icon 501, a battery icon 502, a search box 503, and content 504. As can be seen from Figure 4b, the sizes of the signal icon 501 and the battery icon 502 in each frame are the same as the final displayed size, and there is no scaling effect. The search box 503 and the content 504 in frames 1 and 2 are both smaller than the final display size. This can be understood as the properties of the layers corresponding to the search box 503 and the content 504 include scaling.

[0131] Optionally, the plurality of preset attribute items further include rounded corners, blur effects, special angle rotations, high dynamic range (HDR) and shadow effects, etc. For example, if the attributes of the layer in the first frame include any one or more of scaling, rounded corners, blur effects, special angle rotations, high dynamic range (HDR) and shadow effects, the compositing mode requested for the layer may be determined as GPU compositing.

[0132] Optionally, there may be a case where the attributes of any layer in the first frame do not satisfy the above-mentioned multiple preset attribute items. In this case, this application does not limit how the electronic device synthesizes the multiple layers included in the first frame.

[0133] S304: The image synthesis library determines whether to perform synthesis prediction on the first frame.

[0134] The above briefly describes composition prediction. This section further describes composition prediction with reference to the accompanying figures. As you can see, composition prediction is a method for compositing layers, designed to reduce the time it takes to complete the process. For details, refer to S305-S307 below; this section will not be discussed in detail here.

[0135] In one possible implementation, whether to perform synthesis prediction on the first frame may be determined based on one or more of the following conditions:

[0136] The second frame in the following text refers to the frame before the first frame. In conjunction with FIG4b , if the first frame is “frame 2”, the second frame is “frame 1”. If the first frame is “frame 3”, the second frame is “frame 2”.

[0137] Condition 1: Whether the layers in the first frame include layers for which the requested synthesis mode is GPU synthesis.

[0138] For example, referring to the description of S303 above, it can be determined whether the layers in the first frame include a layer for which the requested synthesis mode is GPU synthesis.

[0139] Condition 2: Are the layer data of the first frame and the second frame the same?

[0140] For example, layer data may include the number of layers, layer size, and layer content. The layer data of the first or second frame can be recorded using a hash value. In one possible implementation, whether the layer data of the first and second frames are identical can be determined by inspecting the BitMaps corresponding to the first and second frames.

[0141] Condition three: Are the synthesis methods of the first frame and the second frame the same?

[0142] It should be noted that the synthesis method of the frame is different from the synthesis method of the layer request. The synthesis method of the frame refers to the set of synthesis methods requested for each layer in the frame. Exemplarily, when the synthesis method of the first frame and the second frame is the same, the layers in the first frame and the second frame are sorted according to the same rules, and the synthesis methods of the layers with the same sequence number in the first frame and the second frame are the same. For example, the first frame and the second frame both include 3 layers, the synthesis method of the second layer request in the first frame and the second frame is GPU synthesis, and the synthesis methods of the other two layer requests are hardware synthesizer synthesis, then it is said that the synthesis method of the first frame and the second frame is the same.

[0143] The image synthesis library in the above S303 can determine the synthesis method of the layer in the first frame based on the properties of the layer. However, whether the remaining layers in the first frame can be synthesized by the hardware synthesizer still needs to be further judged. For example, the hardware synthesizer can determine whether the layer can be processed by the hardware synthesizer based on its own hardware properties. If the layer can be processed by the hardware synthesizer, the synthesis strategy of the layer can be set to hardware processor synthesis. If the layer cannot be processed by the hardware synthesizer and the GPU needs to be used to process the layer, the synthesis strategy of the layer can be set to GPU synthesis. Among them, whether the layer can be processed by the hardware synthesizer can be determined by one or more of the following conditions, such as the bandwidth of the hardware synthesizer, the number of layers input by the hardware synthesizer, the layer size input by the hardware synthesizer, and the layer format input by the hardware synthesizer.

[0144] Condition 4: Whether the first frame is an animation frame.

[0145] In this application, the animation frame refers to the frame that makes up the animation. The animation includes N consecutive frames, where N is greater than or equal to 1; the electronic device starts playing the animation, including: the electronic device starts playing the animation according to the animation attributes; the animation attributes include the animation content, the animation size, the animation duration, or at least one of the animation start position and the animation end position; the animation start position is used to indicate the position of the first frame of the N frames on the display screen, and the animation end position is used to indicate the position of the last frame of the N frames on the display screen. At present, the scenarios for playing animations on electronic devices include one or more of the following: the electronic device starts the target application, the electronic device exits an application, the electronic device enters the multi-tasking interface, the electronic device exits the multi-tasking interface, and switches the screen refresh rate. Therefore, the animation frame includes one or more of the following: the image displayed in the process of the electronic device starting the target application, the image displayed in the process of the electronic device exiting an application, the image displayed in the process of the electronic device entering the multi-tasking interface, the image displayed in the process of the electronic device starting an application from the multi-tasking interface, and the image displayed in the process of the electronic device switching the screen refresh rate.

[0146] Figures 4a and 4b above exemplarily illustrate the images displayed when an electronic device launches a target application. Therefore, the animation frames include frames 1, 2, 3, 4, 5, and 6 in Figure 4b. Specifically, in conjunction with Figures 4a and 4b above, when the user clicks icon 101, the startup animation begins and the image displayed in interface 102 is played; when the image displayed in interface 102 fills the entire screen (or display), the startup animation ends.

[0147] It should be noted that in the context of startup animations, the startup animation can be the animation that plays when the user first launches the app, or it can be the animation that plays when the user launches the app for a non-first time. "First time" means that the app is neither running in the foreground nor in the background. "Non-first time" means that the app is running in the background. In this case, the startup animation can also be understood as the process of the app switching from the background to the foreground.

[0148] Taking the "Communication" application as an example and the animated scene as the scene of exiting the "Communication" application, the mobile phone displays the interface 103 shown in Figure 5 (a), which is the interface after the "Communication" application is launched. For example, as shown in Figure 5 (a), the interface 103 can be, for example, the interface of the user's contact list. Then, in response to the user's exit operation on the "Communication" application, the mobile phone displays the interface 104 shown in Figure 5 (b). Among them, the interface 204 can be, for example, the mobile phone home screen interface.

[0149] In some embodiments, the exit operation can be, for example, a gesture operation, a voice operation, or a touch operation. The touch operation can be, for example, a click operation, a slide operation, etc. For example, if the exit operation is a slide operation, as shown in FIG5( b ), the exit operation can be, for example, a user sliding up on the interface of the "communication" application.

[0150] In the scenario of exiting the "Communication" application, the exit animation of the "Communication" application is the process of the user swiping up on the interface of the "Communication" application to display the main screen interface of the mobile phone. In some embodiments, combined with the above embodiments, the exit animation is the process of the image displayed in the interface 102 changing from large to small. Specifically, when the user swipes up on the interface of the "Communication" application, the exit animation begins; the image in the interface 102 displayed on the mobile phone begins to shrink; when the image displayed in the interface 102 is completely exited and the mobile phone displays the main screen interface, the exit animation ends.

[0151] Taking the dynamic effect scenario of a mobile phone switching from the main screen interface to the multi-tasking interface as an example, for example, the mobile phone displays the interface 105 shown in Figure 6 (a), which is the main screen interface of the mobile phone. Then, in response to the user's operation on the main screen interface, the mobile phone displays the interface 106 shown in Figure 6 (b), which is the multi-tasking interface of the mobile phone. Among them, interface 106 includes the interface of application 1 running in the background, and application 1 is the application with the shortest running time in the background. Among them, in this dynamic effect scenario, the dynamic effect refers to the process from the user's operation on the main screen interface to the mobile phone displaying the multi-tasking interface.

[0152] It should be noted that the user's operation on the home screen interface can refer to the example of the exit operation in the above embodiment, and will not be repeated here. Taking the operation as an example of a sliding operation, as shown in Figure 6 (b), the operation can be, for example, a user sliding up on the home screen interface.

[0153] Taking the example of a mobile phone launching the "Communication" app in the multitasking interface as an example, the phone displays interface 107 as shown in Figure 7(a), which is the multitasking interface of the phone. Then, in response to the user's operation on interface 107, the phone displays interface 108 as shown in Figure 7(b), which is the interface after the "Communication" app is launched. In this animation scenario, the animation refers to the process from the user's operation on interface 107 to the phone displaying interface 108.

[0154] It should be noted that the user's operation on the interface 107 can refer to the example of the start operation in the above embodiment, which will not be repeated here. Taking the operation as a sliding operation as an example, as shown in Figure 7 (b), the operation can be, for example, a click operation.

[0155] The following exemplifies various judgment rules provided by the present application for determining whether to perform synthesis prediction on the first frame, including one or more of the following:

[0156] Judgment rule 1: The first frame includes a layer whose requested synthesis method is GPU synthesis.

[0157] Under this judgment rule, synthesis prediction is performed on the first frame only when the first frame includes a layer whose synthesis method is GPU synthesis. This can increase the probability of performing synthesis prediction on the first frame, save the synthesis time of the first frame, thereby avoiding screen freezes on electronic devices and improving user experience.

[0158] It can be understood that synthesis prediction refers to the asynchronous execution of the GPU synthesis process (S305) and the synthesis strategy judgment process (S306), which allows the electronic device to determine the synthesis strategy in the first frame without affecting the electronic device's execution of GPU synthesis. Therefore, the computing power of the electronic device can be fully utilized for GPU synthesis, thereby reducing the synthesis time of the first frame, avoiding screen freezes, and thus improving the user experience.

[0159] Judgment rule two: the first frame includes a layer whose requested synthesis method is GPU synthesis, and the second frame includes a layer whose requested synthesis method is GPU synthesis.

[0160] It's understandable that since the changes between two frames are typically small, if the previous frame includes a layer whose requested synthesis method is GPU synthesis, the next frame is likely to also include a layer whose requested synthesis method is GPU synthesis. Therefore, if "Judgment Rule 2" is met, synthesis prediction can be performed on the first frame. This can increase the probability of performing synthesis prediction on the first frame, save synthesis time for the first frame, avoid screen freezes on electronic devices, and improve the user experience.

[0161] Judgment rule three: The first frame includes a layer whose requested synthesis method is GPU synthesis, and the first frame is an animation frame.

[0162] Understandably, because motion effects require complex and diverse dynamic images, the synthesis of the frames that make up the effect is more complex than that of ordinary frames, and the synthesis method for their layer requests typically includes GPU synthesis. Therefore, if "Judgment Rule 3" is met, synthesis prediction can be performed on the first frame. This can increase the probability of performing synthesis prediction on the first frame, save synthesis time for the first frame, avoid screen freezes on electronic devices, and improve the user experience.

[0163] Optionally, based on the judgment rule three, it may also be judged whether the second frame includes a layer whose requested synthesis mode is GPU synthesis.

[0164] For example, based on judgment rule three, if the second frame does not include a layer whose requested synthesis method is GPU synthesis, synthesis prediction is performed on the first frame. It can be understood that even if the previous frame (the second frame) of the first frame does not include a layer whose requested synthesis method is GPU synthesis, since the first frame is a motion effect frame, the first frame may also include multiple layers whose requested synthesis method is GPU synthesis. Therefore, in this case, synthesis prediction can be performed on a frame, which can increase the probability of performing synthesis prediction on the first frame, save synthesis time for the first frame, thereby avoiding screen freezes of electronic devices and improving user experience.

[0165] For example, based on judgment rule three, if the second frame includes a layer whose requested synthesis method is GPU synthesis, synthesis prediction is performed on the first frame. It can be understood that in this example, performing synthesis prediction on the first frame not only requires the first frame to be an animation frame, but also requires ensuring that the previous frame of the first frame includes a layer whose requested synthesis method is GPU synthesis. This can avoid performing synthesis prediction on the first frame due to unreasonable judgment, which may cause additional thread overhead and increased CPU load.

[0166] Judgment rule four: the first frame includes a layer whose requested synthesis method is GPU synthesis, and the layer data of the first frame is the same as the layer data of the second frame, and the second frame includes a layer whose requested synthesis method is GPU synthesis.

[0167] It's understandable that the layer data for the first frame is identical to the layer data for the second frame, indicating that the first and second frames are identical. This, combined with the fact that the second frame includes a layer whose requested synthesis method is GPU synthesis, indicates that the first frame likely also includes a layer whose requested synthesis method is GPU synthesis. Therefore, if "Judgment Rule 4" is met, synthesis prediction can be performed on the first frame. This increases the probability of performing synthesis prediction on the first frame, saves synthesis time for the first frame, avoids screen freezes on electronic devices, and improves the user experience.

[0168] Judgment rule five: the first frame includes a layer whose requested synthesis method is GPU synthesis, and the layer data of the first frame is different from the layer data of the second frame, and the second frame includes a layer whose requested synthesis method is GPU synthesis.

[0169] It can be understood that the layer data of the first frame is different from the layer data of the second frame, indicating that the second frame has changed compared to the first frame, combined with the fact that the second frame includes a layer whose requested synthesis method is GPU synthesis. It can be understood that when the previous frame of the first frame includes a layer whose requested synthesis method is GPU synthesis, even if the previous frame of the first frame is different from the first frame, synthesis prediction is performed on the first frame. This implementation method can increase the probability of entering synthesis prediction, thereby reducing the synthesis time of the first frame, avoiding screen freezes, and thus improving user experience.

[0170] Judgment rule six: the first frame includes a layer whose requested synthesis method is GPU synthesis, and the synthesis methods of the first frame and the second frame are the same, and the second frame includes a layer whose requested synthesis method is GPU synthesis.

[0171] Exemplarily, in the process of synthesizing the M first layers of the first frame, the attributes of the first layer satisfy at least one of the above-mentioned multiple preset attribute items, and M is a positive integer. In the process of synthesizing the N second layers of the second frame, the attributes of the second layer satisfy at least one of the above-mentioned multiple preset attribute items, and N is a positive integer, and M is equal to N. For example, M=N=1, or M=N=2. In this example, the synthesis method of the first frame and the second frame is the same. Combined with the fact that the second frame includes a layer whose synthesis method is requested to be GPU synthesis, it indicates that the first frame also most likely includes a layer whose synthesis method is requested to be GPU synthesis. Therefore, synthesis prediction can be performed on the first frame when "Judgment Rule Six" is met, which can increase the probability of performing synthesis prediction on the first frame and save the synthesis time of the first frame, thereby avoiding screen freezes of electronic devices and improving user experience.

[0172] Judgment rule seven: the first frame includes a layer whose requested synthesis method is GPU synthesis, the synthesis methods of the first frame and the second frame are different, and the second frame includes a layer whose requested synthesis method is GPU synthesis.

[0173] Exemplarily, in the process of synthesizing the M first layers of the first frame, the attributes of the first layer satisfy at least one of the above-mentioned multiple preset attribute items, where M is a positive integer. In the process of synthesizing the N second layers of the second frame, the attributes of the second layer satisfy at least one of the above-mentioned multiple preset attribute items, where N is a positive integer, and M is equal to N. For example, M=N=1, or M=N=2. In this example, the synthesis methods of the first frame and the second frame are different. In combination with the fact that the second frame includes a layer for which the synthesis method requested is GPU synthesis, it can be understood that when the previous frame of the first frame includes a layer for which the synthesis method requested is GPU synthesis, even if the number of second layers in the second frame is not equal to the number of first layers in the first frame, synthesis prediction is performed on the first frame. This implementation method can increase the probability of entering synthesis prediction, thereby reducing the synthesis time of the first frame, avoiding screen freezes, and thus improving the user experience.

[0174] It can be understood that the present application proposes a variety of judgment rules for determining whether to perform synthesis prediction on the first frame. When the first frame meets any of the above judgment rules, synthesis prediction can be performed on the first frame. Therefore, the probability of performing synthesis prediction on the first frame can be increased, thereby reducing the synthesis time of the first frame, avoiding screen freezes, and improving user experience.

[0175] It should be noted that, in a specific implementation, one or more of the above judgment rules may be selected to determine whether to perform synthesis prediction on the first frame. The above seven judgment rules may also be combined with each other, which is not limited in this application.

[0176] Next, the first frame satisfies one or more of the above judgment rules and performs synthesis prediction on the first frame, and the subsequent description is given. The execution process of synthesis prediction includes S305, S306 and S307 shown in Figure 2, which are described in turn below.

[0177] S305 : The image rendering library performs GPU synthesis on the M first layers to generate a first target layer.

[0178] S306: The asynchronous synthesizer determines the synthesis mode of the layer requests other than the M first layers in the first frame.

[0179] It should be noted that S305 and S306 are executed asynchronously, that is, the contents shown in S305 and S306 are run through two threads. Specifically, the start time and end time of S305 and S306 are not limited in this application. In one possible implementation, there is a moment when S305 and S306 are running simultaneously in the electronic device. For example, in the case of determining to perform synthesis prediction on the first frame, the electronic device uses the first process to run the content of S305, and uses the second process to run the content of S306, that is, the synthesis process of the M first layers by the GPU and the synthesis method of determining the layer request in the first frame do not affect each other, so that the computing power of the GPU can be fully utilized to synthesize the M first layers, thereby reducing the synthesis time of the first frame, avoiding screen freezes, and thus improving the user experience.

[0180] The image rendering library performs GPU synthesis on the M first layers to generate a first target layer. The first target layer is a single layer synthesized by the GPU from the M first layers. For example, if M is 1, the GPU synthesizes one first layer to generate one layer. For another example, if M is 3, the GPU synthesizes three first layers to generate one layer. By synthesizing the M first layers on the GPU, the first target layer synthesized by the GPU can be used to generate a final image through a hardware synthesizer and displayed on the display screen of the electronic device.

[0181] In a possible implementation, the asynchronous synthesizer may detect whether the first frame meets the asynchronous synthesis conditions.

[0182] In another possible implementation, the asynchronous compositor may also obtain the synthesis mode fed back by the hardware compositor and determine the synthesis mode requested by the layers other than the M first layers in the first frame. For example, based on the synthesis mode fed back by the hardware compositor, the asynchronous compositor determines that the synthesis modes requested by the L layers in the first frame are GPU-synthesized layers (referred to as third layers for short), where L is a positive integer, such as 1, 2, or 3. The asynchronous compositor may then pass the L third layers in the first frame to the layer rendering library, allowing the layer rendering library to perform GPU synthesis on the L third layers and generate a second target layer (S307).

[0183] Alternatively, L may be 0, that is, the asynchronous synthesizer determines 0 third layers based on the synthesis method fed back by the hardware synthesizer. In this case, there is no need to execute S307.

[0184] S307: The image rendering library performs GPU synthesis on the L third layers to generate a second target layer. For details, please refer to the relevant description of S305, which will not be repeated here.

[0185] It should be noted that the thread used by the image rendering library for GPU synthesis of the L third layers can be the same thread used by the image rendering library for GPU synthesis of the M first layers. That is, the thread used to instruct the GPU to synthesize layers sequentially performs GPU synthesis on the M first layers and the L third layers. Furthermore, if L is 0, the second target layer will not be generated. Therefore, when L is 0, the second target layer can also be understood as non-existent or an empty layer.

[0186] When the image rendering library synthesizes the first target layer and the second target layer, the first target layer and the second target layer can be passed to the image synthesis library. The image synthesis library then packages the first target layer, the second target layer and the layer whose synthesis method is requested to be the hardware synthesizer and passes them to the hardware synthesizer.

[0187] S308: The hardware synthesizer performs final synthesis on the first frame and sends it to the display unit.

[0188] Because layers that cannot be processed by the hardware compositor have been processed into layers that the hardware compositor can process through GPU synthesis, the hardware compositor can synthesize the multiple layers sent by the image synthesis library. For example, the hardware compositor can call a display driver to synthesize the multiple layers sent by the image synthesis library. Optionally, the display driver can also send the final synthesized image to the display screen for display.

[0189] To further demonstrate the beneficial effects of performing composition prediction, please refer to Figures 8 and 9, where Figure 8 shows the case where composition prediction is not performed, and Figure 9 shows the case where composition prediction is performed. It should be noted that each row in Figures 8 and 9 can be represented as a process, and the execution order of the processes is to first judge from left to right, and then from top to bottom. Taking Figure 9 as an example, the execution order of the functions in Figure 9 can be prepareFrameAsync, choosecompositionstrategy, REThreaded::drawLayers, etc.

[0190] As shown in Figure 8, a trace without composite prediction involves the image compositor first determining the composite method requested by the layers in the frame, then invoking GPU composite via the image rendering library to perform GPU composite on the layers whose composite method was requested. Finally, the hardware compositor performs the final composite of the layers and sends the result to the display for display.

[0191] As shown in Figure 9, the trace for executing composition prediction involves asynchronous execution between the asynchronous compositor and the image rendering library. Specifically, while the asynchronous compositor is determining the composition method for the layer requests within a frame, the image rendering library also performs GPU composition of the layer. This eliminates the need for the image rendering library to wait for the composition method for the layer requests within a frame to be determined. GPU composition of the layer can proceed synchronously with the determination of the composition method for the layer requests within the frame, thus saving layer composition time, reducing screen lag, and improving the user experience.

[0192] Please refer to Figure 10, which is a workflow diagram of a user interface switching process software provided in the implementation of this application, which is used to demonstrate the workflow of the user interface switching process software when using the frame synthesis method provided by this application. Combined with the trace of executing synthesis prediction shown in Figure 9 above, it can be seen that in the user interface switching process shown in Figure 10, each frame is determined to execute synthesis prediction. It can be seen that the frame synthesis method provided by this application can greatly improve the probability of determining the frame for synthesis prediction, save the time of layer synthesis, thereby reducing the occurrence of screen freezes, and thus improving the user experience.

[0193] In summary, this application uses synthesis prediction technology. By designing a variety of rules for determining the synthesis prediction of frames, the probability of synthesizing predictions for frames can be increased, which can save the time of layer synthesis, thereby reducing the occurrence of screen freezes and improving the user experience. In addition, this application also uses the rendering mode of scaling as a condition for determining whether the synthesis mode requested by the layer is GPU synthesis, which can increase the probability of determining the synthesis mode requested by the layer as GPU synthesis. Combined with the premise of determining the prediction of frame synthesis, that is, the frame includes a layer whose synthesis mode requested is GPU synthesis, the probability of synthesizing predictions for frames can be increased.

[0194] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0195] As used in the above embodiments, the term “when…” may be interpreted to mean “if…” or “after…” or “in response to determining…” or “in response to detecting…”, depending on the context. Similarly, the phrases “upon determining…” or “if (stated condition or event) is detected” may be interpreted to mean “if determining…” or “in response to determining…” or “upon detecting (stated condition or event)” or “in response to detecting (stated condition or event)”, depending on the context.

[0196] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk).

[0197] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A frame synthesis method, which is applied to an electronic device, characterized in that, The method includes: The display screen of the electronic device displays a first user interface; During the process of the electronic device switching from the first user interface to display a second user interface, when synthesizing M first layers of the first frame, if the attributes of the first layer satisfy at least one of a plurality of preset attribute items, where M is a positive integer; the electronic device performs synthesis prediction on the first frame; Wherein, the plurality of preset attribute items include scaling.

2. The method according to claim 1, characterized in that, The plurality of preset attribute items further include: Rounded corners, blur effect, rotation at a special angle, high dynamic range (HDR) synthesis, and shadow effect.

3. The method according to claim 1 or 2, characterized in that, The electronic device performing synthesis prediction on the first frame includes: In the case where the second frame includes a layer with a requested synthesis method of GPU synthesis, the electronic device performs synthesis prediction on the first frame; the second frame is the previous frame of the first frame.

4. The method according to claim 1 or 2, characterized in that The electronic device performing synthesis prediction on the first frame includes: In the case where the first frame belongs to an animation frame, the electronic device performs synthesis prediction on the first frame.

5. The method according to claim 4, characterized in that, The animation frame includes one or more of the following: An image displayed during the process of the electronic device starting a target application; Or, An image displayed during the process of the electronic device exiting the target application; Or, An image displayed during the process of the electronic device entering a multitasking interface; Or, An image displayed during the process of the electronic device entering the target application from the multitasking interface.

6. The method according to claim 4 or 5, characterized in that, During the process of the electronic device switching from the first user interface to display a second user interface, the display screen displays the animation frame.

7. The method according to any one of claims 4 to 6, characterized in that The electronic device performing synthesis prediction on the first frame further includes: In the case where the second frame does not include a layer with a requested synthesis method of GPU synthesis, the electronic device performs synthesis prediction on the first frame; the second frame is the previous frame of the first frame.

8. The method according to any one of claims 1 to 7, characterized in that The layer data of the first frame is the same as the layer data of the second frame; or, the layer data of the first frame is different from the layer data of the second frame; the second frame is the previous frame of the first frame.

9. The method according to any one of claims 1-8, characterized in that, The synthesis method of the first frame is the same as the synthesis method of the second frame; or, the synthesis method of the first frame is different from the synthesis method of the second frame; the second frame is the previous frame of the first frame.

10. An electronic device, characterized in that, The electronic device includes: a display screen, a memory, and a processor coupled to the memory; the display screen is used to display an interface, the memory stores a computer program, and when the processor executes the computer program, the electronic device implements the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, comprising computer instructions, characterized in that, When the computer instructions run on the electronic device, the electronic device executes the method according to any one of claims 1 to 9.

Images