Frame synthesis method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a frame synthesis method, electronic equipment and a computer readable storage medium, and relates to the field of terminals. In the method, when the electronic equipment switches from a first user interface to display a second user interface, in the process of synthesizing M first layers of a first frame, if the attribute of the first layers meets at least one of a plurality of preset attribute items, synthesis prediction is executed on the first frame, and M is a positive integer. The probability of executing synthesis prediction on the frame can be improved, and the synthesis time of the first frame can be saved, so that the image lagging of the electronic equipment is avoided, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of terminals, and in particular to a frame synthesis method, an electronic device, and a computer-readable storage medium. Background Art

[0002] An electronic device displays a picture on a display screen to achieve human-computer interaction with a user. In some scenarios, due to the need to perform processing such as rotation, scaling, and blur effects on layers, frame synthesis needs to go through the GPU, resulting in a longer synthesis time, thus causing the picture to freeze, which affects the user experience. Summary of the Invention

[0003] This application provides a frame synthesis method, an electronic device, and a computer-readable storage medium. When M first layers in a first frame of an electronic device meet a first preset condition, the electronic device directly performs synthesis prediction on the first frame, which can increase the probability of performing synthesis prediction on the frame, save the synthesis time of the first frame, thus avoiding the picture freeze of the electronic device and enhancing the user experience.

[0004] In a first aspect, this application provides a frame synthesis method, which is applied to an electronic device. The method includes: The display screen of the electronic device displays a first user interface. During the process of the electronic device switching from the first user interface to display a second user interface, when synthesizing M first layers of the first frame, if the attribute of the first layer meets at least one of a plurality of preset attribute items, where M is a positive integer; the electronic device performs synthesis prediction on the first frame; where the plurality of preset attribute items include scaling.

[0005] In the frame synthesis method provided by this application, the display screen of the electronic device displays a first user interface, and during the process of the electronic device switching from the first user interface to display a second user interface, the first frame is also synthesized. During the process of the electronic device synthesizing M first layers of the first frame, if the attribute of the first layer meets at least one of a plurality of preset attribute items (including scaling effect), synthesis prediction is performed on the first frame. The probability of performing synthesis prediction on the first frame can be increased, the synthesis time of the first frame can be saved, thus avoiding the picture freeze of the electronic device and enhancing the user experience.

[0006] It can be understood that synthesis prediction refers to a method of asynchronously executing the synthesis method of the GPU synthesizing M first layers and the hardware synthesizer determining the synthesis method of the layers other than the M first layers in the first frame. This method allows the electronic device not to affect the electronic device's execution of GPU synthesis during the process of determining the synthesis method of other layers in the first frame, so that the computing power of the electronic device can be fully utilized for GPU synthesis.

[0007] Optionally, when the composition method of the layer requests outside the M first layers in the first frame is GPU composition, the electronic device performs GPU composition on the layer.

[0008] Optionally, the above first user interface and second user interface may be in the screen-off state.

[0009] Optionally, the rendering attribute of scaling is used to indicate the rendering of the target layer with a magnifying effect or a reducing effect.

[0010] In a possible implementation manner, the first preset condition further includes that the multiple preset attribute items include: rounded corners, blur effect, special angle rotation, high dynamic range (HDR) synthesis, and shadow effect.

[0011] In the implementation manner provided by this application, the multiple preset attribute items further include rounded corners, blur effect, special angle rotation, high dynamic range (HDR) synthesis, and shadow effect. As can be known from the above, when the layer meets at least one of the multiple preset attribute items, the electronic device can directly perform GPU composition on the layer. In the implementation manner of this application, more preset attribute items are shown, so the probability that the layer in the first frame meets at least one of the multiple preset attribute items can be increased, so that the electronic device directly performs GPU composition on more layers.

[0012] In a possible implementation manner, the electronic device performs composition prediction on the first frame, including: when the second frame includes a layer whose requested composition method is GPU composition, the electronic device performs composition prediction on the first frame; the second frame is the previous frame of the first frame.

[0013] In the implementation manner of this application, the second frame includes a layer whose requested composition method is GPU composition. Combining with the fact that the second frame is the previous frame of the first frame, it can be known that the first frame probably also includes a layer whose requested composition method is GPU composition (because the change between the two frames is small. If the previous frame includes a layer whose requested composition method is GPU composition, the subsequent frame probably also includes a layer whose requested composition method is GPU composition). Therefore, in this case, it can be determined that composition prediction is performed on the first frame, which can increase the probability of performing composition prediction on the first frame, save the composition time of the first frame, thus avoiding the screen freeze of the electronic device and improving the user experience.

[0014] In a possible implementation manner, the electronic device performs composition prediction on the first frame, including: when the first frame belongs to an animation frame, the electronic device performs composition prediction on the first frame.

[0015] In the implementation of the present application, the first frame belongs to the motion effect frame. It is understandable that since the motion effect needs to display complex and diverse dynamic images, the motion effect frames that make up the motion effect will be more complicated than the synthesis of ordinary frames during the synthesis process, and its layer request synthesis method usually includes GPU synthesis. Therefore, it can be determined in this case to perform synthesis prediction on the first frame, which can increase the probability of performing synthesis prediction on the first frame, save the synthesis time of the first frame, thereby avoiding screen freezes of electronic devices and improving user experience.

[0016] In one possible implementation, the animation frame includes one or more of the following: an image displayed when the electronic device starts the target application; or, an image displayed when the electronic device exits the target application; or, an image displayed when the electronic device enters a multitasking interface; or, an image displayed when the electronic device enters the target application from the multitasking interface.

[0017] In an embodiment of the present application, the animation frame includes one or more of the following: an image displayed in the process of the electronic device starting the target application, or an image displayed in the process of the electronic device exiting the target application, or an image displayed in the process of the electronic device entering the multi-tasking interface, or an image displayed in the process of the electronic device entering the target application from the multi-tasking interface.

[0018] Optionally, an image displayed during a certain process of the electronic device can be set as a motion effect frame in a preset manner. For example, an image displayed during the process of unlocking the screen of the electronic device can be set as a motion effect frame.

[0019] Optionally, the animation frame also includes an image displayed during the process of switching the screen refresh rate.

[0020] In a possible implementation, the electronic device displays the motion effect frame on the display screen during a period in which the electronic device switches from the first user interface to display the second user interface.

[0021] In the implementation manner of the present application, the electronic device switches from the first user interface to display the second user interface device, and a motion effect frame is displayed on the display screen of the electronic device.

[0022] Optionally, the motion effect frame can also be displayed in the background of the electronic device. For example, the startup process of the target application is performed in the background of the electronic device, and the screen in the startup process of the target application is displayed in the background.

[0023] In a possible implementation, the electronic device performs synthesis prediction on the first frame, further comprising: when the second frame does not include a layer whose requested synthesis method is GPU synthesis, the electronic device performs synthesis prediction on the first frame; and the second frame is a previous frame of the first frame.

[0024] In an embodiment of the present application, when the second frame does not include a layer whose requested composition method is GPU composition and the first frame is an animation frame, the electronic device performs composition prediction on the first frame. It can be understood that even if the previous frame (the second frame) of the first frame does not include a layer whose requested composition method is GPU composition, since the first frame is an animation frame, the first frame may include multiple layers whose requested composition method is GPU composition. Therefore, it can be determined to perform composition prediction on the first frame in this case, which can increase the probability of performing composition prediction on the first frame, save the composition time of the first frame, thereby avoiding screen jitter of the electronic device and improving the user experience.

[0025] In a possible implementation, the layer data of the first frame is the same as the layer data of the second frame; or, the layer data of the first frame is different from the layer data of the second frame; the second frame is the previous frame of the first frame.

[0026] In an embodiment of the present application, the layer data of the first frame may be the same as or different from the layer data of the second frame. When the layer data of the first frame is the same as the layer data of the second frame, it indicates that the first frame is the same as the second frame. Combining with the fact that the second frame includes a layer whose requested composition method is GPU composition, it indicates that the first frame also includes a layer whose requested composition method is GPU composition. Therefore, it can be determined to perform composition prediction on the first frame in this case, which can increase the probability of performing composition prediction on the first frame, save the composition time of the first frame, thereby avoiding screen jitter of the electronic device and improving the user experience.

[0027] The layer data of the first frame is different from the layer data of the second frame, indicating that the second frame has changed compared with the first frame. Combining with the fact that the second frame includes a layer whose requested composition method is GPU composition. It can be understood that when the previous frame of the first frame includes a layer whose requested composition method is GPU composition, even if the previous frame of the first frame is different from the first frame, composition prediction is still performed on the first frame. This implementation can increase the probability of the first frame entering composition prediction, thereby reducing the composition time of the first frame, avoiding screen jitter, and further improving the user experience.

[0028] In a possible implementation, the composition method of the first frame is the same as the composition method of the second frame; or, the composition method of the first frame is different from the composition method of the second frame; the second frame is the previous frame of the first frame.

[0029] In the embodiments of the present application, the synthesis method of the first frame and the synthesis method of the second frame may be the same or different. When the synthesis method of the first frame is the same as that of the second frame, combining the layers in the second frame whose requested synthesis method is GPU synthesis indicates that the first frame also includes the layers whose requested synthesis method is GPU synthesis. Therefore, in this case, it can be determined that synthesis prediction is performed on the first frame, which can increase the probability of performing synthesis prediction on the first frame, save the synthesis time of the first frame, thereby avoiding frame freezing of the electronic device and enhancing the user experience.

[0030] If the synthesis method of the first frame is different from that of the second frame, it indicates that the second frame has changed compared to the first frame. Combining the layers in the second frame whose requested synthesis method is GPU synthesis. It can be understood that when the previous frame of the first frame includes the layers whose requested synthesis method is GPU synthesis, even if the previous frame of the first frame is different from the first frame, synthesis prediction is still performed on the first frame. This embodiment can increase the probability of the first frame entering synthesis prediction, thereby reducing the synthesis time of the first frame, avoiding frame freezing, and further enhancing the user experience.

[0031] In a second aspect, the present application provides an electronic device, which includes a display screen, a memory, and a processor coupled to the memory; the display screen is used to display an interface, the memory stores a computer program, and when the processor executes the above computer program, the electronic device implements the method described in any item of the first aspect above.

[0032] In a third aspect, the present application provides a computer-readable storage medium, which stores a computer program or computer instructions, and when the foregoing computer program or computer instructions are executed by a processor, the method described in any item of the first aspect above is implemented.

[0033] In a fourth aspect, an embodiment of the present application provides a computer program product, and when the computer program product is executed by a processor, the method described in any item of the first aspect above will be implemented.

[0034] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a memory. The memory is used to store a computer program or computer instructions, and the processor is used to execute the computer program or computer instructions stored in the memory, so that the chip executes the method described in any item of the first aspect above.

[0035] The solutions provided in the second to fifth aspects above are used to implement or cooperate with the corresponding methods provided in the first aspect, so they can achieve the same or corresponding beneficial effects as the corresponding methods in the first aspect, and will not be elaborated here. Description of the Drawings

[0036] Figure 1 It is a schematic diagram of the hardware structure of an electronic device 100 provided by an embodiment of the present application;

[0037] Figure 2 It is a software structure block diagram of a terminal device provided by an embodiment of the present application;

[0038] Figure 3 It is a schematic flow diagram of a frame synthesis method provided by an embodiment of the present application;

[0039] Figure 4a It is a schematic diagram of an interface switch provided by an embodiment of the present application Figure 1 ;

[0040] Figure 4b It is a schematic diagram of an animation effect playback provided by an embodiment of the present application;

[0041] Figure 5 It is a schematic diagram of an interface switch provided by an embodiment of the present application Figure 2 ;

[0042] Figure 6 It is a schematic diagram of an interface switch provided by an embodiment of the present application Figure 3 ;

[0043] Figure 7 It is the fourth schematic diagram of an interface switch provided by an embodiment of the present application;

[0044] Figure 8 It is a schematic diagram of not performing synthesis prediction provided by an embodiment of the present application;

[0045] Figure 9 It is a schematic diagram of performing synthesis prediction provided by an embodiment of the present application;

[0046] Figure 10 It is a schematic working flow diagram of the software during the user interface switching process provided by an embodiment of the present application. Detailed implementation manners

[0047] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular forms "a", "one kind", "said", "above-mentioned", "this" and "this one" are also intended to include the plural forms, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more of the listed items.

[0048] Hereinafter, the terms "first" and "second" are for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality" is two or more.

[0049] The frame synthesis processing method provided by the embodiments of the present application can be applied to an electronic device with a display function.

[0050] The electronic device includes a terminal device, and the terminal device can also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device can be a mobile phone, smart TV, wearable device, tablet computer (Pad), a computer with wireless transceiver function, virtual reality (VR) terminal device, augmented reality (AR) terminal device, wireless terminal in industrial control, wireless terminal in unmanned driving wireless terminal in, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, and so on. The embodiments of the present application do not limit the specific technologies and specific device forms adopted by the terminal device.

[0051] In order to better understand the embodiments of the present application, the structure of the terminal device in the embodiments of the present application will be introduced below:

[0052] Figure 1 is a schematic structural diagram of the electronic device 100 provided by the embodiments of the present application.

[0053] The embodiments will be specifically described below taking the electronic device 100 as an example. It should be understood that the electronic device 100 may have more or fewer components than Figure 1 shown in, may combine two or more components, or may have different component configurations. Figure 1 The various components shown in can be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0054] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a camera 193, a display screen 194, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a touch sensor 180B, etc.

[0055] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0056] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0057] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data just used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0058] The charging management module 140 is used to receive charging input from a charger. Among them, the charger may be a wireless charger or a wired charger.

[0059] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives the input from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, etc.

[0060] The electronic device 100 realizes the display function through the GPU, the display screen 194, the application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to execute mathematical and geometric calculations for graphics synthesis. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0061] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0062] The camera 193 is used to capture static images or videos. The electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0063] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0064] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple coding formats.

[0065] The NPU is a neural-network (NN) computing processor. By learning from the biological neural network structure, such as learning from the transmission mode between human brain neurons, it can quickly process input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as image recognition, face recognition, speech recognition, text understanding, etc.

[0066] The external memory interface 120 can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external memory card.

[0067] The internal memory 121 can be used to store computer-executable program code, and the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, applications required for at least one function (such as face recognition function, fingerprint recognition function, mobile payment function, etc.). The data storage area can store data created during the use of the electronic device 100 (such as face information template data, fingerprint information template, etc.). In addition, the internal memory 121 can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0068] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor, etc. For example, music playback, recording, etc.

[0069] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0070] The speaker 170A, also known as the "loudspeaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or hands-free calls through the speaker 170A.

[0071] The receiver 170B, also known as the "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 100 answers a call or a voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0072] The microphone 170C, also known as the "microphone", "transmitter", is used to convert a sound signal into an electrical signal.

[0073] The pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor may include at least two parallel plates having conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions.

[0074] The touch sensor 180B, also known as the "touch panel". The touch sensor 180B may be disposed on the display screen 194, and the touch sensor 180B and the display screen 194 form a touch screen, also known as the "touch screen". The touch sensor 180B is used to detect touch operations acting on or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180B may also be disposed on the surface of the electronic device 100, at a different position from where the display screen 194 is located.

[0075] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture, etc. In the embodiments of this application, taking the Android system with a layered architecture as an example, the software structure of the electronic device 100 is exemplarily described.

[0076] Figure 2 It is the software structure block diagram of the terminal device in the embodiments of this application. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom are the application layer, the application framework layer, Android runtime and system libraries, the hardware abstraction layer, and the kernel layer.

[0077] The application layer may include a series of application program packages. As Figure 2 shown, the application program packages may include application programs such as phone, email, calendar, camera, etc.

[0078] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0079] As Figure 2 shown, the application framework layer may include an image composition system, a window manager, a frame rate control system, a view system, a package manager, an input manager, etc.

[0080] The image composition system is used to control image composition and generate a vertical synchronization (Vsync) signal. The image composition system includes: a composition thread, a Vsync thread, and a buffer thread. The composition thread is used to be awakened by the Vsync signal for composition. The Vsync thread is used to request the generation of the next Vsync signal according to the Vsync signal. The buffer thread is used to store buffers, generate Vsync signal requests, and wake up the composition thread, etc. There is one or more buffer queues in the buffer thread, which are respectively used to store buffers corresponding to different applications.

[0081] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0082] The frame rate control system is used to adjust the screen refresh rate.

[0083] The view system includes visible controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0084] The package manager is used for program management within the system, such as: application installation, uninstallation, and upgrade, etc.

[0085] The input manager is used for programs that manage input devices. For example, the input system can determine input operations such as mouse click operations, keyboard input operations, and touch swipes.

[0086] The application framework layer may also include an activity manager and a resource manager.

[0087] The activity manager is used to manage the life cycles of various applications and the navigation back function. It is responsible for creating the main thread of Android and maintaining the life cycles of various applications.

[0088] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, and so on.

[0089] The Android runtime includes the core libraries and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0090] The core libraries consist of two parts: one is the functional functions that the Java language needs to call, and the other is the core libraries of Android.

[0091] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0092] The system libraries can include multiple functional modules. For example: image rendering library, image composition library, open graphics library, function library, and media library, etc.

[0093] The image rendering library (RenderEngine) is used for the rendering of two-dimensional or three-dimensional images. The image composition library (surfaceflinger) is used for the composition of two-dimensional or three-dimensional images.

[0094] In a possible implementation, the application renders the image through the image rendering library, and then the application sends the rendered image to the cache queue of the image composition system. Whenever the Vsync signal arrives, the image composition system sequentially obtains a frame of image to be composed from the cache queue, and then performs image composition through the image composition library.

[0095] The open graphics library (OpenGL) is a cross-platform and cross-language graphics programming interface. It is widely used for implementing 2D and 3D graphics rendering and is a core component of many applications, games, and web browsers. For example, OpenGL is used to transfer the data drawn by the APP to the GPU for processing.

[0096] The function library provides macros, type definitions, string manipulation functions, mathematical calculation functions, and input / output functions used in the C language, etc.

[0097] The media library supports the playback and recording of various common audio and video formats, as well as static image files, etc. The media library can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG, etc.

[0098] The hardware abstraction layer may include multiple library modules, such as the hardware composer (HWC), the graphics processing unit (GPU), the audio driver, the sensor driver, etc. The Android system can load the corresponding library modules for the device hardware, thereby achieving the purpose of the application framework layer accessing the device hardware. The device hardware may include, for example, the LCD display screen, the camera, etc. in the electronic device.

[0099] The kernel layer is the layer between the hardware and the software. The kernel layer at least includes the display driver, the camera driver, the audio driver, the sensor driver, etc.

[0100] The hardware may be an audio device, a Bluetooth device, a camera device, a sensor device, etc.

[0101] Next, in combination with the scenario of application startup or interface switching occurring in the application, the working processes of the software and hardware of the electronic device 100 will be exemplarily described.

[0102] When the touch sensor 180K in the touch panel receives a touch operation, the kernel layer processes the touch operation into a raw input event (including information such as touch coordinates, touch force, and timestamp of the touch operation). The raw input event is stored in the kernel layer. The kernel layer reports the raw input event to the input manager in the application framework layer through the input processing library. The input manager in the application framework layer parses the information of the raw input event (including: operation type, reporting point position, etc.) and determines the focused application according to the current focus, and sends the parsed information to the focused application. The focus may be the touch point in the touch operation or the click position in the mouse click operation. The focused application is the application running in the foreground of the terminal device or the application corresponding to the touch position in the touch operation. The focused application determines the control corresponding to the raw input event according to the parsed information of the raw input event (for example, the reporting point position).

[0103] Taking the touch operation as a touch swipe operation and the control corresponding to the touch swipe operation being the list control of the WeChat application as an example, the WeChat application calls the image rendering library in the system library through the view system in the application framework layer to draw and render the image. The WeChat application sends the drawn and rendered image to the cache queue of the image composition system. The image composition library in the system library composes the drawn and rendered images in the image composition system into the WeChat interface. The image composition system enables the screen (display screen) to display the corresponding interface of the WeChat application through the display driver in the kernel layer.

[0104] For the convenience of understanding, some descriptions of the concepts related to the embodiments of the present application are given as examples for reference.

[0105] 1. Frame: It refers to a single still image, which is the smallest unit in interface display. A frame can be understood as a static picture, and rapidly and continuously displaying multiple connected frames can create the illusion of object movement. Frame rate refers to the number of frames refreshed in one second, which can also be understood as the number of times the graphics processor in the terminal device refreshes the screen per second. A high frame rate can result in smoother and more realistic animations. The more frames per second, the smoother the displayed actions will be.

[0106] It should be noted that before the interface display frame, processes such as layer drawing, layer rendering, and layer composition are usually required.

[0107] 2. Layer drawing: It refers to the drawing of pictures on the display interface. The display interface can consist of one or more layers (views). Each layer can be drawn by visible controls in the view system. Each layer is composed of sub - layers, and one sub - layer corresponds to a small widget in the layer. For example, one of the sub - layers corresponds to a symbol in the picture layer.

[0108] 3. Layer rendering: It is to perform coloring operations on the drawn layers or add 3D effects, etc. For example, 3D effects can be lighting effects, shadow effects, and texture effects, etc. Another example is that layer rendering can be operations such as scaling, rounding corners, blurring effects, rotating at special angles, high - dynamic range lighting synthesis (HDR), and shadow effects on the layer.

[0109] 4. Layer composition: It is the process of combining one or more of the above - mentioned rendered layers into a display interface.

[0110] 5. Frame composition: It includes operations such as layer drawing, layer rendering, and layer composition.

[0111] In some scenarios, in order to alleviate the problem of animation jitter caused by long GPU composition time, a composition prediction mechanism is proposed to reduce the CPU composition time. However, the existing composition prediction mechanism is prone to the phenomenon of composition prediction failure and cannot effectively alleviate the problem of animation jitter caused by long GPU composition time. In view of this, the present application provides a frame composition method. When M first - layer in the first frame meet the first preset condition, directly perform composition prediction on the first frame, which can increase the probability of performing composition prediction on the frame, save the composition time of the first frame, thus avoiding the screen jitter of the electronic device and improving the user experience.

[0112] Please refer to Figure 3 , Figure 3 which is a schematic flowchart of a frame composition method provided by an embodiment of the present application. As Figure 3 shown, the working process of the software of the electronic device 100 includes but is not limited to steps S301 - S308. Figure 3The execution entities (units) therein include: applications, image rendering libraries, image composition libraries, asynchronous synthesizers, hardware synthesizers, etc. Among them, the application is, for example, the application package in the application layer above Figure 2 such as the application programs in the application layer above, for example, phone, email, calendar, or camera, etc. The image rendering library is, for example, the image rendering library shown above Figure 2 such as the image rendering library shown above Figure 2 such as the image composition library shown above Figure 2 such as the hardware synthesizer shown above. The asynchronous synthesizer is used to perform synthesis prediction on frames. For example, the electronic device 100 creates a first thread and a second thread (the first thread and the second thread execute in parallel and do not affect each other). The first thread is used to process the GPU synthesis layers that need to be processed (such as the content shown in step S305), and the second thread is used to determine the synthesis method requested by other layers in the frame based on the information fed back by the hardware synthesizer. The layers that need to be GPU-synthesized are determined in the second thread and sent to the first thread for synthesis processing (such as the content shown in step S307).

[0113] S301. The application responds to the first operation (optional).

[0114] In response to the first operation, the display screen of the electronic device switches from displaying the first user interface to displaying the second user interface. The operation type of the first operation is, for example, press, move, release, slide, etc. The first operation is, for example, to operate the electronic device through one or more of the following, including: the display screen, physical buttons, image sensors, microphones, etc. It can be understood that the present application does not limit the first operation. The solution provided by the present application is applied during the period when the display screen of the electronic device switches from the first user interface to displaying the second user interface, and the display screen at least also displays a frame of the scene. Of course, in some scenarios, the switch from the first user interface to the second user interface on the display screen will run in the background of the electronic device and is not directly presented on the display screen of the electronic device, and this scenario still falls within the protection scope of the present application.

[0115] Next, a possible implementation of the first operation is provided. Exemplarily, as Figure 4a shown in (a) above, the mobile phone responds to the user's operation on the icon 101 of the "Communication" application in the mobile phone's home screen interface (such as the user clicks on the icon 101), and the mobile phone displays as Figure 4a shown in (b) above, the interface 102. Among them, the interface 102 is the interface after the "Communication" application is launched. In some embodiments, as Figure 4a shown in (b) above, the interface 102 can be the interface of the user's communication list.

[0116] The interface shown in (a) of 4a may be referred to as the first user interface, and the interface 102 may be referred to as the second user interface. During the process of switching from the first interface to the second interface, the electronic device may also display multiple transitional pictures (referred to as frames for short). Exemplarily, when the user clicks on the icon 101, the image displayed in the interface 102 changes from small to large until the image displayed in the interface 102 fills the entire screen (or the display screen).

[0117] Exemplarily, take the example of a mobile phone continuously playing six frames of image frames during the process of switching from the first interface to the second interface. When the user clicks on the icon 101 of the "Communication" application, the mobile phone continuously plays the image frames of Frame 1, Frame 2, Frame 3, Frame 4, Frame 5, and Frame 6 as shown Figure 4b below.

[0118] Optionally, the first user interface and / or the second user interface may also be in the screen-off state. For example, the first user interface is in the screen-off state, and the application responds to a first operation (such as an unlocking operation), causing the display screen to light up and display the second user interface, as shown in (a) of 4a. Another example is that the display screen displays the first user interface, as shown in (a) of 4a, and the application responds to a first operation (such as a locking operation), causing the display screen to be in the screen-off state.

[0119] In a possible implementation, during the process of the application responding to the first operation, the electronic device sends the layer data to be drawn to the layer rendering library. Exemplarily, during the process of the application responding to the first operation, the electronic device can initiate a layer drawing task through the schedule traversals function of the ViewRoot class in the application framework layer, and then pass the data to be drawn to the GPU for processing through the Open Graphics Library in the system library. The GPU completes the drawing of the data. For example, the image rendering library in the system library performs drawing processing on the data through the GPU. The specific processing process can refer to step S302.

[0120] S302. The image rendering library renders the data of the first frame.

[0121] Among them, the first frame can be any frame during the user interface switching process. Taking the above Figure 4b as an example, the first frame can be any one of Frame 1, Frame 2, Frame 3, Frame 4, Frame 5, and Frame 6.

[0122] Exemplarily, an image rendering library can be understood as a rendering thread. The rendering thread can apply for cache space to store the texture data of the application. When the rendering thread detects that the data in the cache space is updated, the rendering thread will read the texture data of the application in the cache space and control the GPU to render the texture data of the application. Exemplarily, the updated data in the cache space maintained by the rendering thread can be referred to as the texture data of the first frame of the application. The rendering thread can read the texture data of the first frame of the application in the cache space and control the GPU to render the texture data of the first frame of the application.

[0123] It should be noted that the embodiments of the present application do not limit how the GPU renders the texture data of the application. It can be understood that the data obtained by GPU rendering exists in units of layers, so multiple layers can be obtained through GPU rendering. The layers obtained by GPU rendering include various types of data, such as scaling, rounded corners, blur effects, rotation at special angles, high dynamic range (HDR) lighting synthesis, and shadow effects.

[0124] In a possible implementation, the data obtained by rendering / drawing is filled into the cache space and passed to the image composition library for the layer composition library to perform the composition operation. It should be noted that the cache space for caching the texture data of the application and the cache space for caching the first frame after rendering can be the same cache space or different cache spaces. The embodiments of the present application do not limit this.

[0125] S303. The image composition library determines the composition method requested for the layers in the first frame.

[0126] In a possible implementation manner, the image composition library can read the first frame after rendering (the first frame includes multiple layers obtained by GPU rendering) from the above cache space, and then perform a composition operation on the multiple layers in the first frame. It can be understood that the electronic device can perform composition on the layers to be composed through the GPU and the hardware compositor. Among them, the composition method of GPU composition can handle complex composition requirements. Exemplarily, the layers that cannot be composed by the hardware compositor can be composed by the GPU. For example, when the rendering effect of the layer includes one or more of scaling, rounded corners, blur effects, rotation at special angles, high dynamic range (HDR) lighting synthesis, and shadow effects, the hardware compositor cannot compose it. In this case, the GPU can be used to compose the layer including the above rendering effects. It should be noted that with the development of technology, the hardware compositor may have more powerful functions, so the functions of the hardware compositor should not be used as a limitation of the present application.

[0127] In a possible implementation, after the image synthesis library reads the first rendered frame, it determines the attributes of each layer in the first frame. Then, based on the attributes of each layer, it determines whether the requested composition method for that layer is GPU composition.

[0128] Exemplarily, by determining whether the attributes of the layer in the first frame satisfy at least one of multiple preset attribute items, the requested composition method of the layer can be determined.

[0129] The above multiple preset attribute items include scaling. When the attributes of the layer in the first frame include scaling, the requested composition method (requestedCompositionType) of the layer is determined to be GPU composition. For the convenience of description, the layer with the requested composition method being GPU composition can be called the first layer. Through the above determination method, M first layers can be determined from the multiple layers included in the first frame, where M is a positive integer. For example, M is 1, 2, 3, or 5, etc. Among them, scaling refers to the situation where the size of the layer obtained by the image rendering library is different from the actual displayed size of the layer. If the size of the layer obtained by the image rendering library is smaller than the actual displayed size of the layer, the attribute of the layer is shrinking. If the size of the layer obtained by the image rendering library is larger than the actual displayed size of the layer, the attribute of the layer is enlarging. Please refer to Figure 4b Frame 1 and Frame 2 shown in, both Frame 1 and Frame 2 include 4 layers, namely signal icon 501, battery icon 502, search box 503, and content 504. From Figure 4b It can be known that the sizes of the signal icon 501 and the battery icon 502 in each frame are the same as the final displayed sizes, and there is no scaling effect. The search box 503 and the content 504 in Frame 1 and Frame 2 are both smaller than the final displayed sizes, and it can be understood that the attributes of the layers corresponding to the search box 503 and the content 504 include scaling.

[0130] Optionally, the above multiple preset attribute items further include rounded corners, blur effect, special angle rotation, high dynamic range (HDR) synthesis, and shadow effect, etc. Exemplarily, when the attributes of the layer in the first frame include any one or more of scaling, rounded corners, blur effect, special angle rotation, high dynamic range (HDR) synthesis, and shadow effect, the requested composition method of the layer can be determined to be GPU composition.

[0131] Optionally, it is possible that the attributes of any layer in the first frame do not satisfy the above multiple preset attribute items. In this case, how the electronic device composes the multiple layers included in the first frame is not limited in this application.

[0132] S304. The image synthesis library determines whether to perform composition prediction on the first frame.

[0133] The above content gives a simple description of synthetic prediction. Here, further introduction to synthetic prediction will be given in combination with the accompanying drawings. It can be known that synthetic prediction is a method for processing synthetic layers, which is used to reduce the time occupied by synthetic layers. For specific details, reference can be made to the description of S305 - S307 in the following text, and it will not be elaborated here for the time being.

[0134] In a possible implementation, to determine whether to perform synthetic prediction on the first frame, one or more of the following conditions can be used:

[0135] The second frame in the following text refers to the previous frame of the first frame. Combining Figure 4b , if the first frame is "Frame 2", then the second frame is "Frame 1". If the first frame is "Frame 3", then the second frame is "Frame 2".

[0136] Condition 1: Whether the layers in the first frame include layers that request GPU synthesis as the synthesis method.

[0137] For example, reference can be made to the description of S303 above to determine whether the layers in the first frame include layers that request GPU synthesis as the synthesis method.

[0138] Condition 2: Whether the layer data of the first frame and the second frame are the same.

[0139] Exemplarily, layer data refers to data such as the number of layers, layer size, layer content, etc. The layer data of the first frame or the second frame can be recorded through a hash value. In a possible implementation, it can be determined whether the layer data of the first frame and the second frame are the same by detecting the BitMap corresponding to the first frame and the second frame.

[0140] Condition 3: Whether the synthesis methods of the first frame and the second frame are the same.

[0141] It should be noted that the synthesis method of a frame is different from the synthesis method requested by a layer. The synthesis method of a frame refers to the set of synthesis methods requested by each layer in the frame. Exemplarily, when the synthesis methods of the first frame and the second frame are the same, when sorting the layers in the first frame and the second frame according to the same rules, the synthesis methods requested by the layers with the same serial number in the first frame and the second frame are the same. For example, if both the first frame and the second frame include 3 layers, and the synthesis method requested by the second layer in the first frame and the second frame is GPU synthesis, and the synthesis methods requested by the other two layers are hardware synthesizer synthesis, then it is said that the synthesis methods of the first frame and the second frame are the same.

[0142] In the above S303, the image synthesis library can determine the synthesis method of the layers in the first frame based on the attributes of the layers. However, further judgment is still needed on whether the remaining layers in the first frame can be synthesized by the hardware synthesizer. For example, the hardware synthesizer can judge whether a layer can be processed by the hardware synthesizer based on its own hardware attributes. If a layer can be processed by the hardware synthesizer, the synthesis strategy of this layer can be set to hardware processor synthesis. If a layer cannot be processed by the hardware synthesizer and the GPU needs to be used to process this layer, the synthesis strategy of this layer can be set to GPU synthesis. Among them, to judge whether a layer can be processed by the hardware synthesizer, one or more of the following conditions can be used. For example, the bandwidth of the hardware synthesizer, the number of layers input to the hardware synthesizer, the size of the layers input to the hardware synthesizer, and the format of the layers input to the hardware synthesizer, etc.

[0143] Condition 4: Whether the first frame belongs to an animation frame.

[0144] In this application, an animation frame refers to a frame that constitutes an animation. An animation includes N consecutive frames of images, where N is greater than or equal to 1. When the electronic device starts playing an animation, it includes: the electronic device starts playing the animation according to the animation attributes; the animation attributes include at least one of animation content, animation size, animation duration, or animation start position and animation end position; the animation start position is used to indicate the position of the first frame of the N frames of images on the display screen, and the animation end position is used to indicate the position of the last frame of the N frames of images on the display screen. Currently, the scenarios of playing animations on the electronic device include one or more of the following: the electronic device starts a target application, the electronic device exits a certain application, the electronic device enters the multitasking interface, the electronic device exits the multitasking interface, and switches the screen refresh rate, etc. Therefore, the animation frames include one or more of the following: the images displayed during the process of the electronic device starting a target application, the images displayed during the process of the electronic device exiting a certain application, the images displayed during the process of the electronic device entering the multitasking interface, the images displayed during the process of the electronic device starting a certain application from the multitasking interface, and the images displayed during the process of the electronic device switching the screen refresh rate.

[0145] The above Figure 4a and Figure 4b exemplarily shows the images displayed during the process of the electronic device starting a target application. Therefore, the animation frames include Figure 4b frames 1, 2, 3, 4, 5, and 6 in Figure 4a and Figure 4b . Specifically, in combination with the above

[0146] It should be noted that in the scenario of the startup animation, the startup animation can be the animation played when the user first starts the application or the animation played when the user does not start the application for the first time. Among them, the first time means that the application is neither running in the foreground nor in the background. Not the first time means that the application is running in the background. In this case, the startup animation can also be understood as the process of the application switching from the background to the foreground.

[0147] Taking the application as the "communication" application and the animation scenario as the scenario of the "communication" application exiting as an example, exemplarily, the mobile phone displays the interface 103 shown in Figure 5 (a) therein. This interface 103 is the interface after the "communication" application is started. Exemplarily, as shown in Figure 5 (a) therein, the interface 103 can be, for example, the interface of the user's communication list. Then, in response to the user's operation to exit the "communication" application, the mobile phone displays the interface 104 shown in Figure 5 (b) therein. Among them, this interface 204 can be, for example, the main screen interface of the mobile phone.

[0148] In some embodiments, the exit operation can be, for example, one of a gesture operation, a voice operation, or a touch operation. Among them, the touch operation can be, for example, a click operation, a swipe operation, etc. Taking the exit operation as a swipe operation as an example, exemplarily, still as shown in Figure 5 (b) therein, this exit operation can be, for example, the operation of the user swiping up the interface of the "communication" application.

[0149] In the scenario of the "communication" application exiting, the exit animation of the "communication" application is the process from the operation of the user swiping up the interface of the "communication" application to the mobile phone displaying the main screen interface. In some embodiments, in combination with the above embodiments, the exit animation is the process of the image displayed in the interface 102 changing from large to small. Specifically, when the user swipes up the interface of the "communication" application, the exit animation starts; the image in the interface 102 displayed on the mobile phone starts to shrink; when the image displayed in the interface 102 completely exits and the mobile phone displays the main screen interface, the exit animation ends.

[0150] Taking the animation scenario as the scenario of the mobile phone switching from the main screen interface to the multi-tasking interface as an example, exemplarily, the mobile phone displays the interface 105 shown in Figure 6 (a) therein. This interface 105 is the main screen interface of the mobile phone. Then, in response to the user's operation on the main screen interface, the mobile phone displays the interface 106 shown in Figure 6 (b) therein. This interface 106 is the multi-tasking interface of the mobile phone. Among them, the interface 106 includes the interface of the application 1 running in the background, and the application 1 is the application with the shortest running time in the background. Among them, in this animation scenario, the animation refers to the process from the user's operation on the main screen interface to the mobile phone displaying the multi-tasking interface.

[0151] It should be noted that the user's operation on the home screen interface can refer to the example of the exit operation in the above embodiment, which will not be repeated here. Taking the sliding operation as an example, the following is still the case. Figure 6 As shown in (b), the operation may be, for example, a swiping up operation of the user on the main screen interface.

[0152] Take the scenario where a mobile phone starts the "communication" application in the multitasking interface as an example. For example, the mobile phone displays the following Figure 7 The interface 107 shown in (a) is a multi-tasking interface of the mobile phone. Then, the mobile phone responds to the user's operation on the interface 107 and displays the following Figure 7 The interface 108 shown in (b) is the interface after the "communication" application is started. In this dynamic effect scene, the dynamic effect refers to the process from the user's operation on the interface 107 to the mobile phone displaying the interface 108.

[0153] It should be noted that the user's operation on the interface 107 can refer to the example of the start operation in the above embodiment, which will not be repeated here. Taking the sliding operation as an example, the following is exemplary. Figure 7 As shown in (b), the operation may be, for example, a click operation.

[0154] Next, various judgment rules for determining whether to perform synthesis prediction on the first frame provided by the present application are exemplarily shown, including one or more of the following:

[0155] Judgment rule 1: The first frame includes a layer whose requested synthesis method is GPU synthesis.

[0156] Under this judgment rule, synthesis prediction is performed on the first frame only when the first frame includes a layer whose synthesis method is GPU synthesis. This can increase the probability of performing synthesis prediction on the first frame and save the synthesis time of the first frame, thereby avoiding screen freezes on electronic devices and improving user experience.

[0157] It can be understood that synthesis prediction refers to the asynchronous execution of the GPU synthesis process (S305) and the synthesis strategy judgment process (S306), which allows the electronic device to determine the synthesis strategy in the first frame without affecting the electronic device's execution of GPU synthesis. Therefore, the computing power of the electronic device can be fully utilized for GPU synthesis, thereby reducing the synthesis time of the first frame, avoiding screen freezes, and thus improving the user experience.

[0158] Judgment rule two: the first frame includes a layer whose requested synthesis method is GPU synthesis, and the second frame includes a layer whose requested synthesis method is GPU synthesis.

[0159] It can be understood that since the changes between two frames are usually small, in the case where the previous frame includes a layer with the requested synthesis method being GPU synthesis, it is very likely that the subsequent frame also includes a layer with the requested synthesis method being GPU synthesis. Therefore, when the "Judgment Rule Two" is satisfied, synthetic prediction can be performed on the first frame, which can increase the probability of performing synthetic prediction on the first frame, save the synthesis time of the first frame, thereby avoiding frame freezing on the electronic device and enhancing the user experience.

[0160] Judgment Rule Three: The first frame includes a layer with the requested synthesis method being GPU synthesis, and the first frame belongs to an animation frame.

[0161] It can be understood that since animations need to display complex and diverse dynamic images, the animation frames that make up the animation are more complex in the synthesis process compared to the synthesis of ordinary frames, and the requested synthesis method for their layers usually includes GPU synthesis. Therefore, when the "Judgment Rule Three" is satisfied, synthetic prediction can be performed on the first frame, which can increase the probability of performing synthetic prediction on the first frame, save the synthesis time of the first frame, thereby avoiding frame freezing on the electronic device and enhancing the user experience.

[0162] Optionally, based on Judgment Rule Three, it is also possible to determine whether the second frame includes a layer with the requested synthesis method being GPU synthesis.

[0163] Exemplarily, based on Judgment Rule Three, if the second frame does not include a layer with the requested synthesis method being GPU synthesis, synthetic prediction is performed on the first frame. It can be understood that even if the previous frame (the second frame) of the first frame does not include a layer with the requested synthesis method being GPU synthesis, since the first frame is an animation frame, the first frame may also include multiple layers with the requested synthesis method being GPU synthesis. Therefore, synthetic prediction can be performed on the first frame in this case, which can increase the probability of performing synthetic prediction on the first frame, save the synthesis time of the first frame, thereby avoiding frame freezing on the electronic device and enhancing the user experience.

[0164] Exemplarily, based on Judgment Rule Three, if the second frame includes a layer with the requested synthesis method being GPU synthesis, synthetic prediction is performed on the first frame. It can be understood that in this example, when performing synthetic prediction on the first frame, it is not only necessary for the first frame to be an animation frame, but also necessary to ensure that the previous frame of the first frame includes a layer with the requested synthesis method being GPU synthesis, which can avoid phenomena such as additional thread overhead and increased CPU load caused by performing synthetic prediction on the first frame due to unreasonable judgment.

[0165] Judgment Rule Four: The first frame includes a layer with the requested synthesis method being GPU synthesis, the layer data of the first frame is the same as the layer data of the second frame, and the second frame includes a layer with the requested synthesis method being GPU synthesis.

[0166] It can be understood that the layer data of the first frame is the same as that of the second frame, indicating that the first frame is the same as the second frame. Combining with the layer including the requested GPU composition method in the second frame, it indicates that the first frame probably also includes the layer with the requested GPU composition method. Therefore, when the "judgment rule four" is satisfied, the composition prediction can be performed on the first frame, which can increase the probability of performing the composition prediction on the first frame, save the composition time of the first frame, thus avoiding the screen freeze of the electronic device and improving the user experience.

[0167] Judgment rule five: The first frame includes a layer with the requested GPU composition method, and the layer data of the first frame is different from that of the second frame, and the second frame includes a layer with the requested GPU composition method.

[0168] It can be understood that the layer data of the first frame is different from that of the second frame, indicating that the second frame has changed compared with the first frame. Combining with the layer including the requested GPU composition method in the second frame. It can be understood that when the previous frame of the first frame includes a layer with the requested GPU composition method, even if the previous frame of the first frame is different from the first frame, the composition prediction is still performed on the first frame. This implementation can increase the probability of entering the composition prediction, thereby reducing the composition time of the first frame, avoiding screen freeze, and further improving the user experience.

[0169] Judgment rule six: The first frame includes a layer with the requested GPU composition method, and the composition methods of the first frame and the second frame are the same, and the second frame includes a layer with the requested GPU composition method.

[0170] Exemplarily, during the process of composing the M first layers of the first frame, the attributes of the first layer satisfy at least one of the above multiple preset attribute items, and M is a positive integer. During the process of composing the N second layers of the second frame, the attributes of the second layer satisfy at least one of the above multiple preset attribute items, N is a positive integer, and M is equal to N. For example, M = N = 1, or M = N = 2. In this example, the composition methods of the first frame and the second frame are the same. Combining with the layer including the requested GPU composition method in the second frame, it indicates that the first frame probably also includes the layer with the requested GPU composition method. Therefore, when the "judgment rule six" is satisfied, the composition prediction can be performed on the first frame, which can increase the probability of performing the composition prediction on the first frame, save the composition time of the first frame, thus avoiding the screen freeze of the electronic device and improving the user experience.

[0171] Judgment rule seven: The first frame includes a layer with the requested GPU composition method, and the composition methods of the first frame and the second frame are different, and the second frame includes a layer with the requested GPU composition method.

[0172] Exemplarily, during the process of synthesizing the M first layers of the first frame, the attributes of the first layers satisfy at least one of the above-mentioned multiple preset attribute items, where M is a positive integer. During the process of synthesizing the N second layers of the second frame, the attributes of the second layers satisfy at least one of the above-mentioned multiple preset attribute items, where N is a positive integer and M is equal to N. For example, M = N = 1, or M = N = 2. In this example, the synthesis methods of the first frame and the second frame are different. Considering that the second frame includes a layer with a requested synthesis method of GPU synthesis, it can be understood that in the case where the previous frame of the first frame includes a layer with a requested synthesis method of GPU synthesis, even if the number of second layers in the second frame is not equal to the number of first layers in the first frame, synthesis prediction is still performed on the first frame. This implementation can increase the probability of entering synthesis prediction, thereby reducing the synthesis time of the first frame, avoiding frame freezes, and improving the user experience.

[0173] It can be understood that this application proposes multiple judgment rules for determining whether to perform synthesis prediction on the first frame. When the first frame meets any of the above judgment rules, synthesis prediction can be performed on the first frame. Therefore, the probability of performing synthesis prediction on the first frame can be increased, thereby reducing the synthesis time of the first frame, avoiding frame freezes, and improving the user experience.

[0174] It should be noted that in specific implementations, one or more of the above judgment rules can be selected to determine whether to perform synthesis prediction on the first frame. The above seven judgment rules can also be combined with each other, and this application does not limit this.

[0175] Next, it is assumed that the first frame meets one or more of the above judgment rules and synthesis prediction is performed on the first frame for subsequent introduction. The execution process of synthesis prediction includes Figure 2 S305, S306, and S307 shown in, which will be introduced below in sequence.

[0176] S305: The image rendering library performs GPU synthesis on the M first layers to generate a first target layer.

[0177] S306: The asynchronous synthesizer determines the requested synthesis methods of the layers in the first frame other than the M first layers.

[0178] It should be noted that S305 and S306 are executed asynchronously, that is, the content shown in S305 and S306 is run through two threads. Specifically, the start time and end time of S305 and S306 are not limited in this application. In one possible implementation, there is a moment when S305 and S306 run simultaneously in the electronic device. For example, in the case of determining to perform synthetic prediction on the first frame, the electronic device uses the first process to run the content of S305 and the second process to run the content of S306, that is, the synthetic process of the M first layers by the GPU and the determination of the synthetic method of the layer requests in the first frame do not affect each other, so that the computing power of the GPU can be fully utilized to synthesize the M first layers, thereby reducing the synthetic time of the first frame, avoiding frame freezes, and improving the user experience.

[0179] The above image rendering library performs GPU synthesis on M first layers to generate a first target layer. Among them, the first target layer is 1 layer obtained by the GPU synthesizing M first layers. For example, when M is equal to 1, the GPU synthesizes 1 first layer to obtain 1 layer. Another example is that when M is equal to 3, the GPU synthesizes 3 first layers to obtain 1 layer. By synthesizing M first layers through the GPU, the first target layer obtained by the GPU synthesis can generate the final image through the hardware synthesizer and be displayed on the display screen of the electronic device.

[0180] In one possible implementation, the above asynchronous synthesizer can detect whether the first frame meets the conditions for asynchronous synthesis.

[0181] In another possible implementation, the above asynchronous synthesizer can also obtain the synthesis method fed back by the hardware synthesizer and determine the synthesis method of the layer requests in the first frame other than the above M first layers. Exemplarily, based on the synthesis method fed back by the hardware synthesizer, the asynchronous synthesizer determines that the synthesis method of L layer requests in the first frame is the layer synthesized by the GPU (abbreviated as the third layer), where L is a positive integer, such as L being 1, 2, or 3, etc. Then, the asynchronous synthesizer can pass the L third layers in the first frame to the layer rendering library, and let the layer rendering library perform GPU synthesis on the L third layers and generate a second target layer (S307).

[0182] Optionally, L can also be 0, that is, the asynchronous synthesizer determines 0 third layers based on the synthesis method fed back by the hardware synthesizer. In this case, S307 does not need to be executed.

[0183] S307. The image rendering library performs GPU synthesis on L third layers to generate a second target layer. For details, reference can be made to the relevant description of S305, which will not be elaborated here.

[0184] It should be noted that the thread used by the image rendering library for GPU composition of L third layers can be the same as the thread used by the image rendering library for GPU composition of M first layers. That is, the thread for instructing GPU to compose layers sequentially composes M first layers and L third layers. In addition, when L is 0, that is, the second target layer will not be generated. Therefore, when L is 0, the second target layer can also be understood as non-existent or an empty layer.

[0185] When the image rendering library composes and obtains the first target layer and the second target layer, the first target layer and the second target layer can be passed to the image composition library, and the image composition library then packages and passes the first target layer, the second target layer, and the layer whose requested composition method is the hardware synthesizer to the hardware synthesizer.

[0186] S308. The hardware synthesizer performs final composition on the first frame and sends it to the display unit.

[0187] Since the layers that cannot be processed by the hardware synthesizer have been processed into layers that can be processed by the hardware synthesizer through GPU composition. Therefore, the hardware synthesizer can compose multiple layers sent by the image composition library. Exemplarily, the hardware synthesizer can call the display driver to compose multiple layers sent by the image composition library. Optionally, the display driver can also send the final composed image to the display screen for display.

[0188] To further demonstrate the beneficial effects of performing synthesis prediction, please refer to Figure 8 and Figure 9 , where Figure 8 is used to show the situation without performing synthesis prediction, Figure 9 is used to show the situation of performing synthesis prediction. It should be noted that Figure 8 and Figure 9 Each row in can be represented as a process, and the execution order of the process is to first judge from left to right and then from top to bottom. Taking Figure 9 as an example, Figure 9 the execution order of the functions in can be prepareFrameAsync, choosecompositionstrategy, REThreaded::drawLayers, etc.

[0189] Such as Figure 8As shown, the trace without performing synthesis prediction includes that the image synthesizer first determines the synthesis method requested for the layers in the frame, and then calls the GPU synthesis through the image rendering library to perform GPU synthesis on the layers whose requested synthesis method is GPU synthesis. Then, the final layer synthesis is performed by the hardware synthesizer and sent to the display screen for display.

[0190] As Figure 9 shown, the trace of performing synthesis prediction includes that the asynchronous synthesizer and the image rendering library perform asynchronously. That is, when the asynchronous synthesizer determines the synthesis method requested for the layers in the frame, the image rendering library also performs the operation of layer GPU synthesis. As a result, the image rendering library does not need to wait for the determination of the synthesis method requested for the layers in the frame to be completed, and can synchronize the GPU synthesis of the layers with the determination of the synthesis method requested for the layers in the frame, thereby saving the synthesis time of the layers, reducing the occurrence of frame freezes, and improving the user experience.

[0191] Please refer to Figure 10 , Figure 10 which is a schematic diagram of the working process of a user interface switching process software provided in an embodiment of the present application, and is used to show the working process of the user interface switching process software in the case of using the frame synthesis method provided in the present application. Combining the above Figure 9 shown trace of performing synthesis prediction, it can be seen that Figure 10 in the user interface switching process shown in, each frame is determined to perform synthesis prediction. It can be seen that the frame synthesis method provided in the present application can greatly increase the probability of performing synthesis prediction on the determined frames, save the synthesis time of the layers, reduce the occurrence of frame freezes, and improve the user experience.

[0192] In summary, the present application uses the synthesis prediction technology. By designing various rules for performing synthesis prediction on frames, the probability of performing synthesis prediction on frames can be increased, the synthesis time of the layers can be saved, the occurrence of frame freezes can be reduced, and the user experience can be improved. In addition, the present application also uses the rendering method of scaling as a condition for determining that the synthesis method requested for the layer is GPU synthesis, which can increase the probability of determining that the synthesis method requested for the layer is GPU synthesis. Combining the premise that the synthesis prediction of the frame is based on the fact that the frame includes a layer whose requested synthesis method is GPU synthesis, the probability of performing synthesis prediction on the frame is improved.

[0193] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the various embodiments of the present application.

[0194] As used in the above embodiments, depending on the context, the term "when..." can be interpreted to mean "if...", or "after...", or "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if detecting (the stated condition or event)" can be interpreted to mean "if determining...", or "in response to determining...", or "when detecting (the stated condition or event)", or "in response to detecting (the stated condition or event)".

[0195] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state drive), etc.

[0196] Those of ordinary skill in the art can understand that all or part of the processes in the above embodiments of the method can be completed by instructing relevant hardware by a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The foregoing storage media include: various media that can store program codes such as ROM or random access memory RAM, magnetic disks, or optical discs.

Claims

1. A frame synthesis method, which is applied to an electronic device, and is characterized in that, The method includes: The display screen of the electronic device displays a first user interface; During the electronic device switches from the first user interface to display a second user interface, in the process of synthesizing M first layers of the first frame, if the attribute of the first layer satisfies at least one of a plurality of preset attribute items, M is a positive integer; the electronic device performs synthesis prediction on the first frame; Wherein, the plurality of preset attribute items include scaling.

2. The method according to claim 1, wherein The plurality of preset attribute items further include: Round corners, blur effect, special angle rotation, high dynamic range (HDR) synthesis, and shadow effect.

3. The method according to claim 1 or 2, characterized in that, The electronic device performing synthesis prediction on the first frame includes: In the case that the second frame includes a layer with a requested synthesis method of GPU synthesis, the electronic device performs synthesis prediction on the first frame; the second frame is the previous frame of the first frame.

4. The method according to claim 1 or 2, characterized in that The electronic device performing synthesis prediction on the first frame includes: In the case that the first frame belongs to an animation frame, the electronic device performs synthesis prediction on the first frame.

5. The method according to claim 4, characterized in that, The animation frame includes one or more of the following: The image displayed during the process of the electronic device starting a target application; Or, The image displayed during the process of the electronic device exiting the target application; Or, The image displayed during the process of the electronic device entering the multitasking interface; Or, The image displayed during the process of the electronic device entering the target application from the multitasking interface.

6. The method according to claim 4 or 5, characterized in that, During the electronic device switches from the first user interface to display a second user interface, the animation frame is displayed on the display screen.

7. The method according to any one of claims 4-6, characterized in that The electronic device performing synthesis prediction on the first frame further includes: In the case that the second frame does not include a layer with a requested synthesis method of GPU synthesis, the electronic device performs synthesis prediction on the first frame; the second frame is the previous frame of the first frame.

8. The method according to any one of claims 1-7, characterized in that, The layer data of the first frame is the same as the layer data of the second frame; or, the layer data of the first frame is different from the layer data of the second frame; the second frame is the previous frame of the first frame.

9. The method according to any one of claims 1-8, characterized in that, The synthesis method of the first frame is the same as the synthesis method of the second frame; or, the synthesis method of the first frame is different from the synthesis method of the second frame; the second frame is the previous frame of the first frame.

10. An electronic device, characterized in that, The electronic device includes: a display screen, a memory, and a processor coupled to the memory; the display screen is used to display an interface, the memory stores a computer program, and when the processor executes the computer program, the electronic device implements the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, comprising computer instructions, characterized in that, When the computer instructions run on the electronic device, the electronic device executes the method according to any one of claims 1 to 9.

Images