Apparatus and method for managing frame loss to enhance user equipment experience

Abstract

A device and method for predicting and preventing frame loss to achieve a smooth user interface (UI) experience on an electronic device are provided. The method includes determining, by the electronic device, one or more display frame parameters for displaying each frame, determining, by the electronic device, one or more device operating parameters indicating a system state of one or more resources in the electronic device, in one embodiment, the method includes estimating, by an electronic device, a readiness time for rendering each frame on the electronic device using one or more display frame parameters and one or more device operating parameters, and adjusting, by the electronic device, one or more resources in the electronic device to reduce the readiness time in response to the estimated readiness time indicating that a frame loss occurs.

Description

Technical Field

[0001] This disclosure relates to frame drop management in electronic devices. More specifically, this disclosure relates to predicting and preventing frame drop to achieve a smooth user interface (UI) experience on electronic devices. Background Technology

[0002] Electronic devices supporting displays with higher refresh rates (HRR) are actively expanding, such as displays ranging from 90 frames per second (FPS) to 120 FPS. Due to the increased refresh rate, multiple applications and devices have less time to prepare and render new frames, resulting in jitter or dropped frames. For example, most Android apps struggle to process input, animation, and rendered content within the 8.3ms vertical sync (Vsync) interval of a 120Hz display. A common approach to addressing dropped frames is to allocate high-performance central processing unit (CPU) cores running at maximum frequency for all user interface (UI) and rendering tasks across all frames. However, this leads to battery drain and increased device temperature.

[0003] Figure 1 The scenario of frame loss based on relevant technologies is described.

[0004] For example, based on the display refresh rate, frames must utilize Vsync pulses to complete execution. For 120Hz, the Vsync pulse duration is 8.33ms. If the frame duration exceeds the Vsync pulse duration (threshold), it is considered a dropped frame causing frame skipping / stuttering. The application frame timeline includes the time spent by several modules, such as input processing time, animation, layout and measurement, inflation, drawing, and GPU completion time, etc. Figure 1 As shown.

[0005] For example, user interaction parameters such as the frequency of touch events can affect frame drops. When touch events are regular (such as slow scrolling) and the number of touch inputs is in a regular pattern, no frame drops will be observed. When the number of touch events is irregular and fast scrolling is applied, the frame drop count can increase.

[0006] Existing methods predict video frame latency after frame execution. Furthermore, Vsync adjustment is implemented based on the video frame rate. However, existing methods have not yet been applied to improve application frame timing.

[0007] Therefore, there is a need in the art for solutions that overcome the aforementioned drawbacks.

[0008] The above information is presented as background information only to aid in understanding this disclosure. No determination or assertion is made regarding whether any of the above content can be used as prior art with respect to this disclosure. Summary of the Invention

[0009] Solution to the problem

[0010] The aspects of this disclosure will at least address the aforementioned problems and / or disadvantages, and provide at least the following advantages. Therefore, one aspect of this disclosure provides apparatus and methods for predicting and preventing one or more dropped frames to enhance the user interface (UI) experience on an electronic device.

[0011] Another aspect of this disclosure provides an apparatus and method for predicting the duration of one or more frames to determine continuous frame drops and implementing proactive decisions (e.g., boosting / scheduling) to reduce frame drops.

[0012] Another aspect of this disclosure provides an apparatus and method for determining one or more display frame parameters (such as input processing duration, animation duration, drawing duration, etc.) for displaying each frame on a device.

[0013] Another aspect of this disclosure provides an apparatus and method for determining one or more operating parameters that indicate the state of resources in the apparatus, such as central processing unit (CPU) load, graphics processing unit (GPU) state, available memory, number of running threads on the CPU, temperature, etc.

[0014] Another aspect of this disclosure provides an apparatus and method for estimating a preparation time for rendering each frame on the apparatus using display frame parameters and operating parameters, and for adjusting resources in the apparatus to reduce the preparation time if the estimated preparation time indicates that frame dropping has occurred.

[0015] Other aspects will be set forth in part in the description which follows, and in part will be apparent from the description, or may be learned by practice of the embodiments presented.

[0016] According to one aspect of this disclosure, a method for managing frame drops in an electronic device is provided. The method includes the electronic device determining one or more display frame parameters for displaying each frame, the electronic device determining one or more device operation parameters indicating the system state of one or more resources in the electronic device, the electronic device estimating a preparation time for rendering each frame on the electronic device using the display frame parameters and the device operation parameters, and, in response to indicating the occurrence of frame drops based on the estimated preparation time, the electronic device adjusting resources in the electronic device to reduce the preparation time.

[0017] According to another aspect of this disclosure, an electronic device is provided. The electronic device includes a memory storing one or more computer programs, and one or more processors communicatively coupled to the memory, wherein the one or more computer programs include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to manage frame dropping. The computer-executable instructions further cause the electronic device to determine one or more display frame parameters for displaying each frame, determine one or more device operating parameters indicating the system state of one or more resources in the electronic device, estimate a preparation time for rendering each frame on the electronic device using the display frame parameters and the device operating parameters, and adjust resources in the electronic device to reduce the preparation time in response to the estimated preparation time indicating the occurrence of frame dropping.

[0018] According to another aspect of this disclosure, one or more non-transitory computer-readable storage media are provided. The one or more non-transitory computer-readable storage media are provided storing one or more computer programs including computer-executable instructions, which, when executed individually or jointly by one or more processors of an electronic device, cause the electronic device to perform operations. These operations include the electronic device determining one or more display frame parameters for displaying each frame, the electronic device determining one or more device operating parameters indicating the system state of one or more resources in the electronic device, the electronic device estimating a preparation time for rendering each frame on the electronic device using the one or more display frame parameters and the one or more device operating parameters, and, in response to an indication of frame dropping caused by the estimated preparation time, the electronic device adjusting one or more resources in the electronic device to reduce the preparation time.

[0019] Other aspects, advantages, and distinctive features of this disclosure will become apparent to those skilled in the art from the following detailed description of various embodiments of the disclosure taken in conjunction with the accompanying drawings. Attached Figure Description

[0020] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0021] Figure 1 The frame loss scenario based on the relevant technology is described;

[0022] Figure 2 A block diagram representation of an electronic device for managing frame dropping according to an embodiment of the present disclosure is shown;

[0023] Figure 3A Examples of frames and frame levels according to embodiments of this disclosure;

[0024] Figure 3BExamples of frames and frame levels according to embodiments of this disclosure;

[0025] Figure 4 A module of an intelligent data collector according to an embodiment of the present disclosure is shown;

[0026] Figure 5 A refresh rate enhancement architecture for managing frame dropping is illustrated according to an embodiment of the present disclosure;

[0027] Figure 6 A method for managing frame dropping in an electronic device according to embodiments of the present disclosure is illustrated;

[0028] Figure 7 A process for managing frame dropping in an electronic device according to embodiments of the present disclosure is illustrated;

[0029] Figure 8 A process for managing frame drops in an electronic device during a scrolling scene, according to an embodiment of the present disclosure, is illustrated.

[0030] Figure 9A Offline modeling of a deep learning (DL) model using one or more display frame parameters and one or more device operation parameters according to embodiments of the present disclosure is illustrated.

[0031] Figure 9B On-device inference using a trained DL method for frame drop prediction according to embodiments of the present disclosure is illustrated;

[0032] Figure 10 The triggering of the frame loss prediction module and the policy decision module according to embodiments of the present disclosure is illustrated;

[0033] Figure 11A Different use cases for detecting and preventing frame drops in several scenarios are illustrated according to embodiments of the present disclosure;

[0034] Figure 11B Different use cases for detecting and preventing frame drops in several scenarios are illustrated according to embodiments of the present disclosure;

[0035] Figure 11C Different use cases for detecting and preventing frame drops in several scenarios are illustrated according to embodiments of the present disclosure; and

[0036] Figure 12 An example implementation of the proposed method for managing frame dropping according to embodiments of this disclosure is shown.

[0037] Throughout the accompanying drawings, the same reference numerals are used to denote the same elements. Detailed Implementation

[0038] The following description with reference to the accompanying drawings is intended to aid in a full understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. It includes various specific details to aid understanding, but these are considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. Additionally, for clarity and brevity, descriptions of well-known functions and structures may be omitted.

[0039] The terms and words used in the following description and claims are not limited to their literal meaning, but are used by the inventors only to enable a clear and consistent understanding of this disclosure. Therefore, those skilled in the art should understand that the following description of various embodiments of this disclosure is for illustrative purposes only and is not intended to limit the disclosure as defined by the appended claims and their equivalents.

[0040] It should be understood that, unless the context clearly specifies otherwise, the singular forms “a,” “an,” and “the” include plural indicators. Thus, for example, a reference to “component surface” includes a reference to one or more such surfaces.

[0041] The embodiments herein provide apparatus and methods for predicting and preventing one or more dropped frames to enhance the user interface (UI) experience on an electronic device. Reference is now made to the accompanying drawings for more specific details. Figure 2 , Figure 3A , Figure 3B , Figures 4 to 8 , Figure 9A , Figure 9B , Figure 10 , Figures 11A to 11C and Figure 12 Similar reference numerals are consistently used throughout the figures to denote corresponding features, illustrating embodiments.

[0042] It should be understood that the boxes in each flowchart and the combination of flowcharts can be executed by one or more computer programs including instructions. The entirety of one or more computer programs can be stored in a single memory device, or one or more computer programs can be divided into different parts stored in multiple different memory devices.

[0043] Any function or operation described herein may be processed by a processor or a combination of processors. A processor or combination of processors is a circuit that performs processing and includes circuits such as: application processors (APs, such as central processing units (CPUs)), communication processors (CPs, such as modems), graphics processing units (GPUs), neural processing units (NPUs) (such as artificial intelligence (AI) chips), Wi-Fi chips, Bluetooth® chips, global positioning system (GPS) chips, near field communication (NFC) chips, connectivity chips, sensor controllers, touch controllers, fingerprint sensor controllers, display driver integrated circuits (ICs), audio codec chips, universal serial bus (USB) controllers, camera controllers, image processing ICs, microprocessor units (MPUs), system-on-a-chip (SoCs), integrated circuits (ICs), etc.

[0044] Figure 2 A block diagram representation of an electronic device 200 for managing frame loss according to an embodiment of the present disclosure is shown.

[0045] Frame drops occur on the display of electronic device 200 when one or more applications fail to render new frames within the vertical synchronization (Vsync) interval of the display due to an increase in refresh rate.

[0046] Figure 3A and Figure 3B Examples of frames and frame levels representing various embodiments of this disclosure.

[0047] A frame represents a display frame on a display that has one or more visual components. For example, a display frame is a combination of multiple application view data rendered onto a physical display, including layouts, desktop applets, resources (images, text), etc. Electronic device 200 includes a processor 202, a communication module 204, and a memory module 206. Electronic device 200 can be, but is not limited to, a smartphone, smartwatch, tablet computer, desktop computer, laptop computer, personal digital assistant, wearable device, etc.

[0048] In the embodiments described herein, processor 202 can evaluate past frame data across multiple scenes and predict the duration of the next frame or consecutive frames. Processor 202 can determine the occurrence of frame drop by predicting the frame duration. Furthermore, if frame drop is determined, processor 202 can make proactive decisions, such as acceleration / scheduling to reduce frame drop.

[0049] The processor 202 also includes an intelligent data collector 208, a frame loss prediction module 210, and a policy decision module 212.

[0050] In embodiments herein, the intelligent data collector 208 may determine one or more display frame parameters for displaying each frame. Display frame parameters may be, but are not limited to, input processing duration, animation duration, rendering duration, tree structure, buffer swapping, enqueue and dequeue duration, layout measurement duration, synchronization duration, swap buffer duration, graphics processing unit (GPU) execution time, and any other frame latency-related parameters. The intelligent data collector 208 may determine one or more device operation parameters indicating the system state of one or more resources in the electronic device 200. Resources may be, but are not limited to, a central processing unit (CPU), scheduler, governor, etc. Device operation parameters may be, but are not limited to, CPU load, GPU state, available memory, number of threads running on the CPU, temperature during the rendering of the current frame, etc. The intelligent data collector 208 may aggregate the determined display frame parameters and device operation parameters. The intelligent data collector 208 further includes a scene detector 402 and a data aggregator 404, such as... Figure 4 As shown in the image.

[0051] Figure 4 A module of an intelligent data collector according to an embodiment of the present disclosure is shown.

[0052] In the embodiments described herein, scene detector 402 can detect scenes with the potential for frame drops based on one or more parameters from at least one touch input received from a user. Parameters from the touch input can be, but are not limited to, frequency, speed, input pressure level, at least one multi-handed gesture, etc. For example, the frequency of touch input can be the number of input events over time (such as aggressive scrolling, aggressive user interaction, etc.). For example, the speed of touch input can be the increment of the x, y coordinates input by the user. Detected scenes can be, but are not limited to, aggressive scrolling, gaming, playing high-resolution content, etc. Scene detector 402 can trigger frame drop management based on the detected scenes.

[0053] In the embodiments described herein, data aggregator 404 can collect information related to the detected scene from scene detector 402. Data aggregator 404 can aggregate display frame parameters, device operation parameters, and window compositor parameters. For example, window compositor parameters may be, but are not limited to, blur, number of layers, etc.

[0054] In the embodiments described herein, the scene detector 402 is responsible for executing the data aggregator 404 during high frame-dropping scenes such as scrolling, application usage, etc.

[0055] In the embodiments described herein, the frame drop prediction module 210 receives information related to the detected scene and aggregated data from the data aggregator 404. The frame drop prediction module 210 can use the aggregated data of display frame parameters, device operation parameters, and window compositor parameters, along with the information related to the detected scene, to estimate the preparation time for rendering each frame on the electronic device 200. A frame duration prediction (FDP) method is used to estimate the preparation time for rendering each frame. The FDP method is a machine learning (ML) method. The FDP method is trained to learn the correlation between display frame parameters, device operation parameters, and the preparation time for rendering each frame. The trained FDP method is configured to predict the duration of consecutive frames based on the current frame duration. The trained FDP method is also configured to determine frame drop if the predicted duration of consecutive frames exceeds a predetermined threshold.

[0056] In the embodiments described herein, if frame drops occur during the estimated preparation time indication, the policy decision module 212 can adjust resources in the electronic device 200 to reduce the preparation time. Resources may include, but are not limited to, CPU, scheduler, speed controller, etc. Resources can be adjusted within a predetermined duration. The window duration can be determined based on frame counts to adjust resources. Methods for adjusting resources may include, but are not limited to, CPU acceleration, scheduling, and task management.

[0057] In the embodiments described herein, processor 202 may process and execute data from multiple modules of electronic device 200, respectively. Processor 202 may be configured to execute instructions stored in memory module 206. Processor 202 may include one or more of a microprocessor, circuitry, and other hardware configured for processing. Processor 202 may be at least one of a single processor, multiple processors, multiple homogeneous or heterogeneous cores, multiple central processing units (CPUs) of different types, a microcontroller, special media, and other accelerators. Processor 202 may be an application processor (AP), a graphics-only unit (such as a graphics processing unit (GPU), a visual processing unit (VPU)), and / or an artificial intelligence (AI) dedicated processor (such as a neural processing unit (NPU)).

[0058] In the embodiments described herein, multiple modules of the processor 202 of the electronic device 200 can communicate via the communication module 204. The communication module 204 can be in the form of a wired network or a wireless communication network module. The wireless communication network can include, but is not limited to, Global Positioning System (GPS), Global System for Mobile Communications (GSM), Wi-Fi, Bluetooth Low Energy, Near Field Communication (NFC), etc. Depending on the usage environment, the wireless communication can also include one or more of Bluetooth, ZigBee, short-range wireless communication (such as Ultra Wideband (UWB)) and medium-range wireless communication (such as Wi-Fi) or long-range wireless communication (such as third-generation (3G) / fourth-generation (4G) / fifth-generation (5G) / sixth-generation (6G) and non-3GPP technologies or Global Microwave Access Interoperability (WiMAX)).

[0059] In embodiments described herein, memory module 206 may include one or more volatile and non-volatile memory components capable of storing data and instructions for a module of electronic device 200 to be executed. Examples of memory module 206 may be, but are not limited to, NAND, embedded multimedia card (eMMC), secure digital card (SD), universal serial bus (USB), serial advanced technology accessory (SATA), solid-state drive (SSD), etc. Memory module 206 may also include one or more computer-readable storage media. Examples of non-volatile storage elements may include magnetic hard disks, optical disks, floppy disks, flash memory, or electrically programmable memory (EPROM) or electrically erasable programmable memory (EEPROM). Additionally, in some examples, memory module 206 may be considered a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or propagating signal. However, the term "non-transitory" should not be construed as meaning that memory module 206 is non-removable. In some instances, non-transitory storage media may store data that may change over time (e.g., in random access memory (RAM) or cache).

[0060] Figure 2 Example modules of electronic device 200 are shown, but it should be understood that other embodiments are not limited thereto. In other embodiments, electronic device 200 may include fewer or more modules. Furthermore, the labels or names of modules are for illustrative purposes only and do not limit the scope of this disclosure. One or more modules may be combined together to perform the same or substantially similar functions in electronic device 200.

[0061] Figure 5 A refresh rate enhancement architecture for managing frame dropping is illustrated according to an embodiment of the present disclosure.

[0062] refer to Figure 5The intelligent data collector 208 is responsible for data aggregation. The data aggregator 404 collects information related to the user interaction scene from the scene detector 402 and aggregates display frame parameters, device operation parameters, and window compositor parameters. The frame drop prediction module 210 retrieves data from the intelligent data collector 208, preprocesses the data, and transmits the processed data to the FDP model. The FDP model predicts whether the next frame will be dropped, such as a dropped frame (FD) or a non-FD, and sends the FD event to the policy decision module 212. The policy decision module 212 is responsible for scheduling and allocating application threads to CPU cores and managing CPU frequency to render the next frame within the Vsync interval. The policy decision module 212 includes a performance manager 502, a scheduler 504, and a speed regulator 506. The performance manager 502 operates the scheduler 504 and the speed regulator 506 based on the FD event to adjust the small CPU, mid CPU, and perf CPU.

[0063] Figure 6 A method 600 for managing frame dropping in an electronic device 200 according to an embodiment of the present disclosure is shown.

[0064] Method 600 includes determining one or more display frame parameters by electronic device 200 for displaying each frame, as shown in operation S602. Method 600 includes determining one or more device operation parameters by electronic device 200 indicating the system state of one or more resources in electronic device 200, as shown in operation S604. Method 600 includes estimating a preparation time for rendering each frame on electronic device 200 using the display frame parameters and the device operation parameters, as shown in operation S606. Method 600 includes adjusting the resources in electronic device 200 to reduce the preparation time if the estimated preparation time indicates that frame dropping has occurred, as shown in operation S608.

[0065] The various actions in method 600 can be performed in the order they are presented, in different orders, or simultaneously. Furthermore, in some embodiments, actions can be omitted. Figure 6 Some of the actions listed in the document.

[0066] Figure 7 A process 700 for managing frame dropping in an electronic device 200 according to an embodiment of the present disclosure is shown.

[0067] As shown in operation S702, input related to one or more touch events is received from the user for scene detection. As shown in operation S704, at least one scene with the possibility of frame dropping is detected based on one or more parameters from at least one touch input received from the user. The scene is detected by determining the device context based on the frequency of the touch input. Subsequently, the system state of one or more resources is verified from the device context, as shown in operation S706. The system state indicates a scene with the possibility of frame dropping. Electronic device 200 initiates frame dropping management based on the system state. For example, if the possibility of frame dropping exists, electronic device 200 initiates frame dropping management.

[0068] Subsequently, as shown in operation S708, one or more display frame parameters and one or more device operation parameters are aggregated by data aggregator 404. The display frame parameters and device operation parameters are determined and compiled into a preprocessed data format for recording in database 710. Display frame parameters such as frame ID and frame duration can be stored in database 710. Frame drop prediction module 210 is configured with a frame duration prediction (FDP) method. The FDP method is trained using training data (such as data recorded in database 710) to determine the duration of one or more frames based on the frequency of touch input. Later, as shown in operation S712, the trained FDP method uses real-time inference data to predict the duration of consecutive frames based on the current frame duration. Frame drop prediction module 210 uses the FDP method to estimate the preparation time for rendering each frame.

[0069] Subsequently, if the predicted duration of consecutive frames exceeds a predetermined threshold, frame dropping is determined. As shown in operation S714, the frame dropping condition is verified. Later, the policy decision module 212 is invoked to adjust resources in the electronic device 200 to reduce the preparation time for rendering each frame on the electronic device 200, thereby avoiding impending frame dropping, as shown in operation S716. For example, CPU acceleration and scheduling are performed to reduce preparation time by determining the window duration or acceleration duration based on the frame count. The policy decision module 212 utilizes a policy window model to determine the window duration or acceleration duration.

[0070] The various actions in method 700 can be performed in the order they are presented, in different orders, or simultaneously. Furthermore, in some embodiments, actions can be omitted. Figure 7 Some of the actions listed in the document.

[0071] Figure 8 A process 800 for managing frame drops in an electronic device 200 during a scrolling scene, according to an embodiment of the present disclosure, is shown.

[0072] As shown in operation S802, input related to scrolling (such as scrolling up or down) is received from the user for scene detection. As shown in operation S804, the scene detector 402 uses one or more parameters of the scrolling received from the user to detect use cases where scrolling has the potential for frame dropping. The scene is detected by determining the device context based on the frequency of scrolling. Later, the system state of one or more resources is verified from the device context, as shown in operation S806. The system state indicates whether scrolling has the potential for frame dropping. If the potential for frame dropping exists, the electronic device 200 initiates frame dropping management.

[0073] Subsequently, as shown in operation S808, one or more display frame parameters and one or more device operation parameters are aggregated by data aggregator 404. The rendered display frame parameters and device operation parameters are sent to frame drop prediction module 210 for inference on the device. Then, as shown in operation S810, the trained FDP method of frame drop prediction module 210 uses real-time inference data to predict the duration of consecutive frames based on the current frame duration. Frame drop prediction module 210 uses the FDP method to estimate the preparation time for rendering each frame.

[0074] If the predicted duration of consecutive frames exceeds a predetermined threshold, frame dropping is determined. As shown in operation S812, the frame dropping condition is verified. Later, as shown in operation S814, CPU acceleration and scheduling are performed to reduce the preparation time for rendering each frame on the electronic device 200, thereby avoiding impending frame dropping. For example, the policy decision module 212 may include a CPU accelerator and / or scheduler 504 that performs acceleration and scheduling when triggered. The policy decision module 212 utilizes a policy window model to determine the window duration or acceleration duration based on the frame count to determine how long the acceleration should be applied, as shown in operation S816.

[0075] The various actions in method 800 can be performed in the order they are presented, in different orders, or simultaneously. Furthermore, in some embodiments, actions can be omitted. Figure 8 Some of the actions listed in the document.

[0076] Figure 9A Offline modeling of a deep learning (DL) model using one or more display frame parameters and one or more device operation parameters is illustrated according to an embodiment of the present disclosure.

[0077] refer to Figure 9A In S902, the display frame parameters and device operation parameters are merged, and the merged data is sent for preprocessing, as shown in S904. The preprocessed data is then sent to train a deep learning (DL) model, as shown in S906. The trained model is stored in database 710, as shown in S908.

[0078] Figure 9B On-device inference using a trained DL method for frame drop prediction according to embodiments of this disclosure is illustrated.

[0079] refer to Figure 9B The trained and stored model is applied to the frame drop prediction module 210 within the Vsync interval of the display to predict frame drops. Later, as shown in S910, acceleration and / or scheduling can be implemented by the policy decision module 212 to reduce frame drops in the user interaction pipeline.

[0080] The frame drop prediction module 210 uses a pre-trained machine learning model called Frame Duration Prediction (FDP) to perform on-device inference for early detection of frame drops. The intelligent data collector 208 invokes the frame drop prediction module 210 by providing input to the FDP method used to calculate the probability of frame drops.

[0081] Figure 10 The triggering of the frame loss prediction module 210 and the policy decision module 212 according to an embodiment of the present disclosure is shown.

[0082] The frame drop prediction module 210 is invoked at the end of each frame. The frame drop prediction module 210 obtains the probability of dropping the next frame based on past history. If the probability is greater than a threshold, the policy decision module 212 is invoked at the beginning of a frame with the expected duration and task ID. The policy decision module 212 is responsible for updating the CPU load, scheduler 504, and speed controller parameters of the UI task to help render the application UI within the Vsync interval.

[0083] Figure 11A , 11B 11C illustrates various use cases according to various embodiments of the present disclosure for detecting and preventing frame drops in several scenarios.

[0084] Figure 11A The illustration shows the viewing of high-definition (HD) video / social media according to an embodiment of this disclosure. Figure 11B An aggressive browser content scrolling method according to an embodiment of this disclosure is illustrated. Figure 11C A consistent frame rate during gameplay is shown according to an embodiment of this disclosure.

[0085] Figure 12 Example implementations of the proposed methods (600, 700, 800) for managing frame loss according to embodiments of the present disclosure are shown.

[0086] First, collect system trajectories and a timeline map application with CPU data on the device. Highlight dropped frames in the tracing and check the CPU frequency at that point. When dropped frames are predicted and preventative measures are applied, spikes can be observed in the CPU frequency.

[0087] For example, when a user begins scrolling through an application, they may notice stuttering due to dropped frames. The proposed methods (600, 700, 800) can predict future dropped frames and apply strategies to avoid them, thereby achieving a smooth and fluid UI experience for end users. Therefore, the proposed devices and methods (600, 700, 800) determine the next dropped frame or consecutive dropped frames to make proactive decisions such as acceleration / scheduleing to reduce dropped frames. Electronic device 200 can be, but is not limited to, a telephone, television (TV), camera, washing machine, refrigerator, smartwatch, etc. Therefore, during key user journeys such as scrolling, application launch, gaming, browsing, and several other use cases, rendering performance can be improved and dropped frames can be reduced, thereby improving the overall user experience on electronic device 200.

[0088] The embodiments disclosed herein can be implemented by at least one software program running on at least one hardware device. Figure 2 The module shown includes a block that may be at least one of a hardware device or a combination of a hardware device and a software module.

[0089] The embodiments disclosed herein describe an electronic device (200) and methods (600, 700, 800) for predicting and preventing one or more dropped frames to enhance the user interface (UI) experience on an electronic device 200. Therefore, it should be understood that the scope of protection extends to programs that, in addition to computer-readable means containing messages, include program code means for implementing one or more steps of the method when the program is run on a server or mobile device or any suitable programmable device. In at least one embodiment, the method is implemented by or with a software program written in another programming language, such as the Very High Speed ​​Integrated Circuit Hardware Description Language (VHDL), or by one or more VHDL or software modules executed on at least one hardware device. The hardware device can be any kind of portable device that can be programmed. The device may also include means that can be, for example, hardware devices such as application-specific integrated circuits (ASICs), or a combination of hardware and software means, such as ASICs and field-programmable gate arrays (FPGAs), or at least one microprocessor and at least one memory having software modules located therein. The method embodiments described herein can be implemented partly in hardware and partly in software. Alternatively, this disclosure can be implemented on different hardware devices, for example, using multiple CPUs.

[0090] It should be understood that the various embodiments of this disclosure described in the claims and specification can be implemented in hardware, software, or a combination of hardware and software.

[0091] Any such software may be stored in a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores one or more computer programs (software modules), the one or more computer programs including computer-executable instructions that, when executed individually or jointly by one or more processors of an electronic device, cause the electronic device to perform the methods of this disclosure.

[0092] Any such software may be stored in the form of volatile or non-volatile storage devices, such as storage devices like read-only memory (ROM), whether erasable or rewritable, or in the form of memory such as random access memory (RAM), memory chips, devices, or integrated circuits, or stored on optical or magnetically readable media, such as optical discs (CDs), digital versatile discs (DVDs), magnetic disks, or magnetic tapes. It should be understood that storage devices and storage media are various embodiments of non-transitory machine-readable storage devices suitable for storing one or more computer programs including instructions that, when executed, implement various embodiments of this disclosure. Therefore, various embodiments provide a program and a non-transitory machine-readable storage device for storing such a program, the program including code for implementing the means or methods claimed as any one of the claims of this specification.

[0093] While this disclosure has been shown and described with reference to various embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope of this disclosure as defined by the appended claims and their equivalents.

Claims

1. A method for managing frame dropping in an electronic device, the method comprising: One or more display frame parameters are determined by the electronic device for displaying each frame; One or more device operating parameters, determined by the electronic device, to indicate the system state of one or more resources within the electronic device; The electronic device uses one or more display frame parameters and one or more device operation parameters to estimate the preparation time for rendering each frame on the electronic device. and In response to a frame drop occurring as indicated by the estimated preparation time, the electronic device adjusts one or more resources within the electronic device to reduce the preparation time.

2. The method according to claim 1, wherein, The method includes: Scenes with the possibility of frame drops are detected by an electronic device based on one or more parameters received from at least one touch input from a user; and Frame drops are managed by electronic devices based on the detected scene.

3. The method according to claim 2, wherein, The detected scenarios include at least one of aggressive scrolling, gaming, and playing high-resolution content.

4. The method according to claim 2, wherein, The parameters from the at least one touch input include at least one of frequency, speed, input pressure level, and at least one multi-hand gesture.

5. The method according to claim 1, wherein, The one or more display frame parameters include at least one of input processing duration, animation duration, and drawing duration.

6. The method according to claim 1, wherein, The one or more device operating parameters include at least one of the following: CPU load, GPU status, available memory, number of running threads on the CPU, and temperature.

7. The method according to claim 1, in, The Frame Duration Prediction (FDP) method is used to estimate the preparation time for rendering each frame. The FDP method is a machine learning (ML) method, and The FDP method is trained to learn the correlation between the one or more display frame parameters, the one or more device operation parameters, and the preparation time for rendering each frame.

8. The method according to claim 7, wherein, The trained FDP method is configured as follows: Predict the duration of consecutive frames based on the duration of the current frame; and Frame loss is determined when the predicted duration of consecutive frames exceeds a predetermined threshold.

9. The method according to claim 1, in, The one or more resources include at least one of a CPU, a scheduler, and a speed regulator, and The method of adjusting the one or more resources includes at least one of CPU acceleration and scheduling and management tasks.

10. The method according to claim 1, in, The strategy decision-making module is used to adjust the one or more resources within a predetermined duration, and The one or more resources are adjusted by determining the window duration based on frame counts.

11. An electronic device, comprising: Memory, which stores one or more computer programs; and One or more processors, which are communicatively coupled to the memory, The one or more computer programs include computer-executable instructions that, when executed individually or jointly by the one or more processors, enable the electronic device to manage frame drops. The computer-executable instructions further enable the electronic device to: Determine one or more display frame parameters to be used to display each frame. Determine one or more device operating parameters that indicate the system state of one or more resources in an electronic device. The preparation time for rendering each frame on the electronic device is estimated using the one or more display frame parameters and the one or more device operation parameters. In response to the estimated preparation time indication of frame loss, one or more resources in the electronic device are adjusted to reduce the preparation time.

12. The electronic device according to claim 11, wherein, The one or more computer programs further include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to: Detecting scenarios with the possibility of frame dropping based on one or more parameters from at least one touch input received from the user; and Frame dropping is managed based on the detected scene.

13. The electronic device according to claim 12, wherein, The detected scenarios include at least one of aggressive scrolling, gaming, and playing high-resolution content.

14. The electronic device according to claim 12, wherein, The parameters from the at least one touch input include at least one of frequency, speed, input pressure level, and at least one multi-hand gesture.

15. The electronic device according to claim 11, wherein, The one or more display frame parameters include at least one of input processing duration, animation duration, and drawing duration.

Images