Sliding frame loss detection method, electronic device, storage medium, and program product

Abstract

Embodiments of the present application provide a sliding frame loss detection method, electronic equipment, storage medium and program product, which relate to the technical field of terminals. The method can obtain a frame loss detection result reflecting frame loss continuity, and based on the frame loss detection result, the method can more accurately analyze the freezing situation in the sliding process. The method is applied to electronic equipment including a screen. During the process of the electronic equipment controlling the page sliding in the screen in response to the sliding operation, the method performs frame loss detection processing on the frame images generated in the page sliding process based on a set sliding window. In the frame loss detection processing, the frame loss images are marked, and the continuity between the frame loss images is analyzed to obtain the frame loss detection result.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to sliding frame dropping detection methods, electronic devices, storage media, and program products. Background Technology

[0002] Because the screen's display area is limited, the information that can be presented to the user is also limited. Therefore, the user can slide their finger across the electronic device's screen, changing the content displayed and browsing more information. Multiple consecutive frames are generated at the beginning and end of the slide to alter the content displayed on the screen.

[0003] However, frame drops are prone to occur during scrolling, and the more frames dropped, the more noticeable the stuttering. Electronic devices can detect frame drops and send the detection results to the cloud for analysis of stuttering during scrolling. However, current frame drop detection results can only identify which frame is missing, resulting in limited information and an inability to accurately analyze stuttering during scrolling. Summary of the Invention

[0004] This application provides a sliding frame dropping detection method, electronic device, storage medium, and program product, which can obtain frame dropping detection results that reflect the continuity of frame dropping, and based on the frame dropping detection results, can more accurately analyze and determine the frame dropping situation and the stuttering situation during the sliding process.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] Firstly, a method for detecting dropped frames during scrolling is provided. This method is applied to an electronic device, which includes a screen. The electronic device can respond to scrolling operations applied to the screen to control the scrolling of pages on the screen. During the page scrolling process, the electronic device can perform dropped frame detection processing on the frame images generated during the page scrolling process based on a set scrolling window. That is, it marks the dropped frame images (frame images with dropped frames) within the scrolling window and analyzes the continuity between the dropped frame images to obtain the dropped frame detection result.

[0007] In the above scheme, the sliding window can perform frame drop detection on the frame images generated during the sliding process and mark the frame images that have dropped frames during the sliding process. After the sliding operation is completed, the continuous frame drop during the sliding process can be determined by judging whether the marked frame images are continuous. Thus, a frame drop detection result that reflects the continuity of frame drop can be obtained. Based on the frame drop detection result, the frame drop situation and the stuttering situation during the sliding process can be analyzed and determined more accurately.

[0008] In one possible implementation of the first aspect, the electronic device can determine the length of the sliding window based on historical data of single sliding operations; that is, the length of the sliding window is determined based on the statistical duration of single sliding operations. The statistical duration of single sliding operations is obtained by statistically analyzing the duration of historical single sliding operations.

[0009] In the above scheme, the length of the sliding window is determined by statistically analyzing the duration of a single sliding operation, making the determined sliding window length more appropriate. Consequently, frame drop detection can be performed more efficiently based on a more appropriate sliding window, and unnecessary processing caused by inaccurate window length is avoided, thereby reducing resource consumption to a certain extent.

[0010] In another possible implementation of the first aspect, the electronic device predetermines the correspondence between the duration of a single sliding operation and the probability value. Based on the correspondence, the electronic device can select the duration of the probability value being greater than a preset probability threshold as the length of the sliding window.

[0011] In the above scheme, by establishing the correspondence between the duration of a single swipe operation and its probability value, it is easier to quickly find a suitable window length (i.e., the length of the swipe window) that can cover most swipe scenarios. Therefore, when a user swipes the screen once, a single window detection can detect all frames generated during the swipe, and the frame drop detection result generated by this single window detection can determine the stuttering situation throughout the swipe process, improving efficiency and avoiding unnecessary resource consumption caused by multiple window detections.

[0012] In another possible implementation of the first aspect, after the page starts to slide, the electronic device can use the first dropped frame image during the page sliding process as the detection starting point of the sliding window and start the current window detection, so as to perform dropped frame detection processing on the frame images located in the sliding window during the current window detection.

[0013] In the above scheme, since a large number of frames are generated during the sliding process, but the number of frame images that can be detected by the sliding window at one time is limited, if the first frame image during the sliding process is used as the starting point for the sliding window detection, the number of detections of the sliding window during the sliding process will increase, wasting computing power. Therefore, using the first dropped frame image during the page sliding process as the starting point for the sliding window detection can reduce the number of detections of the sliding window during the sliding process and reduce the waste of computing power.

[0014] In another possible implementation of the first aspect, if the current window detection ends but the sliding operation does not end, frame images will continue to be generated. Therefore, the electronic device can take the first frame image with frame loss in the frame images generated after the current window detection ends as the detection starting point of the sliding window in the next window detection, and take the next window detection as the new current window detection, and continue to perform frame loss detection processing on the new frame images in the sliding window until the sliding operation ends.

[0015] In the above scheme, when a window detection is completed but the sliding operation is not yet finished, the electronic device can continue to use the first dropped frame image after the current window detection as the detection starting point of the sliding window. This can avoid including the non-dropped frame image after the current window detection into the sliding window, reduce the number of times the sliding window is detected during the sliding process, and reduce the waste of computing power.

[0016] In another possible implementation of the first aspect, in the frame loss detection process, each frame image is judged to determine whether it is a frame loss. Specifically, in the frame loss detection process, for each frame image to be detected, the electronic device can obtain the processing time of the frame image to be detected. The processing time of the frame image to be detected refers to the actual time consumed in processing the frame image to be detected. The processing time of the frame image to be detected is compared with the frame period. When the difference between the two exceeds a preset time threshold, the frame image to be detected is determined to be a frame loss image. The frame period refers to the time required to generate a frame image under normal circumstances.

[0017] In the above scheme, the difference between processing duration and frame period is used to determine whether the frame image is missing a frame. This can more accurately determine whether the frame image is missing a frame, and it is also convenient to determine and analyze the cause of the frame loss through the frame loss detection results.

[0018] In another possible implementation of the first aspect, the continuity between the dropped frames is analyzed to obtain detection information related to the consecutive dropped frames. In addition to the detection information related to the consecutive dropped frames, the dropped frame detection result also includes at least one of the following: information of the dropped frames, global statistical information of the dropped frames, quantization value of the stuttering degree, or length of the sliding window.

[0019] In the above scheme, the frame drop detection results include information from multiple dimensions. Therefore, the frame drop detection results can be used to more comprehensively and holistically determine the frame drop situation during the scrolling process, as well as to more accurately determine the degree of stuttering during the scrolling process.

[0020] In another possible implementation of the first aspect, during frame drop detection processing, the electronic device can obtain a quantification value of the stuttering level as follows: the electronic device can determine the quantification value of the stuttering level based on the processing time of each dropped frame image, the weight of each dropped frame image, and the length of the sliding window. The weight of each dropped frame image is positively correlated with the order in which the dropped frames are processed.

[0021] In the above scheme, each frame drop detection result obtained from the sliding window has a corresponding stuttering level quantification value. The stuttering level quantification value is determined based on the processing time and weight of the dropped frame image during each sliding window detection process, so that the stuttering level quantification value can measure the stuttering level of the sliding scene from the user's perception dimension.

[0022] In another possible implementation of the first aspect, the quantification value of stuttering includes the standardized frame drop time. The electronic device can determine the standardized frame drop time based on the ratio between a second value and the length of the sliding window. The second value is obtained by weighted summation of the processing times corresponding to each dropped frame according to their respective weights.

[0023] In the above scheme, the standardized frame dropping time is determined by the ratio between the weighted sum of the processing times of each dropped frame image and the length of the sliding window. This is equivalent to combining the frame dropping times of multiple dropped frames to more accurately quantify the degree of stuttering. In other words, it can more accurately measure the degree of stuttering in the sliding scene from the user's perception.

[0024] In another possible implementation of the first aspect, the electronic device can determine the weights of each frame image by determining a sliding curve, which characterizes the pixel changes of each frame image compared to the previous frame image during page scrolling. Specifically, in the sliding curve, the amount of pixel change in a frame image is negatively correlated with the order in which the frames are generated during page scrolling. The electronic device can determine the weights of each dropped frame image based on the sliding curve and the order in which the dropped frames are generated during page scrolling.

[0025] In the above scheme, because the pixel changes are greater at the beginning of the swipe, users will be more likely to perceive stuttering if frame drops occur at this time, and the weight of the dropped frame image is greater. Conversely, at the end of the swipe, the pixel changes are smaller, and users are less likely to perceive stuttering if frame drops occur at this time, and the weight of the dropped frame image is smaller. Therefore, determining the weight of each dropped frame image based on the amount of pixel change in the frame image can take into account the different impacts of frame drops on the degree of stuttering. This allows the standardized frame drop time to more accurately measure the degree of stuttering in the swipe scenario from the user's perception perspective.

[0026] Secondly, this application provides an electronic device, which includes a memory, a screen, and a processor; the memory and screen are coupled to the processor. The memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device performs the method described in the first aspect and any possible implementation thereof.

[0027] Thirdly, embodiments of this application provide a computer storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in the first aspect and any possible implementation thereof.

[0028] Fourthly, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform the method as described in the first aspect and any possible implementation thereof. The computer may be an electronic device as described in the second aspect and any possible implementation thereof.

[0029] Understandably, the beneficial effects that the electronic device of the second aspect, the computer storage medium of the third aspect, and the computer program product of the fourth aspect provided above can be referenced to the beneficial effects of the first aspect and any possible implementation thereof, which will not be repeated here. Attached Figure Description

[0030] Figure 1 A schematic diagram of the software architecture of electronic devices provided by traditional methods;

[0031] Figure 2 A schematic diagram of frame loss detection results provided by traditional methods;

[0032] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0033] Figure 4 A schematic diagram of the software architecture of the electronic device provided in the embodiments of this application;

[0034] Figure 5 A flowchart illustrating a sliding frame dropping detection method provided in an embodiment of this application;

[0035] Figure 6 A schematic diagram of a cumulative distribution function curve provided in an embodiment of this application;

[0036] Figure 7 A schematic diagram of a sliding interval provided in an embodiment of this application;

[0037] Figure 8This is a schematic diagram of a frame loss type provided in an embodiment of this application;

[0038] Figure 9 A schematic diagram of a sliding curve provided for an embodiment of this application;

[0039] Figure 10 A schematic diagram of a frame loss detection process provided in an embodiment of this application;

[0040] Figure 11 Another schematic diagram of the sliding interval provided for an embodiment of this application;

[0041] Figure 12 The embodiments provided in this application are related to Figure 11 The corresponding diagram shows the frame loss detection results. Detailed Implementation

[0042] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0043] Before introducing specific embodiments, for ease of understanding, some concepts related to the embodiments of this application are explained by way of example for reference.

[0044] A frame refers to a single, still image, the smallest unit in a user interface display. A frame can be understood as a static image; rapidly displaying multiple consecutive frames can create a dynamic effect of moving objects.

[0045] A swiping scenario refers to the user's ability to slide their finger across the screen of an electronic device during operation, thereby changing the content displayed on the screen. A specific swiping scenario could be: when the electronic device's display shows content from an application, and the screen space is limited due to the application's extensive content, the user can swipe up or down to view content not yet displayed. The swiping action can be an upward swipe, a downward swipe, etc. For example, an upward swipe could be achieved by placing a finger at the bottom of the screen and then swiping upwards from there.

[0046] In a swiping scenario, when an electronic device receives a user's swipe gesture, the content on the screen changes accordingly. The electronic device first determines the pressure applied by the user when swiping, and then converts this pressure data into an initial swipe speed using a preset algorithm or function. At this point, the swipe begins. A preset damping coefficient determines when the swipe ends; that is, the swipe ends when the swipe speed becomes zero. During this entire swipe process (the swipe operation), multiple consecutive frame images are generated, forming a swipe interval. The frame images within this interval have a sequential order and are displayed on the screen in that order. The preset algorithm or function for converting the pressure data into the initial swipe speed, as well as the damping coefficient, are set at the factory and may vary between manufacturers.

[0047] Therefore, when a user performs a swipe operation, the electronic device can generate multiple frame images and display them on the screen, causing the content on the screen to change, presenting new content on the screen, and achieving the screen swiping effect.

[0048] It's understandable that frame images typically require drawing, rendering, and compositing processes to generate. If these processes take too long—meaning the frame image processing time exceeds the screen's refresh cycle—some frames will fail to display on time, resulting in dropped frames and choppy animations or scrolling. Therefore, the more dropped frames, the more severe the perceived stuttering or lag during scrolling.

[0049] In traditional methods, electronic devices can detect frame drops, send the detection results to the cloud, and then determine the cause of the frame drops based on the detection results, thereby reducing the occurrence of frame drops.

[0050] Taking mobile phones as an example, Figure 1 A schematic diagram of the software architecture of electronic devices provided by traditional methods.

[0051] like Figure 1 As shown, a layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into four layers, from top to bottom: the application layer, the application framework layer, the local service layer, and the kernel layer.

[0052] The application layer can include a series of application packages, such as shopping applications, chat applications, browsers, and SMS.

[0053] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0054] like Figure 1 As shown, the application framework layer can include scrollers, synchronization modules, etc. The scroller can be OverScroller, a helper class used to implement smooth scrolling of the view. OverScroller can also be used to identify the start and end of the scroll; the start of the scroll refers to when the frame image on the screen begins to change, and the end of the scroll refers to when the frame image on the screen stops changing. The synchronization module can be hiviewtunnel, used to report frame drop detection results in problem scenarios. hiviewtunnel can obtain the frame drop detection results from the kernel layer and send them to the cloud.

[0055] OverScroller is primarily used to detect scrolling scenarios.

[0056] The local service layer resides between the application framework layer and the kernel layer. It may include a layer integrator, a layer detector, and a fault detection module. The layer integrator manages, composites, and renders the system's graphical interface, ensuring that the application and system interfaces are displayed correctly and efficiently on the screen. The layer detector detects frame images in different layers and records and stores the frame drop detection results in the kernel layer. The fault detection module statistically analyzes the frame drop detection results recorded in the kernel layer and transmits these results to the synchronization module in the application framework layer.

[0057] In some embodiments, the layer integrator may be SurfaceFlinger, the layer detector may be LayerDetector, and the fault detection module may be hiview.

[0058] Among them, the layer integrator and layer detector constitute the smoothness detection model, which is mainly used to detect the smoothness during the sliding process.

[0059] The kernel layer is the layer between hardware and software. It includes drivers for various hardware components.

[0060] In some embodiments, the kernel layer may include a storage driver, a digital signal processor (DSP) driver, and an image processor (IPC) driver. The DSP driver drives the DSP to process images, the IPC driver drives the IPC to process images, and the storage driver ensures effective access and control of the storage device, enabling the layer detector to store the generated frame drop detection results in the ` / dev / hwlog_jank` file and allowing the fault detection module to retrieve the frame drop detection results from the ` / dev / hwlog_jank` file.

[0061] The traditional frame loss detection process is as follows:

[0062] After the electronic device receives the user's swipe operation, the OverScroller in the application framework layer identifies the start and end of the swipe. The OverScroller then transmits this information to the SurfaceFlinger in the local service layer, which determines the layer containing the frame images generated during the swipe. The LayerDetector in the local service layer performs frame drop detection on the frame images generated on that layer during the swipe, for example, detecting the processing time of each frame image—that is, detecting the processing time of drawing, rendering, and compositing processes when generating each frame image—and stores the frame drop detection results in the kernel layer. Finally, the hiview in the local service layer compiles the frame drop detection results from the kernel layer and transmits the frame drop detection results for a single frame image to the hiviewtunnel in the application framework layer. The hiviewtunnel can upload the frame drop detection results for a single frame image to the cloud, allowing developers to determine whether frame drops occur during the swipe, identify the cause of the drops, and make corresponding improvements to reduce the occurrence of frame drops.

[0063] As described above, traditional frame drop detection can only detect which frames have dropped frames among multiple frames, but it cannot detect whether there are consecutive frame drops between these dropped frames. For example, if a sliding effect is displayed using N consecutive frames from frame 1 to frame N, and frames 2 and 3 have dropped frames, traditional methods can only determine that frames 2 and 3 have dropped frames, but cannot determine whether frames 2 and 3 have consecutive dropped frames.

[0064] For example, Figure 2 This is a schematic diagram of frame loss detection results provided by traditional methods, such as... Figure 2 As shown, the frame drop detection results obtained by traditional methods only contain relevant information for each dropped frame image (i.e., the frame image in which frame drop occurs), such as: the layer where the dropped frame image is located, the processing time of the dropped frame image, the frame period, and information about the dropped frame image.

[0065] The layer where the dropped frame image is located can refer to the layer of the application on which the dropped frame image is generated during the swiping process. That is, assuming that the page currently displayed on the screen is a browser page, when the user performs a swiping operation on the screen, a frame image will be generated on the browser layer during the swiping process. If a dropped frame image occurs in the generated frame image, then the layer where the dropped frame image is located is the browser layer.

[0066] The frame period refers to the time required to generate one frame of image under normal circumstances. The frame period can be obtained from the frame rate. For example, the frame period can be the reciprocal of the frame rate, that is, frame period = 1 / frame rate. The frame rate refers to the frequency (rate) at which multiple frames appear on the display continuously, which can also be understood as the number of times the graphics processor refreshes the screen per second.

[0067] Information about dropped frames can include: the ID of the dropped frame and the processing time of the dropped frame, such as the processing time in the rendering stage, the processing time in the compositing stage, and the total processing time in the rendering, compositing, and drawing stages.

[0068] Now Figure 2 Let's take a diagram as an example to explain. For example... Figure 2 As shown, the layer containing the dropped frames is the layer of shopping application A. During the scrolling process, frame images with IDs 11366304, 11366326, and 11366331 experienced frame drops. The processing time for generating these three frame images during the rendering, compositing, and drawing stages is also shown. The total processing time for generating frame image 11366304 was 32 milliseconds, for frame image 11366326 it was 33 milliseconds, and for frame image 11366331 it was 25 milliseconds, with a frame period of 833333 milliseconds for all three.

[0069] Therefore, based on the frame loss detection results obtained from traditional methods, it can be seen that the current detection methods can only obtain frame loss information for a single frame and cannot reflect the continuity of frame loss.

[0070] However, if consecutive frame drops occur during the scrolling process, the stuttering effect is more noticeable. For example, if N consecutive frame images are generated during the scrolling process, namely frame 1, frame 2, frame 3, frame 4, frame 5... frame N, in the first case, frames 1 and 5 are dropped, and in the second case, frames 1 and 2 are dropped. In the second case, the stuttering perceived by the user is more obvious than that perceived in the first case.

[0071] However, traditional methods can only determine the frame loss information of the lost frames. That is, the frame loss detection results can only determine which frame image is missing. The amount of frame loss information obtained is limited and cannot determine the overall frame loss situation during the scrolling process. For example, it is impossible to determine whether there are consecutive frame losses, the cumulative duration of frame losses during the scrolling process, or the maximum number of consecutive frame losses during the scrolling process. Therefore, it is not possible to accurately analyze the frame loss situation and the degree of stuttering during the scrolling process. For example, it is impossible to know whether the frame loss situation affects the smoothness of the interface or whether the stuttering during the scrolling process is obvious. Consequently, developers cannot better find the cause of frame loss through the frame loss detection results.

[0072] Therefore, in the embodiments of this application, a method for detecting dropped frames during sliding is provided. In the process of controlling page sliding on the screen in response to a sliding operation, dropped frame detection processing is performed on frame images generated during page sliding based on a set sliding window. In the dropped frame detection processing, dropped frame images are marked, and the continuity between dropped frame images is analyzed to obtain the dropped frame detection result. Here, a dropped frame image refers to a frame image within the sliding window where dropped frames exist during the dropped frame detection processing.

[0073] Understandably, when an electronic device receives a user's swipe operation and swipes on a page on the screen, the swipe window can detect dropped frames generated during the swipe and mark the dropped frames. After the swipe operation ends, it determines whether there are continuous dropped frames by judging whether the marked frames are continuous. This provides a dropped frame detection result that reflects the continuity of dropped frames. Based on this dropped frame detection result, the dropped frame situation and the stuttering during the swipe can be analyzed more accurately.

[0074] In some embodiments of this application, the electronic device can perform frame drop detection on frame images generated during the sliding process using a sliding window. It is understood that for the entire sliding process of a sliding operation, the sliding window can perform several detections during that process; each detection is called a window detection. In each window detection, frame drop detection processing is performed on the frame images within the sliding window. Specifically, in each window detection, the electronic device can mark the frame images within the sliding window where frame drops occur (i.e., mark the dropped frame images). After the current window detection is completed, the continuity between the marked frame images in this window detection is determined to obtain detection information related to continuous frame drops, and this detection information related to continuous frame drops obtained in this window detection is reported to the cloud as one of the information in the frame drop detection result.

[0075] For example, the aforementioned electronic device may be a mobile phone, tablet computer, smart remote control, wearable device (such as a smart bracelet, smartwatch, or smart glasses), PDA, or augmented reality (AR) / virtual reality (VR) device. Alternatively, the electronic device 300 may also be a portable multimedia player (PMP), media player, or other types of electronic device. This application embodiment does not impose any limitations on the specific type of electronic device.

[0076] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0077] Please refer to Figure 3 This application uses a mobile phone as an example to describe the electronic device 300 provided in this application. Figure 3 As shown, the electronic device 300 may include: a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, antenna 1, antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, a headphone jack 370D, a sensor module 380, buttons 390, an indicator 392, a camera 393, and a display screen 394. The sensor module 380 may include a pressure sensor 380A, a touch sensor 380B, etc.

[0078] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 300. In other embodiments of this application, the electronic device 300 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. For ease of understanding, the various components described above will be briefly described below.

[0079] Processor 310 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. The different processing units may be independent devices or integrated into one or more processors.

[0080] In the embodiments of this application, after the pressure sensor 380A and touch sensor 380B receive a swipe operation performed by the user on the display screen 394, the processor 310 generates multiple consecutive frame images through the image processor and displays these frame images on the display screen 394, so that new page content can be presented on the display screen and a swipe effect can be achieved on the display screen. Furthermore, the processor 310 can detect these frame images to determine whether there are any consecutive dropped frames.

[0081] The controller can be the nerve center and command center of the electronic device 300. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0082] The processor 310 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 310 is a cache memory. This memory can store instructions or data that the processor 310 has just used or that are used repeatedly. If the processor 310 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 310, and thus improves the efficiency of the system.

[0083] In some embodiments, the processor 310 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0084] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 300. In other embodiments of this application, the electronic device 300 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0085] The charging management module 340 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 340 receives charging input from the wired charger via a USB interface 330. In some wireless charging embodiments, the charging management module 340 receives wireless charging input via the wireless charging coil of the electronic device 300. While charging the battery 342, the charging management module 340 can also supply power to the electronic device via the power management module 341.

[0086] The power management module 341 connects the battery 342, the charging management module 340, and the processor 310. The power management module 341 receives input from the battery 342 and / or the charging management module 340, providing power to the processor 310, internal memory 321, external memory, display screen 394, camera 393, and wireless communication module 360. The power management module 341 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 341 may also be located within the processor 310. In other embodiments, the power management module 341 and the charging management module 340 may be housed in the same device.

[0087] The wireless communication function of electronic device 300 can be realized through antenna 1, antenna 2, mobile communication module 350, wireless communication module 360, modem processor and baseband processor, etc.

[0088] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 300 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0089] The mobile communication module 350 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 300. The mobile communication module 350 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 350 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 350 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 350 may be housed in the processor 310. In some embodiments, at least some functional modules of the mobile communication module 350 and at least some modules of the processor 310 may be housed in the same device.

[0090] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 370A, receiver 370B, etc.) or displays images or videos through a display screen 394. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 310 and may be housed in the same device as the mobile communication module 350 or other functional modules.

[0091] The wireless communication module 360 ​​can provide solutions for wireless communication applications on the electronic device 300, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 360 ​​can be one or more devices integrating at least one communication processing module. The wireless communication module 360 ​​receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signal, and sends the processed signal to processor 310. The wireless communication module 360 ​​can also receive signals to be transmitted from processor 310, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0092] In some embodiments, antenna 1 of electronic device 300 is coupled to mobile communication module 350, and antenna 2 is coupled to wireless communication module 360, enabling electronic device 300 to communicate with networks and other devices via wireless communication technology. Wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. GNSS can include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0093] Electronic device 300 implements display functions through a GPU, a display screen 394, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 310 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0094] Display screen 394 is used to display images, videos, etc. Display screen 394 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 300 may include one or N displays 394, where N is a positive integer greater than 1.

[0095] In embodiments of this application, a plurality of consecutive frame images generated by the processor 310 through the image processor can be displayed on the display screen 394.

[0096] Electronic device 300 can achieve shooting function through ISP, camera 393, video codec, GPU, display 394 and application processor.

[0097] The ISP (Image Signal Processor) is used to process data fed back from the camera 393. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's image sensor. The light signal is converted into an electrical signal, and the image sensor transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimizations on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be integrated into the camera 393.

[0098] Camera 393 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 300 may include one or N cameras 393, where N is a positive integer greater than 1.

[0099] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when the electronic device 300 is selecting a frequency, the DSP is used to perform Fourier transforms on the frequency energy.

[0100] Video codecs are used to compress or decompress digital video. Electronic device 300 may support one or more video codecs. Thus, electronic device 300 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0101] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0102] The external storage interface 320 can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of the electronic device 300. The external memory card communicates with the processor 310 through the external storage interface 320 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0103] Internal memory 321 can be used to store computer executable program code, which includes instructions. Processor 310 executes various functional applications and data processing of electronic device 300 by running the instructions stored in internal memory 321. Internal memory 321 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 300 (such as audio data, phonebook, etc.). Furthermore, internal memory 321 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0104] Electronic device 300 can implement audio functions such as music playback and recording through audio module 370, speaker 370A, receiver 370B, microphone 370C, headphone jack 370D, and application processor.

[0105] The audio module 370 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. The audio module 370 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 370 may be located in the processor 310, or some functional modules of the audio module 370 may be located in the processor 310.

[0106] The speaker 370A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. Electronic device 300 can listen to music or make hands-free calls through the speaker 370A.

[0107] The receiver 370B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 300 answers a telephone call or voice message, the receiver 370B can be brought close to the listener's ear to hear the voice.

[0108] Microphone 370C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 370C, inputting the sound signal into microphone 370C. Electronic device 300 may have at least one microphone 370C. In some embodiments, electronic device 300 may have two microphones 370C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 300 may have three, four, or more microphones 370C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0109] The 370D headphone jack is used to connect wired headphones. The 370D headphone jack can be a USB 330 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0110] Pressure sensor 380A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 380A can be disposed on display screen 394. There are many types of pressure sensors 380A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 380A, the capacitance between the electrodes changes. Electronic device 300 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 394, electronic device 300 detects the touch operation intensity based on pressure sensor 380A. Electronic device 300 can also calculate the touch position based on the detection signal from pressure sensor 380A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example: when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0111] Touch sensor 380B, also known as a "touch panel," can be located on display screen 394. The touch sensor 380B and display screen 394 together form a touchscreen, also known as a "touch screen." Touch sensor 380B is used to detect touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 394. In other embodiments, touch sensor 380B may also be located on the surface of electronic device 300, in a different position than display screen 394.

[0112] In the embodiments of this application, the start and end of the sliding operation can be determined by combining the touch sensor 380B and the pressure sensor 380A. Specifically, after the touch sensor 380B receives the user's sliding operation, the pressure sensor 380A determines the force applied by the user when sliding on the display screen 394. The processor 310 then converts the force data into the initial sliding speed, at which point the sliding begins. Subsequently, the timing of the end of the sliding operation is determined by a preset damping coefficient, that is, the end time of the user's sliding operation on the display screen 394 is determined.

[0113] Buttons 390 include a power button, volume buttons, etc. Buttons 390 can be mechanical buttons or touch-sensitive buttons. Electronic device 300 can receive button input and generate key signal inputs related to user settings and function control of electronic device 300.

[0114] In summary, in the embodiments of this application, after the electronic device 300 determines the sliding operation performed by the user on the display screen 394 through the pressure sensor 380A and the touch sensor 380B, the processor 310 generates multiple consecutive frame images and displays multiple frame images on the display screen 394, so as to realize the change of the content on the display screen 394 through the user's sliding operation, and the processor 310 detects the continuous frame loss during the sliding process.

[0115] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0116] Figure 4 This is a schematic diagram of the software architecture of the electronic device provided in the embodiments of this application.

[0117] In the embodiments of this application, a continuous frame dropping detection module may be added to the local service layer to implement the sliding frame dropping detection method. The method in the embodiments of this application will be described in detail below with reference to the above hardware and system structures:

[0118] 1. When the electronic device 300 receives the user's swipe operation, the swipe begins, and the electronic device 300 starts to generate multiple consecutive frame images and displays these frame images on the display screen 394.

[0119] Step 1 is continuous. After the electronic device 300 receives the user's swipe operation on the display screen 394 through the pressure sensor 380A and the touch sensor 380B, the processor 310 generates multiple consecutive frame images and displays these frame images on the display screen 394, thus creating a swipe effect on the display screen 394.

[0120] 2. Electronic device 300 performs frame drop detection on the frame image to determine the continuous frame drop situation during the sliding process.

[0121] Step 2 is continuous. The OverScroller in the application framework layer identifies the start and end of the scrolling. The OverScroller then transmits this information to the SurfaceFlinger in the local service layer. The SurfaceFlinger then determines the layer containing the frame images generated during the scrolling process. The LayerDetector in the local service layer then detects any consecutive frame drops on that layer during the scrolling process. The detection process can be as follows: a consecutive frame drop detection module is set up in the LayerDetector in the local service layer. This module sets up a sliding window that detects the frame images generated during the scrolling process and collects information about dropped frames at each detection. The sliding window then reports this information. Specifically, the electronic device detects the frame images generated during the scrolling process. When the first dropped frame image is detected, it is used as the starting point for the current window detection. The window then performs frame drop detection processing on the frame images within the sliding window. When the current window detection ends, the corresponding frame drop detection result is obtained. If the current window detection ends but the sliding operation continues, the first dropped frame image after the current window detection is used as the starting point for the next window detection, and the next window detection begins. This process is repeated until the sliding operation ends and detection stops. The dropped frame detection results obtained from each window detection are then stored in the kernel layer, for example, written to the ` / dev / hwlog_jank` file. The results are then transmitted from the local service layer (hiview) to the application framework layer (hiviewtunnel), and finally uploaded to the cloud by hiviewtunnel. This allows developers to determine whether continuous frame drops occur during sliding and assess the degree of stuttering during the sliding process based on the generated dropped frame detection results.

[0122] The sliding frame dropping detection method of this application will be specifically described below with reference to the accompanying drawings of specific embodiments.

[0123] Figure 5 The following is a flowchart illustrating a sliding frame dropping detection method provided in an embodiment of this application. This method is applied to an electronic device and specifically includes the following steps:

[0124] S501: Determine the length of the sliding window.

[0125] In some embodiments, the length of the sliding window can be determined based on the statistical duration of a single sliding operation. The statistical duration of a single sliding operation is obtained by statistically analyzing the duration of historical single sliding operations. It can be understood that the statistical duration of a single sliding operation in big data analysis is not limited to the historical duration of single sliding operations by only users of the electronic device, but can be determined by statistically analyzing the historical duration of single sliding operations by multiple users across the entire big data platform. Furthermore, the statistical duration of a single sliding operation in big data analysis can also be determined by statistically analyzing the historical duration of single sliding operations by only users of the electronic device. This allows for the determination of a sliding window length more suited to the sliding habits of different users, thereby improving the accuracy of frame drop detection.

[0126] In some embodiments, determining the length of the sliding window based on the statistically analyzed duration of a single sliding operation includes: obtaining a preset correspondence, which refers to the correspondence between the duration of a single sliding operation and a probability value; the probability value is used to characterize the probability of the duration of a single sliding operation occurring. Based on the preset correspondence, a duration with a probability value greater than a preset probability threshold is selected as the length of the sliding window.

[0127] In this embodiment, the preset correspondence can be represented by a sliding time curve. For example, the sliding time curve can be a cumulative distribution function curve, that is, the preset correspondence between the duration of a single sliding operation and the probability value can be represented by the cumulative distribution function curve.

[0128] In some examples, Figure 6 This is a schematic diagram of a cumulative distribution function curve provided in an embodiment of this application. Figure 6 The diagram shows the cumulative distribution function (CDF) curve of the duration of each user screen swipe, obtained from big data. Figure 6 This shows the specific duration of a user's single screen swipe. Figure 6 The horizontal axis represents the duration of a single swipe by the user, and the vertical axis represents the probability value. Assuming a preset probability threshold of 90%, and from... Figure 6 It can be seen that when the user continuously swipes the screen for less than or equal to 2 seconds, the probability value is 94%, which is greater than the preset probability value of 90%. Therefore, the probability that the user's screen swipe duration is within 2 seconds is 94%, meaning that the majority of the time the user swipes the screen is less than 2 seconds. In the embodiments of this application, the length of the swipe window can be set to 2 seconds.

[0129] Understandably, in this embodiment, based on the duration of a single swipe operation based on statistics, the duration of a single swipe operation with a high probability of occurrence can be selected as the length of the swipe window. Therefore, when the user swipes the screen once, a single window detection can be used to detect all the frame images generated during the swipe process, and the swipe situation during the entire swipe process can be determined by the frame loss detection result generated by the single window detection.

[0130] In other embodiments, the length of the sliding window can be set by the user, or it can be determined by the average time the electronic device takes for the user to swipe the screen. This application does not limit the method of setting the length of the sliding window.

[0131] In other examples, the electronic device counts the duration of each screen swipe by the user within a preset time (i.e., the duration of a single swipe operation), and takes the average of the counted durations to obtain the average screen swipe duration. This average duration is then used as the length of the sliding window. It should be noted that the electronic device determines the length of the sliding window by statistically analyzing the duration of each screen swipe with the user's authorization.

[0132] It can be understood that the length of the sliding window is used to determine the number of frame images detected in each window. That is, the frame images located within the same sliding window are the frame images that need to be detected in one window detection. It should be noted that since the length of the sliding window is determined in the time dimension, there may be a situation where only a portion of the frame images at the end of the sliding window fall within the sliding window. In this case, the frame images that fall into the sliding window at the end, along with all other frame images that fall within the sliding window, can be used together as the frame images that need to be detected in this window detection.

[0133] S502: Frame drop detection processing is performed on the frame images generated during the sliding process through a sliding window. In the frame drop detection processing, the frame drop images are marked and the continuity between the frame drop images is analyzed to obtain the frame drop detection result.

[0134] In some embodiments of this application, the frame loss detection processing of frame images generated during page scrolling via a sliding window includes: after the page starts scrolling, taking the first frame loss image during the page scrolling process as the detection starting point of the sliding window, and starting the current window detection, so as to perform frame loss detection processing on the frame images located within the sliding window during the current window detection.

[0135] Understandably, the first dropped frame image during the page scrolling process can be used as the starting point of the scrolling window. That is, the first dropped frame image during the page scrolling process is the first frame image of the first window detection, and the first dropped frame image and the preset number of frame images after the first dropped frame image can be used as the frame images that need to be detected for dropped frames in the first window detection.

[0136] To facilitate understanding, we will now combine Figure 7 A schematic explanation is provided on how to determine the detection starting point of the sliding window. Figure 7 This is a schematic diagram of a sliding interval provided in an embodiment of this application. Figure 7 A total of 17 frames are shown in the image. Figure 7 In the middle, when the third frame image is detected (i.e. Figure 7 When the frame image marked with the number 3 is detected, the third frame image is dropped. Therefore, the third frame image can be used as the starting point for detection within this sliding window; that is, the third frame image is used as the first frame image to be detected within this sliding window. Subsequently, from... Figure 7 It can be seen that the thirteenth frame image (i.e. Figure 7 If a portion of the frame image marked with the number 13 falls within the sliding window, then the thirteenth frame image can be used as the frame image for which frame loss detection needs to be performed in this window detection. That is, after the detection of the thirteenth frame image is completed, the frame loss detection results of the frame images between the third and thirteenth frames are reported as the frame loss detection results of this window detection.

[0137] In some embodiments, a large number of frames are generated during the scrolling process, but the number of frame images that the scrolling window can detect at one time is limited. If the first frame image generated during the scrolling process is used as the starting point for the scrolling window detection, or if the scrolling window is used to detect all frame images during the scrolling process, the number of times the scrolling window is detected will increase. Therefore, the amount of data to be statistically analyzed for dropped frames will also increase. However, frames that are not dropped during the scrolling process do not affect the stuttering during the scrolling process. Therefore, without affecting the stuttering analysis, the first dropped frame image during the page scrolling process can be used as the starting point for the scrolling window, reducing the number of times the scrolling window is detected and reducing the amount of data to be statistically analyzed for dropped frames, thereby reducing computational power.

[0138] In each window detection, detection information related to consecutive frame drops can be obtained by analyzing the continuity between dropped frame images. In some embodiments of this application, the frame drop detection result obtained by sliding window detection may include, in addition to the detection information related to consecutive frame drops, information about the dropped frame images, global statistical information about the dropped frames, a first value, or the length of the sliding window (which can be represented by detection duration). That is, in each window detection, in addition to analyzing the continuity between dropped frame images, the electronic device may also calculate the first value, calculate the information about the dropped frame images, and perform statistics on the global information of the dropped frames. The first value is a quantification value of the stuttering degree, or simply a stuttering degree quantification value.

[0139] In some embodiments of this application, the detection information related to continuous frame drops may include: frame drop type (which can be represented by ARG3), the longest number of consecutive dropped frames (which can be represented by longestContinuousDropFrames), the longest duration of consecutive dropped frames, etc.

[0140] in, Figure 8 This is a schematic diagram of a frame loss type provided in an embodiment of this application. Figure 8 There are two types of frame loss in China. Figure 8 Image (a) shows a frame image within a sliding window that does not contain consecutive dropped frames. Figure 8 (b) shows a frame image with consecutive frame drops within the sliding window. In the embodiments of this application, the frame drop type can be represented by a numerical value. For example, 0 and 1 can be used to represent the frame drop type during the sliding process. When the value is 0, it means that there is no frame image with consecutive frame drops within the sliding window (i.e., there are no consecutive frame drops). When the value is 1, it means that there is a frame image with consecutive frame drops within the sliding window (i.e., there are consecutive frame drops).

[0141] The longest consecutive frame drop count refers to the maximum number of consecutive frames dropped within a sliding window. Figure 8 In example (b), there are two consecutive frame drops within the sliding window, so the longest consecutive frame drop is 2.

[0142] The longest consecutive frame drop duration can be defined as the sum of the processing times (i.e., the time to generate the frame images) of the longest consecutive dropped frames within the sliding window. Figure 8 Taking (b) as an example, assuming the processing time for two consecutively dropped frames is 32 milliseconds and 35 milliseconds respectively, the longest consecutive dropped frame time is 57 milliseconds.

[0143] In some embodiments of this application, the information of the dropped frame image may include: the layer where the dropped frame image is located (which can be represented by ARG1), and related information of the dropped frame image (which can be represented by FrameInfo), such as: the ID of the dropped frame image and the processing time of each stage of the dropped frame image. Among them, the processing time of each stage of the dropped frame image may include: the processing time of drawing, rendering, compositing and other processes during the generation of the dropped frame image.

[0144] In some embodiments of this application, the global statistics of frame loss may include: cumulative frame loss duration (which can be represented by totalLostFrameTime), number of frame loss (which can be represented by lostFrameCount), etc.

[0145] The cumulative frame loss time refers to the total processing time of all dropped frames within the statistical sliding window. Figure 8 Taking (a) as an example, there are a total of 5 dropped frames in the sliding window. Then, determine the duration of generating these 5 dropped frames, that is, the processing time for generating these 5 dropped frames, and then add up the processing times corresponding to the 5 dropped frames to get the cumulative dropped frame duration.

[0146] The number of dropped frames can refer to the number of dropped images within a sliding window. Similarly, using... Figure 8 Taking (a) as an example, the number of frames dropped in the sliding window is 5.

[0147] In some embodiments of this application, the frame loss detection result may also include a first numerical value.

[0148] In traditional methods, some parameters that measure the degree of stuttering during scrolling differ from the degree of stuttering actually perceived by the user. However, in this embodiment, the degree of stuttering during scrolling can be accurately analyzed and quantified based on the frame drop detection in the scrolling window. That is, a first value can be obtained based on the frame drop detection results reported by the scrolling window, and the quantified result is close to the degree of stuttering actually perceived by the user.

[0149] Therefore, the first metric can measure the degree of lag during screen swiping from the user's perspective. The more frames dropped, the more noticeable the lag in the screen display becomes to the user; thus, frame drops can reflect the user's perceived lag during screen display.

[0150] When the first value is greater than the preset threshold, it means that the lag during the user's current screen swipe is perceptible to the user. When the first value is less than or equal to the preset threshold but greater than 0, it means that frame drops have occurred during the user's current screen swipe, but the lag caused by frame drops is not obvious and is not easily perceived by the user.

[0151] To facilitate understanding, we will now explain the first value with more specific examples. For instance, the preset threshold can be set to 0.25. If the first value corresponding to the current user's screen swipe is 0.56, it means that the user can perceive the lag during the current user's screen swipe. If the first value corresponding to the current user's screen swipe is 0.06, it means that the user is less likely to perceive the lag during the current user's screen swipe.

[0152] In this embodiment, the determination of the first value may include: in the frame loss detection process, determining the first value based on the processing time of each lost frame image, the weight of each lost frame image, and the length of the sliding window. The weight of each lost frame image is positively correlated with the order in which the lost frames are processed.

[0153] Understandably, each frame drop detection result obtained by the sliding window has a corresponding first value. This first value is determined based on the processing time and weight of the dropped frame image during each sliding window detection process, so that the first value can reflect the user's perception of lag during each sliding window detection process.

[0154] In some embodiments, the first value may be a standardized frame drop time, determined based on the processing time of each dropped frame image, the weight of each dropped frame image, and the length of the sliding window. This includes determining the standardized frame drop time based on the ratio between the second value and the length of the sliding window. The second value is obtained by weighted summation of the processing times corresponding to each dropped frame image according to their respective weights.

[0155] The specific calculation formula can be: Where n is the number of dropped frames within the sliding window, t i The processing time τ for each dropped frame image i The weights for each dropped frame are calculated based on the sliding curve, where T is the length of the sliding window. d To standardize the frame drop time.

[0156] Furthermore, the calculation method for the weight of each dropped frame image may include: determining a sliding curve, which characterizes the pixel changes of each frame image compared to the previous frame image during page scrolling. Specifically, in the sliding curve, the amount of pixel change in a frame image is negatively correlated with the order in which the frames are generated during page scrolling. Based on the sliding curve and the order in which the dropped frames are generated during page scrolling, the weight of each dropped frame image is determined.

[0157] Understandably, at the beginning of the swipe, due to the rapid initial swipe speed, the changes in the frame images are more noticeable, and the pixel changes between each frame are also more pronounced. Therefore, at the beginning of the swipe, if frame drops occur, the user will be more likely to perceive the stuttering, and the weight of the dropped frame image is greater. However, as the swipe gradually stops, the frame images on the screen no longer change; that is, they may remain on a certain frame image. At this point, the pixel changes in the frame images cease, and the pixel changes between each frame become less noticeable. Therefore, towards the end of the swipe, the pixel changes in the frame images are less, and the swipe speed gradually slows down. At this point, if frame drops occur, the user is less likely to perceive the stuttering, and the weight of the dropped frame image is smaller.

[0158] Therefore, the order of the frame images generated during the sliding process can be numbered. The earlier the frame number, the greater the weight of the frame image, and the later the frame number, the smaller the weight of the frame image.

[0159] In some examples, the sliding curve of the electronic device can be determined first, and then the frame number of the dropped frame image during the sliding process can be determined. The weight corresponding to the frame number of the frame image can be determined based on the sliding curve. Figure 9 A schematic diagram of a sliding curve provided in an embodiment of this application, such as... Figure 9 As shown, the sliding curve is a decreasing exponential function. Figure 9 The horizontal axis represents the frame number, and the vertical axis represents the number of pixels that changed in the frame. The number of pixels that changed in a frame represents the weight of that frame.

[0160] In some embodiments, the formula for the sliding curve can be y = a·e bx The sliding curve is determined by the manufacturer of the electronic device, meaning that the values ​​of a and b may differ for different electronic devices. For example, a = 16.317, b = -0.046.

[0161] In some embodiments of this application, a single window detection may not be sufficient to detect all frame images generated during the sliding process, so multiple (at least two) window detections are required. Specifically, if the current window detection ends but the sliding operation does not, the electronic device can use the first dropped frame image after the current window detection as the detection starting point for the sliding window in the next window detection, and take the next window detection as the new current window detection, returning to execute the current window detection to perform dropped frame detection processing on the frame images located within the sliding window until the sliding operation ends. Here, the first dropped frame image after the current window detection refers to the first frame image with dropped frames among the frame images generated after the current window detection ends.

[0162] For example, when using a sliding window to sequentially detect frame images generated during the sliding process, the first dropped frame image (i.e., the frame image where the first dropped frame occurs) can be determined as the detection starting point for this window detection. That is, this dropped frame image is used as the first frame image to be detected within the current sliding window. After marking the dropped frame image, the detection of dropped frames in subsequent frame images within the sliding window continues until all frame images within the sliding window have been detected, at which point the current window detection is considered complete. Furthermore, the electronic device can report the dropped frame detection result of this window detection to the cloud and use the sliding window to start the next window detection. That is, the detection starting point of the sliding window is determined for the next window detection, and the dropped frame detection processing of the frame images within the sliding window continues. This will not be elaborated further.

[0163] To facilitate understanding, the following example will illustrate multiple window detections.

[0164] Assuming the user's swipe operation lasts for 3 seconds (from start to finish), and 20 consecutive frames are generated within this timeframe, forming a swipe interval, the swipe window begins detection from the first frame. If the first frame is dropped, it becomes the starting point for the current window detection, i.e., the first frame to be detected within the current swipe window. If, after the eleventh frame is detected, the current detected frame is within a preset swipe window length from the starting point, the current window detection is considered complete. The electronic device then generates the dropped frame detection results for frames between the first and eleventh frames. If the swipe operation is not yet finished and frames twelfth to twentieth are still generated, the electronic device can then determine whether frames after the twelfth frame are dropped. Assuming no frames are dropped from the twelfth to the sixteenth frame, but a frame is dropped in the seventeenth frame, the seventeenth frame is used as the starting point for the sliding window's second detection. Detection is then performed on frames after the seventeenth frame. The sliding operation ends and the detection by the sliding window also ends after the twentieth frame is detected. It can be understood that during this sliding process, the sliding window generates two dropped frame detection results, and the starting point for the second detection is the first dropped frame after the first detection, i.e., the seventeenth frame. The end time of the second detection is determined by the end time of the sliding operation.

[0165] It should be noted that, in the embodiments of this application, the electronic device performs frame drop detection processing in real time during the user's swiping operation. That is, while receiving the user's swiping operation, the electronic device also generates multiple consecutive frame images based on the user's swiping operation, and while these frame images are displayed on the screen, the electronic device also synchronously performs frame drop detection on the displayed frame images.

[0166] Figure 10 This is a schematic diagram of a frame loss detection process provided in an embodiment of this application. Figure 10 In this context, the detection process for a sliding window can be as follows:

[0167] (1) Slide to start.

[0168] Once the swipe operation begins, the electronic device responds to the user's swipe action by generating a series of frames that are displayed on the screen in real time.

[0169] (2) Statistical analysis of information in the current frame.

[0170] The electronic device uses a sliding window to sequentially determine whether a frame is missing from the displayed frames on the screen. In other words, the electronic device can check the current frame (i.e.,...) Figure 10 The information of the "current frame" in the image is statistically analyzed, for example, the processing time of the current frame image is statistically analyzed. It can be understood that the statistical information of the current frame image can be used as the basis for judging whether a frame is lost in (3).

[0171] (3) Determine if the current frame is dropped.

[0172] In some embodiments, whether a frame is lost can be determined based on the information of the current frame image statistically obtained in (2). Specifically, the method for determining whether a frame image is lost may include: in the frame loss detection process, for each frame image to be detected, the processing time of the frame image to be detected is obtained. The processing time of the frame image to be detected refers to the actual time consumed in processing the frame image to be detected. If the difference between the processing time and the frame period exceeds a preset time threshold, the frame image to be detected is determined to be a frame loss image. The preset time threshold may be 25 milliseconds. It can be understood that the actual time consumed in processing the frame image to be detected may refer to the time generated by the process of generating the frame image. The process of generating the frame image may include processes such as drawing, rendering, and compositing. Therefore, the processing time of the frame image to be detected may include the time generated by the processes of drawing, rendering, and compositing when generating the frame image.

[0173] If no frames are dropped in the current frame, determine whether the sliding window detection has started. If not (i.e., the sliding window detection has not started), the electronic device can perform frame drop detection on the next frame of the current frame, that is, take the next frame as the new current frame and continue to return to execute the statistical information of the current frame and subsequent steps; if yes (i.e., the sliding window detection has been detected), the electronic device can execute (4) to determine whether the current sliding window detection has ended.

[0174] If the current frame image is missing a frame, the electronic device can determine whether the missing frame image is the first missing frame image detected by the sliding window in this process, that is, the first missing frame image detected by the sliding window to be started. If it is the first missing frame image, the electronic device can use the missing frame image as the starting point of the sliding window (also called the detection starting point of this sliding window detection), that is, the frame image is the first frame image located in the sliding window during this window detection, and the frame image and the frame images after the frame image and the preset number of frame images are used as the frame images detected in this sliding window, so as to detect whether the frame images in the sliding window are missing frames, and mark the frame images with missing frames as missing frames (that is, mark the missing frame images). Then, the electronic device can execute (4) to determine whether the detection of this sliding window has ended.

[0175] (4) Determine whether the current sliding window detection has ended.

[0176] If the current sliding window detection has ended, report the frame drop detection results of the sliding window, clear the frame drop detection results, and prepare to start the next window detection.

[0177] It is understandable that after preparing to start the next window detection but before officially starting the next window detection, the electronic device can determine whether the sliding has ended, that is, execute (5).

[0178] (5) Determine whether the sliding has ended.

[0179] If the sliding does not end, then the next frame image of the current frame image is subjected to frame loss detection, that is, the next frame image is taken as the new current frame image, and the process continues to return to execute (2) to count the information of the current frame and subsequent steps.

[0180] If the sliding ends, then execute (6).

[0181] (6) Determine whether there are reported frame loss detection results.

[0182] If there are reported frame drop detection results, report the frame drop detection results, and then end the continuous frame drop detection.

[0183] If no frame loss detection results are reported, the continuous frame loss detection will end directly.

[0184] The following example illustrates the execution process of the sliding frame dropping detection method of this application.

[0185] Take swiping on a page in shopping application A as an example. Figure 11 Another schematic diagram of the sliding range provided for an embodiment of this application, as shown below. Figure 11 As shown, when the electronic device receives a swipe gesture from the user onto the page of shopping application A, it controls the page of shopping application A to change accordingly. Therefore, from the start to the end of the swipe, 15 consecutive frame images will be generated, and these 15 frame images will constitute a swipe interval. The electronic device can first determine the processing time for generating the first frame image, and then determine that the difference between the processing time of the first frame image and the frame period is less than 25 milliseconds. Therefore, the first frame image does not drop any frames.

[0186] Since the sliding operation is not yet complete, the processing time for generating the second frame image is determined. At this point, the difference between the processing time of the second frame image and the frame period exceeds 25 milliseconds, meaning that a frame loss is detected for frame image id 11366304. Therefore, it is determined that the second frame image has a frame loss. The electronic device can use the second frame image as the detection starting point of this sliding window; that is, it can use the second frame image as the first frame image to be detected within this sliding window and begin the detection of this sliding window. The electronic device can further mark the second frame image to characterize the frame loss.

[0187] Subsequently, since the window detection and sliding operation have not yet ended, the electronic device can continue to determine the processing time for generating the third frame image, and so on, until the processing time for generating the sixth frame image is determined. The difference between the processing time of the third and sixth frames and the frame period is less than 25 milliseconds; therefore, no frames are dropped in the third and sixth frames.

[0188] Subsequently, since the sliding window detection is not yet complete, the processing time for generating the seventh frame image is determined. At this point, if the difference between the processing time of the seventh frame image and the frame period exceeds 25 milliseconds, it is determined that the seventh frame image has been dropped; that is, a dropped frame image with ID 11366326 is detected. The electronic device can mark the seventh frame image to characterize the dropped frame.

[0189] Subsequently, since the sliding window detection and sliding operation have not yet ended, the electronic device can continue to determine the processing time for generating the eighth frame image. At this point, if the difference between the processing time of the eighth frame image and the frame period exceeds 25 milliseconds, it is determined that the eighth frame image has dropped frames; that is, a dropped frame is detected for frame image with ID 11366331. The electronic device can mark the eighth frame image to characterize the dropped frames. Since both the seventh and eighth frame images have dropped frames, and the frame images are consecutive, the seventh and eighth frame images are considered to have dropped frames consecutively, and are marked as consecutive dropped frames in this window detection.

[0190] The window detection ends when the detected frame image's position exceeds the length of the sliding window from the detection starting point. The electronic device can then analyze the frame images targeted by this window detection to obtain the frame drop detection results. For example, the electronic device can analyze the continuity between the marked frame drop images within the sliding window during this window detection, obtain the frame drop detection results, and report them to the cloud.

[0191] After the current window detection is completed, since the sliding operation is not yet finished, after reporting the frame drop detection results of this window detection, the reporting record is cleared, and the processing time for the next frame image generated during the sliding process is determined. If another frame drop occurs, the dropped frame image is used as the detection starting point for the next window detection, and the next window detection is performed. This process is repeated until the sliding operation is completed.

[0192] Figure 12 The embodiments provided in this application are related to Figure 11 The corresponding diagram shows the frame drop detection results. Since the frame images generated during the sliding process are all generated on the layer of shopping application A, the layer containing the frame drop images detected by the sliding window is the layer of shopping application A.

[0193] Electronic devices can calculate the normalized frame drop time corresponding to the sliding window. Specifically, the formula for the sliding curve can be set as y = a·e bx Given a = 16.317 and b = -0.046, the value is 0.56.

[0194] Depend on Figure 11It can be seen that since the seventh and eighth frame images in the frame drop detection results reported by the sliding window show consecutive frame drops, the frame drop type value is 1, indicating that there are consecutively dropped frame images within the sliding window. Since the second frame image also has frame drops, the number of dropped frames in the sliding window is 3 (i.e., the second, seventh, and eighth frame images are all dropped). The electronic device can add the frame drop times of the second, seventh, and eighth frame images to obtain the cumulative frame drop time, which is 90 milliseconds. The length of the sliding window is 2000 milliseconds (i.e., 2 seconds). The information for all dropped frame images includes the frame IDs corresponding to the second, seventh, and eighth frame images, as well as the processing time for drawing, rendering, and compositing during the generation of the second, seventh, and eighth frame images.

[0195] In some other embodiments of this application, assuming the sliding window performs at least two window detections and obtains at least two frame drop detection results, the frame drop detection results corresponding to the at least two window detections can be analyzed simultaneously to determine whether there are consecutive frame drops between the frame images at the critical position detected in any two window detections, and then the analysis results are uploaded to the cloud. For example: in the first window detection, frame drop detection processing is performed on the second to fifth frame images generated during the sliding process, where the second and fifth frame images are both frame drop images. In the first window detection, frame drop detection processing is also performed on the sixth to ninth frame images generated during the sliding process, where the sixth frame image is a frame drop image. It can be seen that the fifth and sixth frame images are frame images at the critical position. Therefore, from the two frame drop detection results generated by the sliding window, it can be seen that the fifth and sixth frame images are in a relationship of consecutive frame drops.

[0196] S503: Report frame loss detection results.

[0197] In the embodiments of this application, the frame drop detection results generated during the sliding window detection process can be statistically analyzed by hiview and transmitted to hiviewtunnel, which then uploads them to the cloud.

[0198] Other embodiments of this application provide an electronic device that may include: the aforementioned screen (such as a touchscreen or display screen), a memory, and one or more processors. The screen, memory, and processor are coupled. The memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device performs the various functions or steps in the above method embodiments. The structure of the electronic device can be referred to... Figure 3 The structure of the electronic device 300 shown.

[0199] This application also provides a computer storage medium that includes computer instructions. When the computer instructions are executed on the electronic device, the electronic device performs various functions or steps performed by the electronic device in the above method embodiments.

[0200] This application also provides a computer program product that, when run on a computer, causes the computer to perform various functions or steps performed by the electronic device in the above method embodiments.

[0201] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0202] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0203] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0204] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0205] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0206] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of detecting a slide drop frame, the method comprising: The method is applied to an electronic device including a screen, and the method includes: In a process of controlling page sliding in the screen in response to a sliding operation, after the page starts to slide, a first frame loss image in the page sliding process is taken as a detection starting point of a sliding window, and a current window detection is started, so that frame loss detection processing is performed on frame images located in the sliding window in the current window detection; If the current window detection ends but the sliding operation does not end, a first frame loss image after the current window detection is taken as a detection starting point of the sliding window in next window detection, and the next window detection is taken as a new current window detection, and the frame loss detection processing on the frame images located in the sliding window in the current window detection is returned to be performed until the sliding operation ends; wherein the first frame loss image after the current window detection refers to a first frame image with frame loss in frame images generated after the current window detection ends; In the frame loss detection processing, frame loss images are marked, and continuity between the frame loss images is analyzed to obtain a frame loss detection result; The frame loss image refers to a frame image with frame loss in the sliding window in the frame loss detection processing.

2. The method of claim 1, wherein, Before the frame loss detection processing on the frame images located in the sliding window, the method further includes: According to a duration of a single sliding operation, a length of the sliding window is determined; the duration of the single sliding operation is obtained by counting durations of historical single sliding operations.

3. The method of claim 2, wherein, The determination of the length of the sliding window according to the duration of the single sliding operation includes: A preset correspondence is obtained; the preset correspondence refers to a correspondence between a duration of a single sliding operation and a probability value; the probability value is used to represent an occurrence probability of the duration of the single sliding operation; According to the preset correspondence, the duration with a probability value greater than a preset probability threshold is selected as the length of the sliding window.

4. The method of claim 1, wherein, The marking of the frame loss images in the frame loss detection processing includes: In the frame loss detection processing, for each frame image to be detected, a processing duration of the frame image to be detected is obtained; the processing duration of the frame image to be detected refers to an actual duration consumed for processing the frame image to be detected; In a case where a difference between the processing duration and a frame period exceeds a preset duration threshold, the frame image to be detected is determined as a frame loss image; the frame period refers to a duration required for generating a frame image under normal circumstances.

5. The method according to any one of claims 1 to 4, characterized in that, The analysis of the continuity between the frame loss images is used to obtain detection information related to continuous frame loss; the frame loss detection result includes at least one of information of the frame loss images, global statistical information of frame loss, a first value, or the length of the sliding window, in addition to the detection information related to the continuous frame loss.

6. The method according to any one of claims 1 to 4, characterized in that, The frame loss detection result includes the first value; the method further includes: In the frame loss detection processing, the first value is determined based on a processing duration of each of the frame loss images, a weight of each of the frame loss images, and a length of the sliding window; the first value is a quantization value of a frame freezing degree. The weight of each of the frame loss images is positively correlated with a sequence order of each of the frame loss images.

7. The method of claim 6, wherein, The first value includes a standardized frame loss time. The determination of the first value based on the processing duration of each of the frame loss images, the weight of each of the frame loss images, and the length of the sliding window includes: The standardized frame loss time is determined according to a ratio between a second value and the length of the sliding window; the second value is a value obtained by weighting and summing the processing durations of the frame loss images according to the corresponding weights. Before the determination of the first value based on the processing duration of each of the frame loss images, the weight of each of the frame loss images, and the length of the sliding window, the method further includes:

8. The method of claim 6, wherein, A sliding curve is determined, the sliding curve being used to represent a pixel change of each frame image compared with a previous frame image in the page sliding process; in the sliding curve, a pixel change amount of the frame image is negatively correlated with a sequence order of the frame image in the page sliding process; The weight of each of the frame loss images is determined based on the sliding curve and the sequence order of each of the frame loss images in the page sliding process. The electronic device includes a memory, a screen, and a processor; the memory and the screen are coupled with the processor; the memory stores computer program code, the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method in any one of claims 1 to 8.

9. An electronic device, comprising: The computer instructions, when executed on an electronic device, cause the electronic device to execute the method in any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The computer program product, when executed on a computer, causes the computer to execute the method in any one of claims 1 to 8.

11. A computer program product, characterised in that, ​

Images