Video stutter detection method and related apparatus

Abstract

Embodiments of the present application relate to the technical field of terminals, and provide a video stutter detection method and a related apparatus. The method comprises: an electronic device displays a video playback interface, the video playback interface comprising a first control and a second control, the first control being used for displaying a first-type view, the second control being used for displaying a second-type view, the first-type view not comprising a view locally loaded from the electronic device, and the second-type view comprising a view locally loaded from the electronic device; and when the frame interval of the view displayed in the first control is greater than or equal to a preset value, the electronic device determines that a video stutter occurs, wherein the frame interval of the view displayed in the second control is less than, equal to or greater than a preset value. Thus, the accuracy of determining whether a stutter occurs on a video application interface can be improved, thereby reducing stutters on the video application interface.

Description

Video freeze detection method and related device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 15, 2023, with application number 202310715421.1 and application name “Video Freeze Detection Method and Related Devices”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a video freeze detection method and related devices. Background Art

[0003] With the development of electronic technology, some electronic devices support video playback. Users can watch videos, shoot videos, and start live broadcasts through video applications. However, when electronic devices play the content of video applications, the interface of the electronic devices may become stuck, affecting the user experience.

[0004] Summary of the Invention

[0005] The video freeze detection method and related devices provided in the embodiments of the present application enable the electronic device to perform frame rate refresh detection on drawing controls that do not load local data when detecting freezes in the video application interface, thereby being unaffected by controls that load local data, improving the accuracy of determining whether the video application interface is freezed, and further helping to reduce freezes in the video application interface.

[0006] In a first aspect, the embodiments of the present application provide a method for detecting video freeze, the method comprising:

[0007] An electronic device displays a video playback interface, which includes a first control and a second control. The first control is used to display a first type of view, and the second control is used to display a second type of view. The first type of view does not include views loaded locally from the electronic device; the second type of view includes views loaded locally from the electronic device. When the frame interval of the view displayed in the first control is greater than or equal to a preset value, the electronic device determines that the video is stuck. The frame interval of the view displayed in the second control is less than, equal to, or greater than the preset value. This can improve the accuracy of determining whether the video application interface is stuck, thereby helping to reduce stuck video application interfaces.

[0008] In one possible implementation, before the electronic device determines that a video freeze has occurred, the electronic device further determines that no preset events have occurred in the video playback interface. The preset events include one or more of the following: an event in which a progress bar control in the video playback interface is triggered, an event indicating that the video playback interface is paused, and an event indicating that the video playback interface is switched to the background. In this way, it can be determined whether the current interface has been paused due to an event triggered by the user, thereby more accurately determining whether the interface has freezed.

[0009] In one possible implementation, the preset value includes a first threshold value, and before the electronic device determines that the video has been stuck, it also includes: the electronic device determines that the layer texture identifier of the view displayed in the first control has not changed; when the frame interval of the view displayed in the first control is greater than or equal to the preset value, the electronic device determines that the video has been stuck, including: when the frame interval of the view displayed in the first control is greater than or equal to the first threshold value, and no event of the progress bar control being triggered in the video playback interface, and / or an event for indicating the pause of the video playback interface occurs, the electronic device determines that the video has been stuck. In this way, it can be determined that the interval time between two frames of images exceeds the threshold value due to the user's pause operation, thereby accurately judging whether the video has been stuck.

[0010] In one possible implementation, the preset value includes a second threshold value, and before the electronic device determines that the video is stuck, it also includes: the electronic device determines that the layer texture identifier of the view displayed in the first control has changed; when the frame interval of the view displayed in the first control is greater than or equal to the preset value, the electronic device determines that the video is stuck, including: when the frame interval of the view displayed in the first control is greater than or equal to the second threshold value, and no event occurs in the video playback interface to indicate switching the video playback interface to the background, the electronic device determines that the video is stuck. In this way, it can be further determined that the interval time between two frames of images exceeds the threshold value due to the user's background switching operation, thereby accurately judging whether the video is stuck.

[0011] In one possible implementation, the electronic device determines that a video freeze occurs, including: the electronic device records a log related to the freeze, so as to facilitate further analysis and statistics of the freeze.

[0012] In one possible implementation, the logs related to the jamming include one or more of the following: scene resource logs, scene event logs. In this way, the electronic device records different types of logs to facilitate subsequent different types of analysis of the jamming, thereby specifically analyzing and solving certain types of problems.

[0013] In one possible implementation, the electronic device determines that a video freeze has occurred by performing processing related to preset performance indicators, where the preset performance indicators include one or more of the following: CPU utilization of the electronic device's process, memory request duration of the electronic device, and network signal strength of the electronic device. Thus, after the electronic device performs processing related to the preset performance indicators, the performance of the electronic device can be improved, thereby helping to reduce video interface freezes.

[0014] In one possible implementation, the electronic device performs processing related to a preset performance indicator, including: determining that a process CPU utilization is abnormal; performing tests related to the process CPU utilization, including one or more of the following: frequency limit detection, scheduling priority detection, and inter-core runtime detection; recording a log related to the process CPU utilization; increasing the CPU operating frequency and / or raising the priority of the process; determining that a memory request duration is abnormal; performing tests related to the memory request duration, including one or more of the following: available memory detection and slow path request count detection; recording a log related to the memory request duration; clearing the electronic device's memory; determining that a network signal is abnormal; performing tests related to the network signal, including one or more of the following: network delay detection and network packet loss detection; recording a log related to the network signal; and switching the network signal. This can improve the performance of the electronic device, thereby helping to reduce video interface lag.

[0015] In one possible implementation, the method further includes: when the electronic device determines that a preset condition is met, the electronic device records the frame time; when the electronic device determines that a preset event occurs in the video playback interface, the electronic device sets the frame time to a target value; the electronic device determines that buffered data related to the video playback exists, including: when the frame time is not the target value, the electronic device determines that buffered data related to the video playback exists; wherein, meeting the preset condition includes any of the following: no buffered data related to the video playback exists; there is buffered data related to the video playback, and the frame interval of the view displayed in the first control is less than a preset value. In this way, the electronic device determines whether there is buffered data related to the video playback, which can reduce unnecessary execution steps and improve code execution efficiency.

[0016] In one possible implementation, the first control includes a texture view control or an independent layer view control. In this way, the accuracy of determining whether the video application interface is stuck can be improved, which helps to reduce the stuckness of the video application interface.

[0017] In a second aspect, embodiments of the present application provide a device for detecting video freezes. The device may be an electronic device or a chip or chip system within an electronic device. The device may include a processing unit and a display unit. The processing unit is configured to implement any processing-related method performed by the electronic device in the first aspect or any possible implementation of the first aspect. The display unit is configured to implement any display-related method performed by the electronic device in the first aspect or any possible implementation of the first aspect. When the device is an electronic device, the processing unit may be a processor. The device may also include a storage unit, which may be a memory. The storage unit is configured to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement the method described in the first aspect or any possible implementation of the first aspect. When the device is a chip or chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to cause the electronic device to implement the method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit within the chip (e.g., a register, a cache, etc.) or a storage unit within the electronic device located outside the chip (e.g., a read-only memory, a random access memory, etc.).

[0018] Exemplarily, the processing unit is configured to determine whether a video freeze occurs. The display unit is configured to display a video playback interface.

[0019] In a possible implementation, the processing unit is configured to determine that no preset event occurs in the video playback interface.

[0020] In one possible implementation, the processing unit is used to determine that the layer texture identifier of the view displayed in the first control has not changed, and is also used to determine that a video freeze occurs.

[0021] In one possible implementation, the processing unit is used to determine whether a layer texture identifier of a view displayed in the first control has changed, and is also used to determine whether a video freeze has occurred.

[0022] In one possible implementation, the processing unit is used to record logs related to lag.

[0023] In one possible implementation, the logs related to lag include one or more of the following: scene resource logs and scene event logs.

[0024] In a possible implementation, the processing unit is configured to execute processing related to a preset performance indicator.

[0025] In one possible implementation, the processing unit is used to determine whether an abnormality has occurred in the CPU utilization of a process, and is also used to perform detections related to the CPU utilization of the process, and is also used to record logs related to the CPU utilization of the process, and is specifically used to increase the operating frequency of the CPU, and / or increase the priority of the process; is used to determine whether an abnormality has occurred in the memory application duration, and is also used to perform detections related to the memory application duration, and is also used to record logs related to the memory application duration, and is specifically used to clear the memory of the electronic device; is used to determine whether an abnormality has occurred in the network signal, and is also used to perform detections related to the network signal, and is also used to record logs related to the network signal, and is specifically used to switch the network signal.

[0026] In one possible implementation, the processing unit is configured to record the frame time, set the frame time as a target value, and specifically determine whether there is buffered data related to the played video.

[0027] In one possible implementation, the first control includes a texture view control or an independent layer view control.

[0028] In a third aspect, an embodiment of the present application provides a terminal device, comprising a processor and a memory, wherein the memory is used to store code instructions, and the processor is used to run the code instructions to execute the method described in the first aspect or any possible implementation of the first aspect.

[0029] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is run on a computer, the computer executes the method described in the first aspect or any possible implementation of the first aspect.

[0030] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a computer program, which, when the computer program runs on a computer, enables the computer to execute the method described in the first aspect or any possible implementation of the first aspect.

[0031] In a sixth aspect, the present application provides a chip or chip system, comprising at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a line, and the at least one processor is configured to execute a computer program or instruction to perform the method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip may be an input / output interface, a pin, or a circuit.

[0032] In one possible implementation, the chip or chip system described above in this application further includes at least one memory, in which instructions are stored. The memory may be a storage unit within the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0033] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] FIG1 is a schematic diagram of an interface of a video application provided in an embodiment of the present application;

[0035] FIG2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0036] FIG3 is a schematic diagram of a software structure of an electronic device provided in an embodiment of the present application;

[0037] FIG4 is a schematic diagram of module interaction of a video freeze detection method provided by an embodiment of the present application;

[0038] FIG5 is a flow chart of a method for detecting video freeze provided in an embodiment of the present application;

[0039] FIG6 is a schematic diagram of a video freeze detection method provided by an embodiment of the present application;

[0040] FIG7 is a schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] To facilitate a clear description of the technical solutions of the embodiments of the present application, some of the terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0042] 1. Terminology

[0043] In the embodiments of this application, terms such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the terms "first chip" and "second chip" are used solely to distinguish between different chips and do not define their order. Those skilled in the art will understand that terms such as "first" and "second" do not define the quantity or execution order, and do not necessarily define differences.

[0044] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0045] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0046] With the development of electronic technology, some electronic devices support video playback. Users can watch videos, shoot videos, and even start live broadcasts through video applications. However, when playing video application content, the electronic device interface may experience lag, affecting the user experience. Therefore, electronic devices are increasingly focusing on detecting video application lag. This allows electronic devices to take appropriate measures when lag is detected, thereby reducing video application lag.

[0047] In one possible implementation of detecting application jams, the electronic device can detect application jams based on a detection method for software drawing frames. The detection method may include counting the time intervals of frame images drawn on the application interface. For example, the electronic device can call the doframe() method for drawing frame images to determine whether the application is jammed. The electronic device can determine in the doframe() method for drawing frame images whether the time interval between the current frame image and the drawing of the previous frame image exceeds a preset threshold. If it exceeds the preset threshold, it can be considered that the application interface is jammed. Otherwise, it can be considered that the application interface is not jammed.

[0048] Another possible implementation for detecting app lag is for electronic devices to detect app lag based on the layer blending service SurfaceFlinger. For example, when an electronic device cannot deliver frames stably, the update interval between the previous and next layers may be unstable for a period of time, causing the application interface to lag. The SurfaceFlinger service can count the update intervals of the layers. If the update interval of the layer exceeds the preset value, the application interface can be considered to have lag. Otherwise, the application interface can be considered to have no lag.

[0049] However, the two implementations of detecting application freezes can be understood as detecting based on the update frequency of the application interface. However, it is impossible to detect the scenario where the application interface is still refreshing but some content in the application interface is not displayed.

[0050] For example, FIG1a shows the interface of a video application, which includes a video playback control 100 and an avatar display control 101. The avatar display control 101 can display animation effects. When the video application interface freezes, in some scenarios, the video playback control 100 may not display any content, but the avatar display control 101 may still display animation effects, which may be animation effects locally loaded by the video application.

[0051] It is understandable that, as shown in FIG1 b, in the above-mentioned stuck scene, the video playback control 100 may not be able to refresh the video at a certain frame rate due to reasons such as poor network signal, the electronic device's CPU lacking resources to parse network data packets, and problems with the electronic device's codec, thereby causing the video playback control 100 to experience playback stuck. However, since the animation effect of the avatar display control 101 is an animation effect loaded locally by the video application, it does not involve the network environment and does not require the electronic device to parse network data packets. Therefore, the video application interface can still continue to refresh the avatar display control 101 at a certain frame rate. In other words, the refresh rate of the video application interface remains unchanged, and the electronic device is still drawing frame images at a certain time interval.

[0052] When the above-mentioned detection method based on the software drawing framework is used to detect the freeze of the video application, since the detection method determines that the time interval between drawing the frame image of the current video application interface and drawing the previous frame image does not exceed the preset threshold, it is considered that there is no freeze in the interface of the video application.

[0053] When the above-mentioned detection method based on the layer blending service SurfaceFlinger is used to detect the jamming of the video application, since the detection method determines that the update time interval between the previous layer and the next layer of the video application interface does not exceed the preset value, it is considered that there is no jamming in the video application interface.

[0054] It can be seen that the above two detection methods cannot accurately detect whether the interface of the video application is stuck, and thus the stuck is not processed, resulting in more frequent stucks in the video application interface.

[0055] In view of this, the video freeze detection method provided in the embodiment of the present application can detect the frame rate refresh of the drawing controls that do not load local data when the electronic device detects the freeze of the video application interface, so that it will not be affected by the controls that load local data, thereby improving the accuracy of judging whether the video application interface is freezed, and further helping to reduce the freeze of the video application interface.

[0056] It can be understood that the electronic device of the embodiment of the present application can also be a terminal device in any form. For example, the electronic device can include a handheld device, a vehicle-mounted device, etc. For example, some electronic devices include: mobile phones, tablet computers, PDAs, laptop computers, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving cars, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, in-vehicle devices, wearable devices, electronic devices in 5G networks or future evolved public land mobile communication networks (PLMNs). The electronic devices in the network (PLMN) are not limited to this in the embodiments of the present application.

[0057] As an example and not a limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0058] In addition, in the embodiment of the present application, the electronic device can also be an electronic device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0059] The electronic devices in the embodiments of the present application may also be referred to as: user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0060] In the embodiments of the present application, the electronic device or each network device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also known as main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as the Linux operating system, the Unix operating system, the Android operating system, the iOS operating system, or the Windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0061] For example, FIG2 shows a schematic structural diagram of an electronic device.

[0062] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0063] It is understood that the structures illustrated in the embodiments of the present invention do not constitute specific limitations on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0064] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0065] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0066] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0067] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface.

[0068] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present invention is only a schematic illustration and does not constitute a structural limitation of the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0069] The internal memory 121 can be used to store computer executable program code, and the executable program code includes instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor. For example, the method of an embodiment of the present application can be executed.

[0070] The electronic device implements display functionality through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information. The electronic device can implement camera functionality through an ISP, camera 193, video codec, GPU, display screen 194, and application processor.

[0071] Figure 3 is a block diagram of the software structure of an electronic device according to an embodiment of the present application. The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0072] The application layer can include a series of application packages. As shown in Figure 3, the application packages can include applications such as video, live broadcast, and video call. Applications can include system applications and third-party applications.

[0073] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0074] As shown in Figure 3, the application framework layer may include a view system, a window processing device, a performance logging interface, a content provider, and a resource manager. In addition, the application framework layer may also include a drawing management class for view controls, a freeze log system, and the like.

[0075] The view system can be used for application interface drawing and event processing. The view system can include texture view controls and independent layer view controls.

[0076] The window processing device can be used to obtain relevant data of the electronic device window interface, such as the window name, etc. The window processing device can also detect freeze events.

[0077] The performance logging interface can be used to call the jamming related interface to write jamming information to the performance jamming log virtual device in the kernel layer.

[0078] Content providers are used to implement data sharing between different applications, allowing one program to access data in another while ensuring the security of the accessed data.

[0079] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0080] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for scheduling and management of the Android system.

[0081] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0082] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files in the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection. For example, in the embodiments of the present application, the virtual machine can be used to perform functions such as rendering interface frame images, detecting and handling lag events.

[0083] The system library is also called the Native layer. The Native layer can include multiple functional modules, such as function libraries and graphics processing libraries (e.g., OpenGL ES). The Native layer can also include logging, maintenance, and diagnostic services.

[0084] Log maintenance and diagnostic services can include performance log diagnostic services and resource log collection services. Among them, the performance log diagnostic service can be used to detect window interface freeze events, and the resource log collection service can be used to cache system resource logs.

[0085] The function library provides developers with API interfaces for multiple services, making it easy for developers to quickly integrate and implement various functions.

[0086] The graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0087] The kernel layer is the layer between hardware and software. It can include various driver modules, such as display drivers and audio drivers, as well as a performance lag log virtual device. This performance lag log virtual device can be used to save lag information transmitted by the performance logging interface of the application framework layer to the lag log.

[0088] It should be noted that the embodiments of the present application are only illustrated using the Android system as an example. In other operating systems (such as Windows system, IOS system, etc.), as long as the functions implemented by each functional module are similar to those in the embodiments of the present application, the solutions of the present application can also be implemented.

[0089] The following uses Figure 4 as an example to illustrate the interaction between relevant modules in the video freeze detection method in an embodiment of the present application.

[0090] The application layer of an electronic device may include video applications, live broadcast applications, and video call applications. When an application wants to instruct an electronic device to display a certain interface, the application can use the view control of the application framework layer to draw the interface. The view control may include a texture view control and an independent layer view control. For example, the texture view control may include a TextureView control, and the independent layer view control may include a SurfaceView control. The texture view control may also be called a texture control, and the independent layer view control may also be called a canvas control.

[0091] It is understandable that for different video scenarios, applications can use different view controls. For example, in a video playback scenario, an application can use a texture control; in a live broadcast scenario, an application can use a canvas control. The application can decide which view control to use, and this embodiment of the application does not limit this. For ease of description, the following explanation will take the video scenario as an example.

[0092] When an application uses a texture control, it can call related functions in the texture control to draw the interface. For example, the application can call the window-canvas association function, which binds the canvas to the window interface of the electronic device. The application can also call the canvas frame completion function, which displays the content on the canvas on the window interface of the bound electronic device, thereby achieving the effect of video playback. In addition, the application can also call the window-canvas disassociation function to destroy the canvas when the interface drawing is no longer needed.

[0093] Similarly, when an application uses a canvas control, it can call related functions in the canvas control to draw the interface. For example, the application can call the window-canvas association function, which binds the canvas to the window interface of the electronic device. The application can also call the layer update function, which updates the content on the layer to the window interface of the electronic device, thereby achieving the effect of video playback. In addition, the application can also call the window-canvas disassociation function to destroy the canvas when the interface drawing is no longer needed.

[0094] In addition, the canvas frame completion function or layer update function can also call the video frame processing interface in the window processing device to pass the relevant data of each frame of the video playback to the window processing device. In this way, the window processing device can detect whether the current window interface is stuck.

[0095] The window processing device may include a texture control switching device, a texture control frame length calculation device, a canvas control frame length calculation device, etc. The texture control switching device and the texture control frame length calculation device may be used to detect a freeze event in the texture control, and the canvas control frame length calculation device may be used to detect a freeze event in the canvas control.

[0096] Among them, the texture control switching device can be used to detect whether there is a freeze when the window interface switches, wherein the window interface switching may include the electronic device responding to the user's operation of sliding the video and playing the previous video or the next video; the window interface switching may also include the electronic device responding to the user's operation of switching the application to the background and the video interface being cut to the background.

[0097] For example, when the refresh time of the interface reaches a preset time threshold, if the texture control switching device detects an event in which the user slides the video, it can be considered that a video switch has occurred, the refresh time is acceptable, and the interface has not experienced any freezes. If the texture control switching device does not detect an event in which the user slides the video, it can be considered that no video switch has occurred, the refresh time is long, and the interface has experienced any freezes. The specific value of the preset time threshold is not limited in this embodiment of the present application.

[0098] The texture control frame length calculation device can be used to detect whether a video freezes during playback. For example, in the same video, when the refresh time interval between the current and subsequent frames exceeds a preset value, if the texture control frame length calculation device detects that the user clicks to pause playback, it can be considered that the video has not freezed; if the texture control frame length calculation device does not detect that the user clicks to pause playback, it can be considered that the video has freezed. The specific value of the preset value is not limited in this embodiment of the application.

[0099] The canvas control frame length calculation device can be used to detect whether a freeze occurs during the playback of a video. Similarly, the detection process of the canvas control frame length calculation device can refer to the relevant description of the texture control frame length calculation device above, which will not be repeated here.

[0100] It can be understood that the window processing device can set corresponding jamming identification IDs for different types of jamming scenarios. For example, the jamming scene detected by the texture control switching device can correspond to jamming identification ID1, the jamming scene detected by the texture control frame length calculation device can correspond to jamming identification ID2, and the jamming scene detected by the canvas control frame length calculation device can correspond to jamming identification ID3.

[0101] When the window processing device detects that the video interface is stuck, the window processing device can pass the stuck information to the performance logging interface, which can write the stuck information to the stuck log in the performance stuck log virtual device of the kernel layer through the log-related interface of the Native layer and the log-related interface of the kernel layer.

[0102] In the native layer, the performance log diagnostic service within the log maintenance and diagnostic service can analyze and process video interface freeze events based on the freeze logs. The performance log diagnostic service can include a performance log reader, a texture-based video playback freeze handler, and an independent layer-based video playback freeze handler.

[0103] In a possible implementation, the performance log reading device can poll and read the jam log in the performance jam log virtual device of the kernel layer. The performance log diagnostic service can determine whether the current jam log is analyzed and processed by the texture class video playback jam processor or the independent layer class video playback jam processor based on the jam identifier in the jam log. For example, when the jam identifier is ID1 or ID2, the current jam log can be analyzed and processed by the texture class video playback jam processor; when the jam identifier is ID3, the current jam log can be analyzed and processed by the independent layer class video playback jam processor.

[0104] When the texture video playback jamming processor or the independent layer video playback jamming processor analyzes and processes the jamming event, the resource log collection service can be called to obtain the system resource log. It is understandable that the system resource log can record in detail various log information when the electronic device is running, including CPU resource information, input and output information, memory information, network status information and other resource log information. Compared with the system resource log, the above-mentioned jamming log records relatively less information. The jamming log can record jamming time, jamming scene, jamming degree and other information related to jamming.

[0105] For the analysis and processing process of the specific texture video playback jamming processor or the independent layer video playback jamming processor for jamming events, please refer to the relevant description in the (2) video playback jamming detection process of the embodiment corresponding to Figure 5 below, and no further details will be given.

[0106] Figure 5 shows a schematic flow chart of a method for detecting video freezes. The method may include the following processes: (1) a fault collection process, (2) a video playback freeze detection process, and (3) a fault diagnosis and recovery process.

[0107] (1) Fault collection process.

[0108] The fault collection process can be implemented by a fault collection device, which can collect events reported by some system controls in the electronic device. Among them, the fault collection device can be understood as the window processing device of the application framework layer in the embodiment corresponding to Figure 4 above, and the system control can be understood as the texture control or canvas control in the embodiment corresponding to Figure 4 above. The events reported by the system control may include one or more of the following events: Event 1, the event of determining the video scene, Event 2, the event of the visible attribute change of the control, Event 3, the user click event, Event 4, the event of the foreground and background status change of the interface.

[0109] Event 1: Determine the event of the video scene.

[0110] During the interface rendering process of the electronic device, when the system control receives a rendering thread signal from the application, the rendering thread signal can be transmitted to the fault collection device. The fault collection device can determine whether the current scene is a video scene based on the rendering thread signal.

[0111] In a possible implementation, the fault collection device can determine the name of the interface currently displayed by the electronic device and the application package name corresponding to the interface based on the rendering thread signal. It can be understood that a white list of video scenes is set in the electronic device, and the white list may include application package names and interface names. When the interface name determined by the fault collection device matches the interface name in the white list, and the application package name determined by the fault collection device matches the application package name in the white list, it indicates that the current scene is a video scene; otherwise, the current scene is not a video scene.

[0112] If the fault acquisition device determines that the current scene is a video scene, it can record the cache time FrameBufferReady and the layer texture identifier SurfaceLayerID of the current frame image, where the layer texture identifier can also be called the layer texture ID.

[0113] Event 2: The event when the visible property of the control changes.

[0114] The fault collection device can capture visible property change events for controls such as progress bars. These control visible property change events can include events indicating changes in the control's Visible property and AlphaChanged transparency. For ease of description, the following uses a progress bar control as an example to illustrate visible property change events and transparency change events.

[0115] In a possible implementation, the user can click on the video interface to pause the video. In response to the user's click event, the electronic device can change the display state of the progress bar control in the electronic device interface. At this time, the electronic device can report the change value of the visible attribute Visible of the progress bar control. For example, when the value of Visible is true, it means that the progress bar control is visible. When the value of Visible is false, it means that the progress bar control is invisible. In addition, the electronic device can also report the value of the transparency change of the progress bar control. It can be understood that the transparency value can be in the range of 0 to 255. The smaller the value, the more transparent the display of the progress bar control, and the larger the value, the more opaque the display of the progress bar control.

[0116] Event 3: User click event.

[0117] The fault collection device can detect various InputEvents generated by a user on the screen of an electronic device. For example, InputEvents can include ACTION_DOWN, ACTION_MOVE, and ACTION_UP. When a user presses the screen and then lifts their finger, the fault collection device can obtain the ACTION_UP event and record the time of the lift.

[0118] It is understandable that when the electronic device responds to the user clicking to pause the video, the progress bar control on the electronic device interface may become visible. At this point, both the visible property change event of the progress bar and the user click event have occurred on the electronic device. In other words, the fault collection device can detect the occurrence of both Event 2 and Event 3.

[0119] Event 4: The event of the front-end and back-end status change of the interface.

[0120] The fault collection device may register an event of switching the application to the background with the electronic device. When the application switches to the background, the electronic device may report the event of switching the application to the background to the fault collection device.

[0121] (2) Video playback freeze detection process.

[0122] The video playback jam detection process can be implemented by a video playback jam detection device.

[0123] The detection device can be understood as the texture video playback jamming processing of the Native layer in the embodiment corresponding to FIG4 above.

[0124] S501: Determine whether there is buffered data.

[0125] In an embodiment of the present application, a video playback jam detection device can determine whether there is buffered data. In a possible implementation, the video playback jam detection device can obtain the cache time of the previous frame image and determine whether there is buffered data based on the cache time. The video playback jam detection device can obtain the cache time of the previous frame image from the cache time FrameBufferReady of the frame image recorded by the fault acquisition device.

[0126] For example, if the current frame is the first image, the cache time before the video playback freeze detection device obtains the frame can be 0, that is, there is no buffered data before the first frame. 0 can also be set to other preset values, as long as the preset value can be used to indicate that there is no buffered data before the first frame.

[0127] When the cache time of the previous frame image obtained by the video playback jam detection device is 0, it means that there is no buffered data, and step S506 can be executed; when the cache time of the previous frame image obtained by the video playback jam detection device is not 0, it means that there is buffered data, and step S502 can be executed.

[0128] S502: Determine whether the video texture ID has changed.

[0129] When the video playback jam detection device determines that there is buffered data, the video playback jam detection device can further determine whether the video texture ID has changed. If the video texture ID has changed, it means that the current scene is a video switching scene. For example, the video switching scene may include the electronic device responding to the user's operation of sliding the video to play the previous video or the next video, and may also include the electronic device responding to the user's operation of switching the application to the background, and the current video interface switches to the background. If the video texture ID has not changed, it means that the current scene is a video playback scene.

[0130] If the video texture ID changes, step S505 may be executed. If the video texture ID does not change, step S503 may be executed.

[0131] S503: Determine whether the frame interval time exceeds threshold A.

[0132] When the video texture ID does not change, it indicates that the current scene is video playback. The video playback freeze detection device can determine whether the interval between the previous and next two frames exceeds threshold A. Threshold A can be pre-set by the electronic device. For example, threshold A can be set to approximately 80 milliseconds. The specific value of threshold A is not limited in this embodiment of the application.

[0133] When the interval between the current and subsequent image frames exceeds threshold A, it indicates that the refresh interval between the two frames is relatively long, and the video playback freeze detection device needs to further determine whether the current interface has freezes, and then step S505 can be executed. When the interval between the current and subsequent image frames does not exceed threshold A, it indicates that the previous interface is refreshed normally and no freezes have occurred, and then step S506 can be executed.

[0134] S504: Determine whether the frame interval time exceeds a threshold B.

[0135] When the video texture ID changes, it indicates that the current video is switching. The video playback freeze detection device can determine whether the interval between the previous and next two frames exceeds threshold B. Threshold B can be pre-set by the electronic device. For example, threshold B can be set to approximately 155 milliseconds. The specific value of threshold B is not limited in this embodiment of the application.

[0136] When the interval between the current and subsequent image frames exceeds threshold B, it indicates that the refresh interval between the two frames is relatively long, and the video playback freeze detection device needs to further determine whether the current interface is freezed, and then step S505 can be executed. When the interval between the current and subsequent image frames does not exceed threshold B, it indicates that the current interface is refreshed normally and no freeze occurs, and then step S506 can be executed.

[0137] S505: Determine whether any of the following events occurs: a progress bar status change event, a user click event, or an application switching to the background.

[0138] In one possible scenario, in the above step S502, if the video texture ID has not changed, it indicates a video playback scenario. The video playback freeze detection device can further determine whether the interval time between the two frames of images exceeds the threshold A due to an event in which the visible properties of the control are changed and / or a user click event occurs.

[0139] It is understood that the video playback freeze detection device can determine whether a control visible property change event and / or a user click event occurs within a period of time, wherein the period of time may include the time when the video texture ID changes. The specific value of this period of time is not limited in this embodiment of the application.

[0140] If the video playback jam detection device detects an event in which a control's visible property changes, and / or a user click event, it indicates that the interval between two frames exceeds threshold A due to a user pause operation, and it is not considered that the video playback has jammed, and step S506 can be executed. If the video playback jam detection device does not detect an event in which a control's visible property changes, and / or a user click event, it indicates that the video playback has jammed, and step S508 can be executed.

[0141] In another possible scenario, in the above step S502, if the video texture ID changes, indicating a video switching scene, the video playback freeze detection device can further determine whether the interval time between the two frames of images exceeds the threshold B due to an application background switching event.

[0142] If the video playback jam detection device detects that an application has been switched to the background, it indicates that the interval between two frames exceeds threshold B due to the user's switching operation, and it is not considered that the video playback has jammed, and step S506 can be executed. If the video playback jam detection device does not detect that an application has been switched to the background, it indicates that the video playback has jammed, and step S508 can be executed.

[0143] S506: Record frame time.

[0144] The video playback jam detection device can record the frame time, wherein the frame time can be understood as the cache time FrameBufferReady of the frame image reported to the video playback jam detection device by the above-mentioned fault collection process.

[0145] S507: Record the video texture ID.

[0146] The video playback jam detection device can record the video texture ID, wherein the video texture ID can be understood as the video texture ID reported to the video playback jam detection device by the above-mentioned fault collection process.

[0147] S508: Record scene resource logs.

[0148] The scene resource log may be a portion of the system resource log obtained by the video playback jam detection device from the cache of the resource log collection service of the Native layer in the embodiment corresponding to FIG4 . The video playback jam detection device may record the scene resource log to a disk file of the electronic device and may also package the disk file corresponding to the scene resource log and upload it to a server to facilitate subsequent analysis of the scene resource log.

[0149] S509: Record scene event log.

[0150] The scene event log may be a partial jam log obtained from the performance jam log virtual device of the kernel layer in the embodiment corresponding to FIG4 . The video playback jam detection device may record the scene event log to a disk file of the electronic device and may also package the disk file corresponding to the scene event log and upload it to a server to facilitate subsequent analysis of the scene event log.

[0151] S510: Run the fault diagnosis and recovery device.

[0152] The video playback freeze detection device may also include a running fault diagnosis and recovery device. The specific running fault diagnosis and recovery device can refer to the relevant description in the following (3) fault diagnosis and recovery process, which will not be repeated here.

[0153] Optionally, in step S501 above, when the video playback jam detection device detects a progress bar state change event and / or a user click event, the video playback jam detection device may set the cache time corresponding to the previous frame image to 0. In this way, when the video playback jam detection device obtains the cache time of the previous frame image as 0, it can also indicate that there is no buffered data before the current frame image. 0 can also be set to other preset values, as long as the preset value can be used to indicate that there is no buffered data before the current frame image.

[0154] For example, if the N+1 frame image is not the first frame image, assuming that the cache time of the N frame image is 100 milliseconds, the cache time of the N+1 frame image is 200 milliseconds. When a progress bar state change event and / or a user click event occurs at 110 milliseconds, the video playback freeze detection device can set the cache time corresponding to the N frame image to 0, indicating that there is no buffered data before the N+1 frame image. This is because when the electronic device responds to the user clicking to pause playback, the electronic device interface content no longer changes, that is, the electronic device does not refresh the interface, and therefore, it can be considered that there is no buffered data before the N+1 frame image.

[0155] In a possible scenario, when a progress bar state change event and / or a user click event occurs at 110 milliseconds, the video playback jam detection device sets the cache time corresponding to the previous frame image to 0. However, since the electronic device sends frames at a faster speed, the electronic device sends another frame image at 120 milliseconds, causing the cache time corresponding to the previous frame image to become 120 milliseconds, causing the video playback jam detection device to determine that there is currently buffered data and execute step S502. When the video playback jam detection device detects that the frame interval time exceeds the threshold, step S505 can be executed to again determine the progress bar state change event and / or the user click event.

[0156] It is understandable that the video playback jam detection device can deem that the time when the user clicks the screen is the time when the user clicks the screen plus the preset finger lift time. For example, if the user clicks the screen for 110 milliseconds and the preset finger lift time can be 50 milliseconds, the video playback jam detection device can deem that the time when the user clicks the screen is about 160 milliseconds. Therefore, when the video playback jam detection device determines that the 120 milliseconds of the cache time corresponding to the previous frame of image is before 160 milliseconds, it can be considered that a click event has occurred. The cache time corresponding to the previous frame of image can also be set to 0. This can help improve the accuracy of the video playback jam detection device in determining jam events.

[0157] (3) Fault diagnosis and recovery process.

[0158] The fault diagnosis and recovery process can be implemented by a fault diagnosis and recovery device, and the processes that can be executed by the fault diagnosis and recovery device include (3.1) the processing process when the process CPU utilization is abnormal, (3.2) the processing process when the memory application time is abnormal, and (3.3) the processing process when the network signal is abnormal.

[0159] (3.1) Processing flow when process CPU utilization is abnormal.

[0160] When the process CPU utilization is greater than or equal to the preset CPU utilization threshold, the process CPU utilization can be considered abnormal. For example, the preset CPU utilization threshold can be 40%. It is understood that the specific value of the preset CPU utilization threshold is not limited in this embodiment of the application.

[0161] When the CPU utilization of a process is abnormal, the fault diagnosis and recovery device may perform one or more of the following tests: (a) frequency limit test, (b) scheduling priority test, and (c) inter-core running time test.

[0162] (a) Frequency limit detection.

[0163] Frequency limiting can be understood as a thermal frequency limiting strategy in electronic devices. This thermal frequency limiting strategy can be understood as a strategy for electronic devices to limit the CPU frequency based on a preset temperature threshold. The CPU frequency can also be understood as the operating frequency of the CPU.

[0164] For example, when the temperature of an electronic device reaches a preset temperature threshold, the electronic device can reduce the CPU frequency. For example, the preset temperature threshold can be 40 degrees, and the CPU frequency can be reduced to 2.0 GHz. The specific value of the preset temperature threshold and the reduction degree of the CPU frequency are not limited in the embodiment of the present application. It is understandable that frequency limiting can reduce the power consumption of the electronic device while roughly maintaining the basic performance of the electronic device, thereby preventing the temperature of the electronic device from being too high and increasing the battery life of the electronic device.

[0165] When an electronic device executes a thermal frequency limiting strategy, relevant logs can be printed into the system resource log. The fault diagnosis and recovery device can obtain the thermal frequency limiting strategy related logs from the system resource logs, and record the thermal frequency limiting strategy related logs into the scheduling log of the electronic device's disk file, so as to facilitate subsequent analysis and statistics of the thermal frequency limiting problem.

[0166] (b) Scheduling priority detection.

[0167] The fault diagnosis and recovery device can detect the priority of the current video application. If the priority level of the current video application is low, the fault diagnosis and recovery device can send a message to the CPU scheduling module to instruct the CPU scheduling module to increase the priority of the video application.

[0168] In a possible implementation, the fault diagnosis and recovery device can determine whether the priority of the video application is greater than or equal to a preset priority threshold. If the priority of the video application is greater than or equal to the preset priority threshold, it means that the priority level of the video application is low, and the fault diagnosis and recovery device can send a message to the CPU scheduling module, instructing the CPU scheduling module to increase the priority of the video application. The preset priority threshold can be 120, and the specific value of the preset priority threshold is not limited in the embodiment of the present application.

[0169] It is understandable that the fault diagnosis and recovery device can record the relevant logs of the CPU scheduling priority in the above-mentioned scheduling log to facilitate subsequent analysis and statistics of priority scheduling problems.

[0170] (c) Inter-core runtime detection.

[0171] An electronic device may include multiple large CPU cores and multiple small CPU cores, and the fault diagnosis and recovery device may detect the running time and running frequency of each large CPU core or small CPU core. For the convenience of subsequent description, taking the large CPU core as an example, when the application interface freezes due to the low running frequency of a certain large CPU core, the fault diagnosis and recovery device may send a message to the CPU scheduling module, instructing the CPU scheduling module to increase the running frequency of the large CPU core, thereby reducing the freeze of the application interface. Among them, adjusting the running frequency of the large CPU core can also be called frequency modulation.

[0172] For example, the application process currently displayed on the interface is running on a CPU core that runs at a frequency of 1.3GHz. Since the operating frequency of the CPU core is relatively low, causing the application interface to freeze, the fault diagnosis and recovery device can send a message to the CPU scheduling module, instructing the CPU scheduling module to increase the operating frequency of the CPU core. When the CPU scheduling module increases the operating frequency of the CPU core to 2.0GHz, the freeze in the application interface can be reduced.

[0173] It is understandable that the fault diagnosis and recovery device can record the relevant logs of the inter-core running time into the above-mentioned scheduling log to facilitate subsequent analysis and statistics of the inter-core running time problem.

[0174] In the embodiment of the present application, the execution order of (a) frequency limit detection, (b) scheduling priority detection and (c) inter-core running time detection may not be limited.

[0175] (3.2) Processing flow when memory request duration is abnormal.

[0176] When the memory application duration of the electronic device exceeds a preset application duration threshold within a preset time period, the memory application duration may be considered abnormal. For example, the preset time period may be 100 milliseconds, and the preset application duration threshold may be 8 milliseconds. The specific values ​​of the preset time period and the preset application duration threshold are not limited in this embodiment of the application.

[0177] When the memory application duration is abnormal, the fault diagnosis and recovery device may perform one or more of the following tests: (d) available running memory test, (e) slow path application times test.

[0178] (d) Available running memory detection.

[0179] When the memory request duration is abnormal, the fault diagnosis and recovery device can determine the remaining space in the electronic device's running memory. If the remaining space in the running memory is less than a preset running memory threshold, the fault diagnosis and recovery device can perform a background cleanup operation to free up more running memory space.

[0180] For example, for an electronic device with 8GB of running memory, the corresponding preset running memory threshold can be 800MB. When the fault diagnosis and recovery device determines that the remaining space of the running memory is less than 800MB, it can perform a background cleanup operation. It is understandable that different electronic devices have different running memory sizes, so different electronic devices can have different preset running memory thresholds. The specific value of the preset running memory threshold is not limited in the embodiments of the present application.

[0181] It is understandable that the fault diagnosis and recovery device can record the relevant logs of the available memory detection into the memory log of the disk file of the electronic device to facilitate subsequent analysis and statistics of memory problems.

[0182] (e) Apply for slow path detection.

[0183] When an application requests memory from an electronic device, if the electronic device does not have enough memory space, the electronic device needs to clean up the memory first and reclaim some memory before allocating memory to the application. This way of allocating memory can be understood as applying for a slow path.

[0184] In a possible implementation, when the fault diagnosis and recovery device determines that the percentage of times an application applies for a slow path is greater than a preset percentage threshold, and / or the percentage of time an application applies for a slow path is greater than a preset percentage threshold, the fault diagnosis and recovery device can perform a background cleanup operation. For example, the preset percentage threshold can be 80%, and the preset percentage threshold can be 20%. The specific values ​​of the preset percentage threshold and the preset percentage threshold are not limited in this embodiment of the application.

[0185] It is understandable that the fault diagnosis and recovery device can record the relevant logs of the number of slow path application detections in the above-mentioned memory log to facilitate subsequent analysis and statistics of the slow path application problem.

[0186] (3.3) Processing flow when network signal is abnormal.

[0187] In one possible implementation, when the network signal is lower than a preset signal threshold of a full signal, the network signal may be considered abnormal, for example, the preset signal threshold may be 60%. It is understood that the specific value of the preset signal threshold is not limited in this embodiment of the application.

[0188] In another possible implementation, when the network signal strength is not within a preset signal strength range, the network signal may be considered abnormal. For example, the preset signal strength range may be -90dBm to -60dBm. It is understood that the specific value of the preset signal strength range is not limited in this embodiment of the application.

[0189] When the network signal is abnormal, the fault diagnosis and recovery device can perform one or more of the following tests: (f) network delay detection, (g) network packet loss detection.

[0190] (f) Network delay detection.

[0191] When the fault diagnosis and recovery device detects that the network delay is greater than a preset network delay threshold, the fault diagnosis and recovery device may perform a network signal switching operation. The network signal switching operation may include switching from a Wi-Fi signal to a data signal, or switching from a data signal to a Wi-Fi signal. The preset network delay threshold may be pre-set by the electronic device, for example, the preset network delay threshold may be 100 milliseconds. The specific value of the preset network delay threshold is not limited in this embodiment of the present application.

[0192] It is understandable that the fault diagnosis and recovery device can record the relevant logs of network delay detection in the network log of the disk file of the electronic device to facilitate subsequent analysis and statistics of the network delay problem.

[0193] (g) Network packet loss detection.

[0194] When the fault diagnosis and recovery device detects that the network packet loss rate is greater than a preset packet loss rate threshold, the fault diagnosis and recovery device can perform an operation to switch the network signal. The preset packet loss rate threshold can be pre-set by the electronic device, for example, the preset packet loss rate threshold can be 20%. The specific value of the preset packet loss rate threshold is not limited in this embodiment of the application.

[0195] It is understandable that the fault diagnosis and recovery device can record relevant logs of network packet loss detection into the above-mentioned network log to facilitate subsequent analysis and statistics of the network packet loss problem.

[0196] In the embodiment of the present application, the execution order of (3.1) the processing flow when the process CPU utilization is abnormal, (3.2) the processing flow when the memory application duration is abnormal, and (3.3) the processing flow when the network signal is abnormal may not be limited.

[0197] The following describes the method of the embodiment of the present application in detail through specific embodiments. The following embodiments can be combined with each other or implemented independently, and the same or similar concepts or processes may not be repeated in some embodiments.

[0198] FIG6 shows a video freeze detection method according to an embodiment of the present application. The method includes:

[0199] S601. The electronic device displays a video playback interface, where the video playback interface includes a first control and a second control. The first control is used to display a first type of view, and the second control is used to display a second type of view. The first type of view does not include views loaded locally from the electronic device; the second type of view includes views loaded locally from the electronic device.

[0200] In the embodiment of the present application, the first control can be understood as the video playback control 100 in the embodiment corresponding to Figure 1 above, and the second control can be understood as the avatar display control 101 in the embodiment corresponding to Figure 1 above. The first type of view can be a view of the video playback scene displayed in the first control, a view of the video switching scene, etc.

[0201] S602: When the frame interval of the view displayed in the first control is greater than or equal to a preset value, the electronic device determines that the video freezes; wherein the frame interval of the view displayed in the second control is less than, equal to, or greater than the preset value.

[0202] In the embodiment of the present application, the preset value can be pre-set by the electronic device, and the specific value of the preset value is not limited in the embodiment of the present application. For example, the preset value can be understood as the threshold value A or threshold value B in the embodiment corresponding to Figure 5 above.

[0203] When the electronic device determines that the video is stuck, the electronic device can print a log related to the stuck and perform relevant processing on the stuck, which can help reduce the stuck of the video interface.

[0204] When detecting the freeze of the video application interface, the electronic device can detect the drawing controls that do not load local data without being affected by the controls that load local data. This can improve the accuracy of judging whether the video application interface is freezed, thereby helping to reduce the freeze of the video application interface.

[0205] Optionally, based on the embodiment corresponding to Figure 6, before the electronic device determines that the video is stuck, it may also include: the electronic device determines that no preset event occurs in the video playback interface, and the preset event includes one or more of the following: an event in which the progress bar control in the video playback interface is triggered, an event for indicating that the video playback interface is paused, and an event for indicating that the video playback interface is switched to the background.

[0206] In the embodiment of the present application, the event of the progress bar control in the video playback interface being triggered can be understood as event 2 in the embodiment corresponding to FIG5 above, and the event of the visible attribute change of the control. The event used to indicate the pause of the video playback interface can be understood as event 3 in the embodiment corresponding to FIG5 above, and the user click event. The event used to indicate the switching of the video playback interface to the background can be understood as event 4 in the embodiment corresponding to FIG5 above, and the event of the change of the foreground and background status of the interface. The specific preset events can be referred to the relevant description of the fault collection process (1) in the embodiment corresponding to FIG5 above, and will not be repeated here.

[0207] The electronic device determines whether a preset event occurs in the video playback interface, so as to judge whether the current interface is paused due to an event triggered by the user, and thus can more accurately judge whether the interface is stuck.

[0208] Optionally, based on the embodiment corresponding to Figure 6, the preset value includes a first threshold value, and before the electronic device determines that the video is stuck, it may also include: the electronic device determines that the layer texture identifier of the view displayed in the first control has not changed; when the frame interval of the view displayed in the first control is greater than or equal to the preset value, the electronic device determines that the video is stuck, including: when the frame interval of the view displayed in the first control is greater than or equal to the first threshold value, and no event of the progress bar control being triggered occurs in the video playback interface, and / or an event for indicating the pause of the video playback interface occurs, the electronic device determines that the video is stuck.

[0209] In the embodiment of the present application, the first threshold value can be understood as the threshold value A in the embodiment corresponding to Figure 5 above. The specific first threshold value can refer to the relevant description of the threshold value A and will not be repeated here.

[0210] The layer texture identifier can be understood as the layer texture ID in the embodiment corresponding to FIG. 5 . For determining whether the layer texture identifier has changed, reference can be made to the relevant description in S502 of the embodiment corresponding to FIG. 5 , and further details are omitted. Furthermore, for the electronic device to determine whether a video freeze has occurred, reference can be made to the relevant description in S505 of the embodiment corresponding to FIG. 5 , and further details are omitted.

[0211] The electronic device determines the event of the progress bar control being triggered, and / or the event used to indicate the pause of the video playback interface, and can further determine that the interval between two frames of images exceeds the threshold A due to the user's pause operation, thereby accurately determining whether the video is stuck.

[0212] Optionally, based on the embodiment corresponding to Figure 6, the preset value includes a second threshold value, and before the electronic device determines that the video is stuck, it may also include: the electronic device determines that the layer texture identifier of the view displayed in the first control has changed; when the frame interval of the view displayed in the first control is greater than or equal to the preset value, the electronic device determines that the video is stuck, including: when the frame interval of the view displayed in the first control is greater than or equal to the second threshold value, and no event occurs in the video playback interface to indicate switching the video playback interface to the background, the electronic device determines that the video is stuck.

[0213] In the embodiment of the present application, the second threshold value can be understood as the threshold value B in the embodiment corresponding to FIG5 above. For the specific second threshold value, reference can be made to the relevant description of threshold value B, which will not be repeated here. It is understood that the values ​​of the first threshold value and the second threshold value can be the same or different.

[0214] In a possible scenario, since the time it takes to refresh the frame image when the electronic device switches to the previous video or the next video, or switches the video to the background, is longer than the time it takes to refresh the frame image when the same video is played, the second threshold can be appropriately larger than the first threshold, which can be more in line with the judgment logic of the frame time of the specific scenario.

[0215] In addition, the electronic device may determine that a video freeze occurs by referring to the relevant description in S505 of the embodiment corresponding to FIG. 5 above, and will not be repeated here.

[0216] The electronic device determines the event for instructing to switch the video playback interface to the background, and can further determine that the interval between two frames of images exceeds the threshold B due to the user's background switching operation, thereby accurately determining whether the video is stuck.

[0217] Optionally, based on the embodiment corresponding to FIG6 , the electronic device determining that a video freeze occurs may include: the electronic device recording a log related to the freeze.

[0218] In the embodiment of the present application, the logs related to the jamming can be recorded with reference to the relevant descriptions in S508 and S509 of the embodiment corresponding to FIG5 above, and will not be repeated here.

[0219] Electronic devices can record logs related to lags to facilitate further analysis and statistics of the lags. For example, electronic devices can upload logs to a server, and developers can optimize the performance parameters of the electronic devices based on the log information. Electronic devices can also send logs to video application manufacturers, who can further optimize the program code, thereby helping to reduce lags in video playback.

[0220] Optionally, based on the embodiment corresponding to Figure 6, the logs related to the jamming include one or more of the following: scene resource logs, scene event logs.

[0221] In the embodiment of the present application, the scene resource log can refer to the relevant description in S508 of the embodiment corresponding to FIG5 , and will not be repeated here. The scene event log can refer to the relevant description in S509 of the embodiment corresponding to FIG5 , and will not be repeated here.

[0222] Recording different types of logs by electronic devices can facilitate subsequent different types of analysis of the jam, thereby analyzing and solving certain types of problems in a targeted manner.

[0223] Optionally, based on the embodiment corresponding to Figure 6, the electronic device determines that the video is stuck, which may include: the electronic device performs processing related to preset performance indicators; the preset performance indicators include one or more of the following: the process CPU utilization of the electronic device, the memory application time of the electronic device, and the network signal of the electronic device.

[0224] In the embodiments of the present application, the electronic device performs processing related to the preset performance indicators, which can be referred to the relevant description of the fault diagnosis and recovery process (3) in the embodiment corresponding to Figure 5 above, and will not be repeated here. After the electronic device performs relevant processing on the above-mentioned preset performance indicators, the performance of the electronic device can be improved, thereby helping to reduce the lag of the video interface.

[0225] Optionally, based on the embodiment corresponding to Figure 6, the electronic device performs processing related to preset performance indicators, which may include: the electronic device determines that an abnormality has occurred in the process CPU utilization; the electronic device performs detection related to the process CPU utilization, and the detection related to the process CPU utilization includes one or more of the following: frequency limit detection, scheduling priority detection, and inter-core running time detection; the electronic device records logs related to the process CPU utilization; the electronic device increases the CPU operating frequency, and / or increases the priority of the process; the electronic device determines that an abnormality has occurred in the memory application duration; the electronic device performs detection related to the memory application duration, and the detection related to the memory application duration includes one or more of the following: available memory detection, slow path application times detection; the electronic device records logs related to the memory application duration; the electronic device cleans up the memory of the electronic device; the electronic device determines that an abnormality has occurred in the network signal; the electronic device performs detection related to the network signal, and the detection related to the network signal includes one or more of the following: network delay detection, network packet loss detection; the electronic device records logs related to the network signal; the electronic device switches the network signal.

[0226] In the embodiments of the present application, the electronic device performs processing related to the preset performance indicators, which can be referred to the relevant description of the fault diagnosis and recovery process (3) in the embodiment corresponding to Figure 5 above, and will not be repeated here. After the electronic device performs relevant processing on the above-mentioned preset performance indicators, the performance of the electronic device can be improved, thereby helping to reduce the lag of the video interface.

[0227] Optionally, based on the embodiment corresponding to Figure 6, the method may also include: when the electronic device determines that a preset condition is met, the electronic device records the frame time; when the electronic device determines that a preset event occurs in the video playback interface, the electronic device sets the frame time to a target value; the electronic device determines that there is buffered data related to the playback video, including: when the frame time is not the target value, the electronic device determines that there is buffered data related to the playback video; wherein, meeting the preset condition includes any one of the following: there is no buffered data related to the playback video; there is buffered data related to the playback video, and the frame interval of the view displayed in the first control is less than the preset value.

[0228] In an embodiment of the present application, the target value can be used to indicate that there is no buffered data before the current frame image. For example, the target value can be 0 in step S501 of the embodiment corresponding to Figure 5 above, or it can be other values, which is not limited in the embodiment of the present application.

[0229] The electronic device determines whether there is buffered data related to the playing video, which can reduce unnecessary execution steps. For example, in the embodiment corresponding to Figure 5 above, when the electronic device determines that there is no buffered data related to the playing video, the electronic device may not execute steps S502-S505, etc., which can improve the execution efficiency of the code.

[0230] Optionally, based on the embodiment corresponding to FIG6 , the first control includes a texture view (TextureView) control or an independent layer view (SurfaceView) control.

[0231] In the embodiments of the present application, a TextureView control or a SurfaceView control can be a drawing control that does not load local data. By detecting the frame rate refresh of a drawing control that does not load local data, the electronic device can improve the accuracy of determining whether a video application interface is stuck, thereby helping to reduce the stuckness of the video application interface.

[0232] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0233] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the method steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0234] The embodiment of the present application can divide the functional modules of the device implementing the method according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. In actual implementation, there may be other division methods.

[0235] 7 is a schematic diagram of the structure of a chip provided by an embodiment of the present application. The chip 700 includes one or more (including two) processors 701, a communication circuit 702, a communication interface 703 and a memory 704.

[0236] In some embodiments, the memory 704 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0237] The method described in the above embodiment of the present application can be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. During the implementation process, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 701 or an instruction in the form of software. The above-mentioned processor 701 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 701 can implement or execute the methods, steps and logic block diagrams related to each processing disclosed in the embodiment of the present application.

[0238] The steps of the method disclosed in the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read-only memory (EEPROM). The storage medium is located in the memory 704, and the processor 701 reads the information in the memory 704 and performs the steps of the above method in conjunction with its hardware.

[0239] The processor 701 , the memory 704 , and the communication interface 703 can communicate with each other via a communication line 702 .

[0240] In the above embodiment, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory or downloaded and installed in the memory in the form of software.

[0241] The present application also provides a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. For example, the available medium can include magnetic media (e.g., floppy disk, hard disk or tape), optical media (e.g., digital versatile disc (DVD)), or semiconductor media (e.g., solid state disk (SSD)).

[0242] The present application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented in whole or in part via software, hardware, firmware, or any combination thereof. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one location to another. The storage medium can be any target medium that can be accessed by a computer.

[0243] As one possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; computer-readable media may include magnetic disk storage or other magnetic disk storage devices. Moreover, any connecting line may also be appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically using lasers.

[0244] The present application embodiment is described with reference to the flow chart and / or block diagram according to the method, device (system) and computer program product of the embodiment of the present application.It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processing unit of general-purpose computer, special-purpose computer, embedded processing machine or other programmable data processing equipment to produce a machine, so that the instruction executed by the processing unit of computer or other programmable data processing equipment produces the device for realizing the function specified in one flow chart flow chart or multiple flow charts and / or one block or multiple blocks of block diagram.

Claims

1. A video freeze detection method, characterized in that: The method comprises: The electronic device displays a video playback interface, the video playback interface including a first control and a second control, the first control is used to display a first type of view, the second control is used to display a second type of view, the first type of view does not include a view locally loaded from the electronic device; the second type of view includes a view locally loaded from the electronic device; When a frame interval of the view displayed in the first control is greater than or equal to a preset value, the electronic device determines that a video freeze occurs; wherein a frame interval of the view displayed in the second control is less than, equal to or greater than the preset value.

2. The method according to claim 1, characterized in that: Before the electronic device determines that the video freezes, the method further includes: The electronic device determines that no preset event occurs in the video playback interface, and the preset event includes one or more of the following: an event in which a progress bar control in the video playback interface is triggered, an event for indicating that playback of the video playback interface is paused, and an event for indicating that the video playback interface is switched to the background.

3. The method according to claim 1 or 2, characterized in that: The preset value includes a first threshold value, and before the electronic device determines that a video freeze occurs, the electronic device further includes: The electronic device determines that the layer texture identifier of the view displayed in the first control has not changed; When the frame interval of the view displayed in the first control is greater than or equal to a preset value, the electronic device determines that a video freeze occurs, including: When the frame interval of the view displayed in the first control is greater than or equal to the first threshold, and the event of triggering the progress bar control and / or the event for indicating pausing the video playback interface does not occur in the video playback interface, the electronic device determines that the video is stuck.

4. The method according to claim 1 or 2, characterized in that: The preset value includes a second threshold value, and before the electronic device determines that the video freezes, the electronic device further includes: The electronic device determines that a layer texture identifier of a view displayed in the first control changes; When the frame interval of the view displayed in the first control is greater than or equal to a preset value, the electronic device determines that a video freeze occurs, including: When the frame interval of the view displayed in the first control is greater than or equal to the second threshold, and the event for indicating switching the video playback interface to the background does not occur in the video playback interface, the electronic device determines that the video is stuck.

5. The method according to any one of claims 1 to 4, characterized in that: The electronic device determines that a video freeze occurs, including: The electronic device records a log related to the freeze.

6. The method according to claim 5, characterized in that The logs related to the jamming may include one or more of the following: scene resource logs and scene event logs.

7. The method according to any one of claims 1 to 6, characterized in that: The electronic device determines that a video freeze occurs, including: The electronic device performs processing related to the preset performance indicator; the preset performance indicator includes one or more of the following: the process CPU utilization rate of the electronic device, the memory application time of the electronic device, The network signal of the electronic device.

8. The method according to claim 7, characterized in that The electronic device performs processing related to the preset performance indicator, including: The electronic device determines that the CPU utilization of the process is abnormal; The electronic device performs detection related to the CPU utilization of the process, wherein the detection related to the CPU utilization of the process includes one or more of the following: frequency limit detection, scheduling priority detection, and inter-core running time detection; The electronic device records a log related to the CPU utilization of the process; The electronic device increases the operating frequency of the CPU and / or increases the priority of the process; The electronic device determines that an abnormality occurs in the memory application duration; The electronic device performs a detection related to the memory application duration, wherein the detection related to the memory application duration includes one or more of the following: an available memory detection and a slow path application times detection; The electronic device records a log related to the duration of the memory application; The electronic device clears the memory of the electronic device; The electronic device determines that an abnormality occurs in the network signal; The electronic device performs detection related to the network signal, wherein the detection related to the network signal includes one or more of the following: network delay detection and network packet loss detection; The electronic device records a log related to the network signal; The electronic device switches the network signal.

9. The method according to any one of claims 2 to 8, characterized in that: The method further comprises: When the electronic device determines that the preset condition is met, the electronic device records the frame time; When the electronic device determines that the preset event occurs in the video playback interface, the electronic device sets the frame time to a target value; The electronic device determines that there is buffer data related to playing a video, including: When the frame time is not the target value, the electronic device determines that there is buffer data related to the played video; Wherein, the satisfying of the preset condition includes any one of the following: There is no buffering data associated with playing the video; There is buffer data related to playing the video, and the frame interval of the view displayed in the first control is less than the preset value.

10. The method according to any one of claims 1 to 9, characterized in that: The first control includes a texture view control or an independent layer view control.

11. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the computer executes the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that The invention comprises a computer program, which, when being executed, enables an electronic device to execute the method according to any one of claims 1 to 10.

Images