Multimedia content playing lagging detection method and electronic equipment

Abstract

The invention provides a multimedia content playing lagging detection method and electronic equipment. When the electronic equipment receives a preset operation when displaying a multimedia content playing interface, a lagging detection module of the electronic equipment can detect whether a multimedia application reports a playing starting event within a preset time period when the action of the preset operation starts. And if the play starting event reported by the multimedia application is not received in the preset time period, the lagging detection module can indicate the electronic equipment to adjust the network, so that the electronic equipment can use the network with better quality for communication. According to the method, whether playing of the multimedia content is lagged or not when the user triggers the electronic equipment to play the multimedia content can be accurately identified, and the accuracy of multimedia content playing lagging detection is improved.

Description

Technical Field

[0001] This application relates to the field of terminal technologies, and in particular, to a method for detecting stuttering in multimedia content playback and an electronic device. Background Art

[0002] Multimedia content may include audio and / or video. For example, some multimedia content is independent audio, where only sound can be heard during playback without a viewable video image. Some other multimedia content is independent video, where there is a viewable video image during playback but no audible sound. There is also some multimedia content that is a combination of audio and video, where both sound can be heard and a viewable video image is available during playback.

[0003] During the online playback of multimedia content, stuttering often occurs due to issues such as poor network conditions. Currently, parameters such as signal strength and network transmission delay are generally used to detect whether stuttering occurs in multimedia content. However, based on caching technology, an electronic device usually caches a portion of the data for playback during the process of playing multimedia content. When the network is poor, the electronic device may be playing using the cached data, and at this time, the multimedia content is not stuttering. Therefore, the occurrence of stuttering in multimedia content playback and poor network conditions are not necessarily synchronous. The accuracy of the above method of determining audio and video stuttering when detecting poor network conditions is relatively low. Summary of the Invention

[0004] This application provides a method for detecting stuttering in multimedia content playback and an electronic device. By combining user operations on the electronic device with multimedia playback events to detect whether multimedia content is stuttering, the accuracy of detecting stuttering in multimedia content playback can be improved.

[0005] In a first aspect, this application provides a method for detecting stuttering in multimedia content playback. This method is applied to an electronic device, which includes a first application, and the first application can be an application for playing multimedia content. The electronic device receives a first operation on a first interface, where the first interface is an interface in the first application for controlling multimedia content playback, and the first operation is used to play first multimedia content; when the electronic device does not obtain a first start-play event reported by the first application after the first operation, the electronic device adjusts the network for data communication; where the first start-play event is used to indicate the start of playback of the first multimedia content. The first start-play event may be reported by the first application when it obtains the playback data of the first multimedia content.

[0006] The above failure to obtain the first start-play event reported by the first application may indicate that the first multimedia content has not started playing yet.

[0007] The first interface may refer to this application Figure 3AThe user interface 310 shown. The first operation can refer to this application Figure 3C and Figure 3D the upward sliding operation shown. The first multimedia content can refer to Figure 3D the audio - video corresponding to the video picture 322 shown.

[0008] It can be seen that the above - mentioned method can accurately identify whether the electronic device can smoothly play the multimedia content without freezing when the user triggers the electronic device to play the multimedia content. When the electronic device has pre - cached the playback data of the multimedia content, even if the network of the electronic device is poor when the user triggers the electronic device to play the multimedia content, the multimedia application can obtain the pre - cached playback data, and then immediately respond to the user operation to play the multimedia content and report the start - play event. Based on the above user operation and the start - play event, the electronic device can detect that the multimedia content is not frozen. This can reduce the situation where the electronic device mis - detects the multimedia content as frozen based on the parameter indicating poor network when the electronic device smoothly plays the multimedia content in response to the user operation. The above - mentioned method can improve the accuracy of detecting multimedia content playback freezing.

[0009] Moreover, when it is detected that the multimedia content is playing frozen, the electronic device can adjust the network to eliminate the freezing as soon as possible and improve the user experience of listening to and watching the multimedia content.

[0010] Combined with the first aspect, in some embodiments, the first operation matches a preset operation. Among them, the above - mentioned preset operation can be associated with the first application. That is, the electronic device can store a correspondence table between the multimedia application and different preset operations. The preset operation associated with a multimedia application can be an operation used to control the multimedia content playback interface in this multimedia application for triggering the playback of multimedia content. Optionally, the above - mentioned preset operation can be associated with the first interface. That is, the electronic device can store a correspondence table between the multimedia playback interface and different preset operations. The preset operation associated with a multimedia playback interface can be an operation used to trigger the playback of multimedia content on this multimedia playback interface.

[0011] It can be seen that the electronic device can determine whether the first operation is an operation for triggering the playback of multimedia content by judging whether the first operation matches the preset operation, and then decide whether to wait for the start - play event to detect whether the multimedia content is playing frozen. This can reduce the situation of mis - detecting the multimedia content as frozen when the received operation does not match the preset operation and improve the accuracy of detecting multimedia content playback freezing.

[0012] In combination with the first aspect, in some embodiments, the network for data communication is adjusted from a first network to a second network, and a first application in the electronic device obtains playback data of first media content using the second network; the electronic device plays the first multimedia content according to the playback data of the first multimedia content.

[0013] The quality of the second network is superior to that of the first network.

[0014] The adjustment of the network for data communication from a first network to a second network may include, but is not limited to: adjusting from a Wi-Fi network to a cellular mobile network, adjusting from the cellular mobile network corresponding to the primary SIM card to the cellular mobile network corresponding to the secondary SIM card, adjusting from a cellular mobile network of one standard to a cellular mobile network of another standard (such as adjusting from a 4G cellular mobile network to a 5G cellular mobile network), the cellular mobile network moving from one resident cell to another resident cell, and so on.

[0015] It can be seen that when it is detected that the multimedia content playback is stuck, the electronic device can adjust the network so as to use a network with better quality for communication. In this way, the electronic device can obtain the playback data of the multimedia content as soon as possible and eliminate the stuck state of the multimedia content playback.

[0016] In combination with the first aspect, in some embodiments, after the electronic device adjusts the network for data communication, the electronic device changes a first identifier on a first interface to a second identifier, where the first identifier is used to indicate that the network for data communication is the first network, and the second identifier is used to indicate that the network for data communication is the second network.

[0017] The above first identifier may refer to the Wi-Fi identifier 311B shown in this application Figure 3G The second identifier may refer to the cellular mobile network identifier 311E shown in this application Figure 3H shown.

[0018] In combination with the first aspect, in some embodiments, when the electronic device plays the first multimedia content according to the playback data of the first multimedia content, it may specifically include: the electronic device plays a first audio according to the playback data of the first multimedia content, and / or plays a first video.

[0019] In combination with the first aspect, in some embodiments, during the process of playing the first multimedia content, the electronic device has a stuck state when playing the first multimedia content; the electronic device detects a first data exhaustion event and adjusts the network for data communication according to the first data exhaustion event; where the first data exhaustion event is used to indicate that the playback data for playing the first multimedia content in the electronic device has been played out and there is no playback data available for the time being.

[0020] It can be seen that the first data exhaustion event can reflect that the first multimedia content lags during playback. The electronic device can accurately detect whether the first multimedia content lags during playback based on the first data exhaustion event, and adjust the network when the multimedia content lags, and use a network with better quality for communication to eliminate the lag. This can better improve the user experience of listening to audio.

[0021] In combination with the first aspect, in some embodiments, when the electronic device obtains the first start playback event reported by the first application after the first operation, the electronic device plays the first multimedia content; wherein, the playback data of the first multimedia content is obtained by the first application from the memory or is obtained by the first application using the network for data communication.

[0022] In combination with the first aspect, in some embodiments, the electronic device receives a second operation on the first interface, and the second operation does not match the preset operation; or, the electronic device receives a third operation on the second interface, and the second interface is not an interface for controlling multimedia content playback; the electronic device waits to receive subsequent user operations.

[0023] It can be seen that when the received operation does not match the preset operation, the electronic device can detect the lag of multimedia content playback without waiting for the start playback event. This can reduce the situation of misdetecting the lag of multimedia content playback and improve the accuracy of detecting the lag of multimedia content playback.

[0024] In combination with the first aspect, in some embodiments, the first multimedia content pauses playback, the electronic device receives a fourth operation on the first interface, and the fourth operation is used to play the first multimedia content; the electronic device identifies that the fourth operation acts on the first area of the first interface, and the fourth operation matches the preset operation, and the electronic device determines whether the first start playback event reported by the first application is obtained within the second time period; wherein, the second time period is a preset time period starting from the second moment, and the second moment is the time when the second operation acts; when the electronic device does not obtain the first start playback event reported by the first application within the second time period, the electronic device adjusts the network for data communication; when the electronic device obtains the first start playback event reported by the first application within the second time period, the electronic device plays the first multimedia content.

[0025] The above-mentioned first area may be a preset area on the first interface. Different multimedia playback interfaces may correspond to different preset areas. The electronic device can more accurately identify the user operation for triggering the playback of multimedia content by combining the playback state of the multimedia content, the type and acting position of the user operation, so as to detect whether the multimedia content to be played lags in more scenarios where the multimedia content playback is triggered by user operations.

[0026] In combination with the first aspect, in some embodiments, before the electronic device adjusts the network for data communication, the electronic device determines that the first interface is an interface for controlling multimedia content playback according to the activity information of the first interface.

[0027] In combination with the first aspect, in some embodiments, before the electronic device adjusts the network for data communication, the electronic device detects that the data transmission index after the first operation indicates that the network quality is worse than the preset network quality.

[0028] It can be understood that when it is determined that the multimedia content is stuck based on user operations and multimedia playback events, the data transmission index indicating that the network quality is worse than the preset network quality can better explain that the network quality is poor, resulting in the multimedia content being stuck, thereby improving the confidence level of the multimedia content stuck detection result.

[0029] In combination with the first aspect, in some embodiments, the electronic device receives a fifth operation on the second interface, where the second interface is an interface in the first application for controlling multimedia content playback, and the fifth operation is used to play the second multimedia content; when the electronic device does not obtain a second start playback event reported by the first application after the fifth operation, and the data transmission index after the fifth operation indicates that the network quality is better than the preset network quality, the electronic device closes one or more processes, and the one or more processes are not processes of the first application. The second start playback event is used to indicate the start of playback of the second multimedia content. The second start playback event may be reported by the first application when it obtains the playback data of the second multimedia content.

[0030] The above failure to obtain the second start playback event reported by the first application may indicate that the second multimedia content has not started playing yet.

[0031] The above data transmission index is the round-trip time RTT.

[0032] It can be understood that when it is determined that the multimedia content is stuck based on user operations and multimedia playback events, the data transmission index indicating that the network quality is better than the preset network quality can explain that the multimedia content stuck is not caused by poor network quality. For example, the load of the electronic device's processor is too high, and too many background running processes may also cause the multimedia content to be stuck. Therefore, the electronic device can better determine whether the multimedia content stuck is caused by poor network quality in combination with the data transmission index, and then decide whether to adjust the network to eliminate the stuck. This can better eliminate the factors causing the multimedia content to be stuck, restore the smooth playback of the multimedia content, and improve the user experience of listening to and watching multimedia content.

[0033] In combination with the first aspect, in some embodiments, when the electronic device does not obtain the first start-playing event within the first time period, the electronic device adjusts the network for data communication. The first time period is a preset time period starting from the first moment, and the first moment is the time when the first operation takes effect.

[0034] In some embodiments, the data transmission index after the first operation indicates that the network quality is worse than the preset network quality. Specifically, it may include: the RTT within the first time period is greater than the preset RTT threshold, or the average value of the RTT within the first time period is greater than the preset RTT threshold.

[0035] In some embodiments, when the electronic device does not obtain the second start-playing event reported by the first application after the fifth operation, and the data transmission index after the fifth operation indicates that the network quality is better than the preset network quality, the electronic device closes one or more processes. Specifically, when the electronic device does not obtain the second start-playing event reported by the first application within the third time period, and the data transmission index within the third time period indicates that the network quality is better than the preset network quality, the electronic device closes one or more processes. The third time period may be a preset time period starting from the third moment. The third moment may be the time when the fifth operation takes effect.

[0036] Among them, the data transmission index within the third time period indicates that the network quality is better than the preset network quality. Specifically, it may include: the RTT within the third time period is less than or equal to the preset RTT threshold, or the average value of the RTT within the third time period is less than or equal to the preset RTT threshold.

[0037] In the second aspect, the present application provides a method for detecting interface browsing lags. This method can be applied to an electronic device, and the electronic device includes a first application. The electronic device receives a first operation on a first interface in the first application, and the first operation is used to switch the display content on the screen of the electronic device within the first application; the electronic device obtains the data transmission index after the first operation, and the data transmission index is used to indicate the network quality of the electronic device for data communication; if the data transmission index after the first operation indicates that the network quality is worse than the preset network quality, the electronic device adjusts the network for data communication.

[0038] The first interface may be a graphic interface, which displays text and / or pictures.

[0039] The first application may be a multimedia application, or a browser, or a news APP, or a shopping APP, or a reading APP, etc.

[0040] It can be seen that the above method can determine whether the user experiences lag when browsing the interface in the first application. When it detects that there is lag in interface browsing, the electronic device can adjust the network so as to communicate using a network with better quality. In this way, the electronic device can obtain the data required for the display interface as soon as possible, thereby eliminating the lag in interface browsing as soon as possible and improving the user experience of browsing the interface.

[0041] Combined with the second aspect, in some embodiments, the first interface matches the preset interface, and the first operation matches the preset operation.

[0042] That is to say, the electronic device can determine whether the first operation is a user operation for switching the display content on the screen of the electronic device within the first application by determining whether the first operation is a preset operation acting on the preset interface.

[0043] Combined with the second aspect, in some embodiments, the first interface is a long page, and the first operation is used to switch the display content on the screen of the electronic device to the content in different areas of the first interface. For example, the first operation can be an operation of swiping up or down on the first interface. In this way, the electronic device can switch and display the content in different areas of the first interface on the screen according to the first operation. Or, the first operation is used to switch the first interface to the second interface in the first application. For example, the first operation can be an operation of swiping left or right on the first interface. In this way, the electronic device can switch the content displayed on the screen to other interfaces on the left or right of the first interface according to the first operation.

[0044] Combined with the second aspect, in some embodiments, the data transmission metric after the first operation includes the data transmission metric within the first time period, the first time period being a preset time period starting from the first moment, and the first moment being the time when the first operation acts.

[0045] Combined with the second aspect, in some embodiments, the data transmission metric is the round-trip time RTT, and the data transmission metric after the first operation indicates that the network quality is worse than the preset network quality, specifically including: the RTT within the first time period is greater than the preset RTT threshold, or the average value of the RTT within the first time period is greater than the preset RTT threshold.

[0046] Combined with the second aspect, in some embodiments, the network for data communication is adjusted from the first network to the second network. After the electronic device adjusts the network for data communication, the electronic device changes the first identifier on the first interface to the second identifier. The first identifier is used to indicate that the network for data communication is the first network, and the second identifier is used to indicate that the network for data communication is the second network.

[0047] In a third aspect, the present application provides an electronic device. The electronic device includes a memory and a processor. The memory is used to store a computer program, and the processor is used to call the computer program so that the electronic device implements any possible implementation method in the first aspect or the second aspect.

[0048] In a fourth aspect, the present application provides a computer-readable storage medium, including instructions that, when running on an electronic device, cause the electronic device to execute any possible implementation method in the first aspect or the second aspect.

[0049] In a fifth aspect, the present application provides a computer program product that may include computer instructions that, when running on an electronic device, cause the electronic device to execute any possible implementation method in the first aspect or the second aspect.

[0050] In a sixth aspect, the present application provides a chip that is applied to an electronic device. The chip includes one or more processors that are used to call computer instructions to cause the electronic device to execute any possible implementation method in the first aspect or the second aspect.

[0051] It can be understood that the electronic device provided in the above third aspect, the computer-readable storage medium provided in the fourth aspect, the computer program product provided in the fifth aspect, and the chip provided in the sixth aspect are all used to execute the method provided in the embodiments of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a schematic diagram of the hardware structure of an electronic device 100 provided by an embodiment of the present application;

[0053] Figure 2A is a schematic diagram of the software structure of an electronic device 100 provided by an embodiment of the present application;

[0054] Figure 2B is a schematic diagram of the structure of an electronic device 100 provided by an embodiment of the present application;

[0055] Figures 3A - 3H is a schematic diagram of some scenarios for detecting stuttering in multimedia content playback provided by an embodiment of the present application;

[0056] Figure 4A and Figure 4B is a schematic diagram of some scenarios for detecting stuttering in multimedia content playback provided by an embodiment of the present application;

[0057] Figure 5 is a flowchart of a method for detecting stuttering in multimedia content playback provided by an embodiment of the present application;

[0058] Figure 6 It is a flowchart of another method for detecting stuttering in multimedia content playback provided by an embodiment of this application. Detailed implementation manners

[0059] Next, the technical solutions in the embodiments of this application will be described with reference to the accompanying drawings in the embodiments of this application. Among them, in the description of the embodiments of this application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to limit this application. As used in the specification and claims of this application, the singular forms "a", "the", "above-mentioned", "this" and "this one" are also intended to include expressions such as "one or more", unless there is a clear indication to the contrary in the context. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" mean one or more than two (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0060] The reference to "one embodiment" or "some embodiments" etc. described in this specification means that a specific feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of this application. Thus, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments" etc. that appear in different places in this specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "include", "comprise", "have" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connection" includes direct connection and indirect connection, unless otherwise stated. "First" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features.

[0061] In the embodiments of this application, words such as "exemplarily" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0062] The following embodiments of this application relate to a human machine interface (HMI). An HMI is a media interface for interaction and information exchange between an application (APP) or an operating system (OS) and a user, and it realizes the conversion between the internal form of information and the form acceptable to the user. The HMI may include a user interface (UI), physical buttons, a virtual gesture recognition module, etc. The user interface is source code written in a specific computer language such as Java, extensible markup language (XML), etc. The interface source code is parsed and rendered on an electronic device and finally presented as content recognizable by the user. A common manifestation form of the user interface is a graphic user interface (GUI), which refers to a user interface related to computer operations displayed in a graphic manner. It can be visible interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, Widgets, etc. displayed on the display screen of an electronic device. The virtual gesture recognition module can be used to recognize virtual gestures of a user on an electronic device, such as air gestures.

[0063] This application provides a method for detecting multimedia content playback stuttering. By combining user operations on an electronic device with multimedia playback events to detect whether the multimedia content is stuttering, the accuracy of multimedia content playback stuttering detection can be improved. A multimedia application may refer to an APP that can play audio and / or video. For example, multimedia applications may include, but are not limited to: music APPs, video APPs, short video APPs, social APPs, and so on. The embodiments of this application do not limit the multimedia application. The multimedia application may also be referred to by other names such as audio and video playback applications.

[0064] Specifically, when a multimedia application starts playing a multimedia content, it can report a start-playing event to the multimedia processing module in the electronic device. When the interface currently displayed on the electronic device is the multimedia playback interface of the multimedia application, if the electronic device receives a preset operation, the electronic device can detect whether the multimedia application reports a start-playing event after the preset operation. Among them, the above preset operation can include an operation to trigger the multimedia application to play the multimedia content. For example, an operation of swiping up, an operation of swiping down, and so on. If the multimedia application reports a start-playing event after the preset operation, it can indicate that the multimedia content is played smoothly without stuttering. If the multimedia application does not report a start-playing event after the preset operation, it can indicate that the multimedia content is played with stuttering. When it is detected that the multimedia content is played with stuttering, the electronic device can perform operations such as network adjustment, and communicate using a network with better quality. This can reduce the stuttering duration and improve the user experience of listening to and watching the multimedia content.

[0065] Among them, after the above preset operation can include a preset time period starting from the start of the action of the above preset operation. That is to say, the electronic device can detect whether the multimedia application reports a start-playing event within the preset time period starting from the start of the action of the above preset operation.

[0066] It can be understood that after the electronic device receives the above preset operation, the multimedia application needs to respond to the preset operation to obtain the playback data of the multimedia content (such as audio data and / or video data) to play the multimedia content. Among them, the above playback data of the multimedia content can be pre-cached by the electronic device, or can also be obtained by the electronic device in real time based on the communication network. When the multimedia application can immediately obtain the playback data of the multimedia content, the multimedia application can immediately play the multimedia content and report a start-playing event. In this way, when the user triggers the multimedia application to play the multimedia content through an operation, the multimedia application can quickly load the playback data, and the multimedia content is played smoothly without stuttering. Therefore, when the multimedia content is played smoothly without stuttering, the electronic device can usually quickly obtain the start-playing event reported by the multimedia application after receiving the preset operation. However, if the network is poor and the multimedia application cannot obtain the playback data of the multimedia content for a long time, then the multimedia application will not report a start-playing event for a long time. In this way, when the user triggers the multimedia application to play the multimedia content through an operation, the multimedia application cannot quickly load the playback data, and thus cannot respond to the user operation in time to play the multimedia content, and the multimedia content appears stuttering. Therefore, when the multimedia content is played with stuttering, the electronic device may wait for a long time after receiving the preset operation to obtain the start-playing event reported by the multimedia application, and may even not be able to wait for the multimedia application to report the start-playing event.

[0067] The above method can accurately identify whether the electronic device can play the multimedia content smoothly without lag when the user triggers the electronic device to play the multimedia content. When the electronic device has pre-cached the playback data of the multimedia content, even if the network of the electronic device is poor when the user triggers the electronic device to play the multimedia content, the multimedia application can obtain the pre-cached playback data, and then immediately respond to the user operation to play the multimedia content and report the start playback event. Based on the above user operation and the start playback event, the electronic device can detect that the multimedia content does not lag. This can reduce the situation where the electronic device responds to the user operation to play the multimedia content smoothly, but misdetects the lag of the multimedia content playback based on the parameter indicating poor network. The above method can improve the accuracy of detecting the lag of multimedia content playback.

[0068] In some embodiments, the multimedia playback events can be divided into audio events and video events. The audio event can be used to indicate the playback state of the audio. The video event can be used to indicate the playback state of the video. For an independently existing audio, the electronic device can detect whether the audio playback lags according to the audio event. For an independently existing video, the electronic device can detect whether the video playback lags according to the video event. For the multimedia content combining audio and video, since the audio and video are played synchronously, the electronic device can detect whether the audio-video playback lags according to the audio event, or can also detect whether the audio-video playback lags according to the video event.

[0069] The multimedia processing module in the electronic device can include an audio module and a video module. Among them, the multimedia application can report the audio event to the audio module and report the video event to the video module.

[0070] For the convenience of understanding, the "audio event" involved in this application is introduced here.

[0071] When the multimedia application plays the audio, it can report the corresponding audio event to the audio module responsible for audio management and control in the electronic device according to the playback state of the audio. The audio event can include but is not limited to: start playback event, pause playback event, stop playback event, data exhaustion event, data resume event, etc.

[0072] The start event can be reported by the multimedia application when starting to play an audio. The multimedia application starting to play an audio can include: the multimedia application starts playing from the starting position of an audio, and the multimedia application continues to play after an audio is paused.

[0073] The pause event can be reported by a multimedia application when pausing the playback of an audio. In response to a user operation of pausing the audio playback, the multimedia application can report the pause event. When the audio is paused, the electronic device can save the progress of the audio at the time of pausing for resuming playback.

[0074] The stop event can be reported by a multimedia application when stopping the playback of an audio. When an audio playback ends or switches from one audio to another, the multimedia application can report the stop event. In some embodiments, the audio module can determine the stop event independently without the multimedia application reporting it. For example, the audio module can determine the moment when the audio stops playing based on the obtained data for playing the audio, so as to determine the stop event of the audio.

[0075] The timeout event can be an event between the start event and the pause event, or between the start event and the stop event. The timeout event can indicate that during the process of a multimedia application playing an audio, there is no audio data available for playback for a period exceeding a preset duration. The above-mentioned preset duration can take values such as 500 milliseconds (ms), 1 second (s), etc. The embodiments of this application do not limit the value of the preset duration. The multimedia application is in the state of audio playback starting from reporting the start event. The multimedia application can play the audio while obtaining subsequent audio data. Based on the caching technology, the multimedia application can pre-cache the audio data for a period of time subsequent to the current playback position. In this way, in the case where the network is temporarily poor and audio data cannot be obtained, the multimedia application can still use the cached audio data for audio playback. If the cached audio data is played out and the network is still poor and subsequent audio data cannot be obtained, the audio will experience stuttering. Among them, the multimedia application can send the audio data to the audio module. The audio module can play the audio according to the received audio data. If the audio module does not have audio data available for playback for a period exceeding the preset duration after receiving the start event, the audio module can determine the timeout event of the audio.

[0076] The resume event can be an event that occurs after the timeout event. The resume event can indicate that during the playback of audio by the audio module, there is again audio data available for resuming playback. After determining the timeout event of the audio, the audio module can wait for subsequent playback data of the audio. When audio data is obtained again, the audio module can determine the resume event of the audio. In some embodiments, the audio module can generate the resume event of the audio when it continuously obtains audio data within a period of time (such as 500 ms, or 1 s, or 2 s, etc.). The continuous obtaining of audio data within a period of time as described above can indicate that the network is currently stable and in good condition.

[0077] In some embodiments, in a scenario where the played audio is switched from audio a1 to audio a2, the multimedia application can report the stop event of audio a1 and the start event of audio a2. The stop event of audio a1 can indicate that the playback of audio a1 stops. The start event of audio a2 can indicate that the playback of audio a2 starts. This application does not limit the order of reporting the above stop event and start event.

[0078] In a scenario where audio a1 is being played, in response to an operation to pause the playback, the multimedia application can report a pause event. Then, in response to an operation to resume the playback of audio a1, the multimedia application can report a start event.

[0079] It can be seen that each audio can independently correspond to an audio event. The multimedia application can report the audio event corresponding to this audio according to the playback state of an audio. When the played audio is switched to another audio, the multimedia application can report the audio event corresponding to the other audio.

[0080] The above audio events are only exemplary descriptions of this application and should not constitute a limitation to this application. The multimedia application can also report more other types of audio events.

[0081] In some embodiments, video events can also include: start playback event, pause playback event, stop playback event, data exhaustion event, data recovery event, etc. The types of video events can refer to the introduction of the above audio events. Details are not described here again.

[0082] In the subsequent embodiments of this application, the method for the electronic device to detect multimedia content playback jitter based on user operation events and audio events is specifically used as an example to introduce the jitter detection method of this application. The method for the electronic device to detect multimedia content playback jitter based on user operation events and video events can refer to the method for detecting multimedia content playback jitter based on user operation events and audio events.

[0083] The structure of the electronic device involved in this application will be introduced below.

[0084] Figure 1 An exemplary schematic diagram of the hardware structure of the electronic device 100 is shown.

[0085] As Figure 1 shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio processing module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0086] It can be understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0087] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0088] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0089] A memory can also be set in the processor 110 for storing instructions and data. In some examples, the memory in the processor 110 is a cache memory. This memory can store the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0090] The USB interface 130 is an interface that conforms to the USB standard specification. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones.

[0091] The charging management module 140 is used to receive a charging input from a charger. Among them, the charger can be a wireless charger or a wired charger. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0092] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives inputs from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160, etc.

[0093] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0094] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example: the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0095] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through the antenna 1 and radiate it out.

[0096] The wireless communication module 160 may provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSSs), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive the signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 2 for radiation.

[0097] The electronic device 100 realizes the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering.

[0098] The display screen 194 is used to display images, videos, etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0099] The electronic device 100 may realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

[0100] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light passes through the lens and is transmitted to the camera photosensitive element. The optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye.

[0101] The camera 193 is used to capture still images or videos. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0102] The digital signal processor is used to process digital signals. In addition to being able to process digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0103] The NPU is a neural-network (NN) computing processor. By learning from the biological neural network structure, such as learning from the transmission mode between human brain neurons, it can quickly process the input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0104] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external memory card.

[0105] The internal memory 121 can be used to store computer-executable program code, and the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as user operation detection function, audio playback function, video playback function, etc.). The data storage area can store data created during the use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 can include high-speed random access memory and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0106] The electronic device 100 can implement audio functions through the audio processing module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and the application processor, etc. For example, music playback, recording, etc.

[0107] The audio processing module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio processing module 170 can also be used for encoding and decoding audio signals. In some examples, the audio processing module 170 can be disposed in the processor 110, or some functional modules of the audio processing module 170 can be disposed in the processor 110. The speaker 170A, also known as the "loudspeaker", is used to convert an audio electrical signal into a sound signal. The receiver 170B, also known as the "earpiece", is used to convert an audio electrical signal into a sound signal. The microphone 170C, also known as the "microphone" or "transmitter", is used to convert a sound signal into an electrical signal. The headphone jack 170D is used to connect a wired headphone.

[0108] The sensor module 180 may include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a gravity sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0109] The keys 190 include a power-on key, a volume key, etc. The motor 191 can generate a vibration prompt. The indicator 192 can be an indicator light, which can be used to indicate the charging state, the change in battery level, and can also be used to indicate messages, missed calls, notifications, etc.

[0110] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation from the electronic device 100. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communication. In some examples, the electronic device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0111] The electronic device 100 can be a mobile phone, a tablet computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The specific type of the electronic device 100 is not limited in the embodiments of the present application.

[0112] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of the present application, taking the system of the layered architecture as an example, the software structure of the electronic device 100 is exemplarily described.

[0113] Figure 2A Exemplarily shown is a schematic diagram of the software structure of the electronic device 100.

[0114] The layered architecture divides the software into several layers, each layer having a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system is divided into four layers, from top to bottom are the application layer, the application framework layer, Android Runtime and system libraries, and the kernel layer.

[0115] The application layer may include a series of application packages.

[0116] Such as Figure 2A shown, the application packages may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, short message, multimedia applications, etc. Among them, the multimedia application may refer to a type of APP related to audio and video playback in the electronic device 100.

[0117] The application framework layer provides APIs and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0118] Such as Figure 2A shown, the application framework layer may include window manager, content provider, view system, phone manager, resource manager, notification manager, activity manager, etc.

[0119] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0120] The content provider is used to store and obtain data, and make this data accessible to applications. The data may include videos, images, audio, dialed and answered calls, browsing history and bookmarks, phone book, etc.

[0121] The view system includes visible controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

[0122] The phone manager is used to provide the communication function of the electronic device 100. For example, the management of call status (including answering, hanging up, etc.).

[0123] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0124] The notification manager enables an application to display notification information in the status bar (such as the pull-down notification bar). It can be used to convey messages of the notification type, and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that a download is completed, a message is reminded, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or a scroll bar text, such as the notification of a background-running application, or a notification that appears in the form of a dialogue window on the screen. For example, it can prompt text information in the status bar, emit a prompt sound, vibrate the electronic device, blink the indicator light, etc.

[0125] The activity manager is responsible for managing activities. It is responsible for the startup, switching, scheduling of each component in the system, as well as the management and scheduling of applications. The activity manager can be called by upper-layer applications to open the corresponding activity.

[0126] An activity is an application component that can provide a window for users to interact and perform an operation, such as playing audio, taking pictures, making phone calls, etc. Each activity can obtain a window for drawing the user interface. A window can fill the screen of the electronic device 100, or can also be smaller than the screen of the electronic device 100, or float above other windows. That is to say, an activity can be a visual interface or a view interface for user operations. Each user interface of an application is an activity, and different user interfaces are different activities. An application can include one or more activities. For example, the audio and video playback interface in a multimedia application is an activity, and the comment viewing interface is another activity.

[0127] Android Runtime includes core libraries and a virtual machine. Android Runtime is responsible for the scheduling and management of the Android system.

[0128] The core libraries include two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core libraries of Android.

[0129] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to execute functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0130] The system library may include multiple functional modules. For example: a surface manager, Media Libraries, a 3D graphics processing library (such as OpenGL ES), a 2D graphics engine (such as SGL), etc.

[0131] The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0132] The media library supports the playback and recording of multiple common audio and video formats, as well as static image files, etc. The media library can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0133] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0134] The 2D graphics engine is a drawing engine for 2D drawing.

[0135] The kernel layer is the layer between hardware and software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0136] Figure 2A The software structure shown is only an exemplary illustration of this application and should not limit this application. In some embodiments, the electronic device 100 may also have Figure 2A more or fewer software modules than shown, or combine certain software modules, or split certain software modules, or arrange software modules at different locations.

[0137] Figure 2B Another structural schematic diagram of the electronic device 100 is exemplarily shown.

[0138] As Figure 2B shown, the electronic device 100 includes a multimedia application, a stutter detection module, a human-computer interaction module, a network management module, and a multimedia processing module. The multimedia application can refer to the introduction in the foregoing embodiments.

[0139] The human-computer interaction module can be used for a user to interact with the electronic device 100 and determine the information of the interaction between the user and the electronic device 100 through the human-computer interaction interface. For example, the human-computer interaction module can detect user operations on the electronic device 100. The above user operations can include but are not limited to: touch operations on the screen of the electronic device 100 (such as click operations, long-press operations, up-slide operations, down-slide operations, etc.), air gesture operations on the electronic device 100, and operations on the electronic device 100 based on one or more input devices such as a mouse, a keyboard, and a stylus. The embodiments of the present application do not limit the types of user operations that the human-computer interaction module can detect. For example, for a touch operation on the screen of the electronic device 100, the human-computer interaction module can detect information such as the position coordinates and action time of the touch operation on the screen. The embodiments of the present application do not limit the specific method for the human-computer interaction module to detect user operations. The human-computer interaction module can also be called by other names such as a user input module. In some embodiments, the human-computer interaction module can be located in Figure 2A the application framework layer shown.

[0140] The multimedia processing module can be used to process and play multimedia content. The multimedia processing module can include an audio module and a video module. Among them, the audio module can include a software module located in the application framework layer for managing or processing audio. Optionally, the audio module can also include a software program located in the application layer for audio management or audio playback. The audio module can also include a hardware device in the electronic device 100 for playing or processing audio, such as the Figure 1 audio processing module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, etc. shown. The embodiments of the present application do not limit the content of the audio module.

[0141] The audio module can be used to receive audio events reported by one or more multimedia applications in the electronic device 100. The audio events can be referred to the foregoing introduction. The audio module can also receive audio data from a multimedia application and play the audio according to the audio data.

[0142] The video module can include a software module located in the Figure 2A application framework layer shown for managing or processing video. Optionally, the video module can also include a software program located in the application layer for video management or video playback. The video module can also include a three-dimensional graphics processing library, a two-dimensional graphics engine, a media library, etc. in the Figure 2A system library shown to implement the drawing and display of video pictures. The video module can also include a hardware device in the electronic device 100 for playing or processing video. The embodiments of the present application do not limit the content of the video module.

[0143] The video module can be used to receive video events reported by one or more multimedia applications in the electronic device 100. The video module can also receive video data from the multimedia application and play the video according to the video data. In some embodiments, after the video module receives an event indicating the start of video playback, if the video module does not have video data to play for a preset duration, the video module can generate an event indicating that the video data has been exhausted. Then, the video module can wait for subsequent video playback data. When the video data is obtained again, the video module can generate an event indicating the recovery of the video data.

[0144] The stutter detection module can be used to detect stuttering in the playback of multimedia content. In some embodiments, the stutter detection module can register with the human-computer interaction module for reporting the monitoring of user operation events. After the human-computer interaction module detects a user operation, it can send a user operation event to the stutter detection module to indicate information about the user operation on the electronic device 100 to the stutter detection module. The stutter detection module can also register with the audio module for reporting the monitoring of audio events. After the audio module receives an audio event reported by the multimedia application, it can send the audio event to the stutter detection module. In this way, the stutter detection module can detect whether the currently played audio in the electronic device 100 is stuttering based on the above user operation events and audio events.

[0145] If the audio is not stuttering, the stutter detection module can continue to detect and wait for new user operation events and audio events.

[0146] If the audio is stuttering, the stutter detection module can instruct the network management module to adjust the network so that the electronic device 100 can communicate using a network with better quality.

[0147] In some embodiments, the stutter detection module can register with the video module for reporting the monitoring of video events. After the video module receives a video event reported by the multimedia application, it can send the video event to the stutter detection module. In this way, the stutter detection module can detect whether the currently played video in the electronic device 100 is stuttering based on the above user operation events and video events.

[0148] It can be seen that the stutter detection module can detect whether the multimedia content playback of any multimedia application in the electronic device 100 is stuttering at the system level. This can eliminate the need for developers of multimedia applications to perform additional operations to detect stuttering in multimedia content playback, saving their development costs and improving development efficiency.

[0149] The network management module can be used to manage the network accessed by the electronic device 100. In some embodiments, the network management module can use one or more strategies such as Wi-Fi and cellular handover, cell handover, primary and secondary SIM card handover, multi-network handover, multi-network concurrency, Wi-Fi roaming, etc. to adjust the network for the electronic device 100 to perform data communication, so that the electronic device 100 can communicate using a network with better quality.

[0150] Among them, the above-mentioned Wi-Fi and cellular handover can mean that when the electronic device 100 is connected to both a Wi-Fi network and a cellular mobile network, the network management module can switch the network used for data communication with other devices between the Wi-Fi network and the cellular mobile network according to the quality of the Wi-Fi network and the cellular mobile network. When the quality of the Wi-Fi network is better than that of the cellular mobile network, the network management module can adjust the network for the electronic device 100 to communicate with other devices to the Wi-Fi network. When the quality of the cellular mobile network is better than that of the Wi-Fi network, the network management module can adjust the network for the electronic device 100 to communicate with other devices to the cellular network.

[0151] The primary and secondary SIM card handover can mean that the electronic device 100 includes a primary SIM card and a secondary SIM card, and the network management module can switch the network for data communication with other devices between the cellular mobile networks corresponding to the primary SIM card and the secondary SIM card.

[0152] The network management module adjusting the network for the electronic device 100 to communicate from one network to another can include but is not limited to: adjusting from a Wi-Fi network to a cellular mobile network, adjusting from the cellular mobile network corresponding to the primary SIM card to the cellular mobile network corresponding to the secondary SIM card, adjusting from a cellular mobile network of one standard to a cellular mobile network of another standard (such as adjusting from a 4G cellular mobile network to a 5G cellular mobile network), the cellular mobile network moving from one resident cell to another resident cell, and so on.

[0153] The embodiments of the present application do not limit the method for the network management module to adjust the network for the electronic device 100 to communicate.

[0154] In some embodiments, both the lag detection module and the network management module can be located in the aforementioned Figure 2A shown application framework layer.

[0155] In some embodiments, the lag detection module can also forward the aforementioned Figure 2AThe activity manager shown obtains information about the activities in the electronic device 100 to determine whether the current electronic device 100 contains an activity of a multimedia playback interface and whether the multimedia playback interface is currently displayed on the screen. The multimedia playback interface can be used to control the playback of multimedia content. The multimedia playback interface can include a video playback interface and an audio playback interface. Among them, some multimedia content is a combination of video and audio, and the video and audio are played together. When the electronic device 100 plays this some multimedia content, the electronic device 100 can play the audio associated with the video while displaying the video playback interface. Some other multimedia content is an independent video, that is, the video is not associated with audio. When the electronic device 100 plays this some other multimedia content, the electronic device 100 can display the video playback interface without playing audio. Some audio exists independently and is not associated with video. When the electronic device 100 plays this some audio, the electronic device 100 can display the audio playback interface.

[0156] In some embodiments, the electronic device 100 may further include a process management module ( Figure 2B not shown in the figure). The process management module can be used to manage the processes in the electronic device 100. The process management module can create processes, allocate CPU time slice resources for processes, manage the states of processes, and so on. The stutter detection module can obtain information about the processes running in the electronic device 100 from the process management module to determine whether the multimedia application in the electronic device 100 is in a running state and whether the multimedia application is running in the foreground or in the background.

[0157] The following introduces the multimedia content playback stutter detection scenarios provided by this application.

[0158] Figures 3A - 3H Exemplarily shows some schematic diagrams of multimedia content playback stutter detection scenarios.

[0159] Here, an example where the multimedia content is a combination of audio and video is used for illustration.

[0160] As Figure 3A shown, the electronic device 100 can display the user interface 310. The user interface 310 can be a multimedia playback interface (i.e., an audio-video playback interface) of a multimedia application. The user interface 310 may include a status bar 311, a video display area 312, a follow control 313, a like control 314, a comment control 315, and a favorite control 316.

[0161] The status bar 311 may include a signal strength indicator 311A, a Wi-Fi logo 311B, a battery level indicator 311C, and time information 311D.

[0162] The signal strength indicator 311A can be used to indicate the signal strength of the electronic device 100 accessing the cellular mobile network. In some embodiments, one or more signal strength indicators may be included in the status bar 311. For example, in the case where the electronic device 100 includes two SIM cards, the electronic device 100 can display two signal strength indicators in the status bar 311. These two signal strength indicators can respectively represent the signal strengths of the electronic device 100 accessing the cellular mobile network through the two SIM cards.

[0163] The Wi-Fi identifier 311B can be used to indicate that the electronic device 100 accesses the Wi-Fi network. In some embodiments, when the signal strength indicator 311A and the Wi-Fi identifier 311B are simultaneously displayed in the status bar 311, it can indicate that the electronic device 100 is currently performing data communication with other devices through the Wi-Fi network.

[0164] The battery level indicator 311C can be used to indicate the current battery level of the electronic device 100.

[0165] The time information 311D can be used to indicate the current time.

[0166] The video display area 312 can be used to display video images. In the case of video and audio being merged, the video and audio are integrated, and the electronic device 100 can play the audio and video synchronously. In some embodiments, when the audio lags, the video played synchronously with the audio will also lag. That is, the lag of the audio and video occurs synchronously.

[0167] The follow control 313, like control 314, comment control 315, and favorite control 316 can be displayed on top of the video image. The follow control 313 can be used to follow the publisher of the audio and video in the video display area 312. The like control 314 can be used to like the audio and video played in the video display area 312. The comment control 315 can be used to bring up the comment interface for commenting on the audio and video played in the video display area 312. The favorite control 316 can be used to favorite the audio and video played in the video display area 312.

[0168] During Figure 3A the process of playing the audio and video as shown, the electronic device 100 can play the audio and video while online obtaining subsequent play data of the audio and video through a network (such as a Wi-Fi network). Among them, the electronic device 100 can pre-cache the play data of the audio and video for a subsequent period of time at the current play position.

[0169] In response to Figure 3A the click operation acting on the video display area 312 as shown, the electronic device 100 can pause playing the audio and video and display Figure 3BThe user interface 310 shown above. The click operation for pausing the playback of audio and video can be an operation performed at any position in the video display area 312 other than the positions where the follow control 313, like control 314, comment control 315, and favorite control 316 are located.

[0170] As Figure 3B shown, when pausing the playback of audio and video, the electronic device 100 can display a playback control 312A in the video display area 312. The playback control 312A can indicate that the audio and video are currently in a paused playback state and can be used to trigger the electronic device 100 to continue playing the audio and video. In some embodiments, not limited to operating on the playback control 312A to trigger the electronic device 100 to continue playing the audio and video, the electronic device 100 can, in the state where the audio and video are paused, respond to a click operation performed at any position in the video display area 312 other than the positions where there are functional controls (such as the follow control 313, like control 314, comment control 315, and favorite control 316), and continue to play the audio and video.

[0171] In response to Figure 3A the operation of pausing the playback of audio and video shown above, the multimedia application in the electronic device 100 can report a pause event to the audio module, and the user interaction module can send the user operation event corresponding to this operation to the stutter detection module. The user operation event can include information such as the position and time of the operation on the screen. The audio module can also send the pause event to the stutter detection module. The stutter detection module can record the above user operation event and pause event. Optionally, when pausing the playback of audio and video, the multimedia application can also report a pause event indicating that the video is paused to the video module. Then, the video module can send the pause event to the stutter detection module.

[0172] In some embodiments, the stutter detection module in the electronic device 100 can identify the user operation based on the received user operation event and determine whether the user operation is a preset operation. For example, the preset operations include a swipe-up operation and a swipe-down operation, but do not include a click operation. The stutter detection module can identify the Figure 3A user operation shown above as a click operation based on the user operation event. The stutter detection module can determine that this click operation is not a preset operation. When the user operation indicated by the user operation event is not a preset operation, the stutter detection module can continue to wait for new user operation events and audio events.

[0173] As Figure 3B shown, in the state where the audio and video are paused, in response to a click operation on the playback control 312A, the electronic device 100 can continue to play the audio and video and display Figure 3C the user interface 310 shown above.

[0174] As shown Figure 3C in FIG., the electronic device 100 may display a video frame of continued video playback in the video display area 312 and continue to play the audio associated with the video. Among them, in response to Figure 3B the operation of playing audio as shown, the multimedia application in the electronic device 100 may report a start event to the audio module, and the user interaction module may send the user operation event corresponding to the operation to the stuttering detection module. The audio module may also send the start event to the stuttering detection module. It should be noted that the above start event may be reported by the multimedia application after obtaining the playback data starting from the continued playback position in the audio and video corresponding to the content displayed in the video display area 312.

[0175] The stuttering detection module may record the above user operation event and start event. The stuttering detection module may identify the Figure 3B user operation shown in FIG. as a click operation based on the above user operation event, so as to determine that the click operation is not a preset operation.

[0176] When the electronic device 100 plays an audio and video, the electronic device 100 may switch to play another audio and video according to a user operation.

[0177] As shown Figure 3C in FIG., in response to an operation of swiping upward in the video display area 312, the electronic device 100 may display Figure 3D the user interface 320 shown. The user interface 320 may be an audio and video playback interface displayed by the electronic device 100 during the process of the upward swipe operation.

[0178] As shown Figure 3D in FIG., the user interface 320 may include a video frame 321 and a video frame 322. The video frame 321 and the video frame 322 may be video frames corresponding to different audio and videos. The video frame 321 may be Figure 3C the video frame displayed in the video display area 312 shown in FIG.

[0179] Among them, during the process of the user's finger gradually swiping upward, the electronic device 100 may gradually move the video frame 321 and the video frame 322 upward. The video frame 321 gradually moves upward, and the content above the video frame 321 that exceeds the display area is hidden. The video frame 322 gradually moves upward, and the content that was originally hidden below the video frame 321 is gradually displayed. That is to say, as the user's finger swipes upward, the content of the video frame 321 displayed on the screen becomes less and less, and the display area it occupies also becomes smaller and smaller. The content of the video frame 322 displayed on the screen becomes more and more, and the display area it occupies also becomes larger and larger.

[0180] In some embodiments, when the display area occupied by the video frame 322 exceeds a preset area size and the upward sliding operation ends (i.e., the finger performing the upward sliding operation leaves the screen), the electronic device 100 may switch to play the audio-video corresponding to the video frame 322 and display Figure 3E the user interface 330 as shown.

[0181] In some embodiments, the audio-video corresponding to the video frame 322 is the next audio-video to be played after the audio-video corresponding to the video frame 321. The electronic device 100 may pre-obtain and cache the play data of the audio-video corresponding to the video frame 322. For example, based on caching technology, when playing the audio-video corresponding to the video frame 321, in addition to all the play data of the audio-video corresponding to the video frame 321, the data obtained by the electronic device 100 online through the network may also include some or all of the play data of the subsequent audio-video (i.e., the audio-video corresponding to the video frame 322). Alternatively, the electronic device 100 may also Figure 3C obtain the play data of the audio-video corresponding to the video frame 322 online through the network in response to the upward sliding operation as shown. Among them, the electronic device 100 may start obtaining the play data of the audio-video corresponding to the video frame 322 during the process of the upward sliding operation, or may also start obtaining the play data of the audio-video corresponding to the video frame 322 after the upward sliding operation is completed.

[0182] In response to Figure 3C and Figure 3D the upward sliding operation as shown, the multimedia application in the electronic device 100 may report a stop event and a start event to the audio module, and the user interaction module may send the user operation event corresponding to the upward sliding operation to the stutter detection module. The audio module may also send the above stop event and start event to the stutter detection module. The above stop event may be used to indicate the stop of playing the audio-video corresponding to the video frame 321. The above start event may be used to indicate the start of playing the audio-video corresponding to the video frame 322. It should be noted that the above start event may be reported by the multimedia application when it obtains the play data of the audio-video corresponding to the video frame 322. That is to say, when it is necessary to play the audio-video corresponding to the video frame 322, the multimedia application reports the start event only when there is play data that can play the audio-video corresponding to the video frame 322. If it is necessary to play the audio-video corresponding to the video frame 322, but the multimedia application does not obtain the play data of the audio-video corresponding to the video frame 322, the multimedia application will not report the start event.

[0183] After receiving the user operation event corresponding to the above upward sliding operation, the stutter detection module may determine that the user operation is an upward sliding operation according to the user operation event, and thus determine Figure 3C andFigure 3D The user operation shown is a preset operation. When a preset operation is detected, the stuttering detection module can detect whether a start event is received within a preset time period starting from the time when the preset operation takes effect. The preset time period can be 2s, 3s, or other values. The embodiment of the present application does not limit the duration of the preset time period. If the stuttering detection module receives a start event within the preset time period, the stuttering detection module can continue to wait for new user operation events and audio events. If the stuttering detection module does not receive a start event within the preset time period, the stuttering detection module can instruct the network management module in the electronic device 100 to adjust the network so that the electronic device 100 uses a network with better quality for communication. In this way, the electronic device 100 can obtain the playback data of the audio and video corresponding to the video frame 322 as soon as possible, eliminating the stuttering of the audio and video.

[0184] Figure 3D and Figure 3E Exemplarily shows a scenario where the audio and video to be played after switching is not stuttering when switching audio and video. Among them, in response to Figure 3C and Figure 3D the upward sliding operation shown, the multimedia application can quickly obtain the playback data of the next audio and video to be played after switching, and thus play the switched audio and video according to the playback data, displaying Figure 3D the video frame 322 shown and Figure 3E the video display area 331 shown.

[0185] As Figure 3E shown, the user interface 330 may include a video display area 331, a follow control 332, a like control 333, a comment control 334, and a favorite control 335. The video frame displayed in the video display area 331 is the complete frame of the Figure 3D video frame 322 shown. The audio and video corresponding to the video frame in the video display area 331 can be Figure 3C the next audio and video to be played after the audio and video corresponding to the video frame in the video display area 312 shown. The follow control 332 can be used to follow the publisher of the audio and video in the video display area 331. The like control 333 can be used to like the audio and video played in the video display area 331. The comment control 334 can be used to bring up the comment interface for commenting on the audio and video played in the video display area 331. The favorite control 335 can be used to favorite the audio and video played in the video display area 331.

[0186] In some embodiments, when switching audio and video, the audio and video to be played after switching may be stuttering.

[0187] Figures 3F - 3H Exemplarily shows a scenario where the played audio and video is stuttering when switching audio and video.

[0188] Exemplarily, during the process of playing audio and video as shown in Figure 3C , in response to a swiping-up operation, if the electronic device 100 fails to obtain the playing data of the next audio and video to be played due to poor network or other reasons, the electronic device 100 may display Figure 3F the user interface 340 as shown. The user interface 340 may be the audio and video playing interface displayed by the electronic device 100 during the process of the swiping-up operation.

[0189] As Figure 3F shown, the user interface 340 may include a video screen 341 and a display area 342. The video screen 341 may refer to the video screen 321 as shown in the foregoing Figure 3D . The display area 342 may be used to play the next audio and video to be played corresponding to the audio and video of the video screen 341. Since the electronic device 100 fails to obtain the playing data of the next audio and video to be played, the electronic device 100 may display Figure 3F the prompt message as shown. For example, the content of the prompt message may be "Network timeout, please wait...". The prompt message may indicate that the next audio and video to be played is stuck. The present application does not limit the content of the prompt message.

[0190] When the swiping-up operation ends, the electronic device 100 may display Figure 3G the user interface 350 as shown. The user interface 350 may include a video display area 351. Since the electronic device 100 still fails to obtain the playing data of the audio and video to be played, the electronic device 100 may display a prompt message in the video display area 351. The prompt message may refer to Figure 3F the prompt message in the display area 342 as shown.

[0191] In response to Figure 3C and Figure 3F the swiping-up operation as shown, the user interaction module in the electronic device 100 may send the user operation event corresponding to the swiping-up operation to the stuttering detection module. The stuttering detection module may determine that the user operation is a swiping-up operation according to the user operation event, so as to determine Figure 3C and Figure 3F the user operation as a preset operation. When detecting a preset operation, the stuttering detection module may detect whether a start event is received within a preset time period starting from the action time of the preset operation. Since the multimedia application fails to obtain the playing data of the next audio and video to be played for a long time, the stuttering detection module does not receive a start event within the above preset time period. The stuttering detection module may determine that the multimedia content playing is stuck, so as to instruct the network management module to adjust the network.

[0192] For example, the network management module may adjust the network for the electronic device 100 to communicate with other devices from a Wi-Fi network to a cellular mobile network.

[0193] As Figure 3G shown, a Wi-Fi logo 311B may be displayed in the user interface 350. The Wi-Fi logo 311B may indicate that the network for the electronic device 100 to currently communicate with other devices is a Wi-Fi network. When using the Wi-Fi network, the multimedia application fails to obtain the playback data of the next audio / video to be played for a long time. The network management module may, according to the instruction of the stuttering detection module to adjust the network, search for a network with better quality than the Wi-Fi network. When it is found that the quality of the cellular mobile network accessed by the electronic device 100 is better than that of the Wi-Fi network, the network management module may adjust the network for data communication to the cellular mobile network.

[0194] In some embodiments, when adjusting from a Wi-Fi network to a cellular mobile network, the electronic device 100 may display Figure 3H the cellular mobile network logo 311E as shown. The cellular mobile network logo 311E may indicate that the network for the electronic device 100 to currently communicate with other devices is a cellular mobile network. The electronic device 100 may use the cellular mobile network to communicate with other devices (such as an audio / video server) to obtain the playback data of the audio / video to be played online. When the playback data is obtained, the electronic device 100 may play the audio / video according to the playback data and display Figure 3H the video picture as shown in the video display area 351. Among them, after the multimedia application in the electronic device 100 obtains the playback data of the audio / video to be played, it may report a start event to the audio module. Then, the audio module may send the start event to the stuttering detection module.

[0195] In some embodiments, after the stuttering detection module determines that the multimedia content playback is stuttering, it may instruct the network management module to adjust the network. When receiving the start event, the stuttering detection module may stop instructing the network management module to adjust the network.

[0196] The above Figures 3F - 3H scenario of adjusting the network is only an exemplary illustration of the present application and should not limit the present application. The network management module may also use other methods to adjust the network for the electronic device 100 to communicate.

[0197] It should be noted that the above Figures 3A - 3HTaking the scenario shown above where the preset operation is a swiping-up operation and the click operation is not the preset operation as an example, the process of detecting multimedia content playback stuttering is introduced. The embodiments of the present application do not limit the type of the preset operation. In some embodiments, the click operation can also be the preset operation. The criteria for setting the above preset operation may include: setting one or more operations that can trigger the multimedia application to play audio as the preset operation.

[0198] From the scenario shown above Figures 3A - 3H it can be seen that in response to the preset operation for triggering the playback of multimedia content, when the playback data of the multimedia content to be played in the multimedia application in the electronic device 100 can be played, the start event will be reported. If the multimedia application does not report the start event for a long time after the electronic device 100 receives the preset operation, it may indicate that the multimedia application has not obtained the playback data of the multimedia content to be played, and the multimedia content playback is stuttering. Therefore, the present application can use the stuttering detection module to detect whether the start event is received within the preset time period after the preset operation is applied, so as to accurately detect whether the multimedia content to be played is stuttering in the scenario of triggering the multimedia content playback through the preset operation.

[0199] In some embodiments, the stuttering detection module can also combine the playback state of the multimedia content, the type and position of the user operation to determine whether to wait for the start event to detect the multimedia content playback stuttering. Taking the click operation as an example for illustration. Since there may be multiple function controls on the multimedia playback interface. The click operations acting on different positions on the multimedia playback interface may have different purposes. In addition, in different multimedia content playback states, the click operations acting on the same position on the multimedia playback interface may also have different purposes.

[0200] Exemplarily, in Figure 3A the audio-video playback interface shown above, the click operation acting on the area of the video display area 312 except the positions where the function controls (such as the follow control 313, like control 314, comment control 315, favorite control 316) are located can be used to play or pause the audio-video. Among them, when the audio-video is in the playing state, the click operation acting on the area of the video display area 312 except the positions where the function controls are located can be used to pause the audio-video. When the audio-video is in the paused state, the click operation acting on the area of the video display area 312 except the positions where the function controls are located can be used to play the audio-video.

[0201] When the audio is paused, the multimedia application can report a pause event. After receiving the pause event, if the stutter detection module determines, based on the received user operation event, that the user operation is a click operation on a preset area on the screen, the stutter detection module can detect whether a start event is received within a preset time period starting from the time when the click operation takes effect. For example, the above-mentioned preset area can be Figure 3A the area other than the position where the function control is located in the video display area 312 shown in Figure 3A . Among them, if the stutter detection module receives a start event within the preset time period starting from the time when the click operation takes effect, the audio is not stuttering, and the stutter detection module can continue to wait for new user operation events and audio events. If the stutter detection module does not receive a start event within the preset time period starting from the time when the click operation takes effect, the audio is stuttering, and the stutter detection module can instruct the network management module to adjust the network so that the electronic device 100 can communicate using a network with better quality.

[0202] The above-mentioned preset area can be determined according to the audio-video playing interface. The preset areas corresponding to different audio-video playing interfaces can be different. The electronic device 100 can determine whether the user operation is an operation to trigger audio playback based on whether the user operation acts on the preset area of the audio-video playing interface.

[0203] In the above embodiments, the electronic device 100 can not only accurately detect whether the multimedia content to be played is stuttering in the scenario where audio playback is triggered by a swipe-up operation or a swipe-down operation, but also accurately detect whether the multimedia content to be played is stuttering in the scenario where multimedia content playback is triggered by a click operation.

[0204] The above embodiments can more accurately identify the user operation for triggering multimedia content playback by combining the playback state of the multimedia content, the type and position of the user operation, so as to detect whether the multimedia content to be played is stuttering in more scenarios where multimedia content playback is triggered by user operations.

[0205] In some embodiments, in addition to detecting whether the multimedia content is stuttering when triggering multimedia content playback, the stutter detection module can also detect whether the multimedia content is stuttering during the playback process.

[0206] Figure 4A and Figure 4B Exemplarily shows schematic diagrams of other scenarios for detecting stuttering of multimedia content playback.

[0207] Such as Figure 4AAs shown, the electronic device 100 can display a user interface 410. The user interface 410 can be an audio and video playback interface of a multimedia application. The user interface 410 may include a Wi-Fi indicator 411 and a video display area 412. The Wi-Fi indicator 411 can indicate that the network through which the electronic device 100 is currently communicating data with other devices is a Wi-Fi network. The video display area 412 can display a video picture.

[0208] Here, still taking the multimedia content played by the electronic device 100 as an example of the combination of audio and video for illustration. The electronic device 100 can synchronously play a video and the audio associated with the video. That is, when the electronic device 100 displays a video picture in the Figure 4A shown video display area 412, it is also playing audio.

[0209] Among them, when starting to play the audio and video, the multimedia application in the electronic device 100 can report a start event. The stutter detection module can receive the start event. The electronic device 100 can play the corresponding video and audio according to the playback data of the to-be-played audio and video that has been obtained, while obtaining subsequent playback data. During the process of playing an audio and video, if the playback data obtained by the electronic device 100 has been played out, and the electronic device 100 has not obtained subsequent playback data, then the electronic device 100 currently has no playback data available to play the audio and video. At this time, both the audio and the video experience stuttering. The electronic device 100 can display Figure 4A the shown prompt message 412A in the video display area 412. The content of the prompt message 412A can include "Network timeout, please wait...". The prompt message 412A can be used to prompt the user that the audio and video are stuttering. The content of the prompt message 412A in this application is not limited.

[0210] In addition, if there is no playback data available to play the audio and video for more than a preset duration, the audio module can generate a timeout event and send the timeout event to the stutter detection module. When receiving the timeout event, the stutter detection module can determine that the audio is stuttering, and then instruct the network management module to adjust the network so that the electronic device 100 communicates using a network with better quality.

[0211] For example, the network management module can adjust the network through which the electronic device 100 communicates data with other devices from a Wi-Fi network to a cellular mobile network.

[0212] Such as Figure 4BAs shown, the electronic device 100 can display the cellular mobile network identifier 413. The cellular mobile network identifier 413 can indicate that the network through which the electronic device 100 currently communicates data with other devices is a cellular mobile network. The electronic device 100 can use the cellular mobile network to communicate with other devices (such as an audio-video server) and obtain the playback data of the audio-video to be played online. When the playback data is obtained, the electronic device 100 can resume playing the audio-video according to the playback data and display the Figure 4B video picture shown in the video display area 412. In this way, the stuttering of the audio-video is eliminated. Among them, after the multimedia application in the electronic device 100 obtains the playback data of the audio-video to be played, it can send the playback data to the audio module. After receiving the playback data, the audio module can generate a resume event and send the resume event to the stuttering detection module.

[0213] In some embodiments, after receiving the timeout event and before receiving the resume event, the stuttering detection module can instruct the network management module to adjust the network through which the electronic device 100 communicates one or more times. Exemplarily, when receiving the timeout event, the stuttering detection module can first instruct the network management module to adjust the network through which the electronic device 100 communicates. After instructing the network management module to adjust the network through which the electronic device 100 communicates, the stuttering detection module can wait for the resume event within a preset time period. If the resume event is not received within the preset time period after instructing the network management module to adjust the network through which the electronic device 100 communicates, the stuttering detection module can instruct the network management module to adjust the network through which the electronic device 100 communicates again. The stuttering detection module can instruct the network management module to adjust the network through which the electronic device 100 communicates multiple times according to the above method until the resume event is received.

[0214] It can be understood that after the network management module adjusts the network according to an instruction from the stuttering detection module, the adjusted network may still be poor. The electronic device 100 may still be unable to communicate with devices such as an audio and video server based on the adjusted network, and thus unable to obtain the playback data of the audio and video to be played. Then, the audio module will also be unable to generate a resume event for a long time. The stuttering detection module may instruct the network management module again to adjust the network used by the electronic device 100 for communication. Then, the network management module can adjust the network again so that the electronic device 100 communicates using a network with better quality. Exemplarily, the network management module may first adjust the network for data communication between the electronic device 100 and other devices from a Wi-Fi network to a cellular mobile network. When receiving an instruction from the stuttering detection module to adjust the network again, the network management module may use strategies such as cellular cell handover or primary / secondary SIM card handover to enable the electronic device 100 to communicate using a network with better quality. This can facilitate the electronic device 100 to obtain the playback data of the audio and video to be played as soon as possible and eliminate the stuttering of the audio and video.

[0215] From the above Figure 4A and Figure 4B scenarios, it can be seen that the timeout event can reflect that stuttering occurs during the playback of audio. The electronic device 100 can accurately detect whether stuttering occurs during the playback of audio based on the timeout event, and adjust the network when the audio is stuttering, and communicate using a network with better quality to eliminate the stuttering. This can better improve the user experience of listening to audio.

[0216] Based on the above scenarios of multimedia content playback stuttering detection, the method for multimedia content playback stuttering detection provided by this application will be specifically introduced below. Here, an example is given where the electronic device detects multimedia content playback stuttering based on user operation events and audio events.

[0217] Figure 5 An exemplary flowchart of a method for multimedia content playback stuttering detection is shown.

[0218] As Figure 5 shown, the electronic device 100 may include a stuttering detection module 501, a human-computer interaction module 502, an audio module 503, a network management module 504, and a multimedia application 505. The stuttering detection module 501, the human-computer interaction module 502, the audio module 503, and the network management module 504 can all refer to the foregoing Figure 2B description. The multimedia application 505 may be an APP for playing multimedia content in the electronic device 100. Without being limited to the multimedia application 505, the electronic device 100 may also include more multimedia applications.

[0219] S511. The stuttering detection module 501 detects whether there is a multimedia application in a running state.

[0220] Multiple APPs can run simultaneously in the electronic device 100. Among them, some APPs may be in the foreground running state. Some other APPs may be in the background running state. When there is a multimedia application running in the electronic device 100, the electronic device 100 may play multimedia content through the running APP. Therefore, when there is a multimedia application running, the electronic device 100 may experience jitter in multimedia content playback. Then, the electronic device 100 can detect whether there is jitter in multimedia content playback through the jitter detection module 501.

[0221] In some embodiments, the jitter detection module 501 can detect whether there is a multimedia application running by obtaining the process information in the electronic device 100. A process can represent a program running in the system and is an abstraction of a running program. Each running program can constitute a process. If there is a process of a multimedia application (such as the multimedia application 505) in the electronic device 100, the jitter detection module 501 can determine that there is a multimedia application running in the electronic device 100. Among them, a process can include a foreground process and a background process. A foreground process can include a process currently displayed on the screen and some system processes. A foreground process can be a process that is interacting with the user. A background process can include a process that is currently invisible to the user, that is, a process running in the background. The electronic device 100 can determine whether the multimedia application is running in the foreground or in the background according to whether the process of the multimedia application is a foreground process or a background process.

[0222] Not limited to determining whether there is a multimedia application running through a process, the jitter detection module 501 can also use other methods to determine whether there is a multimedia application running. The embodiments of the present application do not limit the method for determining whether a multimedia application is running.

[0223] In some embodiments, when it is detected that there is a multimedia application running, the electronic device 100 can perform the following steps S512 to S513.

[0224] S512. The jitter detection module 501 registers the monitoring and reporting of user operation events to the human-computer interaction module 502.

[0225] The human-computer interaction module 502 can be used to detect user operations on the electronic device 100. The human-computer interaction module 502 can accept the registration of monitoring and reporting of user operation events by other modules or applications in the electronic device 100. When a user operation is detected, the human-computer interaction module 502 can send user operation events to these software or applications that have registered for monitoring and reporting of user operation events.

[0226] The stuttering detection module 501 registers the monitoring and reporting of user operation events to the human-computer interaction module 502, and can obtain information on user operations received by the electronic device 100.

[0227] S513. The stuttering detection module 501 registers the monitoring and reporting of audio events to the audio module 503.

[0228] The audio module 503 can receive audio events reported by a multimedia application and send the audio events to other modules or applications that have registered audio event reporting to the audio module 503.

[0229] The stuttering detection module 501 registers the monitoring and reporting of audio events to the audio module 503, and can obtain the playing state of the audio.

[0230] In some embodiments, without being limited to detecting multimedia applications, when the stuttering detection module 501 detects that at least one APP in the electronic device 100 is in a running state, steps S512 and S513 can be executed. This APP can be any APP.

[0231] In some embodiments, in step S511, when it is detected that no multimedia application is in a running state, the electronic device 100 can repeatedly execute step S511 to wait for at least one multimedia application to run on the electronic device 100.

[0232] Step S511 can be optional. For example, after the electronic device 100 is powered on and the system starts, the stuttering detection module 501 can execute steps S512 and S513. That is, the stuttering detection module 501 can detect whether there is a multimedia application running in the electronic device 100 without performing such detection.

[0233] S514. The user operates the electronic device 100, and the operation 1 is used to play the audio a1.

[0234] In some embodiments, the operation 1 can be an operation of switching the audio a2 to the audio a1 when the audio a2 is being played on the electronic device 100. The operation 1 can refer to the Figure 3D swipe-up operation shown above.

[0235] In some embodiments, the operation 1 can be an operation of triggering the electronic device 100 to continue playing the audio a1 when the audio a1 is in a paused state. The operation 1 can refer to the Figure 3B click operation shown above.

[0236] The embodiments of the present application do not limit the operation 1.

[0237] Here, it is described by taking the operation 1 as a user operation acting on the multimedia playback interface of the multimedia application 505 as an example.

[0238] In some embodiments, the above audio a1 may be associated with a video. When the electronic device 100 plays the audio a1, it can synchronously play the video associated with the audio a1. That is, the multimedia content played by the electronic device 100 is a combination of the audio a1 and the video. When the audio a1 is played, a video picture can be displayed in the multimedia playback interface displayed by the electronic device 100 on the screen. Alternatively, the above audio a1 is independent and not associated with a video. And the multimedia content played by the electronic device 100 is the independent audio a1. The present application does not limit the type of the audio a1.

[0239] The human-computer interaction module 502 can detect the operation 1 and then execute the following steps S515 and S516.

[0240] S515. The human-computer interaction module 502 sends a user operation event corresponding to the operation 1 to the stuttering detection module 501.

[0241] S516. The human-computer interaction module 502 sends a user operation event corresponding to the operation 1 to the multimedia application 505.

[0242] The human-computer interaction module 502 can identify which APP in the electronic device 100 the operation 1 is an operation on. Since the operation 1 is a user operation on the multimedia playback interface of the multimedia application 505, the human-computer interaction module 502 can send a user operation event corresponding to the operation 1 to the multimedia application 505 so that the multimedia application 505 can respond to the operation 1 and perform corresponding operations.

[0243] The user operation event corresponding to the operation 1 may include information such as the position where the operation 1 acts on the screen and the acting time. The user operation event can be used to determine the operation type of the operation 1.

[0244] For example, the user operation event corresponding to the operation 1 may include: [InputEvent1: (x1, y1), 0-0-0, t1], [InputEvent2: (x2, y2), 1-0-0, t2]. Among them, in InputEvent1, 0-0-0 can represent a finger press, (x1, y1) can represent the coordinates where the finger presses on the screen in the operation 1, and t1 can represent the moment when the operation 1 occurs when the finger presses. In InputEvent2, 1-0-0 can represent a finger release, (x2, y2) can represent the coordinates where the finger releases on the screen in the operation 1, and t2 can represent the moment when the operation 1 occurs when the finger releases.

[0245] Among them, if the difference between x1 and x2 does not exceed the distance threshold, and the difference between y1 and y2 does not exceed the distance threshold, the positions where the finger presses and releases in Operation 1 are basically the same. Operation 1 is a click operation or a long-press operation. Further, if the difference between t1 and t2 does not exceed the time threshold, Operation 1 is a click operation. If the difference between t1 and t2 exceeds the time threshold, Operation 1 is a long-press operation.

[0246] If the difference between x1 and x2 does not exceed the distance threshold, and the difference between y1 and y2 exceeds the distance threshold, Operation 1 is an upward swipe operation or a downward swipe operation. Further, if y1 is greater than y2, Operation 1 can be an upward swipe operation. If y1 is less than y2, Operation 1 can be a downward swipe operation.

[0247] If the difference between x1 and x2 exceeds the distance threshold, and the difference between y1 and y1 does not exceed the distance threshold, Operation 1 is a left swipe operation or a right swipe operation. Further, if x1 is greater than x2, Operation 1 is a left swipe operation. If x1 is less than x2, Operation 1 is a right swipe operation.

[0248] The content of the above user operation events is only an exemplary illustration of this application and should not constitute a limitation to this application.

[0249] S517. In response to Operation 1, the multimedia application 505 obtains the playback data of the audio a1.

[0250] The multimedia application 505 can identify the type of Operation 1 (such as Operation 1 is an upward swipe operation) according to the user operation event corresponding to Operation 1. Then, the multimedia application 505 can determine that Operation 1 is used to play the audio a1. The multimedia application 505 can obtain the playback data of the audio a1.

[0251] In some embodiments, before Operation 1, the electronic device 100 obtains and caches the playback data of the audio a1. The multimedia application 505 can directly read the playback data of the audio a1 from the memory and play it.

[0252] In some other embodiments, the electronic device 100 does not store the playback data for playing the audio a1. The multimedia application 505 can communicate with devices such as an audio and video server through the network to obtain the playback data of the audio a1 online.

[0253] The embodiments of this application do not limit the method for the multimedia application 505 to obtain the playback data of the audio a1.

[0254] S518. The multimedia application 505 sends a start event of the audio a1 to the audio module 503, instructing the audio module 503 to play the audio a1.

[0255] When the playback data for playing audio a1 is obtained, the multimedia application 505 can send a start event of audio a1 to the audio module 503 and instruct the audio module 503 to play audio a1. Among them, the multimedia application 505 can send the playback data of audio a1 to the audio module 503.

[0256] S519. The audio module 503 plays audio a1.

[0257] The audio module 503 can start playing audio a1 according to the playback data of audio a1.

[0258] S520. The audio module 503 sends a start event of audio a1 to the jitter detection module 501.

[0259] S521. The jitter detection module 501 detects whether the current interface is a multimedia playback interface.

[0260] In some embodiments, after receiving the user operation event corresponding to operation 1 (i.e., step S515), the jitter detection module 501 can detect whether the current interface is a multimedia playback interface to determine whether operation 1 is a user operation acting on the multimedia playback interface. The multimedia interface can be used to control the playback of multimedia content. For example, the multimedia playback interface can include but is not limited to: short video interface, live video interface, etc.

[0261] It can be understood that if operation 1 is an operation acting on the multimedia playback interface, then operation 1 may be used to control the playback of multimedia content. The jitter detection module 501 can further combine the information of operation 1 to determine whether to wait for the start event. If operation 1 is not an operation acting on the multimedia playback interface, the probability that operation 1 is used to control audio playback is relatively low. To avoid false detection of audio jitter, the jitter detection module 501 does not need to wait for the start event after receiving the user operation event corresponding to operation 1 and continues to wait for new user operation events.

[0262] In some embodiments, the jitter detection module 501 can obtain the information of the activity corresponding to the current interface and determine whether the current interface is a multimedia playback interface based on this activity. Among them, if the activity is the activity of the multimedia playback interface in the multimedia application 505, then the current interface is a multimedia playback interface. Otherwise, the current interface is not a multimedia playback interface. The embodiments of the present application do not limit the method for detecting whether the current interface is a multimedia playback interface.

[0263] If the current interface is a multimedia playback interface, the jitter detection module 501 can execute the following step S522.

[0264] If the current interface is not the multimedia playback interface, the stutter detection module 501 may perform the following step S525.

[0265] In some embodiments, step S521 may be executed before step S515.

[0266] S522. The stutter detection module 501 identifies whether operation 1 is a preset operation.

[0267] The preset operation may refer to an operation for playing multimedia content. For example, the preset operation may include a swiping-up operation.

[0268] It can be understood that if operation 1 is a preset operation for playing multimedia content (such as audio), the multimedia application 505 needs to respond to operation 1 to play the multimedia content, and the stutter detection module 501 may wait for the start event to determine whether the multimedia content is stuttering. If operation 1 is not a preset operation, the probability that the multimedia application 505 reports the start event after operation 1 is relatively low. To avoid mis-detecting stuttering in multimedia content playback, the stutter detection module 501 may not need to wait for the start event after receiving the user operation event corresponding to operation 1, and continue to wait for new user operation events.

[0269] In some embodiments, different multimedia applications may correspond to different preset operations. It can be understood that the operations for triggering multimedia content playback in different multimedia applications may be different. The electronic device 100 may store the preset operations corresponding to different multimedia applications respectively. The stutter detection module 501 may determine the preset operation according to the application to which the activity of the current interface belongs, and then identify whether operation 1 is a preset operation. This can more specifically detect stuttering in multimedia content playback for each multimedia application when playing multimedia content, and improve the accuracy of stutter detection in multimedia content playback.

[0270] In some embodiments, the stutter detection module 501 may further identify whether operation 1 is a preset operation when the multimedia content is in a preset playback state and acts on a preset area of the screen. It can be understood that the purpose of an operation may be different when the multimedia content is in different playback states and / or the acting position is different. The stutter detection module 501 can more accurately identify the user operation for triggering multimedia content playback by combining the playback state of the multimedia content, the type and acting position of the user operation, so as to detect whether the multimedia content to be played is stuttering in more scenarios where multimedia content playback is triggered by user operations.

[0271] If operation 1 is a preset operation, the stutter detection module 501 may perform the following step S523.

[0272] If Operation 1 is not a preset operation, the stuttering detection module 501 may execute the following step S525.

[0273] S523. The stuttering detection module 501 detects whether a start event is received within a preset time period starting from the action time of Operation 1.

[0274] The stuttering detection module 501 may determine the action time of Operation 1 according to the user operation event corresponding to Operation 1. Among them, the action time of Operation 1 may be the moment when Operation 1 ends (for example, the moment when the finger is lifted from the screen in Operation 1). Or, Operation 1 acts continuously for a period of time. For example, operations such as swiping up and long pressing will act continuously on the screen for a period of time. The stuttering detection module 501 may determine a moment during the process of Operation 1 as the action time of Operation 1. The present application does not limit the determination method of the action time of Operation 1.

[0275] In some embodiments, the stuttering detection module 501 may set a timer according to the action time of Operation 1 and determine whether a start event is received before the timer times out. The timing duration of the timer may be the duration of the above preset time period, for example, 1 s, 2 s, etc. The present application embodiments do not limit the value of the timing duration. Among them, the timer may start timing from the action time of Operation 1. The stuttering detection module 501 receiving a start event before the timer times out may indicate that the multimedia content playback is not stuttering. The stuttering detection module 501 not receiving a start event before the timer times out may indicate that the multimedia content playback is stuttering.

[0276] It should be noted that the serial number arrangement of the above steps S521 to S523 and the above steps S517 to S520 does not constitute a limitation on the execution order sequence. For example, steps S521 to S523 and steps S517 to S520 may be performed simultaneously. After the multimedia application 505 receives the user operation event corresponding to Operation 1, if it can quickly obtain the playback data of the audio a1, the multimedia application 505 can report the start event relatively quickly. In this way, the stuttering detection module 501 can receive the start event before the timer times out. After the multimedia application 505 receives the user operation event corresponding to Operation 1, if it needs to wait for a long time to obtain the playback data of the audio a1 or fails to obtain the playback data of the audio a1 for a long time, the multimedia application 505 reports the start event relatively late. In this way, the stuttering detection module 501 does not receive the start event before the timer times out. The stuttering detection module 501 can accurately identify whether the multimedia content is stuttering by combining the above user operation event and the start event.

[0277] If the jitter detection module 501 receives a start event within a preset time period starting from the action time of operation 1, the jitter detection module 501 may perform the following step S525.

[0278] If the jitter detection module 501 does not receive a start event within a preset time period starting from the action time of operation 1, the jitter detection module 501 may perform the following step S524.

[0279] S524. The jitter detection module 501 instructs the network management module 504 to adjust the network through which the electronic device 100 communicates.

[0280] The network management module 504 may adjust the network for data communication between the electronic device 100 and other devices according to the instruction of the jitter detection module 501, so that the electronic device 100 uses a network with better quality for communication. In this way, the multimedia application 505 can obtain the playback data of the audio a1 as soon as possible, thereby eliminating jitter and playing the audio a1.

[0281] S525. The jitter detection module 501 continues to wait for the reporting of user operation events and audio events.

[0282] When it is uncertain whether operation 1 is an operation that triggers the playback of multimedia content, the jitter detection module 501 will not determine that the multimedia content is jittery when it does not receive a start event within a preset time period starting from the action time of operation 1, but will continue to wait for subsequent user operation events and audio events. This can reduce the situation of mis-detecting jitter in multimedia content playback and improve the accuracy of jitter detection in multimedia content playback.

[0283] When it is determined that operation 1 is an operation that triggers the playback of multimedia content, if the jitter detection module 501 receives a start event within a preset time period starting from the action time of operation 1, it can be determined that the multimedia content playback is not jittery, and then continue to wait for subsequent user operation events and audio events. The jitter detection module 501 may detect subsequent received user operation events and audio events according to steps S521 - S525 to identify the situation of jitter in multimedia content playback.

[0284] From the method described above Figure 5 As can be seen, the jitter detection module 501 can obtain user operation events corresponding to user operations on the electronic device 100 and determine whether the user operation is a preset operation for triggering the playback of multimedia content. When it is detected that the user operation is a preset operation for triggering the playback of multimedia content, the jitter detection module 501 can detect whether a start event is received within a preset time period after the preset operation, so as to accurately detect whether the multimedia content to be played is jittery in the scenario of triggering multimedia content playback through a preset operation.

[0285] In some embodiments, the electronic device 100 may detect multimedia content playback stuttering according to user operation events and video events. Among them, the stuttering detection module 501 may register video event monitoring reports with the video module in the electronic device 100. When the playback state of the video changes, the multimedia application 505 may report the corresponding video event to the video module. Then, the video module may send the video event to the stuttering detection module 501.

[0286] Exemplarily, the human-computer interaction module 502 may send the user operation event corresponding to the operation for playing the video on the multimedia playback interface in the multimedia application 505 to the stuttering detection module 501. After determining that the operation is a preset operation for playing the video on the multimedia playback interface according to the user operation event, the stuttering detection module 501 may detect whether a start playback event of the video is received within a preset time period starting from the time when the operation takes effect. The above process may specifically refer to Figure 5 Steps S521 to S523 shown.

[0287] In response to an operation for playing a video on the multimedia playback interface in the multimedia application 505, after obtaining the playback data of the video to be played, the multimedia application 505 may report the start playback event of the video to be played to the video module. The video module may send the start playback event to the stuttering detection module 501. Among them, if the stuttering detection module 501 receives the start playback event of the video to be played within the preset time period starting from the time when the operation for playing the video takes effect, the multimedia content (i.e., the video to be played) is played without stuttering. Otherwise, the stuttering detection module 501 determines that the multimedia content is played stutteringly and may instruct the network management module 504 to adjust the network for the electronic device 100 to communicate.

[0288] Figure 6 Another method flowchart for detecting multimedia content playback stuttering is exemplarily shown.

[0289] As Figure 6 shown, the method may include steps S611 to S615.

[0290] S611. The stuttering detection module 501 records the start event of audio a1.

[0291] From the foregoing Figure 5As can be seen from the method shown, when the electronic device 100 is triggered to play the audio a1, the audio module 503 can forward the start event reported by the multimedia application 505 to the stutter detection module 501. The stutter detection module 501 can record the start event of the audio a1. Among them, when the stutter detection module 501 records the start event, if the stutter detection module 501 does not receive the pause event or stop event of the audio a1, it can indicate that the audio a1 is in the playing state.

[0292] S612. During the playing of the audio a1 by the multimedia application 505, the played data of the audio a1 that has been obtained is played out, and no subsequent playing data of the audio a1 is obtained for more than a preset duration.

[0293] When the playing data of the audio a1 is obtained by the electronic device 100 from other devices through the network, the electronic device 100 can play the audio a1 while obtaining the subsequent playing data of the audio a1 from other devices. The electronic device 100 can cache a part of the subsequent playing data of the audio a1 so that it can continue to play for a period of time even when the network suddenly deteriorates.

[0294] In some embodiments, when the network is in a poor state for a long time, after the playing data of the audio a1 cached in the memory is played out, if the electronic device 100 still fails to obtain the subsequent playing data of the audio a1 from other devices, the multimedia application 505 will have no available playing data. The audio a1 stutters.

[0295] S613. The audio module 503 fails to obtain the subsequent playing data of the audio a1 for more than a preset duration, and generates a timeout event for the audio a1.

[0296] It can be understood that the audio module 503 can obtain the playing data of the audio a1 from the multimedia application 505. As can be seen from the above step S612, the multimedia application 505 fails to obtain the subsequent playing data of the audio a1 for more than a preset duration. Therefore, the audio module 503 will also fail to obtain the subsequent playing data of the audio a1 for more than a preset duration to continue playing the audio a1. When the subsequent playing data of the audio a1 is not obtained for more than a preset duration, the audio module 503 can generate a timeout event for the audio a1.

[0297] The timeout event of the audio a1 can indicate that the audio a1 stutters during the playing process.

[0298] S614. The audio module 503 sends the timeout event of the audio a1 to the stutter detection module 501.

[0299] S615. The stuttering detection module 501 instructs the network management module 504 to adjust the network through which the electronic device 100 communicates according to the timeout event.

[0300] The network management module 504 may adjust the network through which the electronic device 100 communicates with other devices according to the instruction of the stuttering detection module 501, so that the electronic device 100 communicates using a network with better quality. In this way, the multimedia application 505 can obtain the playback data of the audio a1 as soon as possible, thereby eliminating stuttering and playing the audio a1.

[0301] As can be seen from the method Figure 6 described above, the timeout event can reflect that stuttering occurs during the playback of the audio. The electronic device 100 can accurately detect whether the audio in any multimedia application is stuttering during playback based on the timeout event, and adjust the network when the audio is stuttering to communicate using a network with better quality to eliminate stuttering. This can better improve the user experience of listening to audio.

[0302] The method for the electronic device 100 to detect whether the video is stuttering during playback can refer to the method Figure 6 described above for the electronic device 100 to detect whether the audio is stuttering during playback.

[0303] In some embodiments, the electronic device 100 may also combine the user operations on the electronic device 100 and data transmission metrics to detect whether the multimedia content is stuttering. The above data transmission metrics can be used to evaluate the network quality and are metrics reflecting the network quality. For example, the data transmission metrics may include round trip time (RTT). RTT may refer to the duration taken for a network request to go from the starting point to the destination and then back to the starting point. RTT can be used to diagnose the speed and reliability of the network connection. The larger the RTT, the worse the quality of the current network. The specific content of the data transmission metrics in the embodiments of the present application is not limited. For example, in addition to RTT, the data transmission metrics may also include data traffic of data transmission, packet interval size, and other metrics for evaluating network quality. Here, RTT is specifically used as an example for illustration.

[0304] When receiving a user operation acting on the audio-video playback interface for triggering the playback of multimedia content, the electronic device 100 may detect the RTT within a preset time period after the user operation takes effect. If the RTT within the preset time period indicates that the network quality is worse than the preset network quality, the electronic device 100 may determine that the multimedia content is stuttering, and then adjust the network to communicate using a network with better quality.

[0305] Among them, if the RTT within the above-mentioned preset time period is greater than the preset RTT threshold, it can indicate that the network quality is worse than the preset network quality. Or, if the average value of the RTT within the above-mentioned preset time period is greater than the preset RTT threshold, it can indicate that the network quality is worse than the preset network quality. This application does not limit the method for the electronic device 100 to determine whether the RTT within the above-mentioned preset time period indicates that the network quality is worse than the preset network quality.

[0306] This application does not limit the above-mentioned user operation for triggering the playback of multimedia content.

[0307] In some embodiments, the electronic device 100 can detect whether the multimedia content is stuck by combining the user operation on the electronic device 100, the multimedia playback event, and the data transmission index. Among them, the data transmission index can improve the confidence level of the detection result of the stuck multimedia content. And the data transmission index can also be used to determine whether the stuck multimedia content is caused by the network quality.

[0308] Specifically, when receiving a user operation acting on the audio-video playback interface for triggering the playback of multimedia content, the electronic device 100 can obtain the start event reported by the multimedia application for playing the multimedia content after the user operation.

[0309] If the start event is obtained, the electronic device 100 can determine that the multimedia content is not stuck. When it is determined that the multimedia content is not stuck based on the user operation and the multimedia playback event, if the RTT is less than the preset RTT threshold, it can indicate that the current network quality is good, thereby improving the confidence level of the detection result that the above-mentioned multimedia content is not stuck.

[0310] If the start event cannot be obtained, the electronic device 100 can determine that the multimedia content is stuck. When it is determined that the multimedia content is stuck based on the user operation and the multimedia playback event, if the RTT is greater than the preset RTT threshold, it can indicate that the current network quality is poor, thereby improving the confidence level of the detection result that the above-mentioned multimedia content is stuck.

[0311] When it is determined that the multimedia content is stuck based on the user operation and the multimedia playback event, if the RTT is less than the preset RTT threshold, it can indicate that the current network quality is good and the stuck multimedia content is not caused by poor network quality. It can be understood that the stuck multimedia content may be caused by other reasons in addition to poor network quality. For example, the load of the processor of the electronic device 100 is too high, and too many background running processes may also cause the multimedia content to be stuck. Therefore, when the electronic device 100 determines that the multimedia is stuck based on the user operation and the multimedia playback event, and the RTT is less than the preset RTT threshold, the electronic device 100 can close one or more background processes to reduce the load of the processor without adjusting the network for the electronic device 100 to communicate.

[0312] Not limited to detecting whether there is lag in multimedia content playback, the electronic device 100 can also detect whether there is lag in interface browsing by combining user operations and data transmission metrics of the electronic device 100. In some embodiments, in addition to being used to play audio and / or video and display an audio / video playback interface, the multimedia application can also display a graphic interface. The graphic interface can refer to a user interface displaying text and / or pictures. A graphic interface can be a long page. The electronic device 100 can slide the graphic interface up or down in response to a preset user operation (such as a swiping-up operation or a swiping-down operation) to display the content of different regions of the graphic interface on the screen. The electronic device 100 can also jump from one graphic interface to another in response to a preset operation (such as a swiping-left operation or a swiping-right operation). Not limited to the multimedia application, other APPs that can display graphic interfaces can also be included in the electronic device 100, such as browsers, news APPs, shopping APPs, reading APPs, and so on.

[0313] The electronic device 100 can detect whether there is lag when the user browses the graphic interfaces of these APPs.

[0314] Specifically, when receiving a user operation acting on the graphic interface for switching the displayed content, the electronic device 100 can detect the RTT within a preset time period after the user operation takes effect. If the RTT within the preset time period indicates that the network quality is worse than the preset network quality, the electronic device 100 can determine that there is lag in graphic interface browsing, and then adjust the network to communicate using a network with better quality.

[0315] The above-mentioned user operations for adjusting the displayed content can include but are not limited to: user operations for switching the content of different regions of the graphic interface, user operations for jumping the graphic interface, and so on. For example, the user operations for switching the content of different regions of the graphic interface can include a swiping-up operation and a swiping-down operation. The user operations for jumping the graphic interface can include a swiping-left operation and a swiping-right operation.

[0316] As can be seen from the above embodiments, when receiving a user operation acting on an interface, the electronic device 100 can determine whether this interface is a preset interface and whether the user operation is a preset user operation. If it is a preset user operation acting on a preset interface, the electronic device 100 can detect the RTT within a preset time period after the user operation takes effect. If the RTT within the preset time period indicates that the network quality is worse than the preset network quality, the electronic device 100 can determine that there is a lag in browsing this interface, and then adjust the network to communicate using a network with better quality. In this way, the electronic device 100 can obtain the data required for the display interface as soon as possible through a network with better quality, so as to eliminate the lag in interface browsing as soon as possible and improve the user experience of browsing the interface.

[0317] In some embodiments, the electronic device 100 can store a correspondence table of preset interfaces and preset user operations to detect whether the received user operation is a preset user operation acting on a preset interface. Optionally, the electronic device 100 can also obtain the above-mentioned correspondence table of preset interfaces and preset user operations from other devices (such as a cloud server, etc.).

[0318] It can be understood that the various user interfaces described in the embodiments of the present application are only example interfaces and do not limit the solution of the present application. In other embodiments, the user interface can adopt different interface layouts, can include more or fewer controls, and can add or reduce other function options. As long as it is based on the same inventive concept provided by the present application, it is within the protection scope of the present application.

[0319] It should be noted that, without contradiction or conflict, any feature in any embodiment of the present application, or any part of any feature, can be combined, and the combined technical solution is also within the scope of the embodiments of the present application.

[0320] The above is the case. The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting stuttering in multimedia content playback, characterized in that, the method is applied to an electronic device, the electronic device includes a first application, the first application is an application for playing multimedia content, and the method includes: the electronic device receives a first operation on a first interface, the first interface is an interface in the first application for controlling multimedia content playback, and the first operation is used to play first multimedia content; when the electronic device does not obtain a first start playback event reported by the first application after the first operation, the electronic device adjusts the network for data communication; wherein, the first start playback event is used to indicate that the first multimedia content starts to play.

2. The method according to claim 1, characterized in that, the first operation matches a preset operation.

3. The method according to claim 1 or 2, characterized in that, the network for data communication is adjusted from a first network to a second network, and the method further includes: the first application in the electronic device obtains playback data of the first media content by using the second network; the electronic device plays the first multimedia content according to the playback data of the first multimedia content.

4. The method according to claim 3, characterized in that, after the electronic device adjusts the network for data communication, the method further includes: the electronic device changes a first identifier on the first interface to a second identifier, the first identifier is used to indicate that the network for data communication is the first network, and the second identifier is used to indicate that the network for data communication is the second network.

5. The method according to claim 3 or 4, characterized in that, when the electronic device plays the first multimedia content according to the playback data of the first multimedia content, it specifically includes: the electronic device plays a first audio according to the playback data of the first multimedia content, and / or plays a first video.

6. The method according to any one of claims 3-5, characterized in that, the method further includes: during the process of playing the first multimedia content, the electronic device has stuttering when playing the first multimedia content; the electronic device detects a first data exhaustion event, and adjusts the network for data communication according to the first data exhaustion event; wherein, the first data exhaustion event is used to indicate that the playback data for playing the first multimedia content in the electronic device has been played out and there is no playback data available for the time being.

7. The method according to any one of claims 1-6, characterized in that, the method further includes: when the electronic device obtains the first start playback event reported by the first application after the first operation, the electronic device plays the first multimedia content; wherein, the playback data of the first multimedia content is obtained by the first application from a memory, or is obtained by the first application by using the network for data communication.

8. The method according to any one of claims 2-7, characterized in that, the method further includes: The electronic device receives a second operation on the first interface, and the second operation does not match the preset operation; or, The electronic device receives a third operation on a second interface, and the second interface is not an interface for controlling multimedia content playback; The electronic device waits to receive subsequent user operations.

9. The method according to any one of claims 1-8, wherein, The playback of the first multimedia content is paused, and the method further includes: The electronic device receives a fourth operation on the first interface, and the fourth operation is used to play the first multimedia content; The electronic device identifies that the fourth operation acts on a first area of the first interface, and the fourth operation matches the preset operation. The electronic device determines whether the first start playback event reported by the first application is obtained within a second time period; wherein, the second time period is a preset time period starting from a second moment, and the second moment is the time when the second operation acts; When the electronic device does not obtain the first start playback event reported by the first application within the second time period, the electronic device adjusts the network for data communication; When the electronic device obtains the first start playback event reported by the first application within the second time period, the electronic device plays the first multimedia content.

10. The method according to any one of claims 1, wherein, Before the electronic device adjusts the network for data communication, the method further includes: The electronic device determines that the first interface is an interface for controlling multimedia content playback according to the activity information of the first interface.

11. The method according to claim 1, wherein, Before the electronic device adjusts the network for data communication, the method further includes: The electronic device detects that the data transmission index after the first operation indicates that the network quality is worse than the preset network quality.

12. The method according to any one of claims 1-11, wherein, The method further includes: The electronic device receives a fifth operation on a second interface, and the second interface is an interface for controlling multimedia content playback in the first application. The fifth operation is used to play a second multimedia content; When the electronic device does not obtain the second start playback event reported by the first application after the fifth operation, and the data transmission index after the fifth operation indicates that the network quality is better than the preset network quality, the electronic device closes one or more processes, and the one or more processes are not processes of the first application. The second start playback event is used to indicate the start of playback of the second multimedia content.

13. The method according to claim 11 or 12, wherein, The data transmission index is the round-trip time RTT.

14. The method according to any one of claims 1-13, wherein, When the electronic device does not obtain the first start playback event reported by the first application after the first operation, the electronic device adjusts the network for data communication, specifically including: When the electronic device fails to obtain the first start-playing event within the first time period, the electronic device adjusts the network for data communication. The first time period is a preset time period starting from the first moment, and the first moment is the time when the first operation takes effect.

15. A method for detecting interface browsing lag, characterized in that, the method is applied to an electronic device, the electronic device includes a first application, and the method includes: The electronic device receives a first operation on a first interface in the first application, and the first operation is used to switch the display content on the screen of the electronic device within the first application; The electronic device obtains a data transmission metric after the first operation, and the data transmission metric is used to indicate the network quality of the electronic device for data communication; If the data transmission metric after the first operation indicates that the network quality is worse than the preset network quality, the electronic device adjusts the network for data communication.

16. The method according to claim 15, characterized in that, the first interface matches a preset interface, and the first operation matches a preset operation.

17. The method according to claim 15 or 16, characterized in that, the first interface is a long page, and the first operation is used to switch the display content on the screen of the electronic device to the content of different regions on the first interface, or the first operation is used to switch the first interface to a second interface in the first application.

18. The method according to any one of claims 15-17, characterized in that, the data transmission metric after the first operation includes the data transmission metric within the first time period, and the first time period is a preset time period starting from the first moment, and the first moment is the time when the first operation takes effect.

19. The method according to any one of claim 18, characterized in that, the data transmission metric is the round-trip time RTT, and the data transmission metric after the first operation indicating that the network quality is worse than the preset network quality specifically includes: the RTT within the first time period is greater than the preset RTT threshold, or the average value of the RTT within the first time period is greater than the preset RTT threshold.

20. The method according to any one of claims 15-19, characterized in that, the network for data communication is adjusted from the first network to the second network. After the electronic device adjusts the network for data communication, the method further includes: The electronic device changes a first identifier on the first interface to a second identifier. The first identifier is used to indicate that the network for data communication is the first network, and the second identifier is used to indicate that the network for data communication is the second network.

21. An electronic device, characterized in that, the electronic device includes a memory and a processor; wherein, the memory is used to store a computer program, and the processor is used to call the computer program so that the electronic device executes the method according to any one of claims 1-14 or claims 15-20.

22. A computer-readable storage medium stores instructions, characterized in that, when the instructions are run on an electronic device, the electronic device executes the method according to any one of claims 1-14 or claims 15-20.

23. A computer program product, characterized in that, the computer program product includes computer instructions, and when the computer instructions are run on an electronic device, the electronic device executes the method according to any one of claims 1-14 or claims 12-20.

Images