Method and apparatus for stutter detection, device, and storage medium

Abstract

The present application provides a method and apparatus for detecting stuttering, a device, and a storage medium; wherein, the method includes: during the process of playing multimedia data, obtaining operating parameters related to a first memory; wherein, the first memory is used to cache the multimedia data to be played; and predicting the stuttering situation of the multimedia data playback according to the operating parameters.

Description

Technical Field

[0001] This application relates to electronic technology, including but not limited to methods and devices for stutter detection, equipment, and storage media. Background Art

[0002] With the rapid development of multimedia technologies such as audio and video, more and more users tend to communicate through audio and video. For example, a live streamer can interact with the audience in real time through live streaming; friends can chat through instant messaging applications; enterprises can hold online meetings through conferencing applications. However, during the playback of multimedia data, stuttering may occur. In related stutter detection technologies, the accuracy of stutter detection is relatively low. Summary of the Invention

[0003] In view of this, the stutter detection methods, devices, equipment, and storage media provided by this application can improve the accuracy of stutter detection.

[0004] According to one aspect of the embodiments of this application, a stutter detection method is provided, including: during the playback of multimedia data, obtaining working parameters related to a first memory; where the first memory is used to cache multimedia data to be played; predicting the stuttering situation of multimedia data playback according to the working parameters.

[0005] According to one aspect of the embodiments of this application, a stutter detection device is provided, including: an obtaining module configured to obtain working parameters related to a first memory during the playback of multimedia data; a predicting module configured to predict the stuttering situation of multimedia data playback according to the working parameters.

[0006] According to one aspect of the embodiments of this application, an electronic device is provided, including a memory and a processor, where the memory stores a computer program that can run on the processor, and the processor implements the method described in the embodiments of this application when executing the program.

[0007] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, it implements the method provided in the embodiments of this application.

[0008] In the embodiments of this application, stuttering situations are detected based on the obtained working parameters related to the first memory. Compared with predicting the stuttering situation of multimedia data playback only based on the frame loss situation of multimedia data frames, it can not only detect stuttering caused by frame loss, but also detect stuttering caused by other factors. For example, it can detect stuttering caused by application blocking, system scheduling, and lack of timely response at the application layer, etc. Therefore, a higher accuracy of stutter detection can be obtained.

[0009] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings herein are incorporated into and constitute a part of this specification. These drawings illustrate embodiments consistent with this application and, together with the specification, are used to explain the technical solutions of this application. Obviously, the drawings in the following description are only some embodiments of this application, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0011] The flowcharts shown in the drawings are only exemplary illustrations, and do not necessarily include all contents and operations / steps, nor do they necessarily need to be executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined. Therefore, the actual execution order may be changed according to the actual situation.

[0012] Figure 1 Schematic diagram of the implementation process of the stutter detection method provided by the embodiment of this application;

[0013] Figure 2 Schematic diagram of the implementation process of another stutter detection method provided by the embodiment of this application;

[0014] Figure 3 Schematic diagram of the processing flow of multimedia data;

[0015] Figure 4 Schematic diagram of the implementation process of the adjustment method for the first threshold provided by the embodiment of this application;

[0016] Figure 5 Schematic diagram of the display interface provided by the embodiment of this application;

[0017] Figure 6 Schematic diagram of the implementation process of another stutter detection method provided by the embodiment of this application;

[0018] Figure 7 Schematic diagram of the implementation process of another stutter detection method provided by the embodiment of this application;

[0019] Figure 8 Schematic diagram of normal audio;

[0020] Figure 9 Schematic diagram of stuttered audio;

[0021] Figure 10 Schematic diagram of the processing flow of audio data;

[0022] Figure 11Schematic diagram of the time interval for writing interfaces in two calls;

[0023] Figure 12 Schematic diagram of the production and consumption of audio data;

[0024] Figure 13 Schematic diagram of the structure of a stutter detection device provided by an embodiment of the present application;

[0025] Figure 14 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will further describe the specific technical solutions of the present application in detail with reference to the accompanying drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not intended to limit the scope of the present application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0028] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0029] An embodiment of the present application provides a stutter detection method, which is applied to an electronic device. During implementation, the electronic device can be various types of devices with the ability to play multimedia data. For example, the electronic device can include a mobile phone, a tablet computer, a television, a projector, or a personal computer, etc. The functions implemented by this method can be realized by a processor in the electronic device calling program code. Of course, the program code can be stored in a computer storage medium. It can be seen that the electronic device at least includes a processor and a storage medium.

[0030] Figure 1 Schematic diagram of the implementation process of the stutter detection method provided by an embodiment of the present application, as Figure 1 shown, this method can include the following steps 101 to step 102:

[0031] Step 101, during the process of playing multimedia data, obtain working parameters related to a first memory; wherein, the first memory is used to cache the multimedia data to be played;

[0032] Step 102, predict the stutter situation of multimedia data playback according to the working parameters.

[0033] In an embodiment of the present application, on the one hand, based on the obtained operating parameters related to the first memory, the detection of stuttering conditions is performed. Compared with predicting the stuttering conditions of multimedia data playback only based on the frame loss conditions of multimedia data frames, it can not only detect the stuttering caused by frame loss, but also detect the stuttering caused by other factors. For example, it can detect the stuttering caused by application blocking, system scheduling, untimely response of the application layer, etc. Therefore, a higher stuttering detection accuracy rate can be obtained. On the other hand, since the acquisition of the operating parameters related to the first memory does not require complex algorithms to implement, therefore, while improving the stuttering detection accuracy rate, the load and overhead are not increased, and the cost is lower.

[0034] In an embodiment of the present application, the multimedia data can be various types of data. For example, the multimedia data includes audio data and / or video frame data.

[0035] It can be understood that the so-called operating parameters related to the first memory refer to the operating parameters directly and / or indirectly related to the first memory. For example, the operating parameters include the time information of two adjacent calls to the write interface. Since the write interface is used to write the multimedia data cached in the first memory into the second memory, the time information of two adjacent calls to the write interface is an operating parameter indirectly related to the first memory. Another example is that the operating parameters include the data volume of the multimedia data cached in the first memory, which is an operating parameter directly related to the first memory.

[0036] In an embodiment of the present application, there is no limitation on the operating parameters. The operating parameters may include the time information of two adjacent calls to the write interface and / or the data volume of the multimedia data cached in the first memory.

[0037] There is also no limitation on the method for implementing step 102. In some embodiments, the stuttering conditions corresponding to the operating parameters can be predicted by querying the historical record information. For example, Table 1 shows the historical record information, which includes the historical values of the operating parameters and the corresponding stuttering conditions.

[0038] Table 1

[0039] Historical values of working parameters Lag situation Value 1 Lag Value 2 No lag ······ Value N Lag

[0040] In other embodiments, the operating parameters can also be input into a pre-trained neural network model, and the neural network model predicts the stuttering conditions of multimedia data playback based on the input operating parameters. Among them, the pre-trained neural network model is trained based on the historical values of the operating parameters and the corresponding true stuttering conditions.

[0041] In still other embodiments, the electronic device can also pass through Figure 2 、Figure 6 or Figure 7 Implement step 102 according to the steps in the corresponding embodiments of stutter detection respectively, which will not be elaborated here.

[0042] The embodiments of the present application further provide a stutter detection method. Figure 2 It is a schematic diagram of the implementation process of another stutter detection method provided by the embodiments of the present application. As Figure 2 shown, the method may include the following steps 201 to 204:

[0043] Step 201, during the process of playing multimedia data, obtain the working parameters related to the first memory; wherein, the first memory is used to cache the multimedia data to be played; the working parameters include the time information of two adjacent calls to the write interface; the write interface is used to write the multimedia data cached in the first memory into the second memory.

[0044] In the embodiments of the present application, the first memory and the second memory can be any memory used in the process of multimedia data processing. In some embodiments, the first memory is used to cache the multimedia data to be played output by the decoder, and the second memory is the memory at the operating system layer.

[0045] Step 202, according to the time information, determine the time interval between the two adjacent calls to the write interface;

[0046] Step 203, determine whether the time interval is greater than or equal to the first threshold; if so, execute step 204; otherwise, return to execute step 201;

[0047] Step 204, determine that stuttering occurs in the multimedia data playback.

[0048] In a processing flow of multimedia data, as Figure 3 shown, the receiver 301 of the electronic device 30 receives the bitstream from the network side (network), and then caches it in the jitter buffer 302. The decoder 303 decodes the bitstream cached in the jitter buffer 302, outputs the decoded multimedia data to the buffer 304, and then the application layer calls the write interface of the AudioTrack of the operating system, so as to write the multimedia data in the buffer 304 into the buffer 305 of the operating system, thereby realizing the playback of multimedia data. Here, the buffer 304 is an example of the first memory, and the buffer 305 is an example of the second memory.

[0049] Understandably, if the time interval between two adjacent calls to the write interface is greater than the first threshold, it indicates that the multimedia data in the buffer 304 is not timely delivered to the buffer 305 in the system layer. In this case, it may cause stuttering in the multimedia data playback.

[0050] Based on this, in the embodiments of the present application, stuttering detection is performed based on the time interval between two adjacent calls to the write interface, where the write interface is used to write the multimedia data to be played output by the decoder cached in the first memory into the second memory. In this way, on the one hand, since the time information of calling the write interface can be obtained at the application layer without obtaining it from the bottom layer through the Application Programming Interface (API), this method is relatively simple to implement, and thus can improve the efficiency of stuttering detection on the premise of saving resources such as power consumption. On the other hand, since the acquisition of the time information of calling the write interface and the stuttering detection method based on this information do not require complex algorithms to implement, the load and overhead will not be increased while improving the accuracy of stuttering detection, and the cost is lower.

[0051] In some embodiments, the first threshold can be set based on the time length corresponding to the maximum data volume that the second memory can cache. Further, in some embodiments, the first threshold is greater than or equal to the time length corresponding to the maximum data volume that the second memory can cache. For example, assuming that the time length corresponding to the maximum data volume that the second memory can cache is 100 ms of data, then the first threshold can be set to 200 ms. Of course, the first threshold can also be set to any value greater than or equal to 100 ms. In short, it can meet the detection requirements of stuttering detection accuracy.

[0052] Further, in some embodiments, the size of the first threshold can be adaptively adjusted. For example Figure 4 As shown, the method further includes the following steps 401 to 403:

[0053] Step 401, display one or more options reflecting different degrees of stuttering.

[0054] In the embodiments of the present application, there is no limitation on when to display one or more options reflecting different degrees of stuttering. In some embodiments, step 401 can be triggered based on determining that stuttering occurs; in other embodiments, one or more options reflecting different degrees of stuttering can also be displayed when the parameter characterizing the frequency of stuttering meets the display conditions.

[0055] There is no limitation on the parameter for characterizing the frequency of freezes. However, due to the different types of parameters for characterizing the frequency of freezes, the corresponding display conditions are also different. As shown in Table 2 below: In Example 1, the parameter includes the number of freezes within a specific duration, and the corresponding display condition is that the number of freezes within the specific duration is greater than the number threshold; in Example 2, the parameter includes the freeze frequency (i.e., the number of freezes per unit time), and the corresponding display condition is that the freeze frequency is greater than the frequency threshold; in Example 3, the parameter includes the time interval between two adjacent freezes, and the corresponding display condition is that the time interval between two adjacent freezes is less than the first interval threshold; in Example 4, the parameter includes the average time interval of freezes within a specific duration, and the corresponding display condition is that the average time interval of freezes within the specific duration is greater than the second interval threshold; where the first interval threshold and the second interval threshold may be the same or different. The average time interval of freezes within a specific duration can be based on the mean of the time intervals between pairwise adjacent freezes.

[0056] Table 2

[0057]

[0058] It can be understood that in this embodiment, these options are not displayed every time a freeze occurs, but only when the parameter for characterizing the frequency of freezes meets the display condition, the one or more options reflecting different degrees of freezes are displayed; in this way, on the one hand, it can avoid frequent interference to the user; on the other hand, it can obtain accurate feedback information with as few interactions with the user as possible, that is, on the premise of correspondingly saving resources such as the power consumption of the display and the processor, and thus adaptively adjust the first threshold based on this, thereby improving the accuracy of freeze detection.

[0059] Step 402, receive a selection operation on the one or more options; where the selection operation is used to indicate the degree of freeze feedback;

[0060] Step 403, in response to the selection operation, adjust the first threshold according to the degree of freeze selected by the selection operation.

[0061] There is no limitation on the number of options and the degree of freeze corresponding to the options. For example, as Figure 5 shown, a rectangular prompt box 502 is displayed on the currently displayed interface 501, and this box 502 includes three options: very frozen, a bit frozen, and not frozen. The user can click on the area where any of the options is located to thus feedback their personal usage experience.

[0062] Understandably, for different degrees of stuttering, the adjustment for the first threshold is different. For example, when it is determined that the multimedia data is stuttering or the parameter representing the frequency of stuttering meets the display condition, one or more options reflecting different degrees of stuttering are displayed. As shown in Table 3, assuming that the user clicks on the option of "not stuttering", it indicates that there is a misdetection based on the current first threshold, and the non-stuttering is misjudged as stuttering. Therefore, the first threshold can be increased by a preset step size; assuming that the user clicks on the option of "very stuttering", it also indicates that there may be a misdetection based on the current first threshold, and the stuttering is misjudged as non-stuttering. Therefore, the first threshold can be decreased by a preset step size; for another example, if the user clicks on the option of "a little stuttering", it means that the current first threshold is set appropriately, so the first threshold remains unchanged. For example, the current first threshold is adjusted to itself.

[0063] Table 3

[0064] Degree of lag First threshold No lag Increase Very laggy Decrease Somewhat laggy Remain unchanged

[0065] In the embodiment of the present application, one or more options reflecting different degrees of stuttering are displayed, so as to obtain the true stuttering feeling feedback by the user, and then the first threshold is adaptively adjusted based on the true stuttering feeling of the user; in this way, the adjusted first threshold is closer to the situation where the user truly perceives stuttering, thereby improving the accuracy of stuttering detection.

[0066] The embodiment of the present application further provides a stuttering detection method. Figure 6 For another schematic flowchart of the method implementation for stuttering detection provided by the embodiment of the present application, as Figure 6 shown, the method may include the following steps 601 to step 603:

[0067] Step 601, during the process of playing multimedia data, obtain the data volume of the multimedia data to be played cached in the first memory; wherein, the first memory includes any memory used in the multimedia data processing process.

[0068] Step 602, determine whether the data volume is less than or equal to the second threshold; if so, execute step 603; otherwise, return to execute step 601.

[0069] The second threshold can be set to 0 or a value greater than 0. When the second threshold is set to 0, then determine whether the data volume is equal to 0; if so, execute step 603; otherwise, return to execute step 601 to continue obtaining the data volume of the multimedia data to be played cached in the first memory.

[0070] When the second threshold is set to a value greater than 0, it is determined whether the data volume is less than or equal to the second threshold; if so, step 603 is executed; otherwise, return to execute step 601.

[0071] It should be noted that the electronic device may return to execute step 601 when it determines that the data volume is less than or equal to the second threshold, or may also return to execute step 601 after waiting for a preset duration.

[0072] Step 603, based on determining that the data volume is less than or equal to the second threshold, predict the stuttering situation of multimedia data playback.

[0073] In some embodiments, the electronic device may implement step 603 as follows: when the data volume is less than or equal to the second threshold, determine that stuttering occurs in multimedia data playback; in this way, the efficiency of stuttering detection can be improved.

[0074] In other embodiments, the electronic device may also implement step 603 as follows: determine the duration of the state where the data volume is less than or equal to the second threshold; and when the duration of the state is greater than or equal to the third threshold, determine that stuttering occurs in multimedia data playback; where the third threshold is greater than 0.

[0075] It can be understood that in the case where the duration of the state where the data volume is less than or equal to the second threshold is less than the third threshold, the stuttering caused may be imperceptible to the user. Therefore, in this embodiment, the condition for determining that stuttering occurs in multimedia data playback is that the duration of the state is greater than or equal to the third threshold. In this way, the false detection rate of stuttering detection can be reduced, thereby improving the accuracy of stuttering detection.

[0076] In the embodiments of the present application, stuttering detection is performed based on the data volume of the multimedia data to be played cached in the first memory; and the amount of the multimedia data to be played cached in the first memory directly affects the smoothness of multimedia data playback. For example, if the data volume of the multimedia data to be played cached in the first memory is 0, or cannot meet the playback volume requirement for the next moment, then stuttering may occur. Therefore, on the one hand, performing stuttering detection based on the data volume of the multimedia data to be played cached in the first memory can obtain a more accurate stuttering detection result; on the other hand, since stuttering detection is performed based on the data volume of the multimedia data to be played cached in the first memory and can be implemented without complex algorithms, therefore, while improving the accuracy of stuttering detection, the load and overhead are not increased, and the cost is lower.

[0077] The embodiments of the present application further provide a stuttering detection method. Figure 7 It is a schematic flow chart of another method for implementing stuttering detection provided by the embodiments of the present application, as Figure 7As shown, the method may include the following steps 701 to 705:

[0078] Step 701, during the process of playing multimedia data, obtain working parameters related to the first memory; wherein, the working parameters include the time information between two adjacent calls to the write interface and the data volume of the multimedia data cached in the first memory; the write interface is used to write the multimedia data cached in the first memory into the second memory.

[0079] In some embodiments, the first memory is used to cache the multimedia data to be played output by the decoder. Of course, the first memory can also be other memories used in the process of multimedia data processing.

[0080] Step 702, according to the time information, determine the time interval between two adjacent calls to the write interface;

[0081] Step 703, determine whether the time interval is greater than or equal to a first threshold; if so, execute step 704; otherwise, return to execute step 701.

[0082] It should be noted that the electronic device can return to execute step 701 when determining that the time interval is greater than or equal to the first threshold, or can wait for a preset duration and then return to execute step 701.

[0083] Step 704, determine whether the data volume is less than or equal to a second threshold; if so, execute step 705; otherwise, return to execute step 701.

[0084] It should be noted that the electronic device can return to execute step 701 when determining that the data volume is less than or equal to the second threshold, or can wait for a preset duration and then return to execute step 701.

[0085] In the embodiments of the present application, step 704 can also be executed first, and then step 703 can be executed when determining that the data volume is less than or equal to the second threshold. Step 703 and step 704 can also be executed in parallel. In short, in this embodiment, the condition for the multimedia data playback to be stuck is that the time interval is greater than or equal to the first threshold and the data volume is less than or equal to the second threshold.

[0086] Step 705, determine that the multimedia data playback is stuck.

[0087] In some other embodiments, when the duration of the state where the time interval is greater than or equal to the first threshold and the data volume is less than or equal to the second threshold is greater than or equal to a third threshold, determine that the multimedia data playback is stuck.

[0088] In the embodiments of the present application, not only based on the time interval between two adjacent calls to the write interface, but also based on the data volume of the multimedia data cached in the first memory, that is, combining these two parameter information for stutter detection, so as to further improve the accuracy of stutter detection.

[0089] In some embodiments, after determining that stuttering occurs during the playback of multimedia data, the method further includes: performing a preset operation for improving the stuttering. For example, the electronic device may perform the following steps: determining whether the device can be connected to other networks in addition to the already connected network; if it can be connected to other networks, obtaining the performance parameters of the other networks; determining whether the performance of the other networks is better than that of the already connected network according to the performance parameters; if it is better than the already connected network, switching the network connection to the other network. Another example is that the electronic device closes other processes that are irrelevant to the processing and playback of the multimedia data. In short, the preset operations are all measures taken to improve the stuttering.

[0090] In the embodiments of the present application, a method for counting the audio stutter rate in a real-time audio and video (WebRTC) scenario is provided. The audio stutter rate refers to the number of times the sound stutters within a certain period of time, and the sound stutter refers to the phenomenon that the sound is interrupted for a period of time and then resumes. This is determined by the characteristics of the sound. The transmission of the sound is continuous like flowing water. Therefore, when an abnormality occurs and the sound is interrupted for a period of time and then resumes, this is audio stuttering. As Figure 8 shown is a normal audio graph, Figure 9 and is an audio graph with stuttering.

[0091] Comparing Figure 8 it can be seen that in Figure 9 , A and B were originally connected, but there is only 20 ms of silent data in the middle. This is the zero data filled by the system due to the transmission interruption, and the output performance is stuttering. Therefore, it can be defined as follows: when the interruption time between A and B exceeds 200 ms, it is considered that an audio abnormal stuttering phenomenon occurs, and the audio stutter rate is defined by counting the number of AB stutters occurring within a certain period of time.

[0092] Such as Figure 10As shown in the figure, the receiver receives the bitstream from the network side, sends the bitstream into the jitter buffer, the decoder decodes the bitstream in the jitter buffer, and writes the decoded audio data into buffer 1001. The rendering of Webrtc audio data is actually that the application side reads the decoded audio data from buffer 1001 (in the NetEQ module), and then calls the write interface (API AudioTrack write) of the audio track of the Android system, so as to write the audio data into the Android system through this write interface, and thus realize the playback of audio data through the local audio track. The entire writing process is to continuously read the decoded audio data from buffer 1001 and output it to the Android system. After calling the audio track.write interface (AudioTrack.write) to write the audio data to the Android system layer, it is no longer controlled by the application side. If the data written by the application side is timely and uninterrupted, the probability of jamming caused by system layer reasons is extremely small. Therefore, the focus can be on the audio jamming caused by the untimely writing of data by the application side.

[0093] When the time interval between two calls to the write interface exceeds 200 ms, it is defined as a freeze, and the time interval is the freeze duration.

[0094] As Figure 11 shown in the figure, time 1 is the time of the nth call to the write interface, and time 2 is the time of the (n + 1)th call to the write interface. Time 2 - Time 1 is the time interval between two calls to the write interface, denoted as freezeTimeMs. If freezeTimeMs > 200 ms, it is a freeze, and the freeze duration is freezeTimeMs. There are various factors affecting the duration of freezeTimeMs, but from the process of the audio track thread, there are at least two factors: one is whether data can be retrieved from buffer 1001, and the other is whether the write interface returns in time.

[0095] (1) Whether data can be retrieved from buffer 1001:

[0096] This is affected by the video call framework. The influencing factors include: network, the other party's encoding, the other party's sending, our receiving, our decoding, etc. If everything is normal, data can be continuously retrieved from buffer 1001, and the audio data can be written to the Android system layer in time. If data cannot be retrieved, it will wait until there is enough data. If more than 200 ms occurs during the waiting process, an audio freeze will occur.

[0097] (2) Whether the write interface returns in time:

[0098] The process of writing data and consuming data is as Figure 12As shown, the process of writing data is blocking. If the writing is not completed, it will keep waiting until it is finished. The main reason why the writing of data cannot be returned in time is that the shared cache 1201 is full and it is necessary to wait for the consumed data until there is space in the cache for writing.

[0099] In the embodiments of the present application, a method for statistically analyzing the audio stuttering rate in a WebRTC scenario is provided to determine the video quality of the current WebRTC call and monitor it. When data anomalies are detected, this statistical method can be used for anomaly location and problem fixing.

[0100] The above technical solution is to monitor the audio stuttering rate in the WebRTC scenario. This statistical method can actually be extended to other streaming media playback scenarios. In addition, based on the detection of the audio stuttering rate, stuttering optimization can be further performed after stuttering detection, so as to improve the audio stuttering problem and further improve the video call quality.

[0101] It should be noted that although the steps of the methods in the present application are described in a specific order in the drawings, this does not require or imply that these steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution, etc.; or, the steps in different embodiments may be combined into a new technical solution.

[0102] Based on the foregoing embodiments, the embodiments of the present application provide a stuttering detection device. The device includes each module included and each unit included in each module, and can be implemented by a processor; of course, it can also be implemented by specific logic circuits; during implementation, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0103] Figure 13 It is a schematic structural diagram of the stuttering detection device according to the embodiments of the present application. As Figure 13 shown, the stuttering detection device 130 includes:

[0104] An acquisition module 1301, configured to acquire working parameters related to a first memory during the process of playing multimedia data;

[0105] A prediction module 1302, configured to predict the stuttering situation of multimedia data playback according to the working parameters.

[0106] In some embodiments, the first memory is used to cache the multimedia data to be played output by the decoder; the working parameters include the time information of two adjacent calls to the write interface; wherein, the write interface is used to write the multimedia data cached in the first memory into the second memory; the prediction module 1302 is configured to: determine the time interval between the two adjacent calls to the write interface according to the time information; and determine that the multimedia data playback is stuck when the time interval is greater than or equal to the first threshold.

[0107] In some embodiments, the stutter detection device 130 further includes: a display module configured to display one or more options reflecting different stutter levels; a receiving module configured to receive a selection operation on the one or more options; wherein, the selection operation is used to indicate the stutter level of the feedback; a response module configured to, in response to the selection operation, adjust the first threshold according to the stutter level selected by the selection operation.

[0108] Further, in some embodiments, the display module is configured to display one or more options reflecting different stutter levels when a parameter characterizing the frequency of stuttering satisfies a display condition.

[0109] In some embodiments, the first memory includes any memory used in the multimedia data processing flow; the working parameters include the data volume of the multimedia data cached in the first memory; the prediction module 1302 is configured to predict the stutter situation of the multimedia data playback based on determining that the data volume is less than or equal to a second threshold.

[0110] Further, in some embodiments, the prediction module 1302 is configured to: determine that the multimedia data playback is stuck when the data volume is less than or equal to the second threshold; or, the prediction module 1302 is configured to: determine the duration of the state in which the data volume is less than or equal to the second threshold; and determine that the multimedia data playback is stuck when the duration of the state is greater than or equal to a third threshold.

[0111] In some embodiments, the first memory is used to cache the multimedia data to be played output by the decoder; the working parameters include the time information of two adjacent calls to the write interface and the data volume of the multimedia data cached in the first memory; the write interface is used to write the multimedia data cached in the first memory into the second memory; the prediction module 1302 is configured to determine that the multimedia data playback is stuck when the time interval is greater than or equal to a first threshold and the data volume is less than or equal to a second threshold; or, the prediction module 1302 is configured to determine the duration of the state where the data volume is less than or equal to the second threshold; and determine that the multimedia data playback is stuck when the time interval is greater than or equal to the first threshold and the duration of the state is greater than or equal to a third threshold.

[0112] In some embodiments, the prediction module 1302 is configured to determine the stuck situation corresponding to the working parameters by querying the historical record information; wherein, the historical record information includes the historical values of the working parameters and the corresponding stuck situations.

[0113] The description of the above device embodiments is similar to the description of the above method embodiments and has similar beneficial effects to the method embodiments. For the technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0114] It should be noted that in the embodiments of the present application Figure 13 The division of the modules of the stuck detection device shown is schematic, and it is only a logical function division. In actual implementation, there may be other division methods. In addition, each functional unit in the various embodiments of the present application may be integrated in a processing unit, may exist physically alone, or two or more units may be integrated in one unit. The above integrated units may be implemented in the form of hardware, or in the form of software functional units, or in the form of a combination of software and hardware.

[0115] It should be noted that in the embodiments of the present application, if the above method is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the related technology, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing an electronic device to execute all or part of the methods described in the various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), magnetic disks, or optical discs that can store program codes. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0116] An embodiment of the present application provides an electronic device. Figure 14 It is a schematic diagram of the hardware entity of the electronic device according to the embodiment of the present application. As Figure 14 shown, the electronic device 140 includes a memory 141 and a processor 142. The memory 141 stores a computer program that can run on the processor 142. When the processor 142 executes the program, it implements the steps in the method provided in the above embodiment.

[0117] It should be noted that the memory 141 is configured to store instructions and applications executable by the processor 142, and can also cache data to be processed or already processed by each module in the processor 142 and the electronic device 140 (for example, image data, audio data, voice communication data, and video communication data), and can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0118] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps in the method provided in the above embodiment.

[0119] An embodiment of the present application provides a computer program product containing instructions. When it runs on a computer, it causes the computer to execute the steps in the method provided in the above method embodiment.

[0120] It should be pointed out here that the descriptions of the above storage medium and device embodiments are similar to those of the above method embodiments and have similar beneficial effects to the method embodiments. For the technical details not disclosed in the storage medium, storage medium and device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0121] It should be understood that the "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the appearances of "in one embodiment" or "in an embodiment" or "in some embodiments" throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the magnitudes of the sequence numbers of the above processes do not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application. The sequence numbers of the embodiments of the present application above are only for description and do not represent the advantages or disadvantages of the embodiments. The descriptions of the above embodiments tend to emphasize the differences between the embodiments, and their similarities or similarities can be referred to each other. For the sake of brevity, they will not be repeated herein.

[0122] As used herein, the term "and / or" is merely a description of the relationship between associated objects, indicating that there can be three relationships, for example, object A and / or object B can represent: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0123] It should be noted that, as used herein, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0124] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The above-described embodiments are merely illustrative. For example, the division of the modules is only a logical function division, and there can be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling or communication connection between the components shown or discussed with each other can be through some interfaces, and the indirect coupling or communication connection of the devices or modules can be electrical, mechanical or other forms.

[0125] The modules described above as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network elements; some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] In addition, in each embodiment of this application, all the functional modules may be integrated in a processing unit, or each module may be a separate unit alone, or two or more modules may be integrated in one unit; the above integrated modules may be implemented in the form of hardware, or in the form of a combination of hardware and software functional units.

[0127] Those of ordinary skill in the art can understand that all or part of the steps to implement the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: removable storage devices, read-only memory (ROM), magnetic disks, or optical disks and other various media that can store program codes.

[0128] Alternatively, if the above integrated unit of this application is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of this application essentially or the part that contributes to the related art can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable an electronic device to execute all or part of the methods described in various embodiments of this application. And the foregoing storage medium includes: removable storage devices, ROM, magnetic disks, or optical disks and other various media that can store program codes.

[0129] The methods disclosed in several method embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments.

[0130] The features disclosed in several product embodiments provided in this application can be combined arbitrarily without conflict to obtain new product embodiments.

[0131] The features disclosed in several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

[0132] As described above, it is only the implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims described above.

Claims

1. A method for detecting lag, characterized in that, the method includes: During the process of playing multimedia data, obtain working parameters related to a first memory; wherein, the first memory is used to cache multimedia data to be played; wherein, the working parameters include time information of two adjacent calls to a write interface; wherein, the write interface is used to write the multimedia data cached in the first memory into a second memory; According to the time information, determine the time interval between the two adjacent calls to the write interface; In the case where the time interval is greater than or equal to a first threshold, determine that lag occurs in the multimedia data playback; wherein, the method further includes: In the case where a parameter characterizing the frequency of lag meets a display condition, display one or more options reflecting different degrees of lag; Receive a selection operation on the one or more options; wherein, the selection operation is used to indicate the degree of lag for feedback; In response to the selection operation, adjust the first threshold according to the degree of lag selected by the selection operation.

2. The method according to claim 1, characterized in that, the working parameters include the data volume of the multimedia data cached in the first memory; correspondingly, the predicting the lag situation of the multimedia data playback according to the working parameters includes: Based on determining that the data volume is less than or equal to a second threshold, predict the lag situation of the multimedia data playback.

3. The method according to claim 2, characterized in that, the predicting the lag situation of the multimedia data playback based on determining that the data volume is less than or equal to a second threshold includes: In the case where the data volume is less than or equal to the second threshold, determine that lag occurs in the multimedia data playback; or, Determine the duration of the state where the data volume is less than or equal to the second threshold; and in the case where the duration of the state is greater than or equal to a third threshold, determine that lag occurs in the multimedia data playback.

4. The method according to claim 1, characterized in that, the working parameters include the time information of two adjacent calls to the write interface and the data volume of the multimedia data cached in the first memory; the write interface is used to write the multimedia data cached in the first memory into a second memory; correspondingly, the predicting the lag situation of the multimedia data playback according to the working parameters includes: According to the time information, determine the time interval between the two adjacent calls to the write interface; in the case where the time interval is greater than or equal to the first threshold and the data volume is less than or equal to the second threshold, determine that lag occurs in the multimedia data playback; or, Determine the duration of the state where the data volume is less than or equal to the second threshold; in the case where the time interval is greater than or equal to the first threshold and the duration of the state is greater than or equal to the third threshold, determine that lag occurs in the multimedia data playback.

5. The method according to claim 1, characterized in that, the predicting the lag situation of the multimedia data playback according to the working parameters includes: Predict the lag situation corresponding to the working parameter by querying historical record information; wherein, the historical record information includes the historical value of the working parameter and the corresponding lag situation.

6. A lag detection device characterized in that it includes: An acquisition module configured to acquire working parameters related to a first memory during the process of playing multimedia data; wherein, the working parameters include time information of two adjacent calls to a write interface; wherein, the write interface is used to write the multimedia data cached in the first memory into a second memory; A prediction module configured to determine the time interval between two adjacent calls to the write interface according to the time information; and determine that the multimedia data playback lags when the time interval is greater than or equal to a first threshold; The device further includes: A display module configured to display one or more options reflecting different lag degrees when a parameter characterizing the frequency of lags meets a display condition; A receiving module configured to receive a selection operation on the one or more options; wherein, the selection operation is used to indicate the lag degree of feedback; A response module configured to, in response to the selection operation, adjust the first threshold according to the lag degree selected by the selection operation.

7. An electronic device including a memory and a processor, the memory storing a computer program that can run on the processor, characterized in that when the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images