Video stutter prediction method, device, storage medium and program product

Abstract

This application discloses a video stuttering prediction method, device, storage medium, and program product, belonging to the field of computer technology. In this application, during video playback, the rate of decrease of the audio buffer level is continuously detected, and the audio-visual time deviation between the played video frames and audio frames is continuously detected to determine the rate of change of the audio-visual time deviation; the audio buffer level is the number of audio frames to be decoded or played stored in the audio buffer; when the audio-visual time deviation increases, based on at least one of the rate of change and the rate of decrease corresponding to the target time period, a playback stuttering risk prediction result for the video is determined; when the playback stuttering risk prediction result indicates the existence of a playback stuttering risk, playback adjustment measures are executed. This application eliminates the risk before stuttering actually occurs, ensuring smooth video playback.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, device, storage medium, and program product for predicting video stuttering. Background Technology

[0002] With the rapid iteration and development of streaming media technology, users have put forward increasingly stringent requirements for the smoothness and stability of video playback. Video stuttering detection technology has become a core research direction in the field of video playback. Accurate detection of video stuttering is the prerequisite and foundation for subsequent stuttering optimization processing and ensuring playback quality.

[0003] Currently, during video playback, the timestamp difference between the video frames and audio frames is acquired in real time. When the timestamp difference exceeds a preset threshold, it is determined that the video playback is stuttering.

[0004] However, when the timestamp difference exceeds the preset threshold, video stuttering has already occurred. Even if stuttering prevention measures are implemented immediately at this time, it can only prevent the same stuttering problem from occurring again in subsequent playback. It cannot undo the irreversible impact that stuttering has already had on the user's viewing experience. The user has clearly perceived the stuttering, and it is impossible to achieve early stuttering prevention and lossless experience optimization. Summary of the Invention

[0005] This application provides a video stuttering prediction method, device, storage medium, and program product, which can solve the problem of video playback stuttering. The technical solution is as follows: On the one hand, a video stuttering prediction method is provided, the method comprising: During video playback, the rate of decrease of the audio buffer level of the video is continuously detected, and the audio-visual time deviation between the played video frames and audio frames is continuously detected to determine the rate of change of the audio-visual time deviation; the audio buffer level is the number of audio frames to be decoded or played stored in the audio buffer. When the audio-visual time deviation increases, the playback stuttering risk prediction result for the video is determined based on at least one of the rate of change and the rate of decrease corresponding to the target time period; When the playback stuttering risk prediction result indicates that there is a risk of playback stuttering, playback adjustment measures are implemented.

[0006] In one possible implementation, determining the playback stuttering risk prediction result for the video based on at least one of the rate of change and the rate of decrease corresponding to the target time period includes: If the rate of change corresponding to the target time period is greater than or equal to the first specified rate of change, and the rate of decrease corresponding to the target time period is greater than or equal to the first specified rate of decrease, then it is determined that the video has a risk of playback stuttering.

[0007] In another possible implementation, determining the playback stuttering risk prediction result for the video based on at least one of the rate of change and the rate of decrease corresponding to the target time period includes: If the rate of change corresponding to the target time period is greater than or equal to the first specified rate of change and less than the second specified rate of change, and the rate of decrease corresponding to the target time period is greater than or equal to the first specified rate of decrease and less than the second specified rate of decrease, then it is determined that the video has a low risk of playback stuttering, where the second specified rate of change is greater than the first specified rate of change and the second specified rate of decrease is greater than the first specified rate of decrease. If the rate of change corresponding to the target time period is greater than or equal to the second specified rate of change, and the rate of decrease corresponding to the target time period is greater than or equal to the second specified rate of decrease, then it is determined that the video has a high risk of playback stuttering.

[0008] In another possible implementation, determining the playback stuttering risk prediction result for the video based on at least one of the rate of change and the rate of decrease corresponding to the target time period includes: If the rate of change corresponding to the target time period is greater than or equal to the third specified rate of change, it is determined that the video has a high risk of playback stuttering, and the third specified rate of change is greater than the second specified rate of change. If the drop rate corresponding to the target time period is greater than or equal to the third specified drop rate, it is determined that the video has a high risk of playback stuttering, and the third specified drop rate is greater than the second specified drop rate.

[0009] In another possible implementation, determining the playback stuttering risk prediction result for the video based on at least one of the rate of change and the rate of decrease corresponding to the target time period includes: The audio-visual time deviation, rate of change, rate of decrease, CPU usage, GPU usage, memory usage, and battery temperature of the device playing the video are input into the risk prediction model to obtain the stuttering risk probability value output by the risk prediction model. If the probability value of the stuttering risk is greater than or equal to a first specified probability value and less than a second specified probability value, then it is determined that the video has a low risk of playback stuttering, and the second specified probability value is greater than the first specified probability value. If the probability value of the stuttering risk is greater than or equal to the second specified probability value, then it is determined that the video has a high risk of playback stuttering.

[0010] In another possible implementation, the step of performing playback adjustment measures when the playback stuttering risk prediction result indicates the existence of playback stuttering risk includes: If there is a low risk of playback stuttering in the video, increase the preloading time of the video; If there is a high risk of playback stuttering in the video, reduce the bitrate of the video.

[0011] In another possible implementation, the method further includes: In response to the increasing trend of the rate of change, the duration of the target time period is reduced.

[0012] On the other hand, a video stuttering prediction device is provided, the device comprising: The detection module is configured to continuously detect the rate of decrease of the audio buffer level of the video during video playback, and to continuously detect the audio-visual time deviation between the played video frames and audio frames, and determine the rate of change of the audio-visual time deviation; the audio buffer level is the number of audio frames to be decoded or played stored in the audio buffer. The prediction module is configured to determine a playback stuttering risk prediction result for the video based on at least one of the rate of change and the rate of decrease corresponding to the target time period when the audio-visual time deviation increases. The processing module is configured to execute playback adjustment measures when the playback stuttering risk prediction result indicates that there is a risk of playback stuttering.

[0013] In one possible implementation, the prediction module is configured to: If the rate of change corresponding to the target time period is greater than or equal to the first specified rate of change, and the rate of decrease corresponding to the target time period is greater than or equal to the first specified rate of decrease, then it is determined that the video has a risk of playback stuttering.

[0014] In another possible implementation, the prediction module is used for: If the rate of change corresponding to the target time period is greater than or equal to the first specified rate of change and less than the second specified rate of change, and the rate of decrease corresponding to the target time period is greater than or equal to the first specified rate of decrease and less than the second specified rate of decrease, then it is determined that the video has a low risk of playback stuttering, where the second specified rate of change is greater than the first specified rate of change and the second specified rate of decrease is greater than the first specified rate of decrease. If the rate of change corresponding to the target time period is greater than or equal to the second specified rate of change, and the rate of decrease corresponding to the target time period is greater than or equal to the second specified rate of decrease, then it is determined that the video has a high risk of playback stuttering.

[0015] In another possible implementation, the prediction module is used for: If the rate of change corresponding to the target time period is greater than or equal to the third specified rate of change, it is determined that the video has a high risk of playback stuttering, and the third specified rate of change is greater than the second specified rate of change. If the drop rate corresponding to the target time period is greater than or equal to the third specified drop rate, it is determined that the video has a high risk of playback stuttering, and the third specified drop rate is greater than the second specified drop rate.

[0016] In another possible implementation, the prediction module is used for: The audio-visual time deviation, rate of change, rate of decrease, CPU usage, GPU usage, memory usage, and battery temperature of the device playing the video are input into the risk prediction model to obtain the stuttering risk probability value output by the risk prediction model. If the probability value of the stuttering risk is greater than or equal to a first specified probability value and less than a second specified probability value, then it is determined that the video has a low risk of playback stuttering, and the second specified probability value is greater than the first specified probability value. If the probability value of the stuttering risk is greater than or equal to the second specified probability value, then it is determined that the video has a high risk of playback stuttering.

[0017] In another possible implementation, the processing module is used for: If there is a low risk of playback stuttering in the video, increase the preloading time of the video; If there is a high risk of playback stuttering in the video, reduce the bitrate of the video.

[0018] In another possible implementation, the prediction module is used for: In response to the increasing trend of the rate of change, the duration of the target time period is reduced.

[0019] On the other hand, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the method described in any of the above.

[0020] On the other hand, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method described in any of the preceding claims.

[0021] On the other hand, a computer program product is provided, including computer program instructions that, when run on a computer, cause the computer to perform the method described in any of the preceding claims.

[0022] The beneficial effects of the technical solution provided in this application are: the rate of decrease in the audio buffer level and the rate of change of the audio-visual time deviation can reflect the real-time changes in the video playback status. When the audio-visual time deviation increases and the rate of decrease in the audio buffer level and the rate of change of the audio-visual time deviation within the target time period meet the corresponding conditions, it indicates that video playback abnormalities are continuously accumulating and the risk of stuttering is gradually increasing. At this time, determining the existence of playback stuttering risk and implementing processing measures can eliminate the risk before stuttering actually occurs, ensuring the smoothness of video playback. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application; Figure 2 This is a flowchart of the video stuttering prediction method provided in the embodiments of this application; Figure 3 This is a logic block diagram of the video stuttering prediction method provided in the embodiments of this application; Figure 4 This is a logic block diagram of a video stuttering prediction method provided in another embodiment of this application; Figure 5 This is a schematic diagram of the video stuttering prediction device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0026] This disclosure provides a method for predicting video stuttering. This method can be applied to a terminal. For example... Figure 1 As shown, the terminal may include a processor 110, a memory 120, and a communication component 130.

[0027] Processor 110 can be a central processing unit (CPU), graphics processing unit (GPU), microcontroller unit (MCU), accelerated processing unit (APU), neural processing unit (NPU), tensor processing unit (TPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processor (DSP), etc. Processor 110 can be used to detect the rate of decrease of the audio buffer level in a video, the audio-visual time deviation between video frames and audio frames, and so on.

[0028] Memory 120 may include volatile memory and / or non-volatile memory. Volatile memory may include random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, non-volatile random access memory (NVRAM), etc. Memory 120 can be used to store data such as audio-visual timing deviations, their rates of change, and decay rates.

[0029] The communication component 130 can be a wireless communication module (WCM), a subscriber identity module (SIM), a universal subscriber identity module (USIM), an optical network unit (ONU), etc. The communication component 130 can be used for communication between the terminal and a server, or other terminals.

[0030] This application provides a method for predicting video stuttering, such as... Figure 2 As shown, in some embodiments, the method includes: S201. During video playback, continuously detect the rate of decrease of the audio buffer level of the video, and continuously detect the audio-visual time deviation between the played video frames and audio frames, and determine the rate of change of the audio-visual time deviation; the audio buffer level is the number of audio frames to be decoded or played stored in the audio buffer.

[0031] The audio buffer level refers to the number of audio frames temporarily stored in the playback device's audio buffer during video playback, intended for decoding or playback, to ensure audio playback continuity. The rate of decrease in the audio buffer level refers to the magnitude of the decrease in the audio buffer level per unit time, reflecting the relationship between the rate of audio data consumption and the rate of audio data replenishment. Audio-visual timing deviation refers to the difference between the timestamp of the currently playing video frame and the timestamp of the currently playing audio frame during video playback. Under normal playback conditions, video frames and audio frames should remain synchronized, and the audio-visual timing deviation should approach zero or fluctuate within a very small range. The rate of change of the audio-visual timing deviation refers to the magnitude of change in the audio-visual timing deviation per unit time, reflecting the changing trend of the synchronization state between video and audio playback.

[0032] During video playback, the playback device continuously monitors the current level of the audio buffer. The audio buffer level is represented by the duration of the audio data in the buffer. For example, if the buffer stores 500 audio frames, the current level is 500 frames. By sampling the audio buffer level at consecutive time points and calculating the ratio of the difference between two adjacent samples to the sampling time interval, the rate of decrease in the audio buffer level is obtained. If the level is 500 frames at the first sample and 400 frames at the second sample 100 milliseconds later, the rate of decrease is (500-400) / 100 = 1 frame / millisecond, meaning the level decreases by 1 frame every millisecond.

[0033] During video playback, the playback device continuously acquires the timestamps of the currently playing video frames and audio frames. The video frame timestamp indicates the time when the video frame should be played, and the audio frame timestamp indicates the time when the audio frame should be played. The difference between the two is the audio-visual timing deviation. For example, if the timestamp of the currently playing video frame is 1000 milliseconds and the timestamp of the currently playing audio frame is 1020 milliseconds, then the audio-visual timing deviation is 1020 - 1000 = 20 milliseconds (the audio leads the video by 20 milliseconds). By performing differential calculations on the audio-visual timing deviations at consecutive time points, the rate of change of the audio-visual timing deviation can be obtained. For example, if the audio-visual timing deviation is 10 milliseconds at the first moment, and 30 milliseconds at the second moment 100 milliseconds later, then the rate of change is (30 - 10) / 100 = 0.2 milliseconds / millisecond.

[0034] S202. When the audio-visual time deviation increases, determine the playback stuttering risk prediction result for the video based on at least one of the change rate and the decrease rate corresponding to the target time period.

[0035] The target time period refers to a pre-defined time interval used to determine whether the rate of change of audio-visual timing deviation and the rate of decrease of the audio buffer level continuously meet specified conditions. The specific length of the target time period can be set according to the actual application scenario and accuracy requirements, such as within 1 second of history. Specified conditions refer to the criteria used to determine the risk of playback stuttering in the video, including but not limited to thresholds for the rate of change of audio-visual timing deviation and the rate of decrease of the audio buffer level. Playback stuttering risk refers to the risk that abnormal situations such as video stuttering, audio-visual desynchronization, or playback interruption may occur during video playback.

[0036] During video playback, if an increasing trend in audio-visual timing deviation is detected, it indicates that the synchronization between video and audio playback is deteriorating, potentially posing a risk of playback stuttering. Further analysis is needed to determine whether the rate of change of the audio-visual timing deviation and the rate of decrease of the audio buffer level consistently meet specified conditions within a target time period. These specified conditions can be that the rate of change of the audio-visual timing deviation is greater than or equal to a preset rate of change threshold, and the rate of decrease of the audio buffer level is greater than or equal to a preset rate of decrease threshold. The target time period can be a preset time window, such as within 500 milliseconds, 1 second, or 2 seconds historically. If, within the target time period, the rate of change of the audio-visual timing deviation and the rate of decrease of the audio buffer level consistently meet the specified conditions, then the video is deemed to have a risk of playback stuttering. For example, if the target time period is within 1 second historically, and the specified conditions are a rate of change greater than or equal to 0.1 milliseconds / millisecond and a rate of decrease greater than or equal to 3 frames / millisecond, then within 1 second, if the rate of change is consistently greater than or equal to 0.1 milliseconds / millisecond and the rate of decrease is consistently greater than or equal to 3 frames / millisecond, then the video is deemed to have a risk of playback stuttering.

[0037] S203. When the playback stuttering risk prediction result indicates that there is a risk of playback stuttering, playback adjustment measures shall be implemented.

[0038] Specifically, once a video is identified as having a risk of playback stuttering, the playback device can take appropriate measures to eliminate or reduce this risk. These measures may include adjusting the video's preload duration, adjusting the video's bitrate, adjusting the video's resolution, enabling hardware acceleration, and optimizing decoding strategies.

[0039] In this embodiment, by continuously monitoring the rate of decrease in the audio buffer level and the rate of change in the audio-visual timing deviation, and considering the increasing audio-visual timing deviation, the system comprehensively judges whether the rate of change and the rate of decrease continuously meet specified conditions within the target time period, thereby achieving early prediction of video playback stuttering risks. Compared with related technologies that only detect and process stuttering after it occurs, this method can predict stuttering risks before they actually occur, allowing for proactive measures to avoid impacting the user's viewing experience and achieving early stuttering prevention and lossless experience optimization.

[0040] In some embodiments, determining the playback stuttering risk prediction result for the video based on at least one of the rate of change and the rate of decrease corresponding to the target time period includes: If the rate of change corresponding to the target time period is greater than or equal to the first specified rate of change, and the rate of decrease corresponding to the target time period is greater than or equal to the first specified rate of decrease, then it is determined that the video has a risk of playback stuttering.

[0041] Specifically, when the rate of change of the audio-visual timing deviation is consistently greater than or equal to a first specified rate of change within a target time period, and the rate of decrease of the audio buffer level is consistently greater than or equal to a first specified rate of decrease within the target time period, it indicates a dual anomaly of audio-visual asynchrony and a rapid decrease in the audio buffer level during video playback, thus confirming a risk of playback stuttering. For example, setting the first specified rate of change to 0.1 milliseconds / millisecond, the first specified rate of decrease to 3 frames / millisecond, and the target time period to a historical 1 second. If, within a continuous 1-second time period, the rate of change of the audio-visual timing deviation is consistently greater than or equal to 0.1 milliseconds / millisecond, and the rate of decrease of the audio buffer level is consistently greater than or equal to 3 frames / millisecond, then it is determined that the video has a risk of playback stuttering.

[0042] In this embodiment, by setting a first specified rate of change and a first specified rate of decrease as thresholds for determining the risk of stuttering, and requiring that the rate of change and the rate of decrease continuously meet the threshold conditions within the target time period, the accuracy and reliability of stuttering risk determination can be effectively improved. Only when the rate of change of the audio-visual time deviation and the rate of decrease of the audio buffer level both continuously reach the thresholds are stuttering risks determined, avoiding misjudgments caused by instantaneous fluctuations, and also preventing the best prevention opportunity from being missed due to excessively slow response.

[0043] In some embodiments, determining the playback stuttering risk prediction result for the video based on at least one of the rate of change and the rate of decrease corresponding to the target time period includes: If the rate of change corresponding to the target time period is greater than or equal to the first specified rate of change and less than the second specified rate of change, and the rate of decrease corresponding to the target time period is greater than or equal to the first specified rate of decrease and less than the second specified rate of decrease, then it is determined that the video has a low risk of playback stuttering, where the second specified rate of change is greater than the first specified rate of change and the second specified rate of decrease is greater than the first specified rate of decrease.

[0044] If the rate of change corresponding to the target time period is greater than or equal to the second specified rate of change, and the rate of decrease corresponding to the target time period is greater than or equal to the second specified rate of decrease, then it is determined that the video has a high risk of playback stuttering.

[0045] Specifically, when both the rate of change of the audio-visual timing deviation and the rate of decrease of the audio buffer level are between the first and second thresholds, the risk of playback stuttering is relatively low, and the system is classified as having a low risk of playback stuttering. When both the rate of change of the audio-visual timing deviation and the rate of decrease of the audio buffer level reach or exceed the second threshold, the risk of playback stuttering is relatively high, and the system is classified as having a high risk of playback stuttering. For example, a first specified rate of change is set to 0.1 milliseconds / millisecond, a second specified rate of change is set to 0.3 milliseconds / millisecond, a first specified rate of decrease is set to 3 frames / millisecond, a second specified rate of decrease is set to 6 frames / millisecond, and the target time period is within a historical 1 second. If, within 1 second, the rate of change is consistently within the range of [0.1, 0.3) and the rate of decrease is consistently within the range of [3, 6), then the system is classified as having a low risk of playback stuttering. If, within 1 second, the rate of change is consistently greater than or equal to 0.3 milliseconds / millisecond and the rate of decrease is consistently greater than or equal to 6 frames / millisecond, then the system is classified as having a high risk of playback stuttering.

[0046] In this embodiment, by classifying the risk of playback stuttering, the severity of the risk can be assessed more accurately, allowing for targeted and appropriate handling measures. For low-risk scenarios, mild handling measures can be taken to avoid excessive intervention that could negatively impact the user experience; for high-risk scenarios, proactive measures can be implemented promptly to ensure effective stuttering control, minimizing the impact on the user's viewing experience while maintaining effective stuttering prevention.

[0047] In some embodiments, determining the playback stuttering risk prediction result for the video based on at least one of the rate of change and the rate of decrease corresponding to the target time period includes: If the rate of change corresponding to the target time period is greater than or equal to the third specified rate of change, then it is determined that the video has a high risk of playback stuttering, wherein the third specified rate of change is greater than the second specified rate of change.

[0048] If the drop rate corresponding to the target time period is greater than or equal to the third specified drop rate, it is determined that the video has a high risk of playback stuttering, and the third specified drop rate is greater than the second specified drop rate.

[0049] Specifically, if the rate of change of the audio-visual timing deviation is extremely high, it indicates a sharp deterioration in the synchronization between video and audio playback. Even if the rate of decrease in the audio buffer level has not yet reached the second specified rate, it should still be considered a high risk of playback stuttering. If the rate of decrease in the audio buffer level is extremely high (e.g., reaching the third specified rate), it indicates that the rate of audio data consumption is much greater than the rate of replenishment, and audio playback is about to be interrupted. Even if the rate of change of the audio-visual timing deviation has not yet reached the second specified rate, it should still be considered a high risk of playback stuttering. For example, the third specified rate of change is set to 0.5 milliseconds / millisecond, and the third specified rate of decrease is 10 frames / millisecond. If the rate of change is consistently greater than or equal to 0.5 milliseconds / millisecond within the target time period, regardless of the actual rate of decrease, it should be directly considered a high risk of playback stuttering. If the rate of decrease is consistently greater than or equal to 10 frames / millisecond within the target time period, regardless of the actual rate of change, it should be directly considered a high risk of playback stuttering.

[0050] This embodiment achieves rapid assessment of severe playback risks by setting a third specified rate of change and a third specified rate of decrease. When any key indicator reaches an extremely high value, it can be directly identified as high-risk and responded to immediately without waiting for another indicator to reach its threshold. This effectively addresses sudden and severe playback anomalies, avoiding missing the optimal prevention and control opportunity due to waiting for comprehensive assessment of multiple indicators, and further improving the timeliness and effectiveness of playback prevention and control.

[0051] In some embodiments, determining the playback stuttering risk prediction result for the video based on at least one of the rate of change and the rate of decrease corresponding to the target time period includes: The audio-visual time deviation, rate of change, rate of decrease, CPU usage, GPU usage, memory usage, and battery temperature of the device playing the video are input into the risk prediction model to obtain the stuttering risk probability value output by the risk prediction model.

[0052] If the probability value of the stuttering risk is greater than or equal to a first specified probability value and less than a second specified probability value, then it is determined that the video has a low risk of playback stuttering, and the second specified probability value is greater than the first specified probability value.

[0053] If the probability value of the stuttering risk is greater than or equal to the second specified probability value, then it is determined that the video has a high risk of playback stuttering.

[0054] The risk prediction model refers to a model trained using machine learning algorithms to predict the risk of video playback stuttering. This model includes, but is not limited to, neural network models, decision tree models, random forest models, support vector machine models, and logistic regression models. The input features of the risk prediction model include audio-visual time deviation within the target time period, the rate of change of audio-visual time deviation, the rate of decrease in the audio buffer level, device CPU usage, GPU usage, memory usage, and battery temperature. The output of the risk prediction model is a stuttering risk probability value, representing the probability that the video will experience playback stuttering, with a value ranging from [0,1] or [0%,100%]. The first specified probability value and the second specified probability value are probability thresholds used to distinguish between low-risk and high-risk playback stuttering, where the second specified probability value is greater than the first specified probability value.

[0055] The risk prediction model can be trained as follows: Collect a large amount of historical data from video playback, including the true values ​​of the aforementioned features and corresponding stuttering occurrences (labels), to construct a training dataset; use this training dataset to train the initial model, adjusting the model parameters to ensure accurate stuttering risk prediction; after training, deploy the model to the playback device for real-time stuttering risk prediction. For example, set a first specified probability value of 0.3 and a second specified probability value of 0.7. If the risk prediction model outputs a stuttering risk probability value of 0.5, it is considered a low-risk stuttering situation; if the output stuttering risk probability value is 0.8, it is considered a high-risk stuttering situation.

[0056] In this embodiment, a risk prediction model is used to predict stuttering risk by integrating multiple factors, thereby improving the accuracy and comprehensiveness of the prediction. Traditional threshold comparison methods struggle to comprehensively consider the interactions of multiple factors, while machine learning models can automatically learn the complex relationships between these factors, thus making more accurate predictions. Furthermore, by incorporating device status indicators (such as CPU usage, GPU usage, memory usage, and battery temperature), the impact of the playback environment on stuttering risk can be assessed more comprehensively, further improving prediction accuracy.

[0057] In some embodiments, such as Figure 3 , Figure 4 As shown, when the playback stuttering risk prediction result indicates the existence of a playback stuttering risk, the playback adjustment measures are implemented, including: If the video has a low risk of playback stuttering and the video frames lag behind the audio frames, increase the video's preloading duration. If the video has a high risk of playback stuttering and the video frames lag behind the audio frames, decrease the video's bitrate.

[0058] If the video has a low risk of playback stuttering and the audio frames lag behind the video frames, reduce the video playback speed. If the video has a high risk of playback stuttering and the audio frames lag behind the video frames, reduce the video frame rate.

[0059] In video playback, a video frame lagging behind an audio frame means that during video playback, the timestamp of the currently playing video frame is less than the timestamp of the currently playing audio frame. This indicates that the audio playback is ahead of the video playback, resulting in video playback lag. Conversely, an audio frame lagging behind a video frame means that during video playback, the timestamp of the currently playing audio frame is less than the timestamp of the currently playing video frame. This also indicates that the video playback is ahead of the audio playback, resulting in audio playback lag. Since this situation is extremely rare in practical applications, it is usually assumed to be the case of video frames lagging behind audio frames when describing the risk of video playback stuttering. The following discussion will focus on these two scenarios.

[0060] When a video frame lags behind an audio frame, it indicates that the video playback is lagging behind the audio playback. In this case, for low-risk playback stuttering, the preload duration (the duration during which the playback device downloads and caches video data in advance) is increased, allowing the playback device to preload more video data into its local buffer. For example, increasing the preload duration from the default 2 seconds to 5 seconds increases the amount of video data cached in the buffer. Even if there are brief network fluctuations, normal playback can be maintained by relying on the data in the buffer, allowing the video playback to gradually catch up with the audio playback and restore audio-visual synchronization. In some embodiments, the playback device pre-stores a mapping table of preload duration increments. When it is determined that there is a low risk of video stuttering and the video frame lags behind the audio frame, the playback device queries the corresponding mapping table based on the current rate of change or rate of decrease to determine the increment value of the preload duration. The larger the rate of change or rate of decrease, the larger the increment value of the preload duration. This increment value is then added to the current preload duration to obtain the increased preload duration.

[0061] In cases of high risk of playback stuttering, where video data supply is severely insufficient, it is necessary to reduce the video bitrate (bitrate refers to the compression coding rate of video data, i.e., the number of bits of video data per unit time. Reducing the video bitrate means reducing the amount of video data). After reducing the bitrate, the amount of video data decreases, reducing the requirements for network bandwidth and decoding performance, which can fundamentally alleviate the problem of insufficient video data supply. For example, the video bitrate can be reduced from 4Mbps to 2Mbps. In some embodiments, the playback device pre-stores a rate of change-target bitrate mapping table or a rate of fall-target bitrate mapping table. When it is determined that there is a high risk of playback stuttering and the video frame lags behind the audio frame, the playback device queries the corresponding mapping table according to the current rate of change or rate of fall to determine the target bitrate. The higher the rate of change or rate of fall, the lower the target bitrate, in order to further reduce the data processing pressure. After determining the target bitrate, the video bitrate is reduced to the target bitrate.

[0062] When audio frames lag behind video frames, it indicates that the audio playback is lagging behind the video playback. For cases with low risk of playback stuttering, the video playback speed is reduced. Specifically, this means decreasing the playback frequency of video frames while keeping the total number of frames constant, thus extending the video's playback duration. For example, a video with 3000 frames per second (fps) will play for 100 seconds at 30 fps; reducing the playback speed to 28 fps extends the playback duration to approximately 107 seconds. Reducing the playback speed has an effect similar to slowing down the video; the video will appear slightly slower, but all frames will be presented completely without losing any content. When a low risk of playback stuttering is determined and audio frames lag behind video frames, the playback device can directly reduce the current playback speed by a fixed reduction value. For example, if the current playback speed is 30 fps and the fixed reduction value is 2 fps, the reduced playback speed will be 28 fps. In some embodiments, the playback device pre-stores a rate of change-playback speed reduction value mapping table and a rate of decrease-playback speed reduction value mapping table. When a video exhibits a low risk of playback stuttering and audio frames lag behind video frames, the playback device queries the corresponding mapping table based on the current rate of change or rate of fall to determine the reduction in playback speed. A larger rate of change or rate of fall indicates a more severe playback anomaly, and the corresponding reduction in playback speed is greater, subtracted from the current playback speed.

[0063] For situations with a high risk of playback stuttering, the video frame rate is reduced. The playback device reduces the number of video frames that need to be decoded and rendered by skipping frames. For example, reducing the video frame rate from 60 frames per second to 30 frames per second, the playback device plays one frame at a time, skipping half of the video frames. After reducing the frame rate, the processing pressure of decoding and rendering is significantly reduced, system resources are freed up, and it helps audio decoding and playback to proceed normally. At the same time, because the number of frames is reduced, the rendering interval of video frames increases, providing more time for the audio playback progress to catch up. The playback device pre-stores a fixed target frame rate. When it is determined that there is a high risk of playback stuttering and the audio frames lag behind the video frames, the playback device directly reduces the frame rate to the fixed target frame rate. For example, if the current frame rate is 60 frames per second and the fixed target frame rate is 30 frames per second, then the reduced frame rate is 30 frames per second, and the playback device plays one frame at a time, skipping half of the video frames. In some embodiments, the playback device has a pre-set mapping table of rate of change and frame rate reduction value, and a mapping table of descent rate and frame rate reduction value. When a video is determined to have a high risk of playback stuttering and audio frames lag behind video frames, the playback device queries the corresponding mapping table based on the current rate of change or rate of decrease to determine the reduction in frame rate. The larger the rate of change or rate of decrease, the more severe the playback abnormality, and the greater the reduction in frame rate. This reduction is then subtracted from the current frame rate.

[0064] In this embodiment, when video frames lag behind audio frames, it is usually due to insufficient video data supply. Therefore, increasing the preloading time or reducing the bitrate can enhance the video data supply capacity and effectively solve the problem of video playback lag. When audio frames lag behind video frames, it is usually due to insufficient audio data supply or excessively fast video playback speed. In this case, reducing the video frame playback speed or frame rate, by adjusting the video playback rhythm to wait for the audio playback progress to catch up, can more effectively solve the problem of audio playback lag.

[0065] In some embodiments, the method further includes: reducing the duration of the target time period in response to the increasing trend of the rate of change.

[0066] In practice, the trend data of the rate of change can be determined; if the trend data indicates that the rate of change is increasing, the duration of the target time period is reduced. The trend data of the rate of change refers to the trend characteristics of the rate of change of the audio-visual time deviation over time, including but not limited to whether the rate of change is increasing, decreasing, or stable.

[0067] While continuously monitoring the rate of change of audio-visual timing deviation, the playback device also analyzes the trend of this rate of change. This trend can be determined by calculating the increment of the rate of change over a period of time (acceleration of the rate of change) or by using other trend analysis methods. When the trend data shows an increasing rate of change, it indicates that the audio-visual timing deviation is accelerating, and the risk of stuttering may be rising rapidly. In this case, a faster response is needed. Therefore, the response time for determining the risk of stuttering is shortened by reducing the duration of the target time period. For example, the default duration of the target time period is 1 second. When an increasing trend of the rate of change is detected, the duration of the target time period is reduced to 500 milliseconds. As long as the rate of change and the rate of decrease continuously meet the specified conditions within 500 milliseconds, a risk of stuttering can be determined, thereby accelerating the response speed.

[0068] In some embodiments, the playback device pre-stores a mapping table of rate of change acceleration and the reduction in the duration of a target time period. When a rate of change acceleration is detected indicating an increasing trend, the playback device queries the corresponding mapping table based on the current rate of change acceleration to determine the reduction in the duration of the target time period. The greater the rate of change acceleration, the greater the reduction in the duration of the target time period. Subtracting this reduction from the duration of the target time period results in a faster response.

[0069] In this embodiment, dynamic optimization of the lag risk assessment is achieved by adjusting the duration of the target time period according to the changing trend of the rate of change. When the rate of change is increasing, it indicates that the lag risk is rising rapidly. At this time, shortening the assessment time window can speed up the response and allow for more timely preventive measures. When the rate of change is stable or decreasing, maintaining the assessment time window can improve the accuracy of the assessment and avoid misjudgments. While ensuring the accuracy of the assessment, it also improves the responsiveness to rapidly changing scenarios, further optimizing the lag prevention effect.

[0070] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0071] Based on the same inventive concept, corresponding to the video stuttering prediction method provided in the embodiments of this application, this application also provides a video stuttering prediction device.

[0072] refer to Figure 5 The video stuttering prediction device includes: The detection module 501 is configured to continuously detect the rate of decrease of the audio buffer level of the video during video playback, and continuously detect the audio-visual time deviation between the played video frames and audio frames, and determine the rate of change of the audio-visual time deviation; the audio buffer level is the number of audio frames to be decoded or played stored in the audio buffer. The prediction module 502 is configured to determine a playback stuttering risk prediction result for the video based on at least one of the rate of change and the rate of decrease corresponding to the target time period when the audio-visual time deviation increases. The processing module 503 is configured to execute playback adjustment measures when the playback stuttering risk prediction result indicates that there is a risk of playback stuttering.

[0073] In one possible implementation, the prediction module is configured to: If the rate of change corresponding to the target time period is greater than or equal to the first specified rate of change, and the rate of decrease corresponding to the target time period is greater than or equal to the first specified rate of decrease, then it is determined that the video has a risk of playback stuttering.

[0074] In another possible implementation, the prediction module 502 is used for: If the rate of change corresponding to the target time period is greater than or equal to the first specified rate of change and less than the second specified rate of change, and the rate of decrease corresponding to the target time period is greater than or equal to the first specified rate of decrease and less than the second specified rate of decrease, then it is determined that the video has a low risk of playback stuttering, where the second specified rate of change is greater than the first specified rate of change and the second specified rate of decrease is greater than the first specified rate of decrease. If the rate of change corresponding to the target time period is greater than or equal to the second specified rate of change, and the rate of decrease corresponding to the target time period is greater than or equal to the second specified rate of decrease, then it is determined that the video has a high risk of playback stuttering.

[0075] In another possible implementation, the prediction module 502 is used for: If the rate of change corresponding to the target time period is greater than or equal to the third specified rate of change, it is determined that the video has a high risk of playback stuttering, and the third specified rate of change is greater than the second specified rate of change. If the drop rate corresponding to the target time period is greater than or equal to the third specified drop rate, it is determined that the video has a high risk of playback stuttering, and the third specified drop rate is greater than the second specified drop rate.

[0076] In another possible implementation, the prediction module 502 is used for: The audio-visual time deviation, rate of change, rate of decrease, CPU usage, GPU usage, memory usage, and battery temperature of the device playing the video are input into the risk prediction model to obtain the stuttering risk probability value output by the risk prediction model. If the probability value of the stuttering risk is greater than or equal to a first specified probability value and less than a second specified probability value, then it is determined that the video has a low risk of playback stuttering, and the second specified probability value is greater than the first specified probability value. If the probability value of the stuttering risk is greater than or equal to the second specified probability value, then it is determined that the video has a high risk of playback stuttering.

[0077] In another possible implementation, the processing module 503 is used for: If there is a low risk of playback stuttering in the video, increase the preloading time of the video; If there is a high risk of playback stuttering in the video, reduce the bitrate of the video.

[0078] In another possible implementation, the prediction module 502 is used for: In response to the increasing trend of the rate of change, the duration of the target time period is reduced.

[0079] It should be noted that the video stuttering prediction device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video stuttering prediction device and the video stuttering prediction method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0080] Based on the same inventive concept, corresponding to the video stuttering prediction method provided in the embodiments of this application, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the video stuttering prediction method described in the above embodiments.

[0081] Figure 6 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0082] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0083] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0084] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0085] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0086] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0087] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0088] The electronic device described above is used to implement the corresponding video stuttering prediction method in the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0089] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to perform the video stuttering prediction method described above. This computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices, etc.

[0090] In an exemplary embodiment, a computer program product is also provided, including computer program instructions that, when executed on a computer, cause the computer to perform the video stuttering prediction method described in the above embodiments.

[0091] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0092] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0093] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0094] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for predicting video stuttering, characterized in that, include: During video playback, the rate of decrease of the audio buffer level of the video is continuously detected, and the audio-visual time deviation between the played video frames and audio frames is continuously detected to determine the rate of change of the audio-visual time deviation; the audio buffer level is the number of audio frames to be decoded or played stored in the audio buffer. When the audio-visual time deviation increases, the playback stuttering risk prediction result for the video is determined based on at least one of the rate of change and the rate of decrease corresponding to the target time period; When the playback stuttering risk prediction result indicates that there is a risk of playback stuttering, playback adjustment measures are implemented.

2. The method according to claim 1, characterized in that, The determination of playback stuttering risk prediction results for the video based on at least one of the rate of change and the rate of decrease corresponding to the target time period includes: If the rate of change corresponding to the target time period is greater than or equal to the first specified rate of change, and the rate of decrease corresponding to the target time period is greater than or equal to the first specified rate of decrease, then it is determined that the video has a risk of playback stuttering.

3. The method according to claim 1, characterized in that, The determination of playback stuttering risk prediction results for the video based on at least one of the rate of change and the rate of decrease corresponding to the target time period includes: If the rate of change corresponding to the target time period is greater than or equal to the first specified rate of change and less than the second specified rate of change, and the rate of decrease corresponding to the target time period is greater than or equal to the first specified rate of decrease and less than the second specified rate of decrease, then it is determined that the video has a low risk of playback stuttering, where the second specified rate of change is greater than the first specified rate of change and the second specified rate of decrease is greater than the first specified rate of decrease. If the rate of change corresponding to the target time period is greater than or equal to the second specified rate of change, and the rate of decrease corresponding to the target time period is greater than or equal to the second specified rate of decrease, then it is determined that the video has a high risk of playback stuttering.

4. The method according to claim 3, characterized in that, The determination of playback stuttering risk prediction results for the video based on at least one of the rate of change and the rate of decrease corresponding to the target time period includes: If the rate of change corresponding to the target time period is greater than or equal to the third specified rate of change, it is determined that the video has a high risk of playback stuttering, and the third specified rate of change is greater than the second specified rate of change. If the drop rate corresponding to the target time period is greater than or equal to the third specified drop rate, it is determined that the video has a high risk of playback stuttering, and the third specified drop rate is greater than the second specified drop rate.

5. The method according to claim 1, characterized in that, The determination of playback stuttering risk prediction results for the video based on at least one of the rate of change and the rate of decrease corresponding to the target time period includes: The audio-visual time deviation, rate of change, rate of decrease, CPU usage, GPU usage, memory usage, and battery temperature of the device playing the video are input into the risk prediction model to obtain the stuttering risk probability value output by the risk prediction model. If the probability value of the stuttering risk is greater than or equal to a first specified probability value and less than a second specified probability value, then it is determined that the video has a low risk of playback stuttering, and the second specified probability value is greater than the first specified probability value. If the probability value of the stuttering risk is greater than or equal to the second specified probability value, then it is determined that the video has a high risk of playback stuttering.

6. The method according to any one of claims 3 to 5, characterized in that, When the playback stuttering risk prediction result indicates the existence of a playback stuttering risk, playback adjustment measures are implemented, including: If there is a low risk of playback stuttering in the video, increase the preloading time of the video; If there is a high risk of playback stuttering in the video, reduce the bitrate of the video.

7. The method according to any one of claims 1 to 5, characterized in that, The method further includes: In response to the increasing trend of the rate of change, the duration of the target time period is reduced.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method described in any one of claims 1 to 7.

10. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed on a computer, the computer causes the computer to perform the method as described in any one of claims 1 to 7.

Images