A video decoding display method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a video decoding display method and device, electronic equipment and storage medium, wherein the method comprises: in response to detecting that the video decoding display is blocked due to network congestion, adjusting a target buffer parameter of a buffer queue according to the number of blocked video frames; and buffering video frames in the buffer queue for video decoding display according to the adjusted target buffer parameter. The present disclosure sets a suitable target buffer parameter according to the number of blocked video frames when network congestion is detected, so as to cope with possible subsequent network congestion, and realizes the improvement of the real-time performance of the picture on the premise of ensuring the smoothness of the picture.

Description

Technical Field

[0001] This disclosure relates to the field of security, and in particular to a video decoding and display method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of video surveillance technology and the widespread use of smartphones, viewing the live feeds of IPCs (IP Cameras) and NVRs (Network Video Recorders) on mobile apps has become a common application scenario.

[0003] Due to factors such as unstable network environments and jitter, video data transmission speeds can fluctuate during transmission. In severe cases, this can cause noticeable stuttering in live footage, negatively impacting user experience. Therefore, a video decoding and display method is urgently needed to resolve the issue of stuttering caused by network jitter and other factors. Summary of the Invention

[0004] This disclosure provides a video decoding and display method, apparatus, electronic device, and storage medium to solve the problem of live video stuttering caused by factors such as network jitter.

[0005] According to one aspect of this disclosure, a video decoding and display method is provided, the method comprising:

[0006] In response to the detection of video decoding and display stuttering caused by network congestion, the target cache parameters of the cache queue are adjusted according to the number of blocked video frames.

[0007] According to the adjusted target cache parameters, video frames are cached in the cache queue for video decoding and display.

[0008] According to another aspect of this disclosure, a video decoding and display apparatus is provided, the apparatus comprising:

[0009] The first adjustment module is used to adjust the target cache parameters of the cache queue according to the number of blocked video frames in response to the detection of video decoding and display stuttering caused by network congestion.

[0010] The decoding and display module is used to cache video frames in the cache queue according to the adjusted target cache parameters for video decoding and display.

[0011] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0012] At least one processor; and

[0013] A memory that is communicatively connected to at least one processor; wherein,

[0014] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the video decoding and display method of any embodiment of this disclosure.

[0015] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform a video decoding and display method according to any embodiment of this disclosure.

[0016] According to the technology disclosed herein, when network congestion is detected, appropriate target buffer parameters are set based on the number of blocked video frames to cope with subsequent possible network congestion, thereby improving the real-time performance of the video while ensuring smooth playback.

[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0018] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0019] Figure 1a This is an approximate diagram showing the variation of video data receiving speed in an embodiment of this disclosure;

[0020] Figure 1b This is a schematic flowchart of a video decoding and display method provided in an embodiment of this disclosure;

[0021] Figure 2 This is a flowchart illustrating yet another video decoding and display method provided in this embodiment of the present disclosure;

[0022] Figure 3 This is a schematic flowchart of another video decoding and display method provided in an embodiment of this disclosure;

[0023] Figure 4 This is a flowchart illustrating yet another video decoding and display method provided in this embodiment of the present disclosure;

[0024] Figure 5 This is a flowchart illustrating yet another video decoding and display method provided in this embodiment of the present disclosure;

[0025] Figure 6 This is a logical schematic diagram of adjusting target cache parameters provided in an embodiment of this disclosure;

[0026] Figure 7 This is a schematic diagram of the structure of a video decoding display device provided in an embodiment of this disclosure;

[0027] Figure 8 This is a block diagram of an electronic device used to implement the video decoding and display method of the embodiments of this disclosure. Detailed Implementation

[0028] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0029] The method described in this disclosure can be applied to a video surveillance system, which includes at least a video transmitting device (e.g., a network camera) and a video receiving device for video decoding and display. During video decoding and display, network jitter can cause congestion during the transmission of video data from the transmitting end to the receiving end, resulting in changes in the speed at which the receiving end receives the data.

[0030] Figure 1a An approximate graph showing the change in data reception speed is provided. During time intervals (A, B), the data reception speed at the receiving end is less than the data consumption speed. During this time, the amount of data buffered in the receiving device's buffer will gradually decrease. If the buffered data is exhausted before time point B, the live stream will stutter because there is no new data available for decoding and playback; this is known as video decoding and display stuttering. If the buffer has stored enough data before time point B, the live stream will not stutter. During time intervals (B, C), the data reception speed at the receiving end is greater than the data consumption speed, and the amount of data in the buffer will increase.

[0031] Setting a sufficiently large cache to buffer video frames ensures smooth playback with enough data. However, this not only excessively consumes system resources and reduces the system's decoding capabilities, but also introduces significant latency when network jitter is low. Therefore, this paper proposes a video decoding and display method to minimize playback latency while maintaining smoothness. The specific implementation is illustrated in the following example.

[0032] Figure 1b This is a flowchart illustrating a video decoding and display method according to an embodiment of the present disclosure. This embodiment is applicable to situations where the amount of data that will be blocked during the next network congestion is predicted based on the amount of data blocked during the previous network congestion. The method can be executed by a video decoding and display device, which is implemented in software and / or hardware and integrated into an electronic device.

[0033] For details, see Figure 1b The process of video decoding and display is as follows:

[0034] S101. In response to detecting video decoding and display stuttering caused by network congestion, adjust the target cache parameters of the cache queue according to the number of blocked video frames.

[0035] S102. According to the adjusted target cache parameters, cache video frames in the cache queue for video decoding and display.

[0036] In this embodiment of the disclosure, the target buffer parameter refers to the number of video frames to be buffered in the desired buffer queue while ensuring smooth decoding and display of the image and minimizing playback latency. The initial value of the target buffer parameter can be pre-determined based on a set minimum latency time and the current video frame rate.

[0037] It should be noted that video decoding stuttering (i.e., playback stuttering) is caused by the receiving device's insufficient cache of video frames according to the current target cache parameters to cope with the current network congestion. As a result, once the cache is exhausted, no new data is available for playback. Therefore, when video decoding stuttering is detected due to network congestion, the target cache parameters need to be adjusted, for example, by increasing them.

[0038] In this embodiment, when the network is stable and video decoding and display are in progress, the sending end sends data to the receiving end's buffer at a fixed rate based on the current video frame rate. At this time, the data reception speed equals the data consumption speed, and almost no data caching is required. However, when network congestion occurs, video data transmission will be blocked. Video data that the sending end cannot send in time is cached until the congestion ends, at which point the sending end sends the previously cached data to the receiving end's buffer as quickly as possible. Therefore, after the network congestion ends, the peak number of cached video frames in the receiving end's buffer queue during the re-caching process is close to the number of video frames blocked during the network congestion. Thus, the target cache parameter of the buffer queue can be adjusted upwards based on the peak number of re-cached video frames in the buffer queue; for example, the peak number of cached video frames can be directly used as the latest value of the target cache parameter.

[0039] Furthermore, after adjusting the target cache parameters, video frames are cached in the cache queue according to the adjusted parameters for video decoding and display. This ensures that video frame data is cached according to the adjusted target cache parameters before the next network congestion occurs. Therefore, when network congestion occurs again, as long as the congestion level does not exceed that of the previous network congestion, the number of video frames cached according to the adjusted target cache parameters is sufficient to handle the current network congestion, preventing cache exhaustion and thus avoiding video decoding and display stuttering.

[0040] In this embodiment of the disclosure, when network congestion is detected, appropriate target cache parameters are set according to the number of blocked video frames to cope with possible subsequent network congestion, thereby improving the real-time performance of the video while ensuring smoothness.

[0041] Figure 2 This is a flowchart illustrating another video decoding and display method according to an embodiment of the present disclosure. This embodiment is an optimization based on the above embodiment. See also... Figure 2 The specific process of video decoding and display is as follows:

[0042] S201. In response to detecting that the video frames in the buffer queue have been exhausted, periodically check whether the buffer queue should be re-buffered for video frames.

[0043] In this embodiment, network jitter manifests as fluctuations in data transmission speed, causing a discrepancy between the actual data transmission speed at the sending end and the data consumption speed at the receiving end. In the receiving end's buffer, this manifests as an increase or decrease in the number of buffered video frames. Video decoding and display stuttering (i.e., playback stuttering) is caused by insufficient buffer duration to cope with current network congestion, resulting in no new data available for playback after the buffer is exhausted. Therefore, to detect video decoding and display stuttering, it is only necessary to check whether the video frames in the buffer queue have been exhausted.

[0044] Once it is determined that the video frames in the buffer queue have been exhausted, it is necessary to promptly detect whether the network congestion has ended. In one alternative implementation, the network congestion can be determined by periodically checking (e.g., according to the video frame rate) whether the buffer queue has been re-buried for video frames.

[0045] S202. If the detection result is yes, adjust the target cache parameters according to the number of video frames re-cached in the cache queue.

[0046] In this embodiment of the disclosure, if a video frame is detected to be re-cached in the cache queue, it indicates that the network congestion has ended. At this time, the sending end will send the previously cached data (i.e. the blocked video frames) to the receiving end's cache queue as quickly as possible. Therefore, the target cache parameter can be adjusted (i.e. increased) according to the number of video frames re-cached in the cache queue.

[0047] In one alternative implementation, the number of video frames cached in the cache queue is periodically read (e.g., read according to the video frame rate), and it is determined whether the number of video frames is greater than the target cache parameter; if so, the target cache parameter is adjusted upward according to the number of video frames, for example, the number of video frames that is greater than the target cache parameter value is used as the new value of the target cache parameter.

[0048] S203. According to the adjusted target cache parameters, cache video frames in the cache queue for video decoding and display.

[0049] In this embodiment of the disclosure, by increasing the value of the target cache parameter, the receiving end can pre-cache a sufficient number of video frames to cope with possible network congestion. That is, when the number of video frames in the cache queue reaches the value of the target cache parameter, as long as the degree of network congestion does not exceed that of the previous one, the currently cached number of video frames is sufficient to cope with various network congestion situations, and there will be no situation where the cache is exhausted.

[0050] In addition, the solution of this disclosure realizes the offsetting of individual video frame jitter by using real-time cached frame count. As long as the video frames in the cache queue are not exhausted, there is no need to increase the target cache parameters, making the network adaptation and adjustment more timely and smooth, the picture playback smoother, and the system performance consumption elastic and variable, and the release of system resources more timely.

[0051] It should be noted that if a more severe network congestion occurs than before, the cache queue will inevitably run out of resources. In this case, it is necessary to return to and continue executing steps S201-S202 in order to readjust the target cache parameters.

[0052] In this embodiment of the disclosure, during the process of re-caching video frames in the cache queue after the network congestion ends, the value of the target cache parameter is increased once every time the number of video frames in the cache queue is detected to be greater than the target cache parameter, so as to configure an appropriate target cache parameter to cope with possible subsequent network congestion, thereby improving the real-time performance of the picture while ensuring the smoothness of the picture.

[0053] Figure 3 This is a flowchart illustrating another video decoding and display method according to an embodiment of the present disclosure. This embodiment is an optimization based on the above embodiment. See also... Figure 3 The specific process of video decoding and display is as follows:

[0054] S301. Determine the value range of the target buffer parameter based on the delay parameter and the current video frame rate.

[0055] In this embodiment of the disclosure, pre-set delay parameters (including minimum delay time and maximum delay time) can be obtained from the configuration file, and then the value range of the target buffer parameters can be determined in combination with the current video frame rate.

[0056] For example, the minimum value of the target cache parameter Among them, D min R represents the minimum latency, and R represents the video frame rate; the maximum value of the target buffer parameter is... Among them, D max This represents the maximum latency. That is, the target cache parameter's value range is [C]. min C max The minimum value of the target cache parameter constrains setting it too low, reducing the probability of playback stuttering. The maximum value of the target cache parameter constrains setting it too high, preventing excessive latency.

[0057] It should be noted that after determining the range of values ​​for the target buffer parameter, the minimum value can be used as the initial value of the target buffer parameter. Subsequently, the value of the target buffer parameter can be adjusted based on the amount of video frames blocked during video decoding and display stuttering. The adjustment process can be found in steps S302-S303.

[0058] S302. In response to detecting that the video frames in the buffer queue have been exhausted, periodically check whether the buffer queue should be re-buffered for video frames.

[0059] S303. If the detection result is yes, then periodically read the number of video frames cached in the cache queue, and execute S304 or S305 when the number of video frames is greater than the value of the target cache parameter.

[0060] In this embodiment of the disclosure, when adjusting the target cache parameter based on the number of video frames, it can first be determined whether the number of video frames exceeds the value range of the target cache parameter. If yes, then S305 is executed; otherwise, then S304 is executed.

[0061] S304. If the number of video frames does not exceed the preset range of the target cache parameters, adjust the target cache parameters according to the number of video frames.

[0062] Optionally, if the number of video frames does not exceed the preset value range of the target cache parameter, when adjusting the target cache parameter, the number of video frames can be directly used as the new value of the target cache parameter.

[0063] S305. If the number of video frames exceeds the preset range of the buffer parameters, the target buffer parameters are adjusted according to the boundary value of the overflow direction.

[0064] Optionally, if the number of video frames exceeds the maximum value of the preset range of the target buffer parameter, when adjusting the target buffer parameter, the boundary value (i.e. the maximum value) in the overflow direction (i.e. the direction of the maximum value) can be directly used as the new value of the target buffer parameter.

[0065] In this embodiment of the disclosure, a value range of the target cache parameter is preset, and the value of the target cache parameter is limited by the value range to avoid over-adjustment of the target cache parameter.

[0066] Figure 4 This is a flowchart illustrating another video decoding and display method according to an embodiment of the present disclosure. This embodiment is an optimization based on the above embodiment. See also... Figure 4 The specific process of video decoding and display is as follows:

[0067] S401. In response to detecting video decoding and display stuttering caused by network congestion, adjust the target cache parameters of the cache queue according to the number of blocked video frames.

[0068] The target cache parameter refers to the number of video frames to be cached in the desired cache queue while ensuring smooth decoding and display of the screen and minimizing playback latency.

[0069] S402. According to the adjusted target cache parameters, cache video frames in the cache queue for video decoding and display.

[0070] S403. Periodically perform the following operation until video decoding display stuttering is detected: In response to the detection that no video decoding display stuttering has occurred within a preset time period, calculate the minimum number of cached frames in the cache queue within the preset time period, and adjust the target cache parameters according to the minimum value.

[0071] If no cache exhaustion occurs for a period of time, the current target cache parameter value is considered either appropriate or too high, and it can be adjusted downwards. To reduce latency to an appropriate level as quickly as possible, the amount of data blocked during the next network congestion event is predicted based on the amount of data blocked during the previous congestion event. When network congestion occurs, if the cache queue is not empty, the lowest value (i.e., the minimum value) of the remaining video frames in the cache queue during the congestion process is the redundancy of the current target cache parameter. Under the current level of network congestion, even subtracting the redundancy from the target cache parameter value will not result in a cache exhaustion.

[0072] Furthermore, by periodically lowering the value of the target cache parameter, the most suitable value of the target cache parameter can be found quickly. That is, under the most suitable target cache parameter, network congestion can be dealt with and latency can be reduced.

[0073] It should be noted that when periodically lowering the value of the target cache parameter through S403, it is necessary to ensure that the value of the target cache parameter after each adjustment is not less than the preset minimum value of the target cache parameter.

[0074] In this embodiment, the adjustment of the target cache parameters is more precise and timely, and the target cache parameters can be adjusted to a suitable size in a short time, resulting in strong adaptability to network jitter.

[0075] Figure 5 This is a flowchart illustrating another video decoding and display method according to an embodiment of the present disclosure. This embodiment is an optimization based on the above embodiment. See also... Figure 5 The specific process of video decoding and display is as follows:

[0076] S501. Determine the value range of the target buffer parameter based on the delay parameter and the current video frame rate.

[0077] S502. In response to detecting that the video frames in the buffer queue have been exhausted, periodically check whether the buffer queue should be re-buffered for video frames.

[0078] S503. If the detection result is yes, then periodically read the number of video frames cached in the cache queue, and execute S304 or S305 when the number of video frames is greater than the value of the target cache parameter.

[0079] In this embodiment of the disclosure, when adjusting the target cache parameter based on the number of video frames, it can first be determined whether the number of video frames exceeds the value range of the target cache parameter. If yes, S505 is executed; otherwise, S504 is executed.

[0080] S504. If the number of video frames does not exceed the preset range of the target cache parameter, adjust the target cache parameter according to the number of video frames.

[0081] Optionally, if the number of video frames does not exceed the preset value range of the target cache parameter, when adjusting the target cache parameter, the number of video frames can be directly used as the new value of the target cache parameter.

[0082] S505. If the number of video frames exceeds the preset range of the buffer parameters, the target buffer parameters are adjusted according to the boundary value of the overflow direction.

[0083] S506. Periodically perform the following operation until video decoding display stuttering is detected: In response to the detection that no video decoding display stuttering has occurred within a preset time period, calculate the minimum number of cached frames in the cache queue within the preset time period, and adjust the target cache parameters according to the minimum value.

[0084] S507. Update the latency parameters based on the historical prediction data of the target cache parameters and the historical cache data of the cache queue.

[0085] Historical prediction data refers to the number of video frames that can be buffered to overcome network congestion, predicted based on the number of congested video frames each time congestion occurs. The target buffering parameters can be adjusted based on this data. The historical buffering data can optionally be the peak number of video frames actually buffered in the buffer queue, as counted at the end of each network congestion event. Delay parameters include minimum and maximum delay times. In an optional implementation, the delay parameters can be updated in two steps, S5071-S5072:

[0086] S5071. Determine the minimum latency based on the latency time corresponding to the target cache parameter after each reduction, combined with the latency time distribution corresponding to historical cache data.

[0087] Based on the relationship between the target cache parameter values, latency, and video frame rate, the latency corresponding to each adjusted target cache parameter is determined. After rounding the latency to the nearest hundred milliseconds, the most frequently occurring latency (or the maximum if multiple values ​​exist) is selected as the new minimum latency. With relatively stable network jitter, the probability of subsequent frame drops will be reduced. To further increase the reliability of the latency parameters, the latency distribution corresponding to historical cache data can be used to determine the final minimum latency. This historical cache data latency distribution can optionally be based on the latency corresponding to the peak frame count of the video frames cached in the cache queue during network congestion.

[0088] S5072. Determine the maximum latency based on the latency time corresponding to the target cache parameter after each adjustment, combined with the latency time distribution corresponding to historical cache data.

[0089] In this embodiment, the significant acceleration of the video display is due to an excessive number of cached frames, exceeding the maximum latency. To reduce this, the latency corresponding to each adjustment of the target cache parameter is determined based on the relationship between the target cache parameter value, latency time, and video frame rate. The latency time is then rounded down to the nearest hundred milliseconds, and the most frequently occurring latency time (or the minimum value if multiple values ​​are present) is taken as the new maximum latency time. With relatively stable network jitter, the probability of subsequent frame drops will be reduced. To further increase the reliability of the latency parameter, the latency time distribution corresponding to historical cache data can be incorporated into the calculation to determine the final maximum latency time.

[0090] S508. Adjust the value range of the target cache parameter according to the updated delay parameter.

[0091] In this embodiment of the disclosure, after updating the maximum delay time and the minimum delay time, the latest value range of the target cache parameter can be determined by combining the video frame rate. The specific calculation process can be found in the above embodiment, and will not be repeated here.

[0092] In this embodiment of the disclosure, the most suitable value range of the target cache parameter can be automatically updated to avoid over-adjustment during the process of adjusting the target cache parameter upwards or downwards.

[0093] Figure 6 This is a logical schematic diagram of another video decoding and display method according to an embodiment of the present disclosure. This embodiment is an optimization based on the above embodiment. It adjusts the target buffer parameters during video decoding and display by setting a state machine. The state machine has five states: initial state S0, waiting for data state S1, delay increase state S2, delay decrease state S3, and exit state S4, which is the current state of the state machine itself. The values ​​of the target buffer parameters in the buffer queue are adjusted. See also... Figure 6 The logic of the video decoding and display method is as follows:

[0094] First, the state machine is initialized in the initial state S0 by reading the minimum delay D of the current channel stored in the configuration file. min and maximum delay D max (Unit: milliseconds), and calculate the corresponding target buffer parameter F based on the video frame rate R. target Initial value, minimum number of buffered frames C min Maximum number of buffered frames C max And set the parameters that need to be used during operation (such as resolution).

[0095] The following are the initial values ​​for the target cache parameters, the minimum number of cached frames, and the calculation formulas for the maximum number of cached frames:

[0096]

[0097]

[0098] In practical applications, the minimum number of cached frames is usually used as the initial value of the target cache parameter.

[0099] When the state machine detects that the number of video frames in the receiver's buffer queue is 0, it enters the waiting data state S1. It also enters the waiting data state after the initial state S0 ends. In the waiting data state S1, the state machine checks whether there are video frames in the receiver's buffer queue according to the current video frame rate. After detecting that a video frame has been received in the receiver's buffer queue, the state machine ends the waiting data state and enters the delay-up state S2.

[0100] After the state machine enters the delay-increase state S2, it will increase the delay, that is, increase the value of the target buffer parameter, so that the receiving end can buffer a sufficient number of video frames F in advance. now To cope with potential network congestion. Entering the delay-up state indicates that the receiver's buffer queue has run out of space, or that the state machine has just been initialized. In either case, in the delay-up state S2, the state machine will read the number of video frames buffered in the receiver's buffer queue according to the current frame rate R, and record the peak value F of the buffered frame count. top and set the target cache parameter F target The value is updated to the peak value of the cached frames. The target cache parameter ranges from [C] to [the peak value of the cached frames]. min C max When the value exceeds the range, it will be set to the boundary value in the overflow direction.

[0101] The peak value that the current buffered frames in the receiver queue can reach is roughly the amount of data blocked during the last network congestion. That is, in the delay-increase state S1, the target buffer parameter is continuously updated to the current peak value of the buffered frames, and the video buffered frames are increased according to this target buffer parameter. When the buffer reaches the target value, as long as the network congestion does not exceed the severity of the previous one, the currently buffered video frames are sufficient to handle the current network congestion, and the buffer will not run out. If a more severe network congestion than the previous one occurs, the state machine will re-enter the waiting-for-data state S1, then enter the delay-increase state S2, and update the target buffer parameter value again.

[0102] After running in the delay-increase state S2 for a period of time (e.g., 30 seconds, which can be modified according to actual usage), the state machine will switch to the delay-decrease state S3 (if the cache is exhausted, it will enter the waiting-for-data state). Based on the previous principle, if no cache exhaustion occurs for a period of time (e.g., 60 seconds), it indicates that the currently set target cache parameter is appropriate or too large. During this period, the actual number of cached frames reaches a low point C due to network congestion. bottom This refers to the redundancy of the currently set target cache parameters. After subtracting the redundancy from the target cache parameter value, as long as the network congestion level does not exceed that of the previous congestion event, the number of currently cached video frames F will be [value missing]. now This is sufficient to handle the current network congestion and will not result in cache exhaustion. If a more severe network congestion occurs than before, the state machine will re-enter the waiting data state S1, then enter the delay-up adjustment state S2, and update the target cache parameter F again. targetIn the delay-down state S3, the valley value of the actual number of buffered video frames is recorded as the reduction amount of the target buffer parameter after the state switch. After the state switch, the state machine subtracts the reduction amount from the target buffer parameter before the state switch to obtain the target buffer parameter in the current state. However, the target buffer parameter cannot be set lower than the minimum number of buffered frames C. min .

[0103] After entering the delay down-adjustment state, the state machine will continue to run in this state, adjusting the target buffer parameters every certain period of time. To ensure that the target buffer parameters can be reset upon re-entering the delay up-adjustment state S2 or the delay down-adjustment state S3, the peak value F of the actual buffered frame count must also be updated during the delay down-adjustment state. top Valley value F bottom .

[0104] Furthermore, before the state machine exits operation, it will enter the exit state S4. At this time, the state machine will store the data recorded during operation as the initial parameters for the next startup. The initial values ​​are set once initially, and thereafter the state machine will automatically adjust the startup parameters based on the data analysis during operation.

[0105] In practice, the stuttering occurs because of the number of cached frames F. now To reduce the occurrence of this situation, the target cache parameter F is statistically analyzed each time the state machine switches from the delayed downscaling state S3 to the waiting data state S1. target The corresponding delay time, after being rounded to the nearest hundred milliseconds, will be used as the minimum delay D for the next startup, with the most frequent value (or the larger value if there are multiple highest values). min When network jitter is relatively stable, the probability of the next frame stuttering will be reduced. To further increase the reliability of this startup parameter, the distribution of delay values ​​during each run is recorded, and the previously stored delay distribution is added to the statistics each time the startup parameter is calculated in the exit state S4.

[0106] The significant speedup is due to excessive frame buffering, with the buffer duration exceeding the maximum latency D. max To reduce the occurrence of this situation, we statistically analyze the target cache parameter C each time the state machine switches from the delayed up-adjustment state S2 to the delayed down-adjustment state S3. target The corresponding delay time, after being rounded to the nearest hundred milliseconds, will be used as the maximum delay D for the next startup, with the most frequent value (or the minimum value if multiple values ​​are found) being the most frequent one. maxWhen the network is relatively stable, the probability of a significant acceleration in the next frame will decrease. To further increase the reliability of this startup parameter, we record the distribution of delay values ​​during each run, and add the previously stored delay distribution to the calculation each time the startup parameter is calculated in exit state S4.

[0107] In this embodiment, the target cache parameters can be precisely and timely adjusted through the state machine, and can be adjusted to a suitable size in a short time, with strong adaptability to network jitter; at the same time, the range of values ​​of the target cache parameters is further constrained, reducing the probability of playback stuttering and preventing excessive latency.

[0108] Furthermore, this disclosure also provides another optional video decoding and display method, which controls playback speed to cope with network jitter. The specific method is as follows:

[0109] When the actual number of cached frames is lower than the target cache parameter value, playback is slowed down to increase the number of cached frames; when the actual number of frames is higher than the target cache parameter value, playback is accelerated to decrease the number of cached frames, thus controlling the cache and adjusting the latency.

[0110] To facilitate understanding, several scenarios are briefly explained here:

[0111] When the number of buffered video frames at the receiving end is lower than the target buffer parameter, set the slow motion speed (reference range: 85%~90%) to make the buffer grow to the value corresponding to the target buffer parameter as soon as possible, without the audience feeling a significant slowdown in the picture.

[0112] When the number of buffered video frames at the receiving end is higher than the target buffer parameter but does not exceed the maximum buffer frame count, set the playback speed lower limit S. min (Reference range: 105%~110%), so that the cache is reduced to the value corresponding to the target cache parameter as soon as possible, and there is no noticeable screen acceleration.

[0113] When the number of buffered video frames at the receiving end exceeds twice the maximum number of buffered frames, set the playback speed limit S. max (Reference: 140%~150%), quickly reduce the number of buffered frames to prevent exceeding the upper limit of the receiver's buffer queue, without affecting the observation of image details.

[0114] When the number of buffered video frames at the receiving end is higher than the maximum number of buffered frames but does not exceed twice the maximum number of buffered frames, the current acceleration speed S is set to (F... now -F max ) / F max ×(S max -S minWhile reducing the number of cached frames as quickly as possible, the acceleration level should also be gradually reduced to ensure that the acceleration of the image changes smoothly.

[0115] Figure 7 This is a schematic diagram of a video decoding and display device according to an embodiment of the present disclosure. This embodiment is applicable to situations where the amount of data that will be blocked during the next network congestion event is predicted based on the amount of data blocked during the previous network congestion event. This device is configured in a video receiving device, such as... Figure 7 As shown, the device specifically includes:

[0116] The first adjustment module 701 is used to adjust the target cache parameters of the cache queue according to the number of blocked video frames in response to the detection of video decoding and display stuttering caused by network congestion.

[0117] The decoding and display module 702 is used to cache video frames in the cache queue according to the adjusted target cache parameters for video decoding and display.

[0118] Based on the above embodiments, optionally, the first adjustment module includes:

[0119] The detection unit is used to periodically detect whether the buffer queue should re-buffer video frames in response to the detection that the video frames in the buffer queue have been exhausted.

[0120] The adjustment unit is used to adjust the target cache parameters based on the number of video frames re-cached in the cache queue if the detection result is yes.

[0121] Based on the above embodiments, optionally, the adjustment unit includes:

[0122] The read and judge subunit is used to periodically read the number of video frames cached in the cache queue and judge whether the number of video frames is greater than the target cache parameter;

[0123] The adjustment subunit is used to adjust the target buffer parameters based on the number of video frames if the determination is yes.

[0124] Based on the above embodiments, optionally, the adjustment subunit is specifically used for:

[0125] If the number of video frames does not exceed the preset range of the target buffer parameters, the target buffer parameters are adjusted according to the number of video frames; where the preset range is determined based on the delay parameters and the current video frame rate; or

[0126] If the number of video frames exceeds the preset range of the buffer parameters, the target buffer parameters will be adjusted according to the boundary value in the overflow direction.

[0127] In addition to the above embodiments, optionally, the following further includes:

[0128] The second adjustment module is used to periodically perform the following operations until video decoding display stuttering is detected:

[0129] In response to the detection that no video decoding display stuttering occurs within a preset time period, the minimum number of cached frames in the cache queue within the preset time period is counted, and the target cache parameters are adjusted according to the minimum value.

[0130] Optionally, based on the above embodiments, the device further includes:

[0131] The delay parameter update module is used to update the delay parameter based on the historical prediction data of the target cache parameter and the historical cache data of the cache queue.

[0132] The range adjustment module is used to adjust the range of values ​​for the target cache parameters based on the updated delay parameters.

[0133] Based on the above embodiments, optionally, the delay parameter update module includes:

[0134] The first update unit is used to determine the minimum delay time based on the delay time corresponding to the target cache parameter after each reduction, combined with the delay time distribution corresponding to the historical cache data.

[0135] The second update unit is used to determine the maximum latency based on the latency corresponding to the target cache parameter after each adjustment, combined with the latency distribution corresponding to the historical cache data.

[0136] The apparatus provided in this disclosure can execute the video decoding and display method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the video decoding and display method. Content not described in detail in this embodiment can be referred to the description in any method embodiment of this disclosure.

[0137] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. In this embodiment, the electronic device is exemplarily a server device. Figure 8 The structure shown in this embodiment of the electronic device includes: one or more processors 802 and a memory 801; the processor 802 in the electronic device may be one or more. Figure 8 Taking a processor 802 as an example; memory 801 is used to store one or more programs; one or more programs are executed by one or more processors 802, causing one or more processors 802 to implement the video decoding and display method as described in any of the embodiments of this disclosure.

[0138] The electronic device may also include an input device 803 and an output device 804.

[0139] The processor 802, memory 801, input device 803, and output device 804 in this electronic device can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0140] The memory 801 in this electronic device serves as a computer-readable storage medium, which can be used to store one or more programs, including software programs, computer-executable programs, and modules. The processor 802 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 801, thereby implementing the video decoding and display method described in the above method embodiment.

[0141] Memory 801 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, memory 801 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, memory 801 may further include memory remotely located relative to processor 802, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0142] Input device 803 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the electronic device. Output device 804 may include display devices such as a display screen.

[0143] Furthermore, when one or more programs included in the aforementioned electronic device are executed by one or more processors 802, the programs perform the following operations:

[0144] In response to the detection of video decoding and display stuttering caused by network congestion, the target cache parameters of the cache queue are adjusted according to the number of blocked video frames.

[0145] According to the adjusted target cache parameters, video frames are cached in the cache queue for video decoding and display.

[0146] Of course, those skilled in the art will understand that when one or more programs included in the above-described electronic device are executed by one or more processors, the programs can also perform related operations in the video decoding and display method provided in any embodiment of this disclosure.

[0147] One embodiment of this disclosure provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to perform a video decoding and display method, the method comprising:

[0148] In response to the detection of video decoding and display stuttering caused by network congestion, the target cache parameters of the cache queue are adjusted according to the number of blocked video frames.

[0149] According to the adjusted target cache parameters, video frames are cached in the cache queue for video decoding and display.

[0150] The computer storage medium of this disclosure can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable CD-ROM, optical storage device, magnetic storage device, or any suitable combination thereof. The computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0151] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device.

[0152] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, radio frequency (RF), etc., or any suitable combination thereof.

[0153] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (e.g., including local area networks (LANs) or wide area networks (WANs)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0154] Note that the above description is merely a preferred embodiment and the technical principles employed in this disclosure. Those skilled in the art will understand that this disclosure is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this disclosure. Therefore, although this disclosure has been described in detail through the above embodiments, it is not limited to the above embodiments. Many other equivalent embodiments may be included without departing from the concept of this disclosure, and the scope of this disclosure is determined by the scope of the appended claims.

Claims

1. A video decoding and display method, characterized in that, Applied to a video receiving device, the method includes: In response to the detection of video decoding and display stuttering caused by network congestion, the target cache parameter of the cache queue is adjusted according to the number of blocked video frames; the target cache parameter is the number of video frames to be cached in the cache queue; wherein, the number of blocked video frames is the number of video frames cached by the video sender when network congestion occurs; According to the adjusted target cache parameters, video frames are cached in the cache queue for video decoding and display. Among these measures, in response to detecting video decoding and display stuttering caused by network congestion, the target cache parameters of the cache queue are adjusted according to the number of blocked video frames, including: In response to detecting that the video frames in the cache queue have been exhausted, the system periodically checks whether the cache queue should be re-cached for video frames. If so, the network congestion is determined to have ended, and the target cache parameter is adjusted according to the number of video frames re-cached in the cache queue; the number of video frames re-cached is the peak number of video frames that the video receiver's cache queue can cache during the re-caching process after the network congestion ends; and the number of video frames re-cached is close to the number of video frames blocked when the network congestion occurred.

2. The method according to claim 1, characterized in that, Adjusting the target cache parameters based on the number of video frames re-cached in the cache queue includes: Periodically read the number of video frames cached in the cache queue and determine whether the number of video frames is greater than the target cache parameter; If so, the target cache parameters are adjusted according to the number of video frames.

3. The method according to claim 2, characterized in that, Adjusting the target buffer parameters according to the video frame number includes: If the number of video frames does not exceed the preset value range of the target buffer parameter, then the target buffer parameter is adjusted according to the number of video frames; wherein, the preset value range is determined based on the delay parameter and the current video frame rate; or If the number of video frames exceeds the preset value range of the cache parameter, the target cache parameter is adjusted according to the boundary value of the overflow direction.

4. The method according to claim 1, characterized in that, After caching video frames in the cache queue according to the target cache parameters for video decoding and display, the process further includes: Perform the following operation periodically until video decoding stuttering is detected: In response to the detection that no video decoding display stuttering occurs within a preset time period, the minimum number of cached frames in the cache queue within the preset time period is calculated, and the target cache parameters are adjusted according to the minimum value.

5. The method according to claim 3, characterized in that, The method further includes: The delay parameter is updated based on the historical prediction data of the target cache parameter and the historical cache data of the cache queue. Adjust the value range of the target cache parameter according to the updated delay parameter.

6. The method according to claim 5, characterized in that, Based on historical prediction data of the target cache parameters and combined with historical cache data of the cache queue, the latency parameters are updated, including: Based on the latency time corresponding to the target cache parameter after each reduction, and combined with the latency time distribution corresponding to the historical cache data, the minimum latency time is determined. Based on the latency time corresponding to each adjusted target cache parameter, and combined with the latency time distribution corresponding to the historical cache data, the maximum latency time is determined.

7. A video decoding and display device, characterized in that, Configured in a video receiving device, the device includes: The first adjustment module is used to adjust the target cache parameter of the cache queue according to the number of blocked video frames in response to the detection of video decoding and display stuttering caused by network congestion; the target cache parameter is the number of video frames to be cached in the cache queue; wherein, the number of blocked video frames is the number of video frames cached by the video sending end when network congestion occurs. The decoding and display module is used to cache video frames in the cache queue according to the adjusted target cache parameters for video decoding and display. The first adjustment module includes: a detection unit, configured to periodically detect whether the cache queue is re-caching video frames in response to the detection that the video frames in the cache queue are exhausted; and an adjustment unit, configured to determine that the network congestion has ended if the detection result is yes, and adjust the target cache parameter according to the number of re-cached video frames in the cache queue; wherein the number of re-cached video frames is the peak number of video frames that the cache queue of the video receiver can cache during the re-caching process after the network congestion ends; and the number of re-cached video frames is close to the number of video frames blocked when the network congestion occurred.

8. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

Images