Video playback device, video playback method, and computer-readable storage medium

Abstract

The processor of the video playback device performs the following steps: identifying a time limit for acquiring each chunk to play the video without interruption; identifying, for each chunk, the highest quality level at which the chunk can be downloaded by the time limit based on the future throughput; identifying a first path corresponding to the identified highest quality level; identifying a second path by changing the highest quality level included in the first path; estimating a quality of experience value for each of the first and second paths based on the future throughput; identifying the path corresponding to the highest quality of experience value among the estimated quality of experience values ​​as the quality adaptation schedule; and determining, based on the quality adaptation schedule, the quality value of the chunk to be requested in the next future unit time interval of the current unit time interval.

Description

[Technical Field]

[0001] This invention relates to a video playback device, a video playback method, and a program. [Background technology]

[0002] Adaptive bitrate (ABR) video streaming is a widely used application on the internet. ABR video streaming allows users to view video under changing network conditions, such as fluctuations in throughput. In ABR streaming processing, different representations with different throughput conditions are provided to the video playback device. Each of these different representations results in a different quality due to the use of different encoding conditions, such as resolution, frame rate, and bitrate. The entire video is broken down into multiple video segments called chunks, and the video quality can be adjusted at the chunk level. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Kevin Spiteri, Rahul Urgaonkar, and Ramesh Sitaraman. 2016. BOLA: Near- Optimal Bitrate Adaptation for Online Videos. In Proceedings of the 35th Annual IEEE International Conference on Computer Communications. 1-9. [Non-Patent Document 2] X. Yin, A. Jindal,V.Sekar, and B. Sinopoli, “A control-theoretic approach for dynamic adaptive video streaming over HTTP,” in Proceedings of the 2015 Conference of the ACM SIGCOMM, 2015, pp. 325-338.

Outdoor Tools3

Outdoor Tools 4

Direct Environment 5

Outdoor Configuration 6

[0004] Delivering video to a large audience is challenging. Each user accesses the internet with varying levels of available throughput, and this throughput can change over time. Therefore, to deliver video to all users, it is necessary to use different representations (video encoded with various resolutions, bitrates, and frame rates) to adapt the video's bitrate requirements to the user's available throughput. This method of adapting quality to available throughput is called adaptive bitrate video streaming, and the bitrate values ​​corresponding to each possible representation are called a bitrate ladder.

[0005] The process of adapting quality to available throughput is handled by the adaptive bitrate control (ABR) mechanism of the video playback device. The ABR control mechanism relies on handcrafted or learning-based rules for chunk selection. Much research has been done on these rules regarding chunk packet selection, but a limitation of these earlier studies is the time scale considered in chunk selection. In fact, previous studies have focused on the selection of the immediate chunk in relation to the video playback device's problem of "which chunk should be requested next." However, such a narrow perspective may not lead to optimal QoE because it fails to predict network access degradation and can only provide a response to observed degradation events.

[0006] As part of the operation of a video playback device, it is necessary to compare the quality of the results after executing different quality adaptation patterns at each step and select a sequence of chunks that provides the user with the best perceived quality. This sequence of chunks will be referred to as the quality adaptation path below.

[0007] An optimization procedure is needed to identify quality adaptation paths that achieve high QoE while maintaining reasonable computational costs. [Means for solving the problem]

[0008] According to one aspect of the present invention, a video playback device comprising a transmitter; a receiver; a processor; and a memory, wherein the memory provides the processor with the following steps: 1) to predict future throughput during a first future time interval including m future unit time intervals using past throughput values; 2) to set up a search space for finding a quality adaptation schedule, which is defined by k quality levels and a second time interval including the current unit time interval, the m future unit time intervals, and n past unit time intervals; 3) to determine a time limit for acquiring each of the m chunks corresponding to the m future unit time intervals in order to play the video without interruption during the m future unit time intervals; and 4) to acquire each of the m chunks by the time limit based on the future throughput. Steps include: identifying the highest quality level among one or more quality levels for which the chunk can be downloaded; identifying a first path in the search space corresponding to the identified m highest quality levels; identifying one or more second paths by changing one or more of the m highest quality levels included in the first path; estimating perceived quality values ​​for each of the first path and the one or more second paths based on the future throughput; identifying the path corresponding to the highest perceived quality value among the estimated perceived quality values ​​as the quality adaptation schedule; determining the quality value of the chunk to be requested for the next future unit time interval after the current unit time interval based on the quality adaptation schedule; and requesting the chunk with the determined quality value via the transmission unit. ; The step of receiving the determined quality value chunk via the receiving unit. ; and a step in which, depending on the change in throughput, a preset value different from the preset value before the change in throughput is used as the value of the future first time interval. A video playback device is provided, which includes instructions to perform the following actions. [Effects of the Invention]

[0009] According to an embodiment, an optimization procedure is provided for identifying a quality adaptation path that achieves high QoE while maintaining reasonable computational costs.

Brief Description of the Drawings

[0010] [Figure 1] It is a diagram showing an example of the main steps of the chunk scheduling algorithm in an embodiment. [Figure 2] It is a diagram showing an example of comparison of QoE values for different quality adaptations in an embodiment. [Figure 3] It is a diagram showing an example of the hardware configuration of the video playback device 10 in an embodiment. [Figure 4] It is a diagram showing an example of the functional configuration of the video playback device 10 in an embodiment. [Figure 5] It is a flowchart for explaining an example of the processing procedure executed by the video playback device 10. [Figure 6A] It is a diagram showing an example focusing on the selection of the immediately following chunk. [Figure 6B] It is a diagram showing an example of performing chunk selection using long-term throughput prediction. [Figure 7] It is a diagram showing an example of evaluating the perceived quality for each path. [Figure 8] It is a diagram showing an example of limiting the quality difference between adjacent chunks to level A or less. [Figure 9] It is a diagram showing an example of comparing the execution time of the simulation by changing the length (depth) of the path. [Figure 10] It is a diagram showing an example of the iterative processing of the long-term chunk selection algorithm. [Figure 11] It is a diagram showing an example of evaluating the quality of each quality adaptation path based on the time limit for acquiring each chunk. [Figure 12] It is a diagram showing an example of searching for paths around the path of the highest quality chunk that can be downloaded.

Embodiments for Carrying Out the Invention

[0011] The following example describes a fast method for determining which chunk quality level columns should be requested to provide high quality of service to users watching video using adaptive bitrate video streaming. Compared to conventional approaches, chunk selection is achieved by optimizing quality as a path of quality with respect to quality changes among multiple quality levels. This differs from conventional research that selects chunk quality for each chunk.

[0012] However, as path length increases, the number of possible paths with different quality adaptations increases exponentially, depending on the path length. In the following embodiment, this problem is addressed by leveraging long-term throughput forecasts to identify a reasonable subset of quality-adaptive paths that enable high-quality service while reducing the computational complexity of the long-term adaptive bitrate control mechanism.

[0013] To achieve this goal, two main factors must be considered: firstly, a suitable rule must be defined for the chunk request decision mechanism; and secondly, accurate throughput estimation is necessary for appropriate chunk selection, as chunk selection may depend on throughput estimation. Below, we introduce the main algorithms. These main algorithms can be classified into those based on hand-drawn rules and those based on learning-based rules.

[0014] There are two main types of ABR models based on hand-written rules: one that uses only buffer fill rate information, and the other that also uses throughput prediction. The buffer-only method requests chunks based on buffer fill rate information, which also provides final information about chunk quality.

[0015] Methods that take throughput information into account are also being considered. These methods predict future throughput using throughput measurements obtained by downloading chunks from previous time. Chunk selection is based on these predictions and buffer fill rates. Model Prediction Control (MPC) techniques are the core principle of such methods, optimizing a quality function defined by the linear sum of three quality-related factors: the quality of the chunks to be requested, the smoothness of quality changes between consecutive chunks, and the delay time between video pauses and initial loading.

[0016] Instead of handcrafted rule-based ABR models, previous research has explored using reinforcement learning for the chunk selection mechanism. An advancement in such models is Pensieve. The general idea is that rules, such as those defined for MPC, are arbitrary and susceptible to human error. For example, in MPC, the weighting between requesting high-quality chunks and maintaining a certain quality is left as a free parameter, and setting this value is not easy. Furthermore, video pauses are considered only in terms of duration and are linearly coupled with encoding quality-related functions, which is also debatable. Therefore, instead of requiring handcrafted rules for the chunk selection process, data-driven techniques are used to learn the rules that should be used for chunk selection.

[0017] These previous studies have several limitations. Initially, chunk selection was based on the MPC objective function, which did not provide a proper model of the user experience. Also, previous studies focused on the immediate chunk demand and did not address the long-term aspects of the streaming session. To address these issues, it has been proposed to improve the user experience by applying knowledge of future throughput by viewing the chunk selection mechanism as quality path optimization. However, this method requires comparing a number of paths that increases exponentially depending on the path length. Therefore, although the results show that session quality can be significantly improved, an optimized search procedure that can consider longer paths is needed.

[0018] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0019] The adaptation of video bitrate and quality to available throughput is handled by the adaptive bitrate control of the video playback device.

[0020] The ABR control mechanism relies on selecting chunks using either hand-written or learning-based rules, where a chunk may be a portion of an image encoded at a given bitrate.

[0021] Much research has been conducted on chunk packet selection rules. However, these studies limit the timescales considered in chunk selection. In fact, these studies focus on selecting the immediate next chunk in relation to the problem of "which chunk should be requested next" faced by video playback devices.

[0022] However, from such a short-term perspective, it is not possible to predict the decline in network access efficiency, and only a response to observed efficiency declines is obtained, so the resulting QoE may not be optimal.

[0023] As an example, consider a scenario where a user watches video inside a car. At a certain point in time, the user has access to a high-quality network, allowing them to watch high-quality video. However, when the car enters a tunnel, the available throughput drops significantly.

[0024] In this situation, a video playback device with a "short-term perspective" has no choice but to drastically reduce the video quality to avoid stopping video delivery once it detects a decline in throughput; it is not given any other options.

[0025] Implementing such a short-term chunk selection mechanism is problematic because it narrows the options for other quality adaptations under challenging network conditions. To address this issue, we propose performing chunk selection based on long-term throughput forecasts.

[0026] For example, considering the aforementioned case of a car entering a tunnel, the gist of the proposed method is that if the video playback device can predict in advance a significant drop in throughput, it can find solutions other than drastically reducing the video quality.

[0027] For example, a video playback device can buffer video at a faster rate by using the available high throughput before throughput drops.

[0028] This ultimately comes at the cost of slightly lowering the quality. However, the result is that users will be able to watch high-quality video while their car is in a tunnel with restricted network access.

[0029] By performing chunk selection based on long-term throughput forecasts, we provide users with higher quality and more consistent quality, improving the user experience in challenging network environments.

[0030] The proposed algorithm is executed on the device and aims to define a procedure for long-term chunk request scheduling.

[0031] Similar to ABR control methods, the proposed algorithm takes a Media Presentation Description (MPD) as input, which provides information about multiple quality levels available for the video.

[0032] Figure 1 shows the key steps of the proposed method. The key steps are as follows:

[0033] 1) Estimate the temporal variation in throughput over a time interval of up to T seconds (the value of T will be explained below).

[0034] 2) Find the optimal chunk scheduling for a time interval of T seconds (Figure 2).

[0035] 3) Request the first chunk of the found optimal path (optimal scheduling).

[0036] The algorithm then returns to step 1 to re-estimate the temporal variation in throughput over a period of up to T seconds.

[0037] The algorithm runs this process in a loop until it reaches the end of video playback or until the user finishes watching the video.

[0038] Figure 3 shows an example of the hardware configuration of a video playback device 10 in an embodiment of the present invention. The video playback device 10 in Figure 3 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a central processing unit (CPU) 104, and an interface device 105, etc., which are all interconnected by bus B.

[0039] The program that enables processing in the video playback device 10 is provided on a recording medium 101 such as a CD-ROM. When the recording medium 101 containing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101; it may also be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files and data.

[0040] When a program startup command is received, the memory device 103 reads the program from the auxiliary storage device 102 and stores it. The CPU 104 executes the functions related to the video playback device 10 according to the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0041] Figure 4 shows an example of the functional configuration of a video playback device 10 in an embodiment of the present invention. In Figure 4, the video playback device 10 includes a throughput prediction unit 11, a bitrate control unit 12, a communication unit 13, and a buffer 14. Each of these units is realized by processing that one or more programs installed in the video playback device 10 cause the CPU 104 to execute. In other words, each of these units is realized through the cooperation of the hardware resources of the video playback device 10 and the programs (software) installed in the video playback device 10. The communication unit 13 may also include a transmitting device and a receiving device.

[0042] The following describes an example of a processing procedure performed by the video playback device 10. Figure 5 is a flowchart illustrating an example of a processing procedure performed by the video playback device 10.

[0043] Before starting the optimization procedure, the value of T must be determined. T is a parameter defined by the user. A larger value of T results in higher performance. However, increasing the value of T increases computational cost. Therefore, it is necessary to resolve the trade-off between computational cost and quality level. In the simulation, a significant improvement in quality was found when T was set to 20 seconds, but other values ​​can also be considered.

[0044] In step S101 of Figure 5, the throughput prediction unit 11 predicts the future throughput for the next T seconds. Predicting throughput is a difficult task and has attracted much attention.

[0045] By applying one of several different techniques, such as the arithmetic mean, harmonic mean, deep neural networks, or hidden Markov models, it is possible to predict future throughput.

[0046] However, these methods only predict throughput immediately after the event. Therefore, this study utilizes time series forecasting techniques to predict throughput over longer time frames.

[0047] As prediction methods used in the throughput prediction unit 11, different methods can be considered, such as the use of autoregressive models, including autoregressive integrated moving averages (ARIMA) or advanced learning-based techniques.

[0048] In the following, we propose a "transformer-based deep neural network throughput prediction algorithm" (Non-Patent Document 5) as the prediction method used in the throughput prediction unit 11, but alternative models can also be considered.

[0049] When using this method, the throughput prediction unit 11 uses past throughput values ​​to predict future throughput over the next T seconds.

[0050] In step S102 of Figure 5, the bitrate control unit 12 sets a search space to find the optimal quality adaptation schedule for the next T-second time interval.

[0051] Figure 2 shows an example of a search space for finding the optimal quality adaptation schedule. In the example in Figure 2, the horizontal axis of the search space represents time, and the vertical axis represents the video quality level.

[0052] In the example in Figure 2, the search space includes the current time t and the time intervals t+1, t+2, ..., t+4 corresponding to the next T seconds, as well as past times t-2 and t-1. Furthermore, six different quality levels are available for the video, ranging from 144p to a maximum of 1080p. In other words, as shown in the example in Figure 2, the bitrate control unit 12 sets the search space by setting different quality levels (from 144p to a maximum of 1080p in the example in Figure 2) as the vertical axis for each unit time (T / 4 in the example in Figure 2). Note that in the example in Figure 2, the next T seconds are divided into four unit time intervals, but the embodiment is not limited to this example. The next T seconds may be divided into more than four unit time intervals, or fewer than four unit time intervals. Also, the quality levels are not limited to six; there may be more than six levels, or fewer than six levels.

[0053] In the example in Figure 2, the line segment between t-2 and t indicates the previous chunk selected by the bitrate control unit 12. At time t, the bitrate control unit 12 needs to select the quality level of the chunk for the next unit time t+1. To this end, in step S103 of Figure 5, the bitrate control unit 12 selects different paths in the search space corresponding to different quality adaptation schedules for the next T seconds as targets for QoE evaluation.

[0054] In the example in Figure 2, three paths are shown that are expected to result in QoE1, QoE2, and QoE3 as different paths in the search space.

[0055] In step S104 of Figure 5, the bitrate control unit 12 uses the future throughput and QoE model predicted in step S101 to evaluate the QoE for a specific path. The QoE model can be one of those described in Non-Patent Literature 6. However, other QoE models described in ITU-T Rec. P.1203 or ITU-T Rec. P.1204 can also be used.

[0056] These standardized models can be used in different modes, employing different types of information, ranging from less complex models that use only parameters such as bitrate, frame rate, and resolution, to more complex models that even use bitstream or pixel information.

[0057] The bitrate control unit 12 predicts the QoE of a specific path based on the throughput of past P seconds and subsequent T seconds.

[0058] By taking into account previously downloaded chunks, we can address the temporal smoothness of the quality.

[0059] In the example shown in Figure 2, the time interval for one chunk is set to 1 second, P is set to 2 seconds, and T is set to 4 seconds.

[0060] Regarding future chunk requests, Figure 2 shows three different paths, but to enable finding the path that gives the highest QoE, an evaluation of QoE should be performed considering the case where all of the available quality levels are applied.

[0061] The process of calculating the QoE of a specific route is performed as follows: Each chunk in the path has a resolution r and a video bitrate b. v , frame rate f, and audio bitrate b aIt corresponds to different quality levels, characterized by the following:

[0062] By using the characteristics of each chunk within a specific path, we can obtain the per-second value of each of these quality-related parameters.

[0063] Next, based on the QoE model in Non-Patent Document 6, the video quality is evaluated every second using Equation 1, the audio quality is evaluated every second using Equation 4, and the video and audio quality is evaluated every second using Equation 5.

[0064]

number

[0065]

number

[0066]

number

[0067] In equation 1, X and Y are defined by equations 2 and 3.

[0068]

number

[0069]

number

[0070] The temporal aggregation of encoding quality-related scores per second into a score per path is given by equations 6-9.

[0071]

number

[0072]

number

[0073]

number

[0074]

number

[0075] Finally, equation 10 is used to consider the impact of the suspension of video provision.

[0076]

number

[0077] In equation 10, n b This is the number of interruptions in video provision that occurred along the route being considered, and t b This is the total time interval during which video provision is suspended, and a b This is the average time interval between events that interrupted video provision.

[0078] In these formulas, v1-v7, a1-a3, m1-m4, t1-t5, and s1-s3 are model parameters obtained by using regression with a subjective experimental database.

[0079] If L is the number of quality levels and N is the number of chunks to download in the next T seconds, then L N It is necessary to compare the QoE of different routes in the book.

[0080] By considering computational costs and applying limitations to the evaluation of quality adaptation, it is possible to reduce the number of paths that need to be evaluated.

[0081] As an optimization candidate, only an increase or decrease in quality by only level A may be allowed.

[0082] For example, in the example of FIG. 2, A = 1, and at time t, the quality level of the previously requested chunk is 360p.

[0083] At time t + 1, it may be considered that only quality levels 480p, 360p, and 240p can be requested.

[0084] At time t + 2, depending on the selection made at t + 1, the quality value can range from 720p to 144p.

[0085] By performing such optimization, it leads to smooth quality adaptation that results in a higher QoE, and even if a full search for all paths of the book is performed, it is possible to obtain the effect of forcing smooth quality adaptation that will be selected, and reduce the search space to (2 × A + 1) N books of this path. N It can be reduced to this path of the book.

[0086] In step S105 of FIG. 5, the bitrate control unit 12 selects the path in the search space that is expected to provide the highest QoE among the N books of this path. As described above, when only an increase or decrease in quality by only level A is allowed as an optimization candidate, the bitrate control unit 12 selects the path in the (2 × A + 1) N books of this path that is expected to provide the highest QoE.

[0087] Regarding the startup of a video, which is a special case, past throughput data cannot be obtained, so it may not be possible to predict the future throughput for the next T seconds. In this case, different alternatives can be considered.

[0088] 1) The bitrate control unit 12 may select to start at the lowest quality level. In this case, a fast initial load is possible.

[0089] 2) The bitrate control unit 12 may allow the user to select an initial quality value.

[0090] 3) The bitrate control unit 12 may estimate future throughput based on previously viewed video footage by the user.

[0091] For example, when a user finishes watching a video and moves on to a new one, information about the available throughput measured in the previous video can be used to predict the throughput of the next video, making it possible to start the next video at an appropriate quality level.

[0092] Similarly, by using information such as throughput measured during the web interface loading phase as a clue, it is possible to predict the available throughput for the user and select an appropriate initial quality level.

[0093] In step S106 of Figure 5, the bitrate control unit 12 makes a chunk request via the communication unit 13.

[0094] In step S105, after selecting the path that is expected to give the highest QoE, in step S106, the bitrate control unit 12 requests via the communication unit 13 a chunk with a quality level corresponding to the immediately following chunk included in the selected path.

[0095] For example, in Figure 2, let's assume that the path that yields a QoE value of QoE2 provides the highest QoE to the user.

[0096] In this case, the bitrate control unit 12 requests a chunk corresponding to quality level 360p via the communication unit 13.

[0097] Subsequently, the communication unit 13 acquires the quality level chunks requested in step S106 via a network such as the Internet and loads them into the buffer 14. As a result, the video playback device 10 can provide the user with higher quality and more consistent quality video.

[0098] In step S107 of Figure 5, the bitrate control unit 12 determines whether the end of video playback has been reached, or whether the user has finished watching the video.

[0099] In step S107, if it is determined that the end of video playback has been reached, or if it is determined that the user has finished watching the video (S107: YES), the bitrate control unit 12 terminates the processing procedure.

[0100] In step S107, if it is determined that the end of video playback has not been reached and that the user has not finished watching the video (S107: NO), the processing procedure returns to S101, and the throughput prediction unit 11 uses the throughput measured from chunk t-1 to t+1 to predict the throughput from t+2 to t+5.

[0101] In steps S102 to S105, the bitrate control unit 12 recalculates the optimal path between t+1 and t+5 based on the estimated throughput and selects the chunk to be downloaded at t+2.

[0102] This process is repeated in a loop until the user stops watching or the video ends.

[0103] The above-described embodiment demonstrates how long-term throughput predictions using time-series analysis are used in an adaptive bitrate control method that performs long-term chunk request scheduling. This method makes it possible to evolve conventional video playback devices with a narrow field of view into video playback devices that can predict throughput fluctuations and provide users with higher quality.

[0104] Furthermore, the above-described examples demonstrate that the latest perceived quality model can be used to evaluate the quality of a specific chunk selection path. By using knowledge from the QoE model to define appropriate quality adaptation procedures, the user's perceived quality is improved by executing these procedures.

[0105] Furthermore, the above-described examples demonstrate that, in order to reduce computational complexity, quality adaptation is limited to increases or decreases that do not exceed a user-defined threshold. This allows for a significant reduction in computational cost while enforcing temporal smoothness of quality.

[0106] The following details an example of an optimization procedure for identifying a path to achieve high QoE while maintaining reasonable computational costs.

[0107] Figure 6A illustrates an example of a user watching video in a car. At some point, the user enjoys high-quality network access and is able to watch high-quality video. However, when the car enters a tunnel, the available throughput drops significantly. In this context, conventional "narrow-field-of-view" video playback devices have no choice but to significantly degrade video quality to avoid video freezing when the device detects a decrease in throughput. On the other hand, if the video playback device could anticipate this level of throughput reduction, it could identify a better solution. For example, as shown in Figure 6B, by using the high available throughput before entering the tunnel to accelerate re-buffering, high-quality video can be maintained even while the car is in a tunnel with low throughput.

[0108] To apply this principle, the video playback device must compare the quality of the results after executing different quality adaptation patterns at each step and select the sequence of chunks that provides the highest quality to the user. This sequence of chunks will be referred to as the quality adaptation path below. This process is shown in Figure 2. At time t, the video playback device needs to decide which chunks to request. The video playback device then uses long-term throughput forecasts to estimate the results of different quality adaptation paths. Figure 2 compares the quality of three paths, but to identify the optimal quality adaptation, it is necessary to compare many paths to determine which provides the highest QoE. Figure 7 shows the case where all possible quality adaptation paths are evaluated, and the path providing the highest QoE is highlighted with a dotted line. Figure 7 also shows that many paths need to be considered to identify the highest quality path. In a full search approach, the number of paths to evaluate is L. D Here, L is the number of quality levels and D is the length of the path, i.e., the number of future chunks. The method described above introduces a computationally intensive search by considering only quality adaptations limited to a specific range A. For example, when A=1, the video playback device can increase or decrease the quality by one level or maintain the same quality as the previous step. As shown in Figure 8, this significantly reduces the number of paths to compare, resulting in (2A+1) D This is the result.

[0109] Figure 9 shows the simulation execution time for the experimental example. It can be seen that there is an exponential relationship between the execution time value used in the optimization procedure and the depth value (i.e., the number of future chunks). When moving from full search to bounded adaptation, the number of paths to compare can be reduced as the depth increases, but the computational complexity remains high, making it impractical to consider very large depths when optimizing the operation of a video playback device while playing a video streaming session.

[0110] The proposed algorithm runs on the device and aims to define a long-term chunk request scheduling procedure. Similar to conventional ABR control schemes, the proposed algorithm takes a media presentation description (MPD) as input, which provides information about the available quality levels of the video. Figure 10 shows the main steps of the proposed method. First, the temporal variation of throughput up to T seconds is predicted. Next, a path that provides a higher QoE to the user for the next T seconds is identified. This path is represented by a dotted line in Figure 10. Third, the first chunk of the identified optimized path is requested. The algorithm then returns to the first step and re-estimates the temporal variation of throughput up to T seconds. The algorithm loops through this procedure until video playback ends or the user stops watching the video.

[0111] The simulations performed showed that quality improved significantly when T was 20 seconds, but other values ​​may also be considered. The results indicate that quality improves as the value of T increases. However, increasing the value of T increases the computational cost. Considering that performance improvements mostly occur under difficult network conditions, various preset values ​​of T may be used depending on the distribution of network throughput. Subsequently, service providers should pre-define various values ​​of T and use larger values ​​of T when network conditions deteriorate and change significantly. For example, if throughput is stable, T may be kept constant at 8 seconds, but if the network changes significantly, such as for users using public transport, T may be increased up to 20 seconds. This allows for higher quality at a higher computational cost when network conditions become difficult, while maintaining the computational cost and battery of mobile devices when network conditions are stable.

[0112] The following describes the different steps of the proposed method.

[0113] Step 1: Throughput Prediction. Predicting throughput is a challenging task and has attracted much attention. Future throughput has been predicted using various techniques, including arithmetic mean, harmonic mean, deep neural networks, autoregressive approaches, or hidden Markov models. However, only immediate throughput is predicted. Therefore, this study proposes to predict throughput over a larger time frame by leveraging research on time series forecasting. Various methods can be considered, such as the use of autoregressive models like ARIMA or methods based on advanced learning. Below, a transformer-based deep neural network throughput prediction algorithm is proposed, but alternative models are also possible. Using this model, past throughput values ​​are used to predict future throughput for the next T seconds.

[0114] Step 2: Scheduling Long-Term Chunk Requests. The central principle of long-term chunk request scheduling is shown in Figure 2. In this figure, we can see that the video is available at six different quality levels, from 144p to 1080p. The line segment from t-2 to t represents the temporally previous chunk selected by the ABR rate control model. At this stage, it is necessary to select the quality for the next chunk, t+1. For this purpose, different paths of quality adaptation over the next T seconds are considered, and the path that provides the highest QoE is selected.

[0115] To evaluate the QoE of a given path, we use the model described in Non-Patent Literature 6. However, other QoE models, such as those described in ITU-T Rec. P.1203 or ITU-T Rec. P. 1204, can also be used. These standardized models are available in various modes using different types of information, from low-complexity models that use only parameters such as bitrate, frame rate, and resolution, to high-complexity models that also use bitstream or pixel information. The QoE of a given path is predicted based on the preceding P seconds and the following T seconds. By considering previously downloaded chunks, we can address the temporal smoothness aspect of quality. In the example shown in Figure 2, the chunk duration is set to 1 second, and P and T are set to 2 seconds and 4 seconds, respectively. Regarding future chunk requests, Figure 2 shows three different paths to illustrate a common long-term chunk selection mechanism, but it is important to identify which path needs to be evaluated. This will be discussed later.

[0116] Regarding the calculation process for the QoE of a given path, the process is as follows: all chunks within the path correspond to different quality levels that can be characterized by resolution r, video bitrate bv, frame rate f, and audio bitrate ba. Using the characteristics of each chunk from a given path, it is possible to obtain the per-second values ​​of each of these quality-related parameters. Next, based on the model in Non-Patent Literature 6, the video, audio, and audiovisual quality are estimated per second using equations 1, 4, and 5, respectively. The temporal aggregation of the encoding quality-related score per second to the QoE score per path is given by equations 6-9. Finally, the effect of video pauses is considered using equation 10. In this equation, n b This is the number of times video freezes occurred along the route being considered, t b is the total video stop time, a b v is the average time between video stop events. In these equations, v 1-7 a 1-3 , m 1-4 t1-5 , s 1-3 These are model parameters obtained using regression with a subjective experimental database.

[0117] When searching for a path that provides the optimal quality adaptation pattern, it is important to consider an appropriate search procedure. If the video is available at L quality level and a path consisting of D chunks is considered, the sum of comparable paths is L D Therefore, as the path length increases, the number of paths increases exponentially, requiring an optimized search procedure.

[0118] Figure 11 shows the general concept of the proposed method. This figure shows the dependencies between chunks. This figure shows that throughput changes as a function of time. Therefore, the time it takes to download a chunk may differ depending on when the chunk download starts. The "playback" line in Figure 11 corresponds to the constraints of the video playback device for playing back video without interruption. The playback line includes C 1、 C 2、 C 3、 And points marked as C4 are included, which are chunks C that the video playback device uses to play video continuously. 1、 C 2、 C 3、 This indicates the time when C4 needs to be made available. The path exploration line shows the dependencies between consecutive chunks. In path exploration (1), chunk C1 with quality level Q0 is downloaded. To enable consecutive playback, chunk C2 must be downloaded before the time to play C2 begins. Based on the available throughput, chunk 2 (C2) may be downloaded with quality level 0 (C2Q0) or quality level 1 (C2Q1). Quality level 2 (C2Q2) cannot be selected as it will arrive too late. Higher quality values ​​than Q2 simply require more bits, and all paths that use quality 2 or higher for the second chunk may be ignored.

[0119] In the first step, only two possibilities exist: (C2Q0) and (C2Q1). First, the proposed algorithm selects the highest quality chunk that can be reached in time: (C2Q1). This corresponds to pathfinding line (2.1) in Figure 11. If this chunk is selected, then for the third chunk, the next highest quality chunk that can be downloaded is (C3Q1), and for the fourth chunk, it is (C4Q2).

[0120] However, requesting the highest quality chunk available for download does not necessarily guarantee that the user will receive the highest quality. In fact, the example pathfinding line (2.2) shows that, given the low available throughput when downloading chunk C2, requesting a lower quality chunk for C2 allows the download to complete earlier, enabling the download of higher quality chunks for C3 and C4 (C3Q2 and C4Q3, respectively). As a result, the overall user experience improves.

[0121] Therefore, the proposed search procedure consists of two steps. 1) Identify the highest quality chunks that can be downloaded based on the available throughput and time constraints for uninterrupted video playback. 2) Increase or decrease the quality of the requested chunks to search for alternative paths around the initially identified path.

[0122] Figure 12 shows candidate paths considered using the proposed method. First, the path "Path immediately under available throughput and buffer-filling rate" displays a column of the highest quality chunks that can be downloaded in time for smooth playback. Next, other different candidate quality adaptive paths are considered by decreasing or increasing chunk quality compared to the initially identified path.

[0123] The quality range around the initially identified path is constrained by the parameter M. A value of M=1 indicates that chunk quality can be increased or decreased by one level around the original path. This reduces the search space while searching for the optimal path. If L is the number of available quality levels, a value of M=L means that all possible quality-matching paths will be searched.

[0124] Other improvements include restricting the scope of quality adaptation considerations between adjacent chunks within the search space surrounding the original path, allowing only quality changes up to a specified level (A).

[0125] Based on this search procedure, the upper limit of the number of possible paths is shown in Equation 11. It is worth noting that in this equation, the number of paths increases proportionally to the length of the path, rather than exponentially as before. This improvement makes it possible to consider longer paths in the ABR control scheme.

[0126]

number

[0127] Step 3: Chunk Request. Once the optimal path is identified, the chunk immediately following this optimal path is requested. For example, in Figure 2, assume that the path with a QoE value of QoE2 provides the highest QoE to the user. The chunk corresponding to quality level 360p is requested. Next, the measured throughput from chunk t-1 to t+1 is used to predict the throughput from t+2 to t+5. Based on the estimated throughput, the optimal path between t+2 and t+5 is recalculated, allowing for the selection of the chunk to be downloaded at t+2. This process is repeated in a loop until the user stops watching or the video player reaches the end of the video.

[0128] The above-described examples demonstrate that the computational complexity of the long-term adaptive bitrate control mechanism can be improved by considering throughput information that identifies a subset of the paths to be evaluated.

[0129] According to the above embodiment, a “base path” that describes a sequence of the highest quality chunks that can be downloaded, taking into account available throughput, is an efficient starting point for a long-term adaptive bitrate control mechanism.

[0130] In the above-described embodiment, the path length can be made dependent on network conditions. When network throughput is stable, the computational cost and energy consumption of the long-term ABR control model can be reduced. On the other hand, when network conditions become difficult, such as for users on public transport, it is possible to increase the computational cost and energy consumption.

[0131] The above embodiment describes a fast method for finding the necessary chunk quality sequence required to provide users with high service quality. Previous research has shown that long-term throughput forecasting can improve the user's perceived quality by enabling video playback devices to anticipate and proactively take action against poor network conditions, rather than simply reacting to recently observed low-quality network conditions. Long-term ABR control mechanisms optimize quality adaptive paths rather than optimizing chunk selection on a chunk-by-chunk basis. However, as the length of the quality adaptive path increases, the number of possible quality adaptive paths increases exponentially with the number of chunks in the path. The proposed method leverages long-term throughput forecasting to identify reasonable quality adaptive paths and reduce the computational complexity of the long-term chunk selection method. This ultimately allows video playback devices to predict network throughput losses earlier and improve service quality.

[0132] Although embodiments of the present invention have been described in detail above, the present invention is not limited to these specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention as described in the claims.

Claims

1. A video playback device, Transmitter; Receiving unit; Processor; and memory, Equipped with, The memory is provided to the processor, A step of predicting future throughput during a first future time interval that includes m future unit time intervals, using past throughput values; A step of establishing a search space for finding a schedule for quality adaptation, which is defined by k quality levels and a second time interval including the current unit time interval, the m future unit time intervals, and the n past unit time intervals; A step of determining a time limit for acquiring each of the m chunks corresponding to the m future time intervals, in order to play the video without interruption in the m future time intervals; Based on the aforementioned future throughput, for each of the m chunks, the step of identifying the highest quality level among one or more quality levels that allow the chunk to be downloaded by the time limit; Steps include identifying a first path in the search space that corresponds to the m highest quality levels identified; A step of identifying one or more second routes by changing one or more of the highest quality levels among the m highest quality levels included in the first route; A step of estimating perceived quality values ​​based on the future throughput for each of the first route and the one or more second routes; A step of identifying the path corresponding to the highest perceived quality value among the estimated perceived quality values ​​as the quality adaptation schedule; A step of determining the required chunk quality value for the next future unit time interval of the current unit time interval, based on the quality adaptation schedule; A step of requesting the determined quality value chunk via the transmission unit; A step of receiving the determined quality value chunk via the receiving unit; and A step of using a preset value different from the preset value before the change in throughput as the value of the future first time interval in response to the change in throughput, Includes instructions to execute, Video playback device.

2. The step of identifying the one or more second paths involves limiting the scope of the change of the one or more highest quality levels among the m highest quality levels to a predetermined range. The video playback device according to claim 1.

3. The step of identifying the one or more second paths involves changing the one or more highest quality levels among the m highest quality levels such that the difference in quality levels between two adjacent chunks among the m chunks is within A level, where A is an integer less than or equal to k. The video playback device according to claim 1.

4. The step of predicting the future throughput, in the case of video startup, applies the throughput of another previously viewed video or a pre-set lower throughput as the future throughput. The video playback device according to claim 1.

5. The step of estimating the perceived quality value involves evaluating the perceived quality value using a predetermined perceived quality model. The video playback device according to claim 1.

6. A video playback method performed by a computer, A step of predicting future throughput during a first future time interval that includes m future unit time intervals, using past throughput values; A step of establishing a search space for finding a schedule for quality adaptation, which is defined by k quality levels and a second time interval including the current unit time interval, the m future unit time intervals, and the n past unit time intervals; A step of determining a time limit for acquiring each of the m chunks corresponding to the m future time intervals, in order to play the video without interruption in the m future time intervals; Based on the aforementioned future throughput, for each of the m chunks, the step of identifying the highest quality level among one or more quality levels that allow the chunk to be downloaded by the time limit; Steps include identifying a first path in the search space that corresponds to the m highest quality levels identified; A step of identifying one or more second routes by changing one or more of the highest quality levels among the m highest quality levels included in the first route; A step of estimating perceived quality values ​​based on the future throughput for each of the first route and the one or more second routes; A step of identifying the path corresponding to the highest perceived quality value among the estimated perceived quality values ​​as the quality adaptation schedule; A step of determining the required chunk quality value for the next future unit time interval of the current unit time interval, based on the quality adaptation schedule; A step of requesting the chunks of the quality value determined above; A step of receiving the chunk of the quality value determined above; and A step of using a preset value different from the preset value before the change in throughput as the value of the future first time interval in response to the change in throughput, A video playback method that includes the following features.

7. A non-temporary computer-readable storage medium for storing a video playback program, wherein when the video playback program is executed by a computer, the computer shall A step of predicting future throughput during a first future time interval that includes m future unit time intervals, using past throughput values; A step of establishing a search space for finding a schedule for quality adaptation, which is defined by k quality levels and a second time interval including the current unit time interval, the m future unit time intervals, and the n past unit time intervals; A step of determining a time limit for acquiring each of the m chunks corresponding to the m future time intervals, in order to play the video without interruption in the m future time intervals; Based on the aforementioned future throughput, for each of the m chunks, the step of identifying the highest quality level among one or more quality levels that allow the chunk to be downloaded by the time limit; Steps include identifying a first path in the search space that corresponds to the m highest quality levels identified; A step of identifying one or more second routes by changing one or more of the highest quality levels among the m highest quality levels included in the first route; A step of estimating perceived quality values ​​based on the future throughput for each of the first route and the one or more second routes; A step of identifying the path corresponding to the highest perceived quality value among the estimated perceived quality values ​​as the quality adaptation schedule; A step of determining the required chunk quality value for the next future unit time interval of the current unit time interval, based on the quality adaptation schedule; A step of requesting the chunks of the quality value determined above; A step of receiving the chunk of the quality value determined above; and A step of using a preset value different from the preset value before the change in throughput as the value of the future first time interval in response to the change in throughput, A non-temporary, computer-readable storage medium that performs [some action].

Images