Video playback device, video playback method, and program

Abstract

The processor of the video playback device performs the following steps: identifying a time limit for acquiring each chunk to play the video without interruption; identifying, for each chunk, the highest quality level at which the chunk can be downloaded by the time limit based on the future throughput; identifying a first path corresponding to the identified highest quality level; identifying a second path by changing the highest quality level included in the first path; estimating a quality of experience value for each of the first and second paths based on the future throughput; identifying the path corresponding to the highest quality of experience value among the estimated quality of experience values ​​as the quality adaptation schedule; and determining, based on the quality adaptation schedule, the quality value of the chunk to be requested in the next future unit time interval of the current unit time interval.

Description

[Technical Field]

[0001] The present invention relates to a video playback device, a video playback method, and a program. [Background technology]

[0002] Adaptive bitrate (ABR) video streaming is a widely used application on the Internet. ABR video streaming allows users to watch video under changing network conditions, such as varying throughput. In ABR streaming, different representations with different throughput requirements are provided to the video playback device. Each different representation provides different quality by using different encoding conditions, such as resolution, frame rate, and bit rate. The entire video is broken down into multiple video segments called chunks, allowing for chunk-by-chunk video quality adjustment. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Kevin Spiteri, Rahul Urgaonkar, and Ramesh Sitaraman. 2016. BOLA: Near- Optimal Bitrate Adaptation for Online Videos. In Proceedings of the 35th Annual IEEE International Conference on Computer Communications. 1-9. [Non-patent document 2] X. Yin, A. Jindal,V.Sekar, and B. Sinopoli, “A control-theoretic approach for dynamic adaptive video streaming over HTTP,” in Proceedings of the 2015 Conference of the ACM SIGCOMM, 2015, pp. 325-338.

Outdoor Tools3

Outdoor Tools 4

Direct Environment 5

Outdoor Configuration6

[0004] Delivering video to a large audience is challenging. Each user has different capabilities regarding available throughput, which can change as a function of time. Therefore, to deliver video to all users, it is necessary to use different representations (video encoded at various resolutions, bit rates, and frame rate values) so that the video bit rate requirements can be adapted to the user's available throughput. This method of adapting quality to the available throughput is called adaptive bit rate video streaming, and the bit rate values ​​corresponding to each possible representation are called a bit rate ladder.

[0005] The process of adapting quality to the available throughput is handled by the video player's adaptive bitrate control mechanism. ABR control mechanisms rely on handcrafted or learning-based rules for chunk selection. While much research has been done on these rules for chunk packet selection, a limitation of these previous studies is the time scale considered in chunk selection. Indeed, previous studies have focused on selecting the next chunk relative to the video player's problem of "which chunk should I request next." However, this narrow view may result in a suboptimal QoE because it fails to predict network access degradation and can only provide responses to observed degradation events.

[0006] The operation of the video playback device involves comparing the quality of the results after executing different quality adaptation patterns at each step and selecting the sequence of chunks that provides the user with the best quality of experience. This sequence of chunks is hereinafter referred to as the quality adaptation path.

[0007] An optimization procedure is needed to identify quality adaptation paths that achieve high QoE while maintaining reasonable computational costs. [Means for solving the problem]

[0008] According to one aspect of the present invention, a video playback device includes a transmitter; a receiver; a processor; and a memory, wherein the memory causes the processor to: predict future throughput during a first future time interval including m unit time intervals in the future using past throughput values; set a search space defined by k quality levels and a second time interval including the current unit time interval, the m future unit time intervals, and n past unit time intervals, for finding a quality adaptation schedule; specify a time limit for acquiring each chunk among m chunks corresponding to the m future unit time intervals, for playing video without interruption during the m future unit time intervals; and, for each chunk among the m chunks, select the highest quality level among one or more quality levels at which the chunk can be downloaded by the time limit based on the future throughput. a step of identifying a highest quality level in the search space; a step of identifying a first route in the search space corresponding to the identified m highest quality levels; a step of identifying one or more second routes by changing one or more highest quality levels among the m highest quality levels included in the first route; a step of estimating a quality of experience value for each of the first route and the one or more second routes based on the future throughput; a step of identifying the route corresponding to the highest quality of experience value among the estimated quality of experience values ​​as the quality adaptation schedule; a step of determining a quality value of a chunk to be requested in a future unit time interval next to the current unit time interval based on the quality adaptation schedule; a step of requesting a chunk with the determined quality value via the transmitting unit; and a step of receiving a chunk with the determined quality value via the receiving unit. [Effects of the Invention]

[0009] According to an embodiment, an optimization procedure is provided to identify quality adaptation paths that achieve high QoE while maintaining reasonable computational costs. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of the main steps of a chunk scheduling algorithm according to an embodiment. [Figure 2] FIG. 10 is a diagram showing an example of a comparison of QoE values ​​for different quality adaptations in an embodiment. [Figure 3] 1 is a diagram illustrating an example of a hardware configuration of a video playback device 10 according to an embodiment. [Figure 4] 1 is a diagram illustrating an example of a functional configuration of a video playback device 10 according to an embodiment. [Figure 5] 10 is a flowchart illustrating an example of a processing procedure executed by the video playback device 10. [Figure 6A] FIG. 10 is a diagram showing an example of focusing on the selection of the immediately following chunk. [Figure 6B] FIG. 1 illustrates an example of chunk selection using long-term throughput prediction. [Figure 7] FIG. 10 is a diagram illustrating an example of evaluating quality of experience for each route. [Figure 8] FIG. 10 is a diagram showing an example in which the difference in quality between adjacent chunks is limited to A level or less. [Figure 9] FIG. 10 is a diagram showing an example in which the execution time of a simulation is compared by changing the path depth. [Figure 10] FIG. 1 illustrates an example of an iterative process of a long-term chunk selection algorithm. [Figure 11] FIG. 10 illustrates an example of evaluating the quality of each quality adaptation path based on the time limit within which each chunk should be acquired. [Figure 12] FIG. 10 illustrates an example of searching for paths around the path of the highest quality chunk available for download. DETAILED DESCRIPTION OF THE INVENTION

[0011] In the following example, we describe a fast method for finding out what sequence of chunk quality levels needs to be requested to provide a high quality of service to users watching videos using adaptive bitrate video streaming. Compared to traditional approaches, chunk selection is performed by optimizing quality as a quality path for quality variations among multiple quality levels. This is different from traditional work that selects chunk quality on a chunk-by-chunk basis.

[0012] However, as the path length increases, the number of possible paths with different quality adaptations increases exponentially depending on the path length. In the following examples, we address this issue by leveraging long-term throughput predictions to identify a subset of reasonable quality adaptation paths that enable high quality service while reducing the computational complexity of the long-term adaptive bitrate control mechanism.

[0013] To achieve this goal, two main factors need to be considered: first, it is necessary to define appropriate rules for the chunk request decision mechanism; second, since chunk selection may depend on throughput estimation, accurate throughput estimation is necessary to make appropriate chunk selection. In the following, we introduce the main algorithms, which can be classified as either based on handcrafted rules or learning-based rules.

[0014] There are two main types of handcrafted rule-based ABR models: those that use only buffer fill information and those that also use throughput predictions. Buffer-only methods request chunks based on buffer fill information with eventual information about chunk quality.

[0015] Methods that take throughput information into account have also been considered. These methods use throughput measurements obtained by downloading previous chunks in time to predict future throughput. Chunk selection is based on these predictions and the buffer filling rate. Model Predictive Control (MPC) techniques are the core principle of such methods, optimizing a quality function defined by the linear sum of three quality-related factors: the quality of the chunk about to be requested, the smoothness of the quality transition between successive chunks, and the video pause and initial load delay.

[0016] Instead of handcrafted rule-based ABR models, previous research has explored the use of reinforcement learning for chunk selection mechanisms. An advancement in such models is Pensieve. The general idea is that rules like those defined for MPC are arbitrary and subject to human error. For example, in MPC, the weighting between requiring high-quality chunks and maintaining a certain quality is left as a free parameter, and setting this value is not easy. Furthermore, video pauses are considered only in terms of duration and linearly coupled with coding-quality-related features, which is also controversial. Therefore, instead of requiring handcrafted rules for the chunk selection process, data-driven techniques are used to learn the rules that should be used for chunk selection.

[0017] These previous studies have several limitations. Initially, chunk selection was based on an MPC objective function, which does not adequately model the user experience. Furthermore, previous studies focused on immediate chunk requests and did not address the long-term aspects of streaming sessions. To address these issues, it has been proposed to consider chunk selection mechanisms as quality path optimization, thereby applying knowledge of future throughput to improve user experience. However, this approach requires the comparison of a number of paths that grows exponentially depending on the path length. Therefore, while experimental results show that session quality can be significantly improved, an optimized search procedure that can consider longer paths is needed.

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0019] The adaptation of video bit rate and quality to the available throughput is handled by the video playback device's adaptive bit rate control.

[0020] ABR control mechanisms rely on selecting chunks by handcrafted or learning-based rules, where a chunk may be a portion of video encoded at a given bitrate.

[0021] There has been much research into chunk packet selection rules. However, these studies have been limited in the timescales considered in chunk selection. In fact, these studies have focused on selecting the next chunk in relation to the problem of "which chunk should be requested next" faced by video playback devices.

[0022] However, such a short-term perspective cannot predict network access inefficiencies, but only provides a response to observed inefficiencies, and the resulting QoE may not be optimal.

[0023] As an example, consider a user watching video in a car. At some time, the user may have high-quality network access, which allows the user to watch high-quality video. However, when the car enters a tunnel, the available throughput drops significantly.

[0024] In this situation, a video playback device with a "short-term perspective" has no choice but to significantly reduce the quality of the video in order to avoid stopping video provision when it measures a drop in throughput.

[0025] Implementing such a short-term chunk selection mechanism is problematic because it limits options for other quality adaptations under difficult network conditions.To solve the above problem, we propose to base chunk selection on long-term predictions of throughput.

[0026] For example, considering the example of a car entering a tunnel mentioned above, the outline of the proposed method is that if the video playback device can predict a significant drop in throughput in advance, it can find a solution other than significantly reducing the video quality.

[0027] For example, a video playback device can use the higher available throughput to buffer video at a faster rate before the throughput drops.

[0028] This ultimately comes at the cost of a small reduction in quality, but the end result is that users can enjoy high-quality video while their car is in a tunnel where network access is limited.

[0029] By performing chunk selection based on long-term predictions of throughput, higher and more consistent quality is provided to users, improving the user's quality of experience in challenging network environments.

[0030] The proposed algorithm runs on the device and aims to define a long-term chunk request scheduling procedure.

[0031] Similar to ABR control methods, the proposed algorithm takes as input a Media Presentation Description (MPD) that provides information about the multiple quality levels available for the video.

[0032] Figure 1 shows the key steps of the proposed method. The key steps are as follows:

[0033] 1) Estimate the time variation of throughput for a time interval of up to T seconds (the value of T is explained below).

[0034] 2) Find the optimal chunk scheduling for a time interval of T seconds (Figure 2).

[0035] 3) Request the first chunk of the found optimal path (optimal scheduling).

[0036] The algorithm then returns to step 1 and re-estimates the time variation of the throughput for a period of up to T seconds.

[0037] The algorithm performs this process in a loop until the end of the video playback is reached or the user has finished watching the video.

[0038] Fig. 3 is a diagram showing an example of the hardware configuration of a video playback device 10 according to an embodiment of the present invention. The video playback device 10 in Fig. 3 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a central processing unit (CPU) 104, and an interface device 105, all of which are interconnected via a bus B.

[0039] A program that realizes processing in the video playback device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0040] When an instruction to start a program is received, the memory device 103 reads the program from the auxiliary storage device 102 and stores it. The CPU 104 executes functions related to the video playback device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0041] Fig. 4 is a diagram showing an example of the functional configuration of a video playback device 10 according to an embodiment of the present invention. In Fig. 4, the video playback device 10 includes a throughput prediction unit 11, a bit rate control unit 12, a communication unit 13, and a buffer 14. Each of these units is realized by a process executed by a CPU 104 of one or more programs installed in the video playback device 10. In other words, each of these units is realized by cooperation between the hardware resources of the video playback device 10 and the programs (software) installed in the video playback device 10. The communication unit 13 may include a transmitting device and a receiving device.

[0042] The following describes an example of the processing procedure executed by the video playback device 10. Fig. 5 is a flowchart for explaining an example of the processing procedure executed by the video playback device 10.

[0043] Before starting the optimization procedure, the value of T needs to be determined. T is a user-defined parameter. The larger the value of T, the better the performance. However, increasing the value of T increases the computational cost. Therefore, a trade-off between computational cost and quality level needs to be made. In simulations, a value of T of 20 seconds was found to significantly improve the quality, but other values ​​can be considered.

[0044] 5, the throughput prediction unit 11 predicts the future throughput for the next T seconds. Predicting throughput is a difficult task and has attracted much attention.

[0045] It is possible to predict future throughput by applying any of different techniques such as arithmetic mean, harmonic mean, deep neural networks or hidden Markov models.

[0046] However, these methods only predict the immediate future throughput. Therefore, in this study, we utilize time series forecasting techniques to predict throughput for a longer time frame.

[0047] Different prediction methods may be used in the throughput prediction unit 11, such as the use of autoregressive models, such as Autoregressive Integrated Moving Average (ARIMA) or advanced learning-based techniques.

[0048] In the following, a "transformer-based deep neural network throughput prediction algorithm" (Non-Patent Document 5) is proposed as a prediction method used in the throughput prediction unit 11, but alternative models may also be considered.

[0049] When using this method, the throughput predictor 11 uses past throughput values ​​to predict future throughput for the next T seconds.

[0050] In step S102 of FIG. 5, the bit rate control unit 12 sets a search space for finding an optimal quality adaptive schedule for a time interval of T seconds in the future.

[0051] 2 is a diagram showing an example of a search space for finding an optimal quality adaptive schedule. In the example of FIG. 2, the horizontal axis of the search space represents time, and the vertical axis represents the quality level of the video.

[0052] In the example of FIG. 2, the search space includes past times t-2 and t-1 in addition to the current time t and t+1, t+2, ..., t+4 corresponding to a time interval of T seconds in the future. Six different quality levels, ranging from 144p to a maximum of 1080p, are available as video quality levels. That is, as shown in the example of FIG. 2, the bit rate control unit 12 sets the search space by setting different quality levels (ranging from 144p to a maximum of 1080p in the example of FIG. 2) on the vertical axis for each unit time (T / 4 in the example of FIG. 2). Note that, although the example of FIG. 2 divides the future T seconds into four unit times, the embodiment is not limited to this example. The future T seconds may be divided into more than four unit times, or the future T seconds may be divided into fewer than four unit times. The number of quality levels is not limited to six, and may be more than six or fewer than six.

[0053] 2, the line segment from t-2 to t indicates the previous chunk selected by the bitrate control unit 12. At time t, the bitrate control unit 12 needs to select the quality level of the chunk for the next unit time t+1. For this purpose, in step S103 of FIG. 5, the bitrate control unit 12 selects different paths in the search space corresponding to different schedules of quality adaptation for the next T seconds as targets for QoE evaluation.

[0054] In the example of FIG. 2, three routes that are expected to provide QoE1, QoE2, and QoE3 are shown as different routes in the search space.

[0055] 5, the bit rate control unit 12 uses the future throughput predicted in step S101 and the QoE model to evaluate the QoE for a specific path. As the QoE model, the model described in Non-Patent Document 6 can be used. However, it is also possible to use other QoE models described in ITU-T Rec. P.1203 or ITU-T Rec. P.1204.

[0056] These standardized models can be used in different modes that use different types of information, from low-complexity models that use only parameters such as bit rate, frame rate and resolution, to high-complexity models that use bitstream or even pixel information.

[0057] The bit rate control unit 12 predicts the QoE of a specific path based on the throughput for the past P seconds and the subsequent T seconds.

[0058] By taking into account previously downloaded chunks, the temporal smoothness of the quality can be addressed.

[0059] In the example shown in FIG. 2, the time interval of one chunk is set to 1 second, P is set to 2 seconds, and T is set to 4 seconds.

[0060] For future chunk requests, three different paths are shown in Figure 2, but the QoE evaluation should consider all available quality levels to be able to find the path that gives the highest QoE.

[0061] The process for calculating the QoE for a particular route is as follows: Each chunk in the path has a resolution r and a video bitrate b v , frame rate f, and audio bit rate b aThese correspond to different quality levels, characterized by:

[0062] Using the characteristics of each chunk of a particular path, the value of each of these quality-related parameters can be obtained every second.

[0063] Next, based on the QoE model in Non-Patent Document 6, the video quality is evaluated every second using Equation 1, the audio quality is evaluated every second using Equation 4, and the video and audio qualities are evaluated every second using Equation 5.

[0064]

number

[0065]

number

[0066]

number

[0067] In addition, in the formula 1, X and Y are defined by the formulas 2 and 3.

[0068]

number

[0069]

number

[0070] The temporal aggregation of the coding quality related scores per second into a score per path is given by Equations 6-9.

[0071]

number

[0072]

number

[0073]

number

[0074]

number

[0075] Finally, the effect of video outages is considered using Equation 10.

[0076]

number

[0077] In Formula 10, n b is the number of video outages that occurred on the considered route, and t b is the total time interval of video interruption, and a b is the average time interval between video feed outage events.

[0078] In these equations, v1-v7, a1-a3, m1-m4, t1-t5, and s1-s3 are model parameters obtained by using regression with a subjective experimental database.

[0079] If L is the number of quality levels and N is the number of chunks to download in the next T seconds, then L N It is necessary to compare the QoE of different routes.

[0080] Due to computational cost considerations, it is possible to apply restrictions to the evaluation of quality adaptations, thereby reducing the number of paths to be evaluated.

[0081] Only increases or decreases in quality by the A level may be allowed as candidates for optimization.

[0082] For example, in the example of FIG. 2, A=1 and at time t, the quality level of the previously requested chunk is 360p.

[0083] At time t+1, it may be possible to request only quality levels 480p, 360p, and 240p.

[0084] At time t+2, depending on the selection made at t+1, the quality value can range from 720p to 144p.

[0085] Such optimization results in smooth quality adaptation, which leads to higher QoE. N We narrow the search space to (2×A+1) while gaining the benefit of enforcing smooth quality adaptation, which would be selected even if a full search of the book path was performed. N It can be reduced to a book route.

[0086] In step S105 of FIG. 5, the bit rate control unit 12 N Among these paths, the path that is expected to give the highest QoE is selected. As described above, when only an increase or decrease in quality of level A is allowed as a candidate for optimization, the bit rate control unit 12 calculates (2×A+1) N Among these routes, the route that is expected to give the highest QoE is selected.

[0087] For the special case of video activation, no past throughput data is available, so it may not be possible to predict future throughput for the next T seconds. In this case, different alternatives can be considered.

[0088] 1) The bitrate control unit 12 may choose to start at the lowest quality level, which allows for a fast initial load.

[0089] 2) The bit rate control unit 12 may allow the user to select an initial quality value.

[0090] 3) The bit rate control unit 12 may estimate future throughput based on previous videos viewed by the user.

[0091] For example, if a user finishes watching a video and jumps to a new video, information about the available throughput measured in the previous video can be used to predict the throughput of the next video, allowing the video to start at an appropriate quality level.

[0092] Similarly, by using information such as the throughput measured during the loading of the web interface as clues, it is possible to predict the user's available throughput and select an appropriate initial quality level.

[0093] In step S106 of FIG. 5, the bit rate control unit 12 makes a chunk request via the communication unit 13.

[0094] In step S105, after selecting the path that is expected to provide the highest QoE, in step S106 the bit rate control unit 12 requests via the communication unit 13 a chunk of the quality level corresponding to the immediately following chunk included in the selected path.

[0095] For example, in FIG. 2, assume that the path resulting in QoE value QoE2 provides the highest QoE to the user.

[0096] In this case, the bit rate control unit 12 requests, via the communication unit 13, chunks corresponding to a quality level of 360p.

[0097] Thereafter, the communication unit 13 acquires the chunks of the quality level requested in step S106 via a network such as the Internet, and stores them in the buffer 14. As a result, the video playback device 10 can provide the user with video of higher quality and more stable quality.

[0098] In step S107 of FIG. 5, the bit rate control unit 12 determines whether the end of the video playback has been reached or whether the user has finished watching the video.

[0099] In step S107, if it is determined that the video playback has reached the end, or if it is determined that the user has finished watching the video (S107: YES), the bit rate control unit 12 ends the processing procedure.

[0100] In step S107, if it is determined that the end of the video playback has not been reached and that the user has not finished watching the video (S107: NO), the processing procedure returns to S101, and the throughput prediction unit 11 uses the throughput measured from chunks t-1 to t+1 to predict the throughput from t+2 to t+5.

[0101] In steps S102 to S105, the bit rate control unit 12 recalculates the optimal route between t+1 and t+5 based on the estimated throughput, and selects chunks to be downloaded at t+2.

[0102] This procedure repeats in a loop until the user stops watching or the end of the video is reached.

[0103] The above embodiment shows how long-term throughput prediction by time series analysis can be used in an adaptive bitrate control method for long-term chunk request scheduling, which can evolve traditional narrow-scope video playback devices into video playback devices that can predict throughput fluctuations and provide higher quality to users.

[0104] The above examples also show that it is possible to use state-of-the-art quality of experience models to assess the quality of a particular chunk selection path, using knowledge of the QoE model to define appropriate quality adaptation procedures, which in turn improve the user's quality of experience.

[0105] Additionally, in the above examples, in order to reduce computational complexity, it has been shown that quality adaptation can be limited to increases or decreases that do not exceed a user-defined threshold, thereby enforcing temporal smoothness of quality while significantly reducing computational cost.

[0106] In the following, we provide details of an example optimization procedure for identifying routes that achieve high QoE while maintaining reasonable computational costs.

[0107] FIG. 6A illustrates an example of a user watching video in a car. At some point, the user enjoys high-quality network access and is able to watch high-quality video. However, when the car enters a tunnel, the available throughput drops significantly. In this context, a traditional "narrow-minded" video player has no choice but to significantly reduce video quality to avoid stalling when the video player measures the throughput drop. However, if the video player had anticipated this level of throughput drop, it could identify a better solution. For example, as shown in FIG. 6B, by using the available high throughput before entering the tunnel to accelerate rebuffering, high-quality video can be maintained even while the car is in the tunnel with low throughput.

[0108] To apply this principle, the video playback device must compare the quality of the results after executing different quality adaptation patterns at each step and select the sequence of chunks that provides the highest quality to the user. This sequence of chunks is hereafter referred to as the quality adaptation path. This process is illustrated in Figure 2. At time t, the video playback device must decide which chunks to request. The video playback device then uses long-term throughput predictions to estimate the results of different quality adaptation paths. While Figure 2 compares the quality of three paths, identifying the optimal quality adaptation requires comparing multiple paths to identify which one provides the highest QoE. Figure 7 illustrates the case where all possible quality adaptation paths are evaluated, with the path providing the highest QoE highlighted by a dotted line. This Figure 7 also illustrates the need to consider multiple paths to identify the path with the highest quality. In an exhaustive search approach, the number of paths evaluated is L D where L is the number of quality levels and D is the path length, i.e., the number of future chunks. The above method introduces a computationally less intensive search by only considering quality adaptation limited within a certain range A. For example, if A=1, the video playback device can increase or decrease the quality by one level or maintain the same quality as in the previous step. As shown in Figure 8, this significantly reduces the number of paths to be compared, resulting in a number of paths to be compared of (2A+1) D This becomes:

[0109] Figure 9 shows the simulation run time of an experimental example. It can be seen that there is an exponential relationship between the run time value and the depth value (i.e., the number of future chunks) used in the optimization procedure. Moving from full search to bounded adaptation allows us to reduce the number of paths to be compared as the depth increases, but the computational complexity remains high, making it impractical to consider very large depths when optimizing the operation of a video playback device while playing a video streaming session.

[0110] The proposed algorithm runs on the device and aims to define a long-term chunk request scheduling procedure. Similar to traditional ABR control methods, the proposed algorithm takes as input a media presentation description (MPD), which provides information about the available quality level of the video. Figure 10 shows the main steps of the proposed method. First, the time variation of throughput up to T seconds is predicted. Next, a path that provides the user with a higher QoE for the next T seconds is identified. This path is represented by the dotted line in Figure 10. Third, the first chunk of the identified optimized path is requested. Then, the algorithm returns to the first step and re-estimates the time variation of throughput up to T seconds. The algorithm performs this procedure in a loop until the video playback ends or the user finishes watching the video.

[0111] In the simulations conducted, a T value of 20 seconds significantly improved quality, but other values ​​may also be considered. The results show that a larger T value leads to better quality. However, increasing T increases computational costs. Considering that performance improvements mostly occur under difficult network conditions, various presets with different T values ​​may be used depending on the distribution of network throughput. Service providers should then predefine various T values ​​and use larger T values ​​when network conditions deteriorate and change significantly. For example, when throughput is stable, T may be kept constant at 8 seconds. However, when the network changes significantly, such as for users using public transport, T may be increased up to 20 seconds. This allows for maintaining the computational cost and battery life of mobile devices when network conditions are stable, while achieving higher quality at a higher computational cost when network conditions become difficult.

[0112] In the following, the different steps of the proposed method are described.

[0113] Step 1: Throughput Forecasting. Throughput forecasting is a challenging task and has attracted much attention. Different techniques have been used to predict future throughput using averages, either arithmetic mean, harmonic mean, deep neural networks, autoregressive approaches, or hidden Markov models. However, only the immediate future throughput is predicted. Therefore, in this study, we propose to leverage research on time series forecasting to forecast throughput over a larger time frame. Various approaches can be considered, including the use of autoregressive models like ARIMA or advanced learning-based techniques. Below, a Transformer-based deep neural network throughput forecasting algorithm is proposed, although alternative models are also possible. Using this model, past throughput values ​​are used to predict future throughput for the next T seconds.

[0114] Step 2: Scheduling long-term chunk requests. The core principle of long-term chunk request scheduling is illustrated in Figure 2. In this figure, we can see that the video is available at six different quality levels, ranging from 144p to 1080p. The line segment from t-2 to t indicates the temporally previous chunk selected by the ABR rate control model. At this stage, the quality of the next chunk, t+1, needs to be selected. For this purpose, different paths of quality adaptation for the next T seconds are considered and the path offering the highest QoE is selected.

[0115] To evaluate the QoE of a given path, we use the model described in Non-Patent Document 6. However, other QoE models, such as those described in ITU-T Rec. P.1203 or ITU-T Rec. P.1204, can also be used. These standardized models are available in various modes using different types of information, from low-complexity models that use only parameters such as bit rate, frame rate, and resolution, to high-complexity models that also use bitstream or pixel information. The QoE of a given path is predicted based on the previous P seconds and the following T seconds. Taking previously downloaded chunks into account can address the temporal smoothness aspect of quality. In the example shown in Figure 2, the chunk duration is set to 1 second, and P and T are set to 2 and 4 seconds, respectively. Regarding future chunk requests, Figure 2 shows three different paths to illustrate the general long-term chunk selection mechanism. However, it is important to identify which path needs to be evaluated. This will be discussed later.

[0116] Regarding the calculation process of the QoE of a given path, the process is as follows: all chunks in the path correspond to different quality levels that can be characterized by resolution r, video bitrate bv, frame rate f, and audio bitrate ba. Using the characteristics of each chunk from a given path, it is possible to obtain the second-by-second values ​​of each of these quality-related parameters. Then, based on the model in Non-Patent Document 6, the video, audio, and audiovisual quality are estimated second-by-second using Equations 1, 4, and 5, respectively. The temporal aggregation of the coding quality-related scores per second into a QoE score per path is given by Equations 6-9. Finally, the effect of video stalls is considered using Equation 10. In this equation, n b is the number of video stops that occurred on the path under consideration, t b is the total video stop time, a b is the average time between video stop events. In these equations, v 1-7 , a 1-3 , m 1-4 , t1-5 , s 1-3 are the model parameters obtained using regression with the subjective experimental database.

[0117] When searching for a path that provides the best quality adaptation pattern, it is important to consider an appropriate search procedure. If the video is available at L quality levels and paths consisting of D chunks are considered, the total number of comparable paths is L. D Since the number of paths increases exponentially as the path length increases, an optimized search procedure is required.

[0118] The general idea of ​​the proposed method is shown in Figure 11. In this figure, the dependencies between chunks are shown. In this figure, we can see that the throughput varies as a function of time. Therefore, depending on when the chunk download starts, the time it takes to download a chunk may vary. The "playback" line in Figure 11 corresponds to the constraints of the video player to play the video without interruption. The playback line includes C 1、 C 2、 C 3、 and C4, which are chunks C and C6 that the video player needs to play back the video continuously. 1、 C 2、 C 3、 The figure shows the time when C1 and C4 need to be available. Path exploration lines indicate dependencies between consecutive chunks. Path exploration (1) downloads chunk C1 at quality level Q0. To ensure continuous playback, chunk C2 must be downloaded before the time to play C2 begins. Based on available throughput, chunk 2 (C2) may be downloaded at quality level 0 (C2Q0) or quality level 1 (C2Q1). Quality level 2 (C2Q2) cannot be selected because it arrives too late. Quality values ​​higher than Q2 simply require more bits, and all paths using quality 2 or higher for the second chunk may be ignored.

[0119] In the first step, there are only two possibilities: (C2Q0) and (C2Q1). First, the proposed algorithm selects the chunk with the highest quality that can arrive on time: (C2Q1). This corresponds to path search line (2.1) in Figure 11. If this chunk is selected, then for the third chunk, the next highest quality chunk that can be downloaded is (C3Q1), and for the fourth chunk it is (C4Q2).

[0120] However, requesting the highest quality chunk available for download does not necessarily mean that the highest quality is provided to the user. In fact, in the pathfinding example (2.2), given the low available throughput when downloading chunk C2, requesting a lower quality for chunk C2 allows it to complete its download sooner, and allows for the download of higher quality chunks for chunks C3 and C4 (C3Q2 and C4Q3, respectively). This results in an overall improved user experience.

[0121] Therefore, the proposed search procedure consists of two steps. 1) Identifying the sequence of highest quality chunks that can be downloaded based on available throughput and time constraints for uninterrupted video playback. 2) Search for alternative paths around the initially identified path by increasing or decreasing the quality of the requested chunk.

[0122] Figure 12 shows the candidate paths considered using the proposed method. First, the "Path immediately under available throughput and buffer-filling rate" path displays the sequence of the highest quality chunks that can be downloaded on time for smooth playback. Then, other different candidate quality adaptation paths are considered by decreasing or increasing the chunk quality compared to the first identified path.

[0123] The range of quality around the initially identified route is constrained by the parameter M. A value of M=1 indicates that the chunk quality can be increased or decreased by one level around the original route. This allows reducing the search space while still finding the optimal route. If L is the number of available quality levels, a value of M=L will result in searching all possible quality-matching routes.

[0124] Another refinement is to constrain the consideration of quality adaptation between adjacent chunks within the search space around the original path, allowing only quality changes up to a specified level (A).

[0125] Based on this search procedure, the upper bound on the number of paths that can be considered is given in Equation 11. It is worth noting that in this equation, the number of paths increases linearly with the path length, rather than exponentially as before. This improvement allows the ABR control scheme to consider long paths.

[0126]

number

[0127] Step 3: Chunk Request. Once the optimal path is identified, the chunk immediately following this optimal path is requested. For example, in Figure 2, assume that the path with a QoE value of QoE2 provides the highest QoE to the user. The chunk corresponding to quality level 360p is requested. The measured throughput from chunks t-1 to t+1 is then used to predict the throughput from t+2 to t+5. Based on the estimated throughput, the optimal path is recalculated between t+2 and t+5, allowing the selection of the chunk to be downloaded at t+2. This process is repeated in a loop until the user stops watching or the video playback device reaches the end of the video.

[0128] The above examples show that the computational complexity of long-term adaptive bit rate control mechanisms can be improved by considering throughput information that identifies a subset of paths to evaluate.

[0129] According to the above embodiment, a "base path" describing the sequence of highest quality chunks that can be downloaded given the available throughput is an efficient starting point for a long-term adaptive bitrate control mechanism.

[0130] In the above embodiment, the path length can be made dependent on network conditions, reducing the computational cost and energy consumption of the long-term ABR control model when network throughput is stable, while increasing the computational cost and energy consumption when network conditions become difficult, such as in the case of a user on public transport.

[0131] In the above-described embodiment, a fast method for finding the sequence of chunk qualities required to provide a high quality of service to a user has been described. Previous research has shown that long-term throughput prediction can improve a video playback device's quality of experience by enabling it to predict poor network conditions and take proactive action, rather than simply reacting to poor network conditions observed recently. Long-term ABR control mechanisms optimize quality adaptation paths instead of optimizing chunk selection on a chunk-by-chunk basis. However, as the length of the quality adaptation path increases, the number of possible quality adaptation paths increases exponentially with the number of chunks in the path. The proposed method leverages long-term throughput prediction to identify reasonable quality adaptation paths and reduce the computational complexity of the long-term chunk selection method. This ultimately enables a video playback device to predict network throughput losses earlier and improve quality of service.

[0132] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims.

Claims

1. A video playback device, Transmitting section; Receiver; a processor; and memory, Equipped with The memory may include: using the past throughput values ​​to predict future throughput for a first time interval in the future that includes m unit time intervals in the future; establishing a search space for finding a quality adaptation schedule, the search space being defined by k quality levels and a second time interval including a current unit time interval, the m future unit time intervals, and n past unit time intervals; specifying a time limit for acquiring each chunk among the m chunks corresponding to the m future unit time intervals, for playing back video without interruption during the m future unit time intervals; identifying, for each chunk among the m chunks, a highest quality level among one or more quality levels at which the chunk can be downloaded by the time limit based on the future throughput; identifying a first path in the search space that corresponds to the identified m highest quality levels; Identifying one or more second routes by changing one or more highest quality levels among the m highest quality levels included in the first route; estimating a quality of experience value for the first path and each of the one or more second paths based on the future throughput; identifying a route corresponding to the highest quality of experience value among the estimated quality of experience values ​​as the quality adaptation schedule; determining a quality value of a chunk to be requested in a future unit time interval next to the current unit time interval based on the quality adaptation schedule; requesting a chunk of the determined quality value via the sender; and receiving a chunk of said determined quality value via said receiver; including instructions to execute Video playback device.

2. the step of identifying the one or more second paths limits a range of change of the one or more highest quality levels among the m highest quality levels to a predetermined range; 2. The video playback device according to claim 1.

3. and when changing the one or more highest quality levels among the m highest quality levels, the step of identifying the one or more second paths includes changing the one or more highest quality levels such that a difference in quality levels between two adjacent chunks among the m chunks is within A levels, where A is an integer equal to or less than k.

2. The video playback device according to claim 1.

4. and in the step of predicting the future throughput, in the case of video activation, applying a throughput of another video that has been previously viewed or a preset low throughput as the future throughput.

2. The video playback device according to claim 1.

5. the step of estimating a quality of experience value comprises evaluating the quality of experience value using a predetermined quality of experience model; 2. The video playback device according to claim 1.

6. 1. A computer-implemented video playback method, comprising: using the past throughput values ​​to predict future throughput for a first time interval in the future that includes m unit time intervals in the future; establishing a search space for finding a quality adaptation schedule, the search space being defined by k quality levels and a second time interval including a current unit time interval, the m future unit time intervals, and n past unit time intervals; specifying a time limit for acquiring each chunk among the m chunks corresponding to the m future unit time intervals, for playing back video without interruption during the m future unit time intervals; identifying, for each chunk among the m chunks, a highest quality level among one or more quality levels at which the chunk can be downloaded by the time limit based on the future throughput; identifying a first path in the search space that corresponds to the identified m highest quality levels; Identifying one or more second routes by changing one or more highest quality levels among the m highest quality levels included in the first route; estimating a quality of experience value for the first path and each of the one or more second paths based on the future throughput; identifying a route corresponding to the highest quality of experience value among the estimated quality of experience values ​​as the quality adaptation schedule; determining a quality value of a chunk to be requested in a future unit time interval next to the current unit time interval based on the quality adaptation schedule; requesting chunks of said determined quality values; and receiving chunks of said determined quality values; A video playback method comprising:

7. A non-transitory computer-readable storage medium storing a video playback program, wherein when the video playback program is executed by a computer, the computer: using the past throughput values ​​to predict future throughput for a first time interval in the future that includes m unit time intervals in the future; establishing a search space for finding a quality adaptation schedule, the search space being defined by k quality levels and a second time interval including a current unit time interval, the m future unit time intervals, and n past unit time intervals; specifying a time limit for acquiring each chunk among the m chunks corresponding to the m future unit time intervals, for playing back video without interruption during the m future unit time intervals; identifying, for each chunk among the m chunks, a highest quality level among one or more quality levels at which the chunk can be downloaded by the time limit based on the future throughput; identifying a first path in the search space that corresponds to the identified m highest quality levels; Identifying one or more second routes by changing one or more highest quality levels among the m highest quality levels included in the first route; estimating a quality of experience value for the first path and each of the one or more second paths based on the future throughput; identifying a route corresponding to the highest quality of experience value among the estimated quality of experience values ​​as the quality adaptation schedule; determining a quality value of a chunk to be requested in a future unit time interval next to the current unit time interval based on the quality adaptation schedule; requesting chunks of said determined quality values; and receiving chunks of said determined quality values; A non-transitory computer-readable storage medium for executing the method.

Images