Streaming media qoe prediction method based on convolutional recurrent neural network under attention mechanism

Abstract

The application discloses a streaming media QoE prediction method based on a convolution recurrent neural network under an attention mechanism, and has the characteristics that the method comprises two methods, one of which is a streaming media QoE prediction method based on a convolution recurrent neural network under an attention mechanism for solving time sequence QoE prediction. Four input factors are relied on for continuous QoE prediction, namely, short-term subjective quality STSQ, current whether stuttering PI, current bit rate BITRATE and rebuffering time T R ; the other is a global QoE prediction method based on a BP network combining subjective and objective factors for verifying the universality of input indexes and solving global QoE prediction. CNN is used to capture context information and short sequence information, GRU is used to capture global long sequence dependency, and Attention is used to capture key information, so that the accuracy of prediction is improved. The CNN with parallel processing is added to improve the training speed. The current bit rate, an objective QoS index, is introduced and mixed with PI to reduce the calculation complexity, and the current bit rate can better reflect the actual network environment than the commonly used PI.

Description

Technical Field

[0001] This invention relates to the field of user experience quality prediction and analysis technology, and in particular to a streaming media QoE prediction method based on convolutional recurrent neural networks under the attention mechanism. Background Technology

[0002] In recent years, with the continuous development of 5G communication and network technologies, network operation models have successfully shifted from single voice services to multi-service models. Streaming media has become a key technology for future data networks, significantly impacting people's lives. The types and scale of streaming media services such as online teaching and video conferencing have grown unprecedentedly, profoundly affecting people's work and learning methods. Faced with the massive and complex demands of streaming media services, it is clear that traditional network performance indicators are insufficient to assess user satisfaction with services, and Quality of User Experience (QoE) has become a widely recognized and important standard for evaluating the quality of streaming media services.

[0003] QoE is an evaluation method based on user satisfaction, directly reflecting users' level of approval of a service. Many factors influence user QoE, primarily encompassing three levels: service, user, and environment, covering both subjective and objective factors. However, current predictive models are not yet perfect.

[0004] For example, in 2012, Lin et al. discussed models and evaluation methods for user experience quality based on statistics, psychology and artificial intelligence. They proposed a hidden Markov-based user experience quality model in the context of video streaming and established an evaluation method based on a stochastic model, but it was not put into practice.

[0005] CGBampis et al. used the Hammerstein-Wiener model to model time-varying subjective quality (TVSQ). Zhou, Wang et al. also only considered subjective factors and did not consider objective QoS factors [7][8][9].

[0006] In 2017, Yao et al. established a global QoS-QoE mapping model based on BP neural network, without considering subjective factors.

[0007] While LSTM-QoE, proposed in 2020, considers both subjective and objective factors, it only selects PI as the objective factor, neglecting bit rate variations. In terms of network implementation, it also uses only LSTM networks, whose serial nature and chain structure result in high computational complexity. Meanwhile, CNN-QoE, proposed by TNDuc in the same year to solve the temporal QoE task, does not differ from the above in its choice of input metrics, but it uses a Temporal Convolutional Network (TCN). Although it solves the computational cost problem, it only performs well on specific datasets. Summary of the Invention

[0008] This invention establishes two prediction models for the User Experience Journey (QoE): a time-series model and a global model, to meet prediction needs under different conditions.

[0009] The adopted technical solution includes two models, one of which is a streaming media QoE prediction method based on a convolutional recurrent neural network with an attention mechanism to solve the problem of time series QoE prediction. The attention mechanism, proposed by Google Mind, allows the model to focus on and fully learn and absorb important information, and it has also shown good performance in time series prediction models. This model relies on four input factors for continuous QoE prediction: short-term subjective quality (STSQ), current buffering status (PI), current bit rate (BITRATE), and rebuffering time (T). R Another method for global QoE prediction is based on a BP network, which combines subjective and objective factors to verify the universality of the input indicators and solve the problem of global QoE prediction.

[0010] A streaming media QoE prediction method based on a convolutional recurrent neural network under an attention mechanism is proposed. The method includes the following steps: Step 1, generating a video QoE database with subjective and objective data. A database containing subjective and objective QoE influencing factors is selected, specifically the LFQVIA database. Each video is 120 seconds long, and the current bitrate (BITRATE), current STSQ, current buffering (PI), and the time elapsed since the last buffering (T) are recorded per second. R The data, along with the current QoE value, forms five matrices of length 120. STSQ uses readily available Video Quality Assessment (VQA) metrics for measurement, including Peak Signal-to-Noise Ratio (PSNR), SSSIM, MS-SSIM, STRED, and NIQE. Experiments showed that NIQE performed best in prediction results, therefore it was chosen as the STSQ metric.

[0011] Step 2: Re-encode some of the indicators.

[0012] Step 2.1: Since PI is represented as logic 1 when BITRATE is 0 and 0 in other cases, PI and BITRATE are mixed and encoded as BIT-PI, that is, the value of BITRATE is normalized, and if it is 0, it is set to 1, otherwise its reciprocal is taken.

[0013] Step 2.2: Normalize all input and output metrics to the range of 0-1.

[0014] Step 3: Train the real-time QoE prediction network and test it synchronously using cross-validation.

[0015] Step 3.1: A 3*120 matrix is ​​formed by combining one subjective influencing factor (STSQ) and two objective factors (TR and BIT-PI) as input variables, and the real-time QoE value is used as the output variable. The input variables are batched into the network, with each batch having a size of [1, 4, 3].

[0016] Step 3.2: Construct a streaming QoE prediction network based on a convolutional recurrent neural network under the attention mechanism, such as... Figure 1 As shown.

[0017] Step 3.3: Input the input variables from Step 2.1 into the C-GRU network in batches, with the corresponding QoE values ​​as labels, and train using S-fold cross-validation.

[0018] Step 4, evaluate the network

[0019] The QoE prediction performance of the model is quantified by the average of the following five metrics: linear correlation coefficient (LCC), Spearman order correlation coefficient (SROCC), normalized root mean square error (RMSE), over-the-line rate (OR)

[20] , and time taken to run (TIME). LCC and SROCC provide a quantification of the correlation between the predicted QoE score and the true QoE score. RMSE and OR measure the closeness between the predicted score and the true score. OR is the probability of exceeding twice the confidence space. In addition, in order to achieve high accuracy while keeping computational costs low, the time TIME for the model to perform one training and prediction operation on this dataset is included.

[0020] A global QoE prediction method based on BP networks and combining subjective and objective factors includes the following steps:

[0021] Step 1: Generate a video QoE database containing both subjective and objective data.

[0022] Step 2: Train the global QoE prediction network. The network is constructed using the rebuffering count, average rebuffering time, average bit rate, and MS-SSIM of each of the 36 video segments as input factors, and the global QoE value of each video segment as the output factor.

[0023] Step 2.1: Divide all data into training and testing groups in an 8:2 ratio.

[0024] Step 2.2: Construct a BP neural network with a learning rate of 0.05 and 10 hidden layers.

[0025] Step 2.3: The input metrics from Step 2.1 are arranged into a 4*36 matrix and input into the BP network. The global QoE of size 1*36 is used as the label for training.

[0026] Step 3: Test the global QoE prediction network.

[0027] Step 3.1: Input the remaining 20% ​​of the data into the network as the test group.

[0028] Step 3.2, compare the Pearson correlation coefficient (LCC), Spearman correlation coefficient (SROCC), and the trend line fit index R. 2 The output is used as an evaluation metric for the network; some results are as follows: Figure 2 As shown.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] 1. A continuous QoE prediction model, CGRU-QoE, based on an attention mechanism, is proposed. CNN can capture short-sequence contextual information, GRU can capture global long-sequence dependencies, and attention can capture key information. The combination of these three improves prediction accuracy. Furthermore, because the serial nature and chain structure of LSTM networks result in high computational complexity, the approach of using a single LSTM network is abandoned, and a parallel CNN is added to improve training speed.

[0031] 2. To address the issue that previous predictions rarely considered QoS metrics, we introduced the current bit rate as an objective QoS metric and co-encoded it with PI to reduce computational complexity. This approach better reflects the actual network environment than the commonly used PI.

[0032] 3. The selected input metrics were applied to global QoE prediction, and the constructed CGRU-QoE network was applied to multiple datasets, all of which yielded good results, proving that the model has universality. Attached Figure Description

[0033] Figure 1 This is the overall architecture of CGRU-QoE.

[0034] Figure 2 It achieves excellent performance for global QoE prediction methods. Detailed Implementation

[0035] To better understand the purpose, structure, and function of this invention, the following detailed description of the streaming media QoE prediction method based on convolutional recurrent neural networks under the attention mechanism of this invention is provided in conjunction with the accompanying drawings.

[0036] like Figure 1 The image shows the overall architecture of CGRU-QoE.

[0037] like Figure 2 The results show that the global QoE prediction method achieves excellent performance.

[0038] The streaming QoE prediction method based on convolutional recurrent neural networks under the attention mechanism provided in this embodiment includes the following steps:

[0039] Training the network:

[0040] Step 1: Process the information in the QoE database. Since QoE is composed of both subjective evaluation and objective factors, select the current BITRATE, current STSQ, current PI (pixel lag), and the time T since the last lag. R In particular, the current network conditions also contribute significantly to QoE, as the current bit rate is incorporated into the prediction. All the above data is derived from every second of information within each 120-second video.

[0041] Step 2: The data is re-encoded. To reduce time complexity, the current BITRATE and whether there is a lag are combined and encoded into an index BIT-PI. That is, the value of BITRATE is normalized. If it is 0, it is set to 1. If it is not 0, its reciprocal is taken.

[0042] Step 3: Normalize all input and output metrics using the following formula: Where X represents the original data.

[0043] At this point, the normalized input index X nom ={BIT-PI, STSQ, T} R The output index Y = {QoE_T} is the real-time QoE value, which ranges from 0 to 1.

[0044] Step 4, establish the CGRU-QoE network

[0045] The first layer of the network is a CNN layer, used to extract short-order dependencies and contextual information. It contains a convld layer with a filter size of 64 and a kernel size of 1, and a maxpooling layer with a maximum pooling window of 1.

[0046] The second and third layers are two GRU layers with 22 units each and a dropout rate of 0.5 to extract long-order dependencies.

[0047] Next is an Attention layer, used to extract key information, and its calculation method is as follows: Where α n The nth factor x in the input vector X of this layer n The level of attention received;

[0048] The fifth layer passes through a Dropout layer with a dropout rate of 0.5 to randomly discard items and prevent overfitting; finally, it passes through a fully connected layer followed by a sigmoid function. Output.

[0049] Step 5: Train the CGRU-QoE network.

[0050] Training is performed using S-fold cross-validation. That is, in each round, S-1 portions are randomly selected as the training set, and the remaining portion is used as the test set. After this round, S portions are randomly selected again for training. After several rounds (less than S), a loss function is chosen to evaluate the optimal model and parameters.

[0051] The training step size is set to 4, the batch size to 1, and no_feat to 3. The data is divided into multiple batches of size [1, 4, 3] and input into the network to obtain the predicted scores.

[0052] Step 6, according to the formula Calculate the Pierce correlation coefficient (PCC). (where n is the number of samples) Calculate the Spearman correlation coefficient (SROCC). Calculate the normalized root mean square error (RMSE), determine the out-of-line rate (OR) based on the probability of exceeding twice the confidence interval, and record the training run time as TIME. Use this as the network evaluation metric.

[0053] The experimental results of applying this invention are given below.

[0054] To evaluate the correctness of the added input indicator BITRATE, the experimental results before and after its addition were compared, as shown in Table 1. The method after its addition is superior.

[0055] Table 1. Comparison of results before and after adding BITRATE

[0056] Before After LCC 0.845 0.861 SROCC 0.785 0.805 RMSE(%) 8.261 7.930 OR(%) 9.145 8.976

[0057]

[0058] To evaluate the prediction results of CGRU-QoE, different VQA methods were selected, and the average values ​​of LCC, SROCC, RMSE, and OR were used as the basis for cross-model comparison. The optimal results of each model are shown in bold. As can be seen from the table below, CGRU-QoE has the advantage.

[0059] Table 2. Horizontal comparison of various continuous QoE models

[0060]

[0061] A comparison of the computational complexity of LSTM-QoE and CGRU-QoE shows that the present invention saves more computational resources.

[0062] Table 3. Comparison of computational complexity

[0063]

[0064]

[0065] To demonstrate the universality of CGRU-QoE, it was also validated on the LIVE QoE dataset (containing only subjective metrics) and the LIVE Mobile Video Stall Database-II dataset (containing only objective metrics such as rebuffering). CGRU-QoE performed better on the Mobile Video Stall Database-II dataset, while LIVE QoE showed improvements in all performance metrics except for a slight decrease in the correlation coefficient. The data proves that CGRU-QoE is applicable to various dataset scenarios.

[0066] Table 4. Comparison of LIVEQoE prediction results

[0067] QoE Model LCC SROCC RMSE(%) OR(%) TIME(S) LSTM-QoE 0.892 0.889 10.76 21.42 3032.81 CGRU-QoE 0.889 0.890 4.48 9.22 2416.43

[0068] Table 5. Comparison of prediction results from LIVE Mobile Video Stall Database-II

[0069] QoE Model LCC SROCC RMSE(%) OR(%) TIME(s) LSTM-QoE 0.869 0.854 9.95 35.74 132761.6 CGRU-QoE 0.894 0.889 6.87 31.75 143150.1

[0070] Prediction Instance

[0071] Step 1: Select a 120-second video and record the current bitrate (BITRATE), whether there is a stutter (PI), and the time elapsed since the last stutter (T) every second. R , so that according to the formula (Where v1 and Σ1 are the mean vector and covariance matrix obtained from a set of clear images, and v2 and Σ2 are the mean vector and covariance matrix obtained from the input low-quality images) to obtain NIQE.

[0072] Step 2: If BITRATE = 1, then BIT_PI = 0; otherwise... All indicators according to the formula Normalize the matrix and form a 5*36 matrix X_pre.

[0073] Step 3: Input the matrix X_pre into CGRU-QoE, and you will get a predicted value Y_pre between 0 and 1.

[0074] Step 4: Denormalize it to obtain its predicted QoE value.

Claims

1. A streaming media QoE prediction method based on convolutional recurrent neural networks under an attention mechanism, characterized in that, Includes the following steps, Step 1: Process the information in the QoE database. QoE is composed of both subjective evaluation and objective factors. Select the current BITRATE, current STSQ, current PI lag, and the time elapsed since the last lag. The current network conditions contribute to QoE, and are included in the current bit rate for prediction; the above QoE data comes from information from every second of each 120s video. Step 2: Perform secondary encoding on the QoE data, combining the current BITRATE and whether there is lag into an index BIT-PI, that is, normalize the value of BITRATE, set it to 1 if it is 0, and take its reciprocal if it is not 0. Step 3: Normalize all input and output metrics using the following formula: Where X represents the original data; the normalized input index ={BIT-PI,STSQ, The output metric Y={QoE_T} is the real-time QoE value, which ranges from 0 to 1; Step 4: Establish the CGRU-QoE network; The first layer of the CGRU-QoE network is a CNN layer used to extract short-order dependencies and contextual information; it includes a conv1d layer with a filter size of 64 and a kernel size of 1, and a maxpooling layer with a maximum pooling window of 1. The second and third layers are two GRU layers with 22 units each and a dropout rate of 0.5 to extract long-order dependencies; Next is an Attention layer, used to extract key information, and its calculation method is as follows: ,in The vector input to this layer The nth factor The level of attention received; The fifth layer is randomly dropped by a Dropout layer with a dropout rate of 0.5 to prevent overfitting; Finally, the sigmoid function is passed through the fully connected layer. Output; Step 5: Train the CGRU-QoE network; Training is performed using S-fold cross-validation: each time, S-1 parts are randomly selected as the training set, and the remaining part is used as the test set; after one round, S parts are randomly selected again to train the data; after several rounds, a loss function is selected to evaluate the optimal model and parameters. The training step size is set to 4, the batch size to 1, and no_feat to 3. The data is divided into multiple batches of size [1, 4, 3] and input into the network to obtain the predicted scores. ; Step 6, according to the formula Calculate the Pierce correlation coefficient (PCC). Where n is the number of samples. = Calculate the Spearman correlation coefficient (SROCC). Calculate the normalized root mean square error, calculate the overshoot rate (OR) based on the probability of exceeding twice the confidence interval, and record the training run time as TIME, which serves as the network evaluation metric.

Images