Ultra-high-definition streaming media experience quality perception method

Abstract

The invention discloses an ultra-high-definition streaming media experience quality perception method, and provides a QoE index integrating video terminal playing equipment quality evaluation, video picture quality evaluation and service network quality evaluation, namely an MUSQM model, based on characteristics of ultra-high-definition streaming media service and problems existing in an existing QoE perception model. And the MUSQM model dynamically perceives the experience quality of the user in different service scenes through a deep learning model. Three video perception quality parameters including two subjective parameters and one objective parameter are used, and comprehensive and accurate QoE perception is realized in a subjective and objective combination mode. Meanwhile, a part of dynamic parameters used for describing a historical service state and a current service state are introduced, so that the real-time performance and memorability are achieved in the time sequence. With the development of deep learning DL, the method has the end-to-end learning capability and excellent performance in processing large-scale and high-dimensional complex problems, so that the method has the possibility of being applied to the QoE perception problem.

Description

Technical Field

[0001] The present invention relates to a method for perceiving quality of experience, and in particular to a method for perceiving quality of experience of ultra-high-definition streaming media, and belongs to the technical field of video streaming media service networks. Background Art

[0002] With the rapid development of network technology in recent years, more and more consumers choose 5G as the network service in their lives. Among all network services, the network traffic consumed by streaming-related services accounts for more than 60% of the total service traffic. The continuous advancement of video technology has enabled ultra-high-definition video to gradually expand from professional fields to wider fields and gain wider applications. With the expansion of the market, users' demand for streaming services is also changing. The emergence of technologies such as adaptive bitrate algorithm (ABR) and high dynamic range (HDR) has made users' requirements for ultra-high-definition streaming services change faster than ever before. However, ultra-high-definition streaming services not only require high-quality video sources, but also require network operators to ensure the quality of network services during video transmission. For mobile network operators and mobile service providers, how to more efficiently and accurately understand how users experience video quality will help meet the ever-increasing user expectations and expand the number of ultra-high-definition video service users, which will bring huge benefits to mobile network operators and mobile service providers. Therefore, it is very necessary to develop an accurate and efficient ultra-high-definition streaming user experience quality model for operators to improve services and meet user needs.

[0003] Regarding how to improve the quality of video services according to user needs, the main challenge is how to perceive the user experience level based on the QoS data held by mobile network operators and mobile service providers. The commonly used experience level quantification standard is the mean opinion score (MOS) developed by ITU-T. The MOS rating standard divides user experience into five levels, and users rate the video based on their personal subjective feelings about the video quality, which represents the user's experience quality of the video. At the same time, ITU-T also provides a model structure from video stream to terminal MOS in the P.1203 standard to guide the construction of the perception model.

[0004] Existing QoE perception methods are mainly divided into subjective perception methods and objective perception methods. In the study of subjective perception methods, a method for applying subjective perception of user experience quality in multimedia scenarios is given. A video quality assessment method based on crowdsourcing is introduced, which is different from the traditional subjective perception quality research in the laboratory. Although the final score of the subjective perception method is directly given by the end user, which makes it more persuasive and credible, it also has disadvantages. A large number of subjects are required in subjective research, and the same video needs to be repeatedly rated. This makes it have a large time cost and personnel cost, and is not suitable for large-scale multi-sample research. Therefore, researchers began to pay attention to objective perception methods. This type of method mainly constructs a functional relationship between it and the user terminal experience quality through network and video data, so as to reduce the perception cost while calculating the terminal QoE score with high credibility.

[0005] In the study of objective perception methods, according to the characteristics of services such as video on demand (VoD) and interactive Internet TV (IPTV), a model for calculating user terminal QoE based on network QoS parameters is given. In the prior art, the impact of the initial delay, freeze and bit rate of the video on the user terminal experience in the dynamic adaptive streaming media service (DASH) is analyzed, and an overall user experience model is formulated for DASH video based on the impact of each parameter on the terminal. With the development of artificial neural networks (ANN), researchers select different QoS parameters according to the video scenes of different videos and use ANN to build perception models. It enables operators to obtain guidance information from the perception model, thereby improving network quality and user QoE.

[0006] Based on parameter selection, QoE perception models can be divided into four categories: image fidelity-based models, application QoS-based models, network QoS-based models, and hybrid parameter models. Among the models based on signal fidelity, VQM and SSIM have been widely used. This type of model evaluates the fidelity of the video based on the brightness, contrast, and structure of the video image, so it pays more attention to the video image itself than other models. The application QoS-based model mainly perceives QoE based on video playback QoS parameters such as the duration and number of freezes. This type of model pays less attention to the video image itself than the previous model. The network QoS-based model focuses on analyzing the impact of network quality on terminal video quality, and does not consider the proportion of video playback QoS parameters such as the number of freezes and terminal image quality in QoE prediction. It mainly predicts terminal QoE based on network QoS parameters such as network bandwidth, packet loss rate, and jitter during video playback. Compared with other models, this type of model aims to build a direct mapping relationship between network QoS parameters and user QoE. The hybrid model actually combines the first two models, introducing QoS parameters of video playback while evaluating the picture quality. The hybrid model has a better QoE perception effect than the other three models.

[0007] With the continuous development of consumer electronics, the devices used by users to enjoy ultra-high-definition video services are no longer limited to TVs and computer monitors. Mobile phones and tablets have gradually become the main terminal devices for users. ITU-R pointed out in Recommendation BT.500-14 that there is evidence that the size of the screen used to watch videos may affect the subjective evaluation results. Therefore, it is necessary to incorporate the parameters describing the characteristics of user terminal devices into the QoE model system for accurate perception of user experience. Most existing QoE perception models are based on the construction of QoE models based on complete video service data sets, which results in the parameters used in these models including indicators recorded after the service ends, such as the number of video freezes and the duration of freezes. In the application scenario of ultra-high-definition video, in order to provide a better user experience, operators need to perceive the QoE of user terminals in real time during the service and optimize it. This requires the perception model to calculate QoE using dynamic parameters and time series structure. In summary, the current QoE model is not suitable for direct application in ultra-high-definition video services in terms of parameter selection and model structure, and there is still room for improvement in accurately perceiving user QoE. Summary of the invention

[0008] In view of the problems mentioned in the above background technology, four subjective and objective quality parameters are proposed according to the characteristics of QoS parameters. The present invention proposes a multi-parameter ultra-high-definition streaming media experience quality perception method. MUSQM uses three video perception quality parameters, including two subjective parameters and one objective parameter, to achieve comprehensive and accurate QoE perception through a combination of subjective and objective methods. At the same time, some dynamic parameters used to describe the historical service status and the current service status are introduced to make it real-time and memorable in terms of timing. In the model structure implemented by this method, with the rapid development of deep learning (DL), its end-to-end learning ability and its outstanding performance in dealing with large-scale, high-dimensional complex problems make it possible to be applied to QoE perception problems. Therefore, the gated recurrent unit (GRU) is selected to construct the MUSQM operation unit.

[0009] Based on the characteristics of ultra-high-definition streaming services and the problems of existing QoE perception models, a QoE index that integrates video terminal playback device quality evaluation, video picture quality evaluation, and service network quality evaluation, namely the MUSQM model, is proposed. The MUSQM model uses a deep learning model to dynamically perceive the user's experience quality in different service scenarios. Specifically, it includes the following steps:

[0010] Step 1: First, the MUSQM model builds a multi-level video service quality evaluation system. The MUSQM model divides service parameters into four categories: video image presentation quality, terminal player equipment quality parameters, video service damage parameters, and video service process network quality parameters. By establishing a multi-level evaluation system, the MUSQM model's perception of video service scenarios is improved, which is used to guide the parameter selection of the subsequent QoE calculation model.

[0011] Step 2: Based on the video service quality evaluation system constructed in Step 1, the generation algorithms of three video perception quality parameters, PSNR, SSIM and VMAF, are selected for the MUSQM model according to the image characteristics of different streaming media service scenarios such as traditional video services and ultra-high-definition video services. The MUSQM model selects the most appropriate video quality assessment VQA algorithm according to different service types to accurately perceive the video picture quality and then accurately evaluate the user experience.

[0012] Step 3: The MUSQM model uses the gated recurrent unit as the calculation unit to model the sequence and time correlation of the video service. The calculation unit inputs the service parameters selected in the first two steps to calculate the output QoE of the MUSQM model. In order to better capture the impact of the previous and next video blocks on the user experience in the video service, the MUSQM model designs a recency effect simulator to simulate the impact of human memory on the generation of service quality, which improves the correlation between the output QoE of the MUSQM model and the user's subjective experience.

[0013] Furthermore, the MUSQM model is divided into three parts: quality parameter conversion model, recency effect model and QoE calculation model.

[0014] Quality parameter conversion model: In the discussion of QoE Assessment Model Research, the input parameters of existing models and their methods of perceiving QoE were analyzed. Different industry indicators were summarized, and the QoS parameters were merged and summarized according to their characteristics and their application in existing QoE models. They are divided into the following four quality parameters for evaluating video service quality.

[0015] 1) Video playback impairment parameter PIP: It is calculated from the QoS parameters describing playback jamming and playback buffering phenomena using the initial buffering duration and the number of jamming times. PIP is only related to the video playback during the service and is an objective quality parameter.

[0016] 2) Service Network Quality Parameter NQP: It consists of QoS parameters used to measure network quality, represented by network jitter and network bandwidth. NQP is related to the network status at the time of service, while DQP is an objective quality parameter.

[0017] 3) Video presentation quality parameter PQP: used to describe the quality of video presentation, and is calculated using the VQA algorithm or the video playback bit rate in the QoE model. PQP attempts to simulate the user's intuitive feeling about the picture quality and is a subjective quality parameter.

[0018] 4) Terminal device quality parameter DQP: It consists of parameters describing the physical characteristics of the video terminal playback device, such as screen size and screen resolution. DQP is a parameter for evaluating the quality of video services and is an objective quality parameter.

[0019] Furthermore, NQP is used to replace PIP commonly used in existing models to participate in the calculation of QoE, and the MUSQM model also selects DQP as one of the evaluation parameters of service quality.

[0020] Furthermore, QoS parameters among the three types of quality parameters, PQP, NQP and DQP, are selected as the input of the MUSQM model, where DQP and NQP are objective quality parameters and PQP is a subjective quality parameter.

[0021] Furthermore, the QoE perception model is divided into a subjective perception model and an objective perception model. The subjective perception model perceives the user experience based on subjective parameters. The objective model is based on objective parameters, which helps to reduce the perception cost of the QoE perception model; three types of QoS parameters, DQP, PQP and NQP, are selected as the input of the established QoE perception model.

[0022] In the calculation of DQP, the parameters of the video playback device are screen size, screen resolution, screen refresh rate and maximum screen brightness. The higher the screen refresh rate, the better the user experience when watching the video. Screen size will affect the user experience. In order to integrate the impact of screen size and screen refresh rate on user experience, the physical impact PI is defined to measure the impact of screen size and screen refresh rate on user experience. The calculation formula is as follows (1)

[0023]

[0024] Among them, QoE std The best MOS values ​​for different screen sizes at different video resolutions are given. ScreenSize and ScreenFrameRate represent the screen size and refresh rate of the player respectively. The relative relationship between screen resolution and video resolution can measure the video presentation effect of the terminal.

[0025] To measure this relationship, the MOS data in WaterlooSQoE-IV is linearly mapped to [1,5] and the standardized resolution NR is defined, which is calculated by formula (2).

[0026]

[0027] Where A=-0.6471,B=80.69,C=15.124. VR x , VR y , SR x , SR y Represents the resolution of the video and screen in the x and y directions respectively.

[0028] When the display resolution is smaller than the video resolution, that is As the display resolution increases, R gradually decreases and NR increases, indicating that when the display resolution gradually matches the video resolution, the image distortion caused by the merged pixels is alleviated and the user experience is improved. When the screen and video resolutions lose match, NR drops rapidly, indicating that pixel interpolation causes distortion and leads to a decline in user experience. PI and NR are selected as parameters to measure the video presentation effect of the terminal.

[0029] After obtaining the physical impact PI and the normalized resolution NR, the calculation formula (3) of DQP is defined.

[0030]

[0031] In the generation of PQP, VQA has peak signal-to-noise ratio PSNR, VMAF and SSIM. PSNR is a VQA indicator whose calculation is based on the mean square error between the original image and the distorted image, and is used to evaluate the performance of image and video coding. SSIM is a structured VQA indicator that considers three aspects of information: brightness, contrast and structure. By comparing the mean, variance and covariance of the image, SSIM can capture the human eye's perception of image quality. The value of SSIM is between -1 and 1, 1 means exactly the same, and -1 means completely different. VMAF is a VQA indicator that combines multiple visual features, including brightness, contrast, color and motion information, to evaluate video quality; the calculation of VMAF is based on a machine learning model, using training data of manual subjective ratings, and is expressed in the form of scores, with higher scores indicating higher video quality.

[0032] For NQP, network QoS parameters include bandwidth, network packet loss rate and jitter. Define bandwidth variation coefficient BCV and average bandwidth It is used to describe the fluctuation of the network and the average bandwidth size near each time. The BCV at time k is given by (4).

[0033]

[0034] Where BW i is the network bandwidth at time i before time k. When i = 0, BW i =BW k N = 5, that is, BCV calculates the coefficient of variation of the network bandwidth at time k and within the 4 seconds before it to measure the network fluctuation around time k.

[0035] After clarifying the calculation parameters of NQP, the calculation formula (5) of NQP at time k is defined, which is composed of real-time bandwidth, BCV and Calculated.

[0036]

[0037] The PLCC and SRCC between DQP and user MOS are both greater than 0.5. DQP is a perception parameter that reflects the impact of the user terminal on the user experience. Since the value ranges of DQP, PQP, NQP and MOS are different, the values ​​of the three parameters are uniformly processed according to formula (6) to achieve stable QoE calculation.

[0038]

[0039] Where μ DQP ,μ PQP ,μ NQP and μ MOS is the mean of each parameter, σ DQP ,σPQP ,σ NQP and σ MOS is the standard deviation of each parameter. Set the transformed target threshold to 10 -5 ,When the error between the QoE score and MOS_z is less than the threshold, it is considered that the model achieves good perception.

[0040] Recency effect model: To simulate the impact of the recency effect on the user's QoE score, a loop structure is introduced into the MUSQM model, that is, the QoE score at the previous moment is input into the QoE calculation model together with DQP, PQP, and NQP to calculate the QoE score.

[0041] The weight of the video quality at the past moment in the evaluation is consistent with the length of human memory; according to the time interval between the past moment and the current moment, the weight of the video quality at that moment is calculated through the Hermann-Ebbinghaus forgetting curve, and the QoE score of the past moment after simulating the recency effect is generated. The relative strength of memory is set to 2, that is, the recency effect generated by the QoE scores of the two moments before the current moment is considered.

[0042]

[0043] A n is the memory retention rate, N is the relative strength of memory, n is the time interval, QoE n and QoE' n is the QoE score and its recency effect value at time interval n from the current time.

[0044] The workflow of the recency effect model is as follows: If the current moment is i, the QoE scores of moments i-1 and i-2 are stored in the model. If i-1 or i-2 is less than 0, the QoE memory value at that moment is 0. When calculating the QoE score at the current moment, the model outputs the two stored historical QoE scores to the memory curve model, generates a historical score after recency effect simulation, and outputs it to the input end of the QoE calculation model. After generating the QoE score at this moment, the recency effect model reads the current output QoE from the QoE score output end. i And execute formula (8) to achieve real-time update of the model.

[0045]

[0046] Video QoE calculation model; QoS parameter conversion model and recency effect model are designed to perceive user experience, and the video QoE calculation model is used to quantify user experience. The MUSQM model is divided into two parts: the calculation training model and the calculation application model. The parameter input part and the calculation part of the two models are the same. The training model uses the multi-layer GRU structure in deep learning and introduces a hybrid optimization algorithm to achieve accurate and fast perception of user experience based on a small amount of data; by separating the training and application modules, the MUSQM perception speed is improved.

[0047] Furthermore, the main structures of the computational training model and the computational application model are both composed of multi-layer GRUs. At each time step, the QoS parameter model simultaneously inputs the computational parameters of the current moment into the multi-layer GRUs, and after the computations of each layer of GRUs are completed, they are uniformly output to different GRU integration modules for processing.

[0048] The GRU layer consists of H, X, R, Z and It consists of five parts. H is the hidden layer state, X is the input of the model. R is the update gate, which controls the update amplitude of the past state and the current input at the next moment, allowing the network to selectively retain or ignore information. Z is the reset gate, which determines how to fuse the past state and the current input information. is the candidate hidden state determined by the current input, which is fused with the input of other gates to generate the current hidden state H i , the specific calculation method of each part is shown in formula (9).

[0049]

[0050] In the above formula, W xr ,W hr ,W xz ,W hz ,W xh ,W hh Respectively represent the transfer matrix between input and update gate and hidden layer and update gate, input and reset gate and hidden layer and reset gate, input and hidden layer, and hidden layer and candidate hidden layer. b r ,b z ,b h They are update gate, reset gate, and hidden layer bias respectively.

[0051] GRU is selected as the basic unit of the video QoE calculation model, and a multi-layer GRU structure is constructed to capture the long-term dependencies in the input sequence. Each layer of GRU uses an independent gating mechanism to retain and update useful information in the time step and pass it to the next layer.

[0052] Each layer of GRU outputs the current hidden state H i . Use formula (10) to construct H of each layer of GRUi With the model output y i The relationship between QoS and output QoE is realized.

[0053] y i =W hy ·H i +b y (10)

[0054] Use MUSQM with different numbers of GRU layers for testing, and select the optimal number of layers based on the results.

[0055] Furthermore, the computational training model uses the BP-Ada optimizer as a GRU integration module, which is responsible for calculating the QoE of each time step and calculating the sequence perception error after the entire video sequence is processed. The BP-Ada optimizer is composed of the BP algorithm and the Adagrad algorithm. The BP algorithm calculates the gradients used to adjust the GRU weights based on the perception error. The Adagrad algorithm uses these gradients to generate the optimized values ​​of the weights of each layer, and then re-injects the optimized weights into the multi-layer GRU to achieve the optimization of MUSQM.

[0056] In the later stage of training, when the gradient gradually stabilizes, the learning rate will gradually decrease, which helps to stabilize the training process and prevent excessive parameter updates. The core formula of Adagrad is shown in formula (11).

[0057]

[0058] In the above formula, G is the weight before Adagrad adjustment, g is the gradient of G. r is the cumulative square gradient, and G' is the adjusted weight. ε is the global learning rate, set ε = 0.3. δ is a very small positive number used to ensure that the denominator of ▽G is not zero during the calculation process.

[0059] In the calculation of weight gradients in Adagrad, the BP algorithm is used to reversely process a video sequence according to the output error after processing it, calculate the weights and their gradients that are initially optimized according to the error, and pass the gradients into the Adagrad algorithm for further optimization. The process of calculating each weight and deviation gradient using the BP algorithm is completed by formulas (12)-(14). Loss is the error between the average QoE score of the current video and the user MOS.

[0060]

[0061]

[0062] Where N is the current time, and gradually moves backwards from the end of the sequence to the starting point. next gW is the change value of the hidden layer at time N-1.xr wxya hr ,gW xz ,gW hz ,gW xh ,gW hh ,gW hy ,gb r ,gb z ,gb h ,gb y is the gradient of each parameter. Substituting the gradient into (10), we get the BP-Adagrad optimization value of the weight. Through multiple iterations, the error is reduced, and the user QoE is accurately perceived.

[0063] By combining BP and Adagrad algorithms for GRU optimization, adaptive learning rate adjustment is provided for gradient changes at different time steps and complex parameter updates, thereby improving efficiency and stability during training and helping MUSQM better capture patterns and dependencies in sequence data.

[0064] Furthermore, the computational training model uses the output calculator as the GRU integration module, whose core function is to calculate the H output of each layer. i Compute the output of MUSQM directly.

[0065] The WaterlooSQoE-IV dataset is brought into the MUSQM model for testing to generate real-time QoE scores for video data. After completing the perception of each complete sequence, the average QoE score of the sequence is calculated. After all sequences in the entire dataset are calculated, the correlation coefficient between the average QoE score of each sequence and the corresponding MOS is calculated to evaluate the perceived effect of the QoE model output on the user's MOS.

[0066] Furthermore, the number of GRU layers is related to the perceptual effect of the model. The PSNR with the highest correlation coefficient with MOS is used as the test PQP in the layer selection experiment, and the global learning rate is set to 0.1. In the range of GRU layers from 10 to 20, the optimal number of GRU layers is selected by calculating the minimum MAE of each layer in 500 iterations.

[0067] As mentioned in the PQP generation section, PSNR, SSIM and VMAF are all correlated with user MOS. When PSNR, SSIM and VMAF are used as PQP, the PLCC, SRCC, KRCC and MAE between the model output and user MOS are used. PLCC and MAE are used to evaluate the perceptual accuracy, and SRCC is used to evaluate the monotonicity of the perceptual score. Therefore, the output of a good QoE perception model should have higher PLCC and SRCC values ​​and lower MAE values.

[0068] In terms of perceived quality, the correlation parameters of PSNR, SSIM, and VMAF decrease in turn, while MAE increases in turn. This shows that using PSNR as PQP has the best effect, while using VMAF has the worst effect. However, even for the QoE perception model that uses VMAF as PQP, its QoE score and the PLCC and SRCC of MOS are still around 0.8, which still shows that there is a strong correlation between the two. This proves that the structural design of the model can better perceive the user terminal experience, and has more VQA options while achieving accurate perception, and has better versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is the MUSQM structure diagram.

[0070] Figure 2 This is a specific segmentation effect diagram.

[0071] Figure 3 This is the data graph after interpolation.

[0072] Figure 4 It is a QoS parameter conversion model.

[0073] Figure 5 is the standardized resolution.

[0074] Figure 6 For DQP.

[0075] Figure 7 For NQP.

[0076] Figure 8 For PQP Fig. 9 Correlation heat map of DQP, NQP and three PQPs.

[0077] Fig.10 This is a recency effect model.

[0078] Fig.11 Training models for computation.

[0079] Fig.12 Model for computational applications.

[0080] Fig.13 Iteration effect of different GRU layers.

[0081] Fig.14 It is PSNR-MUSQM.

[0082] Fig.15 It is SSIM-MUSQM.

[0083] Fig.16 It is VMAF-MUSQM.

[0084] Fig.17 For PLCC.

[0085] Fig.18 For SRCC.

[0086] Fig.19 For KRCC. DETAILED DESCRIPTION

[0087] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0088] With the rapid changes in users' requirements for video services in recent years, researchers have proposed a variety of QoE models to perceive the quality of user experience. According to the QoE model, the characteristics of its parameter selection and the type of model are summarized. Next, the proposed QoE model is further analyzed to explore the potential relationship between QoS and QoE.

[0089] By analyzing the impact of stall events and initial buffering duration on user QoE, it is found that stall events have the greatest impact on user experience, and stall frequency and duration are important parameters for perceiving user QoE. It is very difficult to perceive stall events on the user side using network traffic from the network level. The quantization parameter (QP) is used to measure the instantaneous video picture quality, and the buffering duration is used to perceive the impact of buffering events on user experience. A QoE perception model integrating instantaneous quality and cumulative quality is constructed. However, only two QP parameters are selected for analysis, so it is believed that the research on the relationship between QP and perceived quality is not comprehensive enough. By exploring the selection strategies of different content delivery networks (CDN), it is proved that dynamically adjusting video parameters can effectively improve the user experience quality. They constructed a user experience quality model based on buffering duration ratio and video bit rate, and changed the buffering duration ratio and video bit rate through different CDN strategies to test their impact on user experience quality. The researchers used four parameters, network throughput, video bit rate, initial loading time and loading time, to predict user QoE, and proposed a configuration optimization algorithm to adjust resource configuration in real time according to user needs. proposed a buffer control algorithm called BOLA. The algorithm controls the video buffer size based on network tracking, thereby reducing the occurrence of buffering. In terms of user experience perception, BOLA uses video bitrate and buffering duration to calculate QoE. Bentaleb et al. used VQA to perceive video picture quality and combined the initial buffering duration to perceive user experience quality. They also proposed a dynamic resource allocation and management architecture based on software-defined networking (SDN) for network resource configuration during video services. The proposed VideoATLAS and KSQI both use VQA and buffering duration to perceive user QoE. Compared with KSQI, VideoATLAS is more inclined to objective evaluation. It generates QoE based on objective quality features, replay perception features, and memory-driven features, while KSQI is based on a corpus of subjective research, and analyzes and calculates KSQI based on the human visual system and human annotation data.

[0090] Based on the above analysis of existing user quality perception models, it is found that most of them perceive user experience through the following methods:

[0091] 1) Video Presentation Quality: Since video data is ultimately presented to users in the form of images, the user's perception of service quality can be achieved by analyzing whether the images have distortion such as blur and tearing. The more commonly used QoS parameters in this regard are video bitrate and VQA.

[0092] 2) Network condition perception during video service (Network): Through network parameters such as network jitter and bandwidth, the impact of network quality on user QoE is measured, and user QoE is perceived. The more commonly used QoS parameters in this regard are network bandwidth and latency.

[0093] 3) Rebuffering during video playback: When the video playback speed is faster than the data loading speed into the buffer, the playback is paused to load the data, and rebuffering occurs. Common QoS parameters in this regard include initial buffer length and rebuffering times.

[0094] 4) Stalling during video playback: When the data in the buffer has been played and the new video data has not been loaded into the buffer, the player interrupts the playback to load data, which causes stalling. Common QoS parameters in this regard include the number of stalls and the duration of the stall.

[0095] It is worth noting that although both buffering and stuttering are related to the speed of data loading into the buffer, buffering is manifested as a short pause in playback, while stuttering is a long interruption in playback. Compared with buffering, this difference makes stuttering give users a worse experience.

[0096] Table 1 Common QoS parameters of QoE model

[0097]

[0098] Table 1 gives the QoS parameters used by the model and the quality parameters to which they belong, in order to intuitively analyze the parameter selection characteristics of different models. It is not difficult to see that most of the existing models construct the relationship between user QoE and QoS parameters from three aspects: playback stagnation, video buffering, and presentation quality. Since presentation quality describes human feelings about the video picture, it is a subjective parameter. Parameters such as stagnation and buffering describe the service process of the video, which are objective parameters. This proves that most of the existing QoE models use a mixed subjective and objective parameter selection logic. However, only one model perceives user QoE from the network aspect, and no model considers the characteristics of the terminal playback device when calculating QoE.

[0099] As QoE evaluation models have been widely studied in recent years, a considerable number of QoE video datasets can be used for training and testing QoE models. For the QoE perception in the ultra-high-definition scenario under study, the dataset needs to contain ultra-high-definition video data and the network data used when testing the dataset. The datasets established by researchers have common characteristics. Their experimental equipment is all high-definition television (HDTV), and they ignore the impact of the network on QoE and focus on the impact of video parameters such as bit rate, frame rate and compression resolution on user QoE. However, the terminal devices of the above datasets cannot support ultra-high-definition video services, and the network data corresponding to the video services are not recorded, so they are not suitable for use as datasets for this study.

[0100] The experimental terminal devices are all mobile phones, and the video data used includes 4K ultra-high-definition video. However, the terminal devices they used only support 1080p video playback, and like the previous study, they did not include network quality data in the data set. Therefore, these two data sets are also not suitable for testing ultra-high-definition QoE perception models.

[0101] The constructed LIVE-NFLX-I dataset uses 420 video data, and downsamples the videos with resolutions exceeding 1080p to 1080p. Considering the impact of network conditions on user QoE, the researchers used 7 network traces to simulate the impact of different networks on video services, and used HDTV as the terminal device for testing. The established dataset contains 1,350 video data, covering nine types of video resolutions from 320p to 4K to simulate the quality of different video sources. In the network environment, nine network traces were used to simulate the impact of various network conditions on videos of different resolutions. At the same time, considering the increasing diversification of current video service terminals, the researchers used 1080p HDTV, 2K resolution mobile devices, and 4K resolution ultra-high-definition televisions (UHDTVs) as terminal devices for experiments.

[0102] In summary, the WaterlooSQoE-IV dataset meets the requirements of UHD video QoE research in terms of video resolution, network data, and terminal devices, so it is used as the training and testing dataset for the model. The LIVE-NFLX-II dataset contains more complete network data than WaterlooSQoE-IV, so it is selected as a supplementary dataset for the model. It should be noted that since the construction of the QoE dataset requires real people to participate in the experiment, it cannot have a dataset of the same scale as problem problems such as target recognition. Therefore, the UHD video QoE perception model must have the ability to perform well in a resource-constrained environment.

[0103] The idea and method of creating MUSQM are discussed in detail. Its main logic is to extract the evaluation parameters of video services through different types of QoS parameters, and build the relationship between the parameters and the user QoE score based on deep learning models such as GRU to achieve direct mapping between the two. The structure diagram of the model is shown in the figure Figure 1 shown.

[0104] The main structure of the MUSQM model can be divided into three parts: quality parameter conversion model, recency effect model and QoE calculation model. The following will discuss the various tasks involved in MUSQM.

[0105] Network sequence reconstruction and segmentation

[0106] When studying the relationship between network data and video sequences, it was found that there is a time difference between network data and video data, that is, most network trace data is significantly longer than video data. In order to achieve the correspondence between network data and video data at each moment, it is necessary to intercept data with a length of video playback time from the network trace corresponding to the video as the network data when the user watches the video. In order to maximize the correlation between the intercepted data and the user's MOS value, a complete network data segment is divided into the beginning, middle and end, and the length of each segment is the length of the video playback time. The specific segmentation effect is as follows Figure 2 shown.

[0107] Since WaterlooSQoE-IV does not provide real-time MOS data, we rely on the LIVE-NFLX-II dataset to perform relevant analysis on network data interception, and intercept the network data of WaterlooSQoE-IV based on the analysis results. By calculating the Pearson correlation coefficient (PLCC) and Spearman correlation coefficient (SRCC) between each segment of network data and user MOS, and taking the absolute value of all the results of the dataset and then averaging them, we obtained the average absolute correlation between different network segments and user terminal experience, and the calculation results are shown in Table 2 below.

[0108] Table 2 SRCC and PLCC between each segment of network data and user MOS

[0109] Front Mid End SRCC 0.303 0.431 0.328 PLCC 0.327 0.436 0.288

[0110] According to Table 2, the middle segment of the network data is more correlated with the user MOS, which is consistent with the result that the correlation between the network data in the middle of the video and the user MOS score is significantly higher. Therefore, the middle segment of the WaterlooSQoE-IV network data is intercepted as the network data of the subsequent model.

[0111] When processing the network data of WaterlooSQoE-IV, it was found that the step length between adjacent data of trace5 and trace6 in the network data was 5s, while the step length between the data of the other 7 traces was 1s. In order to unify the data length of the network parameters, the piecewise cubic Hermite interpolation algorithm (PCHIP) was used to interpolate trace5 and trace6. PCHIP interpolation takes into account the data points and the derivative information of these data points. Its core idea is to use a cubic polynomial to interpolate in a small interval between each adjacent data point, while ensuring the smoothness and continuity of the interpolation curve. The interpolated data is as follows Figure 3 As shown, the original data and the interpolated data are marked.

[0112] Since the WaterlooSQoE-IV dataset is designed for three different service terminals, the relevant parameters of the three terminals, namely FHD display (ViewSonic VA2452SM), mobile phone (Apple iPhone XS Max) and UHDTV (Sony XBR55X800H), were found, and their screen size, screen resolution and screen refresh rate were added to the WaterlooSQoE-IV dataset.

[0113] After data capture and completion, the complete WaterlooSQoE-IV data set for MUSQM testing was obtained. It contains 1,350 video data, each of which is 28 seconds long, and each data has corresponding terminal parameters. At the same time, each video data also corresponds to a piece of real-time network bandwidth data of the same length to study the impact of real-time network quality on terminal QoE.

[0114] Quality Parameter Transformer

[0115] Quality parameter study

[0116] In the discussion of QoEAssessmentModel Research, the input parameters of existing models and their methods of perceiving QoE are analyzed. Different industry indicators are summarized and classified into the following four quality parameters for evaluating video service quality based on their characteristics and their application in existing QoE models.

[0117] 1. Video playback impairment parameters (Playback Impairment Parameters, PIP): This parameter is mainly calculated from QoS parameters such as initial buffering duration and number of freezes that describe playback freezes and playback buffering. Since PIP is only related to the playback of the video during the service process, it is an objective quality parameter.

[0118] 2. Service Network Quality Parameter (NQP): This parameter mainly consists of QoS parameters used to measure network quality, represented by network jitter and network bandwidth. NQP is only related to the network status at the time of service, so DQP is an objective quality parameter.

[0119] 3. Video Presentation Quality Parameter (PQP): This parameter is mainly used to describe the quality of video presentation. In existing QoE models, it is often calculated using the VQA algorithm or the video playback bit rate. PQP attempts to simulate the user's intuitive feeling of picture quality, so it is a subjective quality parameter.

[0120] 4. Device Quality Parameter (DQP): This parameter is composed of parameters that describe the physical characteristics of the video terminal playback device, such as screen size and screen resolution. Existing QoE models often ignore the characteristics of user terminal devices. However, in actual service scenarios, user terminal devices affect the user experience to a certain extent. Therefore, DQP is added as a parameter for evaluating the quality of video services. Since it is only related to the characteristics of the user's terminal device, DQP is an objective quality parameter.

[0121] Table 3 lists the four quality parameters and the QoS parameters they contain. By comparing Table 1 with Table 2, most of the existing QoE models use QoS in the two types of parameters, PIP and PQP, as input parameters.

[0122] Table 3 Quality parameters and the QoS parameters they contain

[0123]

[0124] Video presentation quality is one of the important factors affecting user QoE. In the previous discussion of QoE models, several models that did not take video presentation quality into consideration performed poorly in QoE perception. Therefore, PQP was selected as one of the evaluation parameters of MUSQM. It was found that most QoS parameters in PIP are parameters recorded after the video is watched, and the ultra-high-definition video scene has high requirements for the real-time performance of the QoE perception model. Therefore, it is not appropriate to continue to use such QoS parameters in ultra-high-definition scenes. QoS parameters represented by bandwidth have good real-time performance, and it is mentioned in the literature that the decline in network quality during video playback will cause video freezes and picture distortion and other playback damage phenomena, which means that the deterioration of network conditions is the root cause of buffering and freezes. Therefore, NQP is used to replace the PIP commonly used in existing models to participate in the calculation of QoE, and MUSQM also selects DQP as one of the evaluation parameters of service quality.

[0125] In summary, QoS parameters among the three types of quality parameters, PQP, NQP and DQP, are selected as the input of MUSQM. Among them, DQP and NQP are objective quality parameters, and PQP is a subjective quality parameter. This parameter selection makes up for the problem that the existing model lacks perception of user terminal characteristics, and also conforms to the parameter selection logic of most existing models that use a combination of subjective and objective QoS parameters.

[0126] Quality parameter calculation method

[0127] The existing QoE perception models can be divided into subjective perception models and objective perception models. The subjective perception model is based on subjective parameters and can better perceive the user experience. The objective model is based on objective parameters, which helps to reduce the model perception cost. The above divides the commonly used QoS parameters into four categories, and selects three types of QoS parameters: DQP, PQP and NQP as the input of the established model. Next, we will discuss the specific QoS parameter selection and quality parameter calculation methods of the three types of quality parameters, and give a quality parameter conversion model. The internal structure of the model is as follows Figure 4 As shown, it will serve as the first level of MUSQM, responsible for converting the input QoS into the quality parameter input calculation model.

[0128] In the calculation of DQP, the main parameters of video playback devices are screen size, screen resolution, screen refresh rate and maximum screen brightness. The higher the screen refresh rate, the better the user experience when watching videos. Screen size will affect user experience. Therefore, in order to integrate the impact of screen size and screen refresh rate on user experience, physical impact (PI) is defined to measure the impact of screen size and screen refresh rate on user experience. Its calculation formula is as follows (1)

[0129]

[0130] Among them, QoE std The best MOS values ​​for different screen sizes at different video resolutions are given. ScreenSize and ScreenFrameRate represent the screen size and refresh rate of the player respectively.

[0131] When a terminal device plays a video, if the screen resolution is not equal to the video resolution, the device will scale the video image to achieve normal playback. Specifically, when the screen resolution is smaller than the video resolution, the device will merge some pixels in the video to fit the screen size. This may cause black edges to appear when the video is displayed on the screen. When the screen resolution is larger than the video resolution, the device will interpolate the video pixels to meet the screen requirements, which may cause the video to appear blurry. Therefore, the relative relationship between the screen resolution and the video resolution can measure the terminal's presentation effect on the video.

[0132] To measure this relationship, the MOS data in WaterlooSQoE-IV is linearly mapped to [1,5] and the normalized resolution (NR) is defined. Its calculation formula is given by (1) and its function graph is shown in Figure 5 .

[0133]

[0134] Where A=-0.6471,B=80.69,C=15.124. VR x , VR y , SR x , SR y Represents the resolution of the video and screen in the x and y directions respectively

[0135] When the display resolution is smaller than the video resolution, that is As the display resolution increases, R gradually decreases and NR increases. This result shows that when the display resolution gradually matches the video resolution, the picture distortion caused by the merged pixels is alleviated and the user experience is improved. When the screen and video resolutions lose match, NR drops rapidly, indicating that pixel interpolation causes distortion and leads to a decline in user experience. The above characteristics of NR are consistent with the previous discussion on resolution matching. Therefore, PI and NR are selected as parameters to measure the video presentation effect of the terminal.

[0136] After obtaining the physical impact PI and the normalized resolution NR, the calculation formula (3) of DQP is defined.

[0137]

[0138] In the generation of PQP, the commonly used VQAs are Peak Signal-to-Noise Ratio (PSNR), VMAF, and SSIM. PSNR is a traditional VQA metric whose calculation is based on the mean square error between the original image and the distorted image. This metric is often used to evaluate the performance of image and video coding. SSIM is a structured VQA metric that takes into account three aspects of information: brightness, contrast, and structure. By comparing the mean, variance, and covariance of the image, SSIM can better capture the human eye's perception of image quality. The value of SSIM is between -1 and 1, with 1 indicating exactly the same and -1 indicating completely different. VMAF is an emerging VQA metric that combines multiple visual features, including brightness, contrast, color, and motion information, to evaluate video quality. The calculation of VMAF is based on a machine learning model using training data for manual subjective ratings. It is usually expressed in the form of a score, with a higher score indicating higher video quality.

[0139] In practical applications, PSNR is suitable for the performance evaluation of traditional image and video coding, SSIM is more suitable for capturing image structure information, and VMAF combines multiple factors and is suitable for actual video quality evaluation, especially in streaming media and video content publishing. The three VQAs have their own suitable service scenarios, so three calculation methods are given to PQP, allowing operators to choose the appropriate VQA to calculate PQP according to different service characteristics. It should be noted that network damage is considered to be the main cause of video damage on user terminals. Therefore, it is believed that PQP does not change within the same video segment, and PQP changes to a new value when a new video segment is played.

[0140] For NQP, commonly used network QoS parameters include bandwidth, network packet loss rate, and jitter. In video services, if the network bandwidth becomes smaller, the terminal will not be able to download enough data for buffering in time. This will cause loading delays and freezes during video playback. Similarly, when the network bandwidth is insufficient, the network transmission speed may not meet the frame rate requirements of the video, resulting in partial frame loss. This will cause freezes and discontinuous images during video playback. In order to adapt to the low network bandwidth, the terminal device may automatically adjust the video quality, which may include reducing resolution, reducing details, etc., making the picture blurry or distorted. Therefore, the freeze phenomenon of the video service at any time depends on the network quality at that time and before. Researchers pointed out that bandwidth fluctuations are the main cause of video freezes. Therefore, network bandwidth is a key factor in measuring network quality. To this end, the bandwidth coefficient of variation (BCV) and average bandwidth are defined. It is used to describe the fluctuation of the network and the average bandwidth around each time. The BCV at time k is given by (4).

[0141]

[0142] Where BW i is the network bandwidth at time i before time k. When i = 0, BW i =BW k N = 5, that is, BCV calculates the coefficient of variation of the network bandwidth at time k and within the 4 seconds before it to measure the network fluctuation around time k.

[0143] After clarifying the calculation parameters of NQP, the calculation formula (5) of NQP at time k is defined, which is calculated from the real-time bandwidth, BCV and BW.

[0144]

[0145] Down Figure 6-8 The scatter plots of three quality parameters and MOS in the WaterlooSQoE-IV dataset are listed. Fig. 9 Their correlation heat maps are listed. The correlation between PLCC, SRCC and KRLL and MOS is analyzed, and the relevant results are shown in Table 4.

[0146] Table 4 Correlation coefficients between quality parameters and MOS

[0147]

[0148] Figure 6-8 It is proved that only PSNR and MOS have a nonlinear relationship. This may be due to the fact that it only considers brightness information, ignoring the structure and the sensitivity of the human eye to details, so it may not be accurate enough in some cases. The remaining quality parameters are all approximately linear with MOS within the definition domain. This linear relationship helps to achieve accurate perception of user experience while reducing model complexity and improving perception efficiency. From the data in Table 5, it can be seen that the PLCC and SRCC between DQP and user MOS are both greater than 0.5, which proves that there is a strong correlation between the two. Therefore, DQP can be used as a perception parameter to reflect the impact of user terminals on user experience. For PQP, the correlation test proves that the three parameters are correlated with user MOS, so PSNR, VMAF and SSIM can all perceive the impact of the presentation quality of the picture on user experience. The PLCC and SRCC between NQP and user MOS are both above 0.4, which proves that there is a correlation between the proposed NQP indicator and user MOS, so NQP can replace the PIP parameter to reflect the impact of network quality on user experience. Since the value ranges of DQP, PQP, NQP and MOS are different, the values ​​of the three parameters are uniformly processed according to (6) to achieve stable QoE calculation.

[0149]

[0150] Where μ DQP ,μ PQP ,μ NQP and μ MOS is the mean of each parameter, σ DQP ,σ PQP ,σ NQP and σ MOS is the standard deviation of each parameter. Set the transformed target threshold to 10 -5 ,When the error between the QoE score and MOS_z is less than the threshold, it is considered that the model achieves good perception.

[0151] Recency Effect Simulator

[0152] When users evaluate video services, it depends not only on the video quality at the current moment, but also on the quality of the videos they watched previously, which is the recency effect. Researchers have studied the recency effect when users watch videos and found that when subjects evaluate videos, the video sequences they have recently seen also participate in the evaluation process and obtain a larger weight. Therefore, when perceiving the quality of user experience in ultra-high-definition video scenes, the recency effect cannot be ignored. In order to simulate the impact of the recency effect on the user's QoE score, a loop structure is introduced into MUSQM, that is, the QoE score at the previous moment is input into the QoE calculation model together with DQP, PQP, and NQP to calculate the QoE score.

[0153] The weight of the video quality at the past moment in the evaluation is consistent with the length of human memory. Therefore, according to the time interval between the past moment and the current moment, the weight of the video quality at that moment is calculated through the Hermann-Ebbinghaus forgetting curve, and the QoE score of the past moment after simulating the recency effect is generated. The relative strength of memory is set to 2, that is, the recency effect generated by the QoE scores of the two moments before the current moment is considered.

[0154]

[0155] A n is the memory retention rate, N is the relative strength of memory, n is the time interval, QoE n and QoE' n is the QoE score and its recency effect value at time interval n from the current time.

[0156] After clarifying the simulation method of the recency effect, the recency effect model was proposed, and its structure is as follows: Fig.10 shown.

[0157] The workflow of the model is as follows: If the current moment is i, the QoE scores of moments i-1 and i-2 are stored in the model. If i-1 or i-2 is less than 0, the QoE memory value at that moment is 0. When calculating the QoE score at the current moment, the model outputs the two stored historical QoE scores to the memory curve model, generates a historical score after recency effect simulation, and outputs it to the input end of the QoE calculation model. After generating the QoE score at this moment, the recency effect model reads the current output QoE from the QoE score output end. i And execute (8) to achieve real-time update of the model.

[0158]

[0159] Video QoE Calculator

[0160] In the previous article, the QoS parameter conversion model and the recency effect model were designed to perceive the user experience, while the QoE calculation model is used to quantify the user experience. Different from the existing QoE calculation model, MUSQM is divided into two parts: a calculation training model and a calculation application model. The parameter input part and the calculation part of the two models are the same. The training model uses the multi-layer GRU structure in deep learning and introduces a hybrid optimization algorithm to achieve accurate and fast perception of user experience based on a small amount of data. The application model uses the results of the training model to calculate the user QoE in real time. Since all the weights in the model have been calculated in the training model, the application model only involves mathematical calculations. By separating the training and application modules, the MUSQM perception speed is improved. The internal structures of the two models are as follows: Figure 11-12 As shown in the figure, the following will analyze the various parts of the two models.

[0161] Multi-layer GRU architecture

[0162] The main structure of the two models is composed of multi-layer GRU. At each time step, the QoS parameter model simultaneously inputs the calculation parameters of the current moment into the multi-layer GRU. After the calculation of each layer of GRU is completed, it will be uniformly output to different GRU integration modules for processing.

[0163] In recent years, the problem of time series processing has been widely studied. The structural advantages of deep learning models such as recurrent neural networks (RNN) and long short-term memory networks (LSTM) have made them widely used in sequence processing. RNN retains historical information through recurrent connections, but it is difficult to capture long-term dependencies. LSTM solves this problem through a gating mechanism, allowing for the selective integration of new information, forgetting, and output. LSTM can be seen as the evolution of RNN. By introducing a gating mechanism, the ability to model sequences is improved, and better application results have been achieved than RNN.

[0164] Compared with LSTM, the Gated Recurrent Unit (GRU) solves the long-term dependency problem in traditional recurrent neural networks through new gating mechanisms such as update gates, reset gates, and hidden gates while maintaining powerful sequence modeling capabilities. The specific operation process is as follows: At each time step, GRU calculates an update gate and a reset gate. The output range of the update gate is between 0 and 1, which determines the degree of fusion between the past hidden state and the candidate hidden state. More specifically, the update gate combines the past hidden state with the current input information, and then outputs a value between 0 and 1 through a Sigmoid function, indicating how much past state information is retained. Next, the reset gate determines how to combine the past hidden state with the candidate hidden state. The output of the reset gate is also between 0 and 1, which determines the degree of influence of the past hidden state in calculating the candidate hidden state. Similar to the update gate, the reset gate also outputs a value between 0 and 1 through a Sigmoid function. Finally, the candidate hidden state is combined with the past hidden state by applying a reset gate, and then interpolated with the output of the update gate to obtain the current hidden state. This mechanism allows GRU to balance the influence of the past hidden state and the candidate hidden state according to the weights controlled by the above gates when generating a new hidden state. This design enables GRU to adapt to different sequence patterns and effectively capture long-term dependencies, while also reducing the number of parameters, making it more suitable for smaller data sets while improving computational efficiency.

[0165] The GRU layer consists of H, X, R, Z and It consists of five parts. H is the hidden layer state, X is the input of the model. R is the update gate, which controls the update amplitude of the past state and the current input at the next moment, allowing the network to selectively retain or ignore information. Z is the reset gate, which determines how to fuse the past state and the current input information. is the candidate hidden state determined by the current input, which is fused with the input of other gates to generate the current hidden state H i, the specific calculation methods of each part are as shown in (9).

[0166]

[0167] In the above formula, W xr ,W hr ,W xz ,W hz ,W xh ,W hh Respectively represent the transfer matrix between input and update gate and hidden layer and update gate, input and reset gate and hidden layer and reset gate, input and hidden layer, and hidden layer and candidate hidden layer. b r ,b z ,b h They are update gate, reset gate, and hidden layer bias. These parameters determine the performance of GRU and the perceived accuracy of the model.

[0168] Since GRU has the above advantages such as being suitable for small data sets, low computing cost, and a wide range of applications, it meets the requirements of low perception cost of ultra-high-definition video QoE models and still performs well in resource-constrained environments. Therefore, GRU is selected as the basic unit of the video QoE calculation model.

[0169] In order to solve the long-term dependency problem in sequence processing, a multi-layer GRU structure is constructed to better capture the long-term dependency in the input sequence. Each layer of GRU can retain and update useful information in time steps with independent gating mechanisms and pass it to the next layer. This helps the model better capture the temporal dependency when processing long sequences.

[0170] Each layer of GRU outputs the current hidden state H i . Use (10) to construct the H of each layer of GRU i With the model output y i The relationship between them realizes the complete mapping from input QoS to output QoE.

[0171] y i =W hy ·H i +b y (10)

[0172] However, the choice of the number of GRU layers is an issue that needs to be considered. As the number of layers increases, the model training time increases and the model complexity will be greatly increased, and the data volume requirement will also increase. Therefore, it is necessary to test MUSQM with different numbers of GRU layers and select the optimal number of layers based on the results.

[0173] BP-Ada optimizer

[0174] The training model uses the BP-Ada optimizer as the GRU integration module, which is mainly responsible for calculating the QoE at each time step and calculating the sequence perception error after the entire video sequence is processed. The BP-Ada optimizer is a combination of the BP algorithm and the Adagrad algorithm. The BP algorithm calculates the gradients used to adjust the GRU weights based on the perception error. The Adagrad algorithm uses these gradients to generate the optimized values ​​of the weights of each layer, and then re-injects the optimized weights into the multi-layer GRU to achieve MUSQM optimization.

[0175] The BP algorithm was proposed by David E. Rumelhart et al. in 1986 and has become one of the most basic and important optimization algorithms in machine learning. Adagrad is an adaptive learning rate optimization algorithm for deep learning models, proposed by researchers in 2011. It is not difficult to find from (9) and (10) that GRU has a large number of parameters involved in the calculation of the final result, and its gating mechanism makes the parameter update more complicated. Therefore, the optimization process needs to consider the different characteristics of the parameters. Adagrad can adaptively adjust the learning rate according to the historical gradient of each parameter in GRU, so as to better adapt to the update of different parameters. For sequence analysis models such as GRU, the gradients at different time steps may be quite different because the patterns and dependencies in the sequence data may change over time. Adagrad's adaptive learning rate can make the model converge quickly in the early stage of training, because a larger learning rate can cope with the situation of large initial gradients. At the same time, in the later stage of training, when the gradient gradually stabilizes, the learning rate will gradually decrease, which helps to stabilize the training process and prevent excessive parameter updates. The core formula of Adagrad is shown in (11).

[0176]

[0177] In the above formula, G is the weight before Adagrad adjustment, g is the gradient of G. r is the cumulative square gradient, and G' is the adjusted weight. ε is the global learning rate, set ε = 0.3. δ is a very small positive number used to ensure that the denominator of ▽G is not zero during the calculation process.

[0178] In the calculation of weight gradients in Adagrad, the idea of ​​BP algorithm is used. After a video sequence is processed, the sequence is processed in reverse according to the output error, and the weights and their gradients that are initially optimized according to the error are calculated. The gradients are then passed into the Adagrad algorithm for further optimization. The process of calculating each weight and deviation gradient using BP algorithm is completed by (12)-(14). Loss is the error between the average QoE score of the current video and the user MOS.

[0179]

[0180]

[0181] Where N is the current time, and gradually moves backwards from the end of the sequence to the starting point. next gW is the change value of the hidden layer at time N-1. xr wxya hr ,gW xz ,gW hz ,gW xh ,gW hh ,gW hy ,gb r ,gb z ,gb h ,gb y is the gradient of each parameter. Substituting the gradient into (10), we get the BP-Adagrad optimization value of the weight. Through multiple iterations, the error can be reduced, and the user QoE can be accurately perceived.

[0182] In summary, by combining BP and Adagrad algorithms for GRU optimization, adaptive learning rate adjustment can be provided for gradient changes at different time steps and complex parameter updates, thereby improving efficiency and stability during training and helping MUSQM better capture patterns and dependencies in sequence data.

[0183] Output Calculator

[0184] The training model uses the output calculator as the GRU integration module, whose core function is to calculate the H output of each layer. i Compute the output of MUSQM directly.

[0185] A major feature of the application model is that it does not adjust the weight parameters, but only performs mathematical calculations between input and output. This makes the application model lower in both model complexity and computational complexity, thus enabling faster perception. This feature is particularly important in ultra-high-definition video scenarios because it meets the high real-time requirements for QoE perception models.

[0186] After completing the data selection and model construction, the WaterlooSQoE-IV dataset was brought into MUSQM for testing to generate real-time QoE scores for video data. Since the WaterlooSQoE-IV dataset only provides the average MOS data for each sequence, in order to accurately evaluate the model effect, after completing the perception of each complete sequence, the average QoE score of the sequence was calculated. After all sequences in the entire dataset were calculated, the correlation coefficient between the average QoE score of each sequence and the corresponding MOS was calculated to evaluate the perceived effect of the QoE model output on the user's MOS. This evaluation method can not only objectively evaluate the performance of the model, but also provide valuable information about the relationship between the model output and the user experience.

[0187] The number of GRU layers is closely related to the perceptual effect of the model. The PSNR with the highest correlation coefficient with MOS is used as the test PQP in the layer selection experiment, and the global learning rate is set to 0.1. In the range of GRU layers from 10 to 20, the optimal number of GRU layers is selected by calculating the minimum MAE of each layer in 500 iterations. Fig.13 The iteration curves of different numbers of layers are given, and the MAE and error standard deviation of 500 iterations of different numbers of layers are given in Table 4. In the ultra-high-definition video scene, the QoE perception model should have good transferability and relatively stable perception effect, so its MAE and error standard deviation σ should be as small as possible.

[0188] Layers 10 11 12 13 14 15 16 17 18 19 20 MAE 0.076 0.137 0.041 0.109 0.052 0.067 0.081 0.063 0.091 0.038 0.041 σ 0.074 0.152 0.038 0.104 0.054 0.084 0.080 0.069 0.101 0.040 0.038

[0189] In Table 4, the computational model with 19 GRU layers has the smallest MAE and has a better perceptual effect. The models with 12 and 20 layers have the smallest standard deviation and the perceptual effect is more stable. It is worth noting that the MAE and σ of GRU layers with 12, 19, and 20 are relatively close, and the 19-layer GRU model with better perceptual effect is given priority as the basic structure of the MUSQM computational model.

[0190] As mentioned in the PQP generation section, PSNR, SSIM and VMAF are all correlated with user MOS. Table 5 shows the PLCC, SRCC, KRCC and MAE between the model output and user MOS when PSNR, SSIM and VMAF are used as PQP. PLCC and MAE are used to evaluate the perception accuracy, and SRCC is used to evaluate the monotonicity of the perception score. Therefore, the output of a good QoE perception model should have higher PLCC and SRCC values ​​and lower MAE values.

[0191] In terms of perceived quality, the correlation parameters of PSNR, SSIM, and VMAF decrease in turn, while MAE increases in turn. This shows that using PSNR as PQP has the best effect, while using VMAF has the worst effect. However, even for the QoE perception model that uses VMAF as PQP, its QoE score and the PLCC and SRCC of MOS are still around 0.8, which still shows that there is a strong correlation between the two. This proves that the structural design of the model can better perceive the user terminal experience, and has more VQA options while achieving accurate perception, and has better versatility.

[0192] Table 5 MAE and standard deviation corresponding to different GRU layers

[0193]

[0194] Table 5 shows the PLCC, SRCC, KRCC and MAE of MUSQM output QoE and MOS using different PQPs

[0195] exist Fig.13 In the paper, the relationship between QoE and MOS in the WaterlooSQoE-IV dataset is given, and the fitting curves of the two are drawn.

[0196] from Fig.13 It was found that the three data points using MUSQM have good compactness and linearity, which means that it has achieved stable and accurate perception of user MOS. Figure 15-17 The PLCC, SRCC and KRCC between the QoE and user MOS of the above model are given. A good objective QoE perception model should have higher PLCC and SRCC values. The PLCC and SRCC of the proposed PSNR-QoE are 0.9032 and 0.8858 respectively, indicating that the QoE and MOS output by MUSQM have a strong correlation and can better reflect the user's subjective experience. By comparing the other models horizontally, it is not difficult to find that MUSQM has the best effect among all models and achieves a good perception of user experience.

[0197] Through the analysis of the experimental results, MUSQM, which uses NQP instead of the commonly used PIP, does not show inaccurate user MOS perception compared to other models, which proves that the method of using NQP instead of PIP is feasible. MUSQM integrates the VQA algorithm in the process of evaluating QoE, which makes the linearity of its perception effect significantly better than the model that does not use VQA. In the model that also uses the VQA algorithm, because MUSQM introduces the evaluation of user device quality, the perception effect of user subjective experience is more accurate and stable. This can be easily seen from the results that MUSQM's data points are closer than other models and have better linearity.

[0198] The present invention studies the QoE perception problem in ultra-high-definition video scenarios, and proposes a multi-parameter QoE perception model named MUSQM for the key issue of sequence processing in real-time QoE perception. It makes up for the problem that the existing QoE model ignores the impact of user terminal devices on terminal experience, and designs a variety of evaluation parameters for multi-angle perception of user experience quality. This design has been proven to be feasible in tests. Experiments also prove that the proposed MUSQM performs well in ultra-high-definition data sets compared to existing QoE models, and has a more accurate user terminal experience perception capability.

Claims

1. A method for perceiving ultra-high-definition streaming media quality of experience, characterized in that: Based on the characteristics of ultra-high-definition streaming media services and the QoE perception model, a QoE index that integrates video terminal playback device quality evaluation, video picture quality evaluation, and service network quality evaluation, namely the MUSQM model, is proposed. The MUSQM model uses a deep learning model to dynamically perceive the user's experience quality in different service scenarios. Specifically, it includes the following steps: Step 1: First, the MUSQM model builds a multi-level video service quality evaluation system. The MUSQM model divides service parameters into video image presentation quality, terminal player equipment quality parameters, video service damage parameters, and video service process network quality parameters. By establishing a multi-level evaluation system, the MUSQM model's perception of video service scenarios is improved, which is used to guide the parameter selection of the subsequent QoE calculation model. Step 2: Based on the video service quality evaluation system constructed in step 1, the generation algorithms of three video perception quality parameters, namely PSNR, SSIM and VMAF, are selected for the MUSQM model according to the image characteristics of different streaming service scenarios of video services and ultra-high-definition video services. The MUSQM model selects the most appropriate video quality assessment (VQA) algorithm according to different service types to accurately perceive the video picture quality and thus accurately evaluate the user experience. Step 3: The MUSQM model uses the gated recurrent unit as the computing unit to model the sequentiality and temporal correlation of video services. The computing unit inputs the service parameters selected in the first two steps to calculate the output QoE of the MUSQM model. In order to better capture the impact of the previous and next video blocks in the video service on the user experience, the MUSQM model designs a recency effect simulator to simulate the impact of human memory on the generation of service quality, thereby improving the correlation between the output QoE of the MUSQM model and the user's subjective experience.

2. The method for perceiving ultra-high-definition streaming media quality of experience according to claim 1, characterized in that: The MUSQM model consists of three parts: quality parameter conversion model, recency effect model, and QoE calculation model; Quality parameter conversion model: Based on the characteristics of QoS parameters and their application in the QoE model, they are summarized and divided into the following four quality parameters for evaluating video service quality: 1) Video playback impairment parameter PIP: It is calculated from the QoS parameters describing playback jamming and playback buffering phenomena based on the initial buffering duration and the number of jamming times. PIP is only related to the video playback during the service process and is an objective quality parameter. 2) Service Network Quality Parameter NQP: It consists of QoS parameters used to measure network quality, represented by network jitter and network bandwidth. NQP is related to the network status during service, while DQP is an objective quality parameter. 3) Video presentation quality parameter PQP: used to describe the quality of video presentation. It is calculated using the VQA algorithm or the video playback bit rate in the QoE model. PQP attempts to simulate the user's intuitive feeling about the picture quality and is a subjective quality parameter. 4) Terminal device quality parameter DQP: It consists of parameters that describe the physical characteristics of the video terminal playback device, such as screen size and screen resolution. DQP is a parameter for evaluating the quality of video services and is an objective quality parameter.

3. The method for perceiving ultra-high-definition streaming media quality of experience according to claim 1, characterized in that: NQP is used to replace PIP in the calculation of QoE. The MUSQM model can also use DQP as the evaluation parameter of service quality.

4. The method for perceiving ultra-high-definition streaming media quality of experience according to claim 2, characterized in that: The QoS parameters among the three types of quality parameters, PQP, NQP and DQP, are selected as the input of the MUSQM model; among them, DQP and NQP are objective quality parameters, and PQP is a subjective quality parameter.

5. The method for perceiving ultra-high-definition streaming media quality of experience according to claim 2, characterized in that: The QoE perception model is divided into a subjective perception model and an objective perception model. The subjective perception model perceives the user experience based on subjective parameters. The objective model is based on objective parameters to reduce the perception cost of the QoE perception model. Three types of QoS parameters, DQP, PQP and NQP, are selected as the input of the established QoE perception model. In the calculation of DQP, the parameters of the video playback device are screen size, screen resolution, screen refresh rate and maximum screen brightness; the higher the screen refresh rate, the better the video viewing experience; the physical impact PI is defined to measure the impact of screen size and screen refresh rate on user experience, and the calculation formula is as follows (1) Among them, QoE std The best MOS value for different screen sizes at different video resolutions. ScreenSize and ScreenFrameRate represent the screen size and refresh rate of the player, respectively. The MOS data in WaterlooSQoE-IV is linearly mapped to [1,5], and the standardized resolution NR is defined, and its calculation formula is given by formula (2); Among them, A=-0.6471, B=80.69, C=15.124; VR x , VR y , SR x , SR y Represents the resolution of the video and screen in the x and y directions respectively; When the display resolution is smaller than the video resolution, that is As the display resolution increases, R gradually decreases and NR increases, indicating that when the display resolution gradually matches the video resolution, the image distortion caused by the merged pixels is alleviated and the user experience is improved; when the display resolution continues to increase, When the screen and video resolutions lose match, NR drops rapidly, indicating that pixel interpolation causes distortion and leads to a decline in user experience. PI and NR are used as parameters to measure the video presentation effect of the terminal. After obtaining the physical impact PI and the normalized resolution NR, the calculation formula of DQP is defined (3); In the generation of PQP, VQA has peak signal-to-noise ratio PSNR, VMAF and SSIM; PSNR is based on the mean square error between the original image and the distorted image, and is used to evaluate the image and video coding performance; SSIM is a structured VQA metric that considers information in three aspects: brightness, contrast, and structure; By comparing the mean, variance, and covariance of images, SSIM captures the human eye’s perception of image quality; The value of SSIM is between -1 and 1, with 1 indicating exact similarity and -1 indicating complete difference. VMAF is a VQA metric that combines multiple visual features, including brightness, contrast, color, and motion information, to assess video quality. VMAF is calculated based on a machine learning model using training data from human subjective ratings and is expressed in the form of a score, with a higher score indicating higher video quality. For NQP, network QoS parameters include bandwidth, network packet loss rate and jitter; bandwidth variation coefficient BCV and average bandwidth are defined It is used to describe the fluctuation of the network and the average bandwidth size near each time. The BCV at time k is given by (4); Where BW i is the network bandwidth at time i before time k. When i = 0, BW i =BW k ; N = 5, that is, BCV calculates the coefficient of variation of the network bandwidth at time k and within the 4 seconds before it to measure the network fluctuation around time k; After clarifying the calculation parameters of NQP, the calculation formula (5) of NQP at time k is defined, which is composed of real-time bandwidth, BCV and calculated; The PLCC and SRCC between DQP and user MOS are both greater than 0.5; DQP is a perception parameter that reflects the impact of user terminals on user experience; Since the value ranges of DQP, PQP, NQP and MOS are different, the values ​​of the three parameters are uniformly processed according to formula (6) to achieve stable QoE calculation; Where μ DQP ,μ PQP ,μ NQP and μ MOS is the mean of each parameter, σ DQP ,σ PQP ,σ NQP and σ MOS is the standard deviation of each parameter; set the transformed target threshold to 10 -5 , perception is achieved when the error between the QoE score and MOS_z is less than the threshold; Recency effect model: To simulate the impact of recency effect on user QoE score, a loop structure is introduced into the MUSQM model, that is, the QoE score at the previous moment is input into the QoE calculation model together with DQP, PQP, and NQP to calculate the QoE score; The weight of the video quality at the past moment in the evaluation is consistent with the length of human memory. According to the time interval between the past moment and the current moment, the weight of the video quality at that moment is calculated through the Hermann-Ebbinghaus forgetting curve to generate the QoE score of the past moment after simulating the recency effect. The relative strength of memory is set to 2, that is, the recency effect generated by the QoE scores of the two moments before the current moment is considered. A n is the memory retention rate, N is the relative strength of memory, n is the time interval, QoE n and QoE' n is the QoE score and its recency effect value at time interval n from the current time; The workflow of the recency effect model is as follows: if the current moment is i, the QoE scores of moments i-1 and i-2 are stored in the model; if i-1 or i-2 is less than 0, the QoE memory value at that moment is 0; when calculating the QoE score at the current moment, the model outputs the two stored historical QoE scores to the memory curve model, generates a historical score after recency effect simulation, and outputs it to the input end of the QoE calculation model; after generating the QoE score at this moment, the recency effect model reads the current output QoE from the QoE score output end i And execute formula (8) to achieve real-time update of the model; Video QoE calculation model; QoS parameter conversion model and recency effect model are designed to perceive user experience, and the video QoE calculation model is used to quantify user experience; the MUSQM model is divided into two parts: a computing training model and a computing application model. The parameter input part and the computing part of the two models are the same; the training model uses the multi-layer GRU structure in deep learning to separate the training and application modules, thereby improving the MUSQM perception speed.

6. The method for perceiving ultra-high-definition streaming media quality of experience according to claim 1, characterized in that: The computation training model and the computation application model are composed of multi-layer GRUs. At each time step, the QoS parameter model simultaneously inputs the computation parameters at the current moment into the multi-layer GRUs. After the computation of each layer of GRU is completed, the GRUs are uniformly output to different GRU integration modules for processing. The GRU layer consists of H, X, R, Z and It consists of five parts; H is the hidden layer state, X is the input of the model; R is the update gate, which controls the update amplitude of the past state and the current input at the next moment; Z is the reset gate, which determines how to fuse the past state and the current input information; is the candidate hidden state determined by the current input, which is fused with the input of other gates to generate the current hidden state H i , the specific calculation method of each part is as shown in formula (9); In the above formula, W xr ,W hr ,W xz ,W hz ,W xh ,W hh Respectively represent the transfer matrix between input and update gate and hidden layer and update gate, input and reset gate and hidden layer and reset gate, input and hidden layer, and hidden layer and candidate hidden layer; b r ,b z ,b h They are update gate, reset gate, and hidden layer bias respectively; GRU is selected as the basic unit of the video QoE calculation model, and a multi-layer GRU structure is constructed to capture the long-term dependencies in the input sequence. Each layer of GRU uses an independent gating mechanism to retain and update useful information in the time step and pass it to the next layer. Each layer of GRU outputs the current hidden state H i ; Use formula (10) to construct H of each layer of GRU i With the model output y i The relationship between input QoS and output QoE is realized. y i =Wh y ·H i +b y (10) Use MUSQM with different numbers of GRU layers for testing, and select the optimal number of layers based on the results.

7. The method for perceiving ultra-high-definition streaming media quality of experience according to claim 1, characterized in that: The computational training model uses the BP-Ada optimizer as a GRU integration module, which is responsible for calculating the QoE of each time step and calculating the sequence perception error after the entire video sequence is processed; the BP-Ada optimizer is composed of the BP algorithm and the Adagrad algorithm; the BP algorithm calculates the gradient for adjusting the GRU weights based on the perception error; the Adagrad algorithm uses these gradients to generate the optimized values ​​of the weights of each layer, and then re-injects the optimized weights into the multi-layer GRU to achieve MUSQM optimization; In the later stage of training, when the gradient gradually stabilizes, the learning rate will gradually decrease, which helps to stabilize the training process and prevent excessive parameter updates. The core formula of Adagrad is shown in formula (11); In the above formula, G is the weight before Adagrad adjustment, g is the gradient of G; r is the cumulative square gradient, G' is the adjusted weight; ε is the global learning rate, set ε = 0.3; δ is a very small positive number used to ensure that The denominator of is not zero; In the calculation of weight gradients in Adagrad, the BP algorithm is used to reversely process the sequence according to the output error after a video sequence is processed, calculate the weights and their gradients that are initially optimized according to the error, and pass the gradients into the Adagrad algorithm for further optimization; the process of calculating each weight and deviation gradient using the BP algorithm is completed by formulas (12)-(14); Loss is the error between the average QoE score of the current video and the user MOS; Where N is the current time, and gradually moves backward from the end of the sequence to the starting point; dH next is the change value of the hidden layer at time N-1; gW xr wxya hr ,gW xz ,gW hz ,gW xh ,gW hh ,gW hy ,gb r ,gb z ,gb h ,gb y is the gradient of each parameter; substituting the gradient into (10), we get the BP-Adagrad optimization value of the weight; through multiple iterations, the error is reduced, and the user QoE is accurately perceived; The BP and Adagrad algorithms are combined for GRU optimization, providing adaptive learning rate adjustment for gradient changes at different time steps and complex parameter updates, thereby improving efficiency and stability during training and helping MUSQM better capture patterns and dependencies in sequence data.

8. The method for perceiving ultra-high-definition streaming media quality of experience according to claim 1, characterized in that: The calculation training model uses the output calculator as the GRU integration module, according to the H output of each layer i Directly calculate the output of the MUSQM model; The WaterlooSQoE-IV dataset is brought into the MUSQM model for testing to generate real-time QoE scores for video data; after completing the perception of each complete sequence, the average QoE score of the sequence is calculated; After all sequences in the entire data set have been calculated, the correlation coefficient between the average QoE score of each sequence and the corresponding MOS is calculated to evaluate the perceived effect of the QoE model output on the user's MOS.

9. The method for perceiving ultra-high-definition streaming media quality of experience according to claim 8, characterized in that: The number of GRU layers is related to the perceptual effect of the model; the PSNR with the highest correlation coefficient with MOS is used as the test PQP in the layer selection experiment, and the global learning rate is set to 0.1; in the range of GRU layers from 10 to 20, the optimal number of GRU layers is selected by calculating the minimum MAE of each layer in 500 iterations.

Images