Video dynamic quality assessment system based on perception and memory

Abstract

This paper proposes a video dynamic quality assessment system based on perception and memory. This system, powered by a computer system, employs a PAM‑DVQI (Peripheral Video Dynamic Quality Assessment) model, comprising a perceptual feature extraction module and a long-short-term memory (LSTM) module, to output a user experience quality score for input videos. This model is constructed and trained using a subjective video database, taking into account the visual attention mechanism of the High-Level Visual Score (HVS) and the temporal dependence of memory effects. The proposed model combines a ResNet and LSTM network to achieve significant performance improvements over existing technologies, as demonstrated by comparative algorithms and ablation experiments.

Description

Technical Field

[0001] The present invention belongs to the technical field of video quality evaluation, and in particular relates to a video dynamic quality evaluation system based on perception and memory. Background Art

[0002] Due to the variability of network environments, video playback is prone to freezes and bitrate fluctuations, seriously affecting the end-user experience. Accurately assessing video quality is crucial to optimizing network resource allocation and improving the user viewing experience.

[0003] Existing video quality evaluation schemes are mainly targeted at short videos, and generally focus on the visual perception characteristics of the human eye, with less consideration of the human memory characteristics' ability to store and express visual information, as well as the interaction between visual perception and memory characteristics.

[0004] Compared with short videos, users watch a relatively large proportion of long videos, and the quality requirements for long videos are relatively higher. The video quality evaluation schemes in existing technologies are not effective in judging long videos and are inconsistent with the actual user experience quality. Summary of the Invention

[0005] In view of this, the present invention proposes a dynamic evaluation scheme for videos. In addition to considering perception factors, memory factors are also introduced to better measure the quality evaluation of long videos. To this end, the present invention proposes a video dynamic quality evaluation scheme based on perception and memory, and constructs an evaluation system with practical value to improve the user experience quality (QoE), especially for the prediction performance of long videos.

[0006] During the design process, this paper first explored the impact of visual perception and human memory on user experience quality under different video playback modes through subjective experiments, and constructed a video quality database with perception and memory (PAM-VQD). Secondly, based on the PAM-VQD database, a deep learning method was used, combined with a visual attention mechanism, to extract deep perceptual features of the video to accurately assess the impact of perception on user experience quality. The perceptual quality score, playback status, and self-stuttering interval output by the front-end network were input as three features into a long-short-term memory network to establish a temporal dependency between visual perception and memory characteristics. Finally, a comparative algorithm and ablation experiments were conducted to analyze the performance of the PAM-DVQI model.

[0007] The technical solution adopted by the present invention to solve the technical problem is:

[0008] A video dynamic quality assessment system based on perception and memory:

[0009] Based on a computer system, a video dynamic quality assessment model PAM-DVQI, which includes a perceptual feature extraction module and a long short-term memory module, is used to score the user experience quality of the input video output;

[0010] The video dynamic quality evaluation model is constructed and trained based on a subjective video database, taking into account the visual attention mechanism of HVS and the temporal dependency of the memory effect, and combining a ResNet network and an LSTM network.

[0011] Furthermore, the perceptual feature extraction module uses the residual network ResNet as the front-end network, performs data normalization on the input video frame, and uses four residual units as feature extractors. Its calculation formula is shown in (1):

[0012]

[0013] Among them, x and y represent the input and output feature maps of the residual module, W is the parameter of the convolution layer, and the function represents the residual mapping obtained through training;

[0014] An effective channel attention module (ECA) is set after the four residual units to optimize the feature expression capability of the ECA-ResNet network.

[0015] Perform a global average pooling operation on the feature map output by the ECA-ResNet network to obtain the perceptual features corresponding to all video frames; establish a mapping between the perceptual features and the user's instantaneous perceptual quality:

[0016] The long-term short-term memory module is modeled using three features: IPQ, PS, and TS. IPQ is the user's instantaneous perceived quality, PS is a binary variable used to identify whether the video is in a playing state or a stuck state, and TS is a variable for the time interval between two stuck events.

[0017] In order to capture the user's memory characteristics, LSTM is used l,d The network is modeled on the back end, which is composed of multiple LSTM units cascaded together to build a mapping between perception and memory and user experience quality. The specific calculation formulas are shown in (2) to (4):

[0018] p(y(t)|y(t-1),y(t-2),…,y(1))=p(y(t);g(c(t))) (2)

[0019]

[0020]

[0021] The conditional probability p(y(t)|y(t-1),y(t-2),…,y(1)) indicates that the user experience quality has a high-order time dependency. g(·) is a differentiable function that maps the memory state c(t) in the LSTM unit to the potential QoE distribution. c(1:t-1) is the memory state at time t-1. is the QoE prediction value at time t-1, c(t) is the memory state at time t; is the predicted value of user experience quality at time t, x(t) is the input feature vector, and c(t-1) is the memory state at time t-1.

[0022] Furthermore, the LSTM network of the long short-term memory module is configured as 2 LSTM layers and 11 units.

[0023] Furthermore, the video subjective database is specifically a video subjective database constructed by taking into account the impact of visual perception characteristics and memory characteristics on user experience quality, and the duration of the video meets preset conditions.

[0024] Furthermore, the videos in the subjective video database include multiple play modes, each play mode includes multiple videos, and corresponds to the user's subjective quality ratings of the videos in different play modes.

[0025] Compared with the existing technology, the present invention and its preferred solution have significant advantages in terms of the performance analysis results of the proposed model based on comparative algorithms and ablation experiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0027] Figure 1 A schematic diagram of the process of constructing a video dynamic quality assessment system based on perception and memory provided by an embodiment of the present invention;

[0028] Figure 2 Schematic diagram of the overall structure of a video dynamic quality assessment model based on perception and memory according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] To make the features and advantages of this patent more clearly understood, the following embodiments are specifically described in detail as follows:

[0030] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by those skilled in the art to which this application belongs.

[0031] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0032] like Figure 1 As shown, this embodiment first introduces the construction scheme of the core model of the video dynamic quality evaluation system based on perception and memory of the present invention, which mainly considers the visual perception characteristics and human memory characteristics, and includes the following steps:

[0033] Step S1: Considering the impact of visual perception and memory characteristics on user QoE, a subjective database of videos with a long duration is constructed;

[0034] Specifically:

[0035] Step S11: Select high-quality videos with a wide range of scenes from the online video website YouTube as experimental source videos;

[0036] Step S12: Use Adobe Premiere Pro software to process the source video and design test sequences with different playback modes. In this embodiment, 8 video playback modes are set, with 10 videos in each mode.

[0037] Step S13: Design a subjective experiment to obtain users' subjective video quality scores under different playback modes, and establish a perception and memory-based video subjective database (PAM-VQD).

[0038] Step S2: Based on the constructed database, considering the visual attention mechanism of HVS and the temporal dependency of memory effect, a video quality assessment model (PAM-DVQI) is constructed by combining the ResNet network and the LSTM network;

[0039] Specifically:

[0040] Step S21: Use the ResNet as the front-end network. Perform data normalization on the input video frame and use four residual units as feature extractors. The calculation formula is shown in (1):

[0041]

[0042] Among them, x and y represent the input and output feature maps of the residual module, W is the parameter of the convolution layer, and the function Represents the residual mapping to be learned.

[0043] Step S22: To simulate the guiding effect of saliency information on user visual memory, an effective channel attention (ECA) module is placed after the four residual units to optimize the feature expression capability of the ECA-ResNet network.

[0044] Step S23: performing a global average pooling operation on the feature map output by the ECA-ResNet network to obtain the perceptual features corresponding to all video frames;

[0045] Step S24: establishing a mapping between the perception feature and the user's instantaneous perception quality;

[0046] Step S25: Three features (IPQ, PS, and TS) are used for modeling, where IPQ is the user's instantaneous perceived quality, PS is a binary variable indicating whether the video is in a playing state or a stuck state, and TS is a variable indicating the time interval between two stuck events.

[0047] Step S26: User QoE is not only affected by the current playback event, but also by the user's own memory. In order to capture the user's memory characteristics, this embodiment uses LSTM l,d The network is used for back-end modeling. A single LSTM cannot effectively model the problem, so multiple LSTM units are cascaded to build a mapping between perception and memory and user QoE. The specific calculation formulas are shown in (2) to (4):

[0048] p(y(t)|y(t-1),y(t-2),…,y(1))=p(y(t);g(c(t))) (2)

[0049]

[0050]

[0051] The conditional probability p(y(t)|y(t-1),y(t-2),…,y(1)) indicates that user QoE has a high-order temporal dependency. g(·) is a differentiable function that maps the memory state c(t) in the LSTM unit to the potential QoE distribution. c(1:t-1) is the memory state at time t-1. is the QoE prediction value at time t-1, c(t) is the memory state at time t; is the QoE prediction value at time t, x(t) is the input feature vector, and c(t-1) is the memory state at time t-1.

[0052] This constructs the Figure 2 Evaluation model shown.

[0053] The database constructed in step S1 is used to train the model obtained in step S2 to optimize and adjust the parameters of the model.

[0054] The key points include evaluating the impact of the number of LSTM layers and units on QoE prediction performance. Experiments in this embodiment found that the best QoE prediction performance was achieved when the LSTM network was configured with 2 LSTM layers and 11 units.

[0055] Step S3: Finally, a comparative algorithm and ablation experiment are conducted to perform performance analysis on the PAM-DVQI model obtained in this embodiment to verify its effectiveness.

[0056] Among them, the database PAM-VQD and LIVE, Waterloo SQoE-Ⅰ and LFOVIA QoE video databases were selected for comparative experiments. The comparison algorithms were the common FR-VQA algorithms: PSNR, SSIM, MS-SSIM, RR-VQA algorithms: SpEED-QA and STRRED, and NR-VQA algorithm: BRISQUE

[33] , NIQE and BIQI, as well as deep learning-based algorithms: VGG, AlexNet and DeepVQA; analyze the performance of each algorithm under the four databases;

[0057] First, we analyzed the overall model performance and calculated the weighted average PLCC, KRCC, and SRCC of each algorithm across four databases. Experimental results show that the PAM-DVQI model proposed in this example outperforms other VQA comparison algorithms and exhibits high correlation across different databases, making it significantly superior for user QoE assessment.

[0058] Secondly, ablation experiments were conducted to analyze the performance gain of the ECA module. The PAM-DVQI model performance was analyzed by removing the ECA module. Results showed that the ECA module improved overall performance. Furthermore, the ECA module was removed and other attention modules were embedded in the model. The overall performance was analyzed, and the results showed that the ECA module improved performance even more significantly. The experiments showed that embedding the ECA module effectively improved the PAM-DVQI model's performance in evaluating user QoE.

[0059] Finally, an ablation experiment analyzes the performance gain of the PAM-DVQI model using IPQ, PS, and TS features. The following feature combinations are input to the network: (a) IPQ, (b) PS, (c) TS, (d) IPQ+PS, (e) IPQ+TS, (f) PS+TS, and (g) IPQ+PS+TS. The results show that the IPQ+PS+TS feature combination used in this embodiment achieves the best prediction performance.

[0060] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0061] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0062] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0063] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0064] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

[0065] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive various other forms of video dynamic quality evaluation systems based on perception and memory under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of this invention should be covered by this patent.

Claims

1. A video dynamic quality assessment system based on perception and memory, characterized by: Based on a computer system, a video dynamic quality assessment model PAM-DVQI, which includes a perceptual feature extraction module and a long short-term memory module, is used to score the user experience quality of the input video output; The video dynamic quality evaluation model is constructed and trained based on a subjective video database, taking into account the visual attention mechanism of HVS and the temporal dependency of the memory effect, and combining the ResNet network and the LSTM network. The perceptual feature extraction module uses the residual network ResNet as the front-end network, performs data normalization on the input video frame, and uses four residual units as feature extractors. Its calculation formula is shown in (1): （1） in, and represents the input and output feature maps of the residual module, is the parameter of the convolution layer, function represents the residual mapping obtained through training; An effective channel attention module (ECA) is set after the four residual units to optimize the feature expression capability of the ECA-ResNet network. Perform a global average pooling operation on the feature map output by the ECA-ResNet network to obtain the perceptual features corresponding to all video frames; establish a mapping between the perceptual features and the user's instantaneous perceptual quality: The long-term short-term memory module is modeled using three features: IPQ, PS, and TS. IPQ is the user's instantaneous perceived quality, PS is a binary variable used to identify whether the video is in a playing state or a stuck state, and TS is a variable for the time interval between two stuck events. In order to capture the user's memory characteristics, LSTM is used l,d The network is modeled on the back end and is composed of multiple LSTM units cascaded to build a mapping between perception and memory and user experience quality. The specific calculation formulas are shown in (2) to (4): （2） （3） （4） Among them, the conditional probability Indicates that the user experience quality has a high-order time dependency, Is a differentiable function used to convert the memory state in the LSTM unit Mapping to potential QoE distribution; for The memory state of the moment, for QoE prediction value at time for The state of memory at the moment; for The predicted value of user experience quality at the moment, is the input feature vector, for The memory state of the moment.

2. The video dynamic quality assessment system based on perception and memory according to claim 1, characterized in that: The LSTM network of the long short-term memory module is configured as 2 LSTM layers and 11 units.

3. The video dynamic quality assessment system based on perception and memory according to claim 1, characterized in that: The video subjective database is specifically a video subjective database whose duration meets preset conditions and is constructed by considering the influence of visual perception characteristics and memory characteristics on user experience quality.

4. The video dynamic quality assessment system based on perception and memory according to claim 3, characterized in that: The videos in the video subjective database include multiple play modes, each play mode includes multiple videos, and corresponding users have subjective quality scores of the videos in different play modes.

Images