A two-stage full-reference video quality assessment method based on adaptive sampling and multi-scale temporal sequencing

Abstract

The present invention belongs to the field of image and video processing, and discloses a full-reference video quality evaluation method based on two stages of adaptive sampling and multi-scale temporal sequence, comprising the following steps: Step 1: obtaining a training sample set Xtrain and a test sample set Xtest; Step 2: for each video V in the training set, d With V r Adaptive sampling is performed: Step 3: Construct a multi-scale spatiotemporal feature extraction network (MTN); Step 4: Construct a quality decision module based on a multi-channel self-attention mechanism; Step 5: Iteratively train the full-reference video quality assessment model (MTN); Step 6: Obtain quality assessment results for the full-reference video. The solution of the present invention can measure the perceptual differences between original and distorted videos and can be used for quality detection and assessment in video compression, transmission, and processing.

Description

Technical Field

[0001] The present invention belongs to the field of image and video processing, and in particular relates to a full-reference video quality evaluation method based on two stages of adaptive sampling and multi-scale timing. Background Art

[0002] As a vital medium for information transmission on the internet, video has a vast array of applications. As video plays an increasingly crucial role in people's lives, improving the efficiency of video encoding, storage, and network data transmission is crucial. This is a key goal of video sharing and streaming platforms. While most modules in the entire video chain can be accurately measured, such as acquisition, upload, preprocessing, transcoding, and distribution, the unknown component—the user's video viewing experience—is crucial. Because subjective video quality assessment (VQA) requires professional observers to score videos, it is labor-intensive and inefficient, making it impractical for industrial production.

[0003] Based on the above problems, researchers are looking for objective computational models to automatically evaluate video quality, replacing inefficient manual subjective judgment. Existing objective video quality evaluation methods can be divided into three categories based on the amount of reference video information required:

[0004] The first category is non-reference video quality assessment methods. These methods do not require the original reference video and directly use computational models to predict the quality of the test video. Due to the lack of reference video information, the model's performance is poor, limiting its practical application scenarios.

[0005] The second category is partial-reference video quality assessment methods. These methods extract a certain number of features from the original reference video and evaluate the quality of the test video by comparing the feature differences between the reference and test videos. Because only a portion of the reference video information is available, the evaluation results, while somewhat improved compared to no-reference evaluation, still differ significantly from subjective evaluation, limiting their application.

[0006] The third category is the full-reference video quality assessment method. This method requires the use of all original reference video information and compares the difference between the original reference video and the distorted video to give a quality score for the test video. Because it uses the entire reference video information, its assessment accuracy is the best of the three methods. Researchers have proposed many technical solutions for this type of method, such as:

[0007] Kim et al. proposed DeepVQA, which uses CNN and Convolutional Neural Aggregation Networks (CNAN) to quantify spatiotemporal visual perception. Drawing on the concept of the "attention mechanism," CNAN was proposed to weight the predicted quality score of each frame. In its temporal pooling approach, the model considers the overall distribution of predicted scores, rather than the quality scores of individual frames, to improve model evaluation. Compared to models without CNAN, the model using CNAN improved overall prediction performance. However, 2D convolutions struggle to preserve temporal information. To better capture temporal features in videos, 3D convolutions were proposed to process video information.

[0008] Xu et al. proposed the C3DVQA method. They use 3D convolution to learn spatiotemporal features, capture temporal masking effects in videos, and simulate the HVS quality assessment process. The introduction of 3D convolution improves model performance, but further scaling to larger datasets and conducting more comprehensive experiments are needed to validate the model's performance.

[0009] Li et al., believing that object motion affects human visual experience, innovatively proposed a 3D-CNN network model for evaluating human motion quality in complex motion scenes. Feature maps were used to extract local similarities between reference and distorted videos, and weights were then combined to derive an overall image quality score. As a full-reference model, its evaluation performance was mediocre, likely due to the fact that it only considered the quality effects of objects in motion, which carries many uncertainties and is difficult to fully account for. Further data preprocessing is needed to improve the model's performance. This approach uses the residual frame between the reference and distorted frames as input, and adds different types of input to obtain more features.

[0010] In summary, FR-VQA is garnering increasing attention, and this field urgently needs a model that aligns with human visual perception. However, most existing FR-VQA models perceive video sequences at a fixed time scale. Extracting only a single fixed-scale spatiotemporal feature is insufficient for characterizing the diverse content of video sequences in a dataset. Intuitively, videos with frequent scene changes require a smaller-scale LSTM network to capture short-term dependencies, while videos with fewer scene changes require a larger scale to capture long-term dependencies. Therefore, establishing a prediction model that can effectively extract and aggregate multi-scale spatiotemporal features is crucial for video quality assessment. Summary of the Invention

[0011] The purpose of the present invention is to provide a full-reference video quality assessment method based on two stages of adaptive sampling and multi-scale timing to solve the above-mentioned technical problems.

[0012] To solve the above technical problems, the specific technical solutions of the full-reference video quality assessment method based on two stages of adaptive sampling and multi-scale temporal sequence of the present invention are as follows:

[0013] A full-reference video quality assessment method based on two stages of adaptive sampling and multi-scale temporal sequence includes the following steps:

[0014] Step 1: Get the training sample set Xtrain and the test sample set Xtest:

[0015] Step 2: For each video V in the training set d With V r Perform adaptive sampling:

[0016] Step 3: Construct a multi-scale spatiotemporal feature extraction network MTN:

[0017] Step 4: Construct a quality decision module based on the multi-channel self-attention mechanism:

[0018] Step 5: Iteratively train the full-reference video quality assessment model MTN:

[0019] Step 6: Obtain the quality evaluation results of the full reference video.

[0020] Furthermore, the step 1 includes the following specific steps:

[0021] A public dataset of natural videos is obtained. For each of the V distorted video frame sequences, a portion of the frame sequences and their corresponding true quality scores are randomly selected to form the training sample set Xtrain. The remaining frame sequences and their corresponding true quality scores are combined into the test sample set Xtest. Once a reference video is assigned to the training set or test set, all distorted videos generated by it will be placed in the corresponding dataset to ensure that there is no intersection between the training set and the test set.

[0022] Furthermore, the step 2 includes the following specific steps:

[0023] Step 2a: Determine the number of segments and the length of each sequence:

[0024] For each training video V d With V r The test video is usually between 300 and 600 frames, and the length is T = 18 frames. The test video is divided into n sequence segments: V d ={V1 d ...V i d ...V n d}, V r ={V1 r ...Vi r ...V n r}Where V i d With V i r represents the i-th test distorted segment and the reference segment, i = 1, 2, ..., n, n ≥ 2 and n is an integer;

[0025] Step 2b: Determine the starting point of each subsequence:

[0026] For each test segment in Indicates the frame with the largest PMD index in the i-th test sequence, V i d Indicated as this frame is the sequence segment from the starting point to T frames later, V i r The same applies;

[0027] Step 2c: Calculate spatial distortion:

[0028] The spatial distortion is presented in the form of an image of each frame. The static image quality evaluation methods include multi-scale structural similarity MS-SSIM, feature similarity FSIM and gradient similarity deviation GMSD. Gradient similarity deviation is used for static quality estimation, and character FQ is used for static quality evaluation. S (t) indicates;

[0029] Step 2c1: The calculation of the gradient magnitude requires the use of two 3x3 Prewitt operators in orthogonal directions to convolve with the reference image r and the distorted image d respectively to obtain the horizontal and vertical gradients of r and d. The root mean square of the gradients in the two orthogonal directions represents the image gradient magnitude. The 3x3 Prewitt operators h in the horizontal and vertical directions are x , h y is defined as:

[0030]

[0031] m r (i) and m d (i) represents the gradient magnitude at position i, Represents the convolution operation:

[0032]

[0033]

[0034] Step 2c2: Based on the calculation results of step 2c1, calculate the gradient amplitude similarity of each region i to measure its distortion level GMS:

[0035]

[0036] Step 2c3: GMS average pooling of K regions is performed to obtain the average value of gradient magnitude similarity GMSM to measure the quality of the entire image:

[0037] Step 2c4: Perform standard deviation pooling on each region to obtain the image quality GMSD(t) at time t:

[0038]

[0039] Step 2d: To fully characterize the impact of temporal masking, describe both the intensity and direction of motion:

[0040] Step 2d1: The video frame is divided into N blo The blocks do not overlap the 48×48 macroblock size, and the mean value of the macroblock-level motion vector amplitude is used to represent the motion intensity of the current frame:

[0041]

[0042] where m i,j (t) refers to the motion vector amplitude of the macroblock at position j in frame i at time t;

[0043] Step 2d2: By calculating the optical flow histogram HOF(t,k) of the video frame, the optical flow amplitude distributed in the k-angle range at time t is obtained. The dominant motion direction θ of the motion vector within the frame is determined by the optical flow with the largest amplitude. The dominant motion direction of the t-th frame is obtained by counting all the directional optical flows:

[0044]

[0045] Divide 2π into 8 angle ranges;

[0046] Step 2d3: Define the temporal masking level FM T (t) is the product of the frame-level motion intensity and the dominant motion direction:

[0047]

[0048] Represents the normalization operation, the degree of motion FM T (t) is linearly normalized between 0 and 1, and the closer its value is to 1, the more obvious the movement;

[0049] Step 2d4: Calculate the PMD for each frame:

[0050]

[0051] To ensure that the denominator is non-zero, γ is taken as a constant with a value of 1.

[0052] Furthermore, the step 3 includes the following specific steps:

[0053] A multi-scale spatiotemporal feature extraction network (MTN) was constructed, which serially connected the spatial feature extraction module (SFE) and the multi-scale temporal feature extraction module (MTFE). The spatial feature extraction module (SFE) adopted the VGG-16 network structure with the last fully connected layer removed. The temporal multi-scale module (MTFE) consisted of three parallel stacked LSTM network layers, two downsampling modules consisting of multiple average pooling layers, and an upsampling module consisting of two adjacent interpolation layers.

[0054] Step 3a: Construct the spatial feature extraction module SFE:

[0055] The spatial feature extraction module SFE includes 5 convolution modules and 2 fully connected layers. Its specific structure is: convolution module 1 → convolution module 2 → convolution module 3 → convolution module 4 → convolution module 5 → fully connected layer 1 → fully connected layer 2;

[0056] Convolutional module 1 and convolutional module 2 both consist of two convolutional layers and one maximum pooling layer. The specific structure is: convolutional layer 1 → convolutional layer 2 → maximum pooling layer 1. The convolution kernel size of each convolutional layer is 3×3, and the stride is 1. The number of convolution kernels in both convolutional layers of convolutional module 1 is 64, and the number of convolution kernels in both convolutional layers of convolutional module 2 is 128.

[0057] Convolution modules 3, 4, and 5 are composed of three convolutional layers and one maximum pooling layer. The specific structure of the convolution module is: convolution layer 1 → convolution layer 2 → convolution layer 3 → the first maximum pooling layer. The convolution kernel size of the convolution layer is 3×3, and the stride is 1. The number of convolution kernels in the three convolution layers of convolution module 3 is set to 256, and the number of convolution kernels in the three convolution layers of convolution modules 4 and 5 is set to 512.

[0058] Step 3b: Construct the multi-scale temporal feature extraction module MTFE:

[0059] The time domain multi-scale module MTFE includes three parallel stacked LSTM network layers, two downsampling modules consisting of average pooling layers, and an upsampling module consisting of two adjacent interpolation layers. The sampling time steps {α1, α2, α3} are {0, 2, 4}, so the length of the LSTM network layer also decreases successively. To ensure that the feature dimensions between every two LSTM layers are the same, when aggregating from fine-grained features to coarse-grained features, a downsampling module is added to reduce the number of data channels; when aggregating from coarse-grained features to fine-grained features, the upsampling module increases the dimension of the data.

[0060] Furthermore, step 4 includes the following specific steps:

[0061] The attention analyzer included in the quality decision module consists of 3 fully connected layers and an average pooling layer.

[0062] Furthermore, the step 5 includes the following specific steps:

[0063] Step 5a: The initial learning rate is set to 0.0008, the Adam optimizer is used, the loss function is the L1 norm loss function, and the number of iterations is t d , the maximum number of iterations is T d , t d The video quality evaluation model of the iteration is M t , T d =200, and let t d =1,M t =M;

[0064] Step 5b: The training sample set Xtrain is used as the input of the full reference video quality assessment model Mt. The spatial feature extraction module SFE extracts spatial features from each sequence in Xtrain frame by frame to obtain the spatial distortion feature map S t =[γ sat ,γ blo ,γ sha ,γ con ,γ noi ];

[0065] Step 5c: The two downsampling layers and downsampling modules in the time domain multi-scale module MTFE downsample the input spatial feature map St respectively: in the first downsampling layer, St is temporally downsampled by temporal average pooling with a scale of 2; in the second downsampling layer, St is temporally downsampled by temporal average pooling with a scale of 4; in the sampling module, the local temporal distortion feature matrices of three scales are obtained by temporal average pooling with scales of 2 and 4 respectively. The three local spatiotemporal distortion feature matrices are defined as The three matrices are concatenated using the concat method to obtain the local distortion information of each frame sequence at different spatiotemporal scales. The spatiotemporal feature map is perceived using stacked LSTM layers. LSTM layers of different scales are represented as Therefore, the perceptual feature map is

[0066] Step 5d: Aggregate multi-scale perceptual feature maps:

[0067] Simulate the complex feedforward and feedback process in visual perception to fuse spatiotemporal features.

[0068]

[0069]

[0070]

[0071] In the following formula, h′ is the initial state of the network hidden layer, and the default settings are 0.1. δ(·), Represents the fully connected layer, ReLu activation layer, average pooling layer, q′ t represents the predicted score of the t-th sub-video;

[0072] Step 5e: Quality decision process based on multi-channel self-attention mechanism:

[0073] The semantic features of a complete subsequence can be expressed as The semantic features extracted at the worst perception moment are composed of characters Indicates that the scaling dot product method is used to compare and The similarity of the feature distribution of each video in the video is used to adaptively complete the weight distribution, where d h is the size of the hidden layer,

[0074]

[0075] From this we get i The quality of all test sequence fragments under the scale in is the quality of the i-th sequence at the corresponding scale, and the adaptive weight The final video score Q is expressed as the mean of the predicted scores at different time scales:

[0076]

[0077] Furthermore, step 6 includes the following specific steps:

[0078] The test sample set Xtest is used as the full reference video quality evaluation model M t The input is forward propagated to obtain the predicted quality score set of Xtest in Represents the video sequence V i The predicted quality score of the video is: since the calculated quality value Q of each test sample ranges from 0 to 1, the larger the value of Q, the more serious the pollution of the test sample. Therefore, the quality of the test video can be judged according to the size of the quality value Q: if Q = 0, it means that the test video is not polluted by noise; if 0 < Q ≤ 0.2, it means that the test video is slightly polluted by noise; if 0.2 < Q ≤ 0.5, it means that the test video is moderately polluted by noise; if Q > 0.5, it means that the test video is heavily polluted by noise.

[0079] The present invention's two-stage full-reference video quality assessment method, based on adaptive sampling and multi-scale temporal sequencing, has the following advantages: It integrates the perceptual mechanisms of visual neurons and uses spatiotemporal distortion metrics to propose an adaptive sampling strategy for distorted frames. It leverages the powerful modeling capabilities of recurrent neural networks (RNNs) for time series and utilizes stacked LSTM networks to extract spatiotemporal dependency features at multiple time scales. Furthermore, based on the self-attention mechanism in biological neuroscience, it adaptively assigns weights to the perceptual results at each scale according to the human eye's perceptual mechanisms. This approach can measure the perceptual differences between original and distorted videos and can be used for quality assessment in video compression, transmission, and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a flow chart of the full-reference video quality assessment method based on two stages of adaptive sampling and multi-scale timing of the present invention;

[0081] Figure 2 It is the overall block diagram of the model of the present invention. DETAILED DESCRIPTION

[0082] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of the full-reference video quality assessment method based on adaptive sampling and multi-scale timing in two stages of the present invention in conjunction with the accompanying drawings.

[0083] The full-reference video quality assessment method based on adaptive sampling and multi-scale temporal two-stages of the present invention includes the following three parts:

[0084] 1. Adaptive sampling strategy based on perception-dependent mechanism

[0085] Our adaptive sampling strategy is guided by the perceptual continuity theory of HVS, which states that an observer's perception of a video is context-dependent. The worst-perceived frame significantly impacts the quality of subsequent frames. Therefore, our sampling process explicitly considers the frame with the worst perceptual quality.

[0086] Compared to image quality assessment, when evaluating the perceptual quality of video frames, frames with intense motion can accommodate more distortion due to the motion masking effect. Therefore, it is necessary to comprehensively consider the degree of temporal motion and spatial distortion to characterize the perceptual quality of video frames.

[0087] Perceived Motion Energy (PME) is a comprehensive measure of motion intensity and characteristics, defining the degree of temporal motion in a video frame by frame. In the motion vector field of a video frame, the magnitude of the motion vector reflects the speed of macroblock motion and is commonly used to calculate the frame's motion energy. The angle of the motion vector indicates the possible motion direction of the macroblock, and the consistency of the motion vector angle reflects the overall motion intensity. Spatial motion consistency is determined by calculating the percentage of dominant motion directions across the entire coordinate system. The greater the consistency of the angle, the greater the overall motion intensity.

[0088] Gradient Magnitude Similarity Deviation (GMSD) is a high-performance IQA algorithm that can evaluate the degree of spatial distortion in video frames. Natural images often have a variety of local structures. When an image is distorted, different local structures will suffer varying degrees of distortion.

[0089] The PMD metric, which characterizes perceptual distortion, comprehensively considers the temporal motion index (PME) and the spatial distortion index (GMSD). PME is normalized to a range of 1-2. When PME = 1, the video frame exhibits no motion and the content remains static. The perceptual quality of the video at this time is the GMSD. When PME is greater than 1, the video frame exhibits motion masking, concealing some noise. Therefore, the perceptual quality should be superior to the image quality evaluation metric, resulting in a PMD value smaller than the GMSD.

[0090] Based on the above PMD index and combined with the perception continuity dependency mechanism, the test video V d And its reference video V r Divide into n sequence segments of the same length: V d ={V1 d ...V i d ...V n d}, V r ={V1 r ...V i r ...V n r}, where V i d represents the i-th test segment, V i d Denotes the i-th reference sequence segment, i=1,2,...,n, n≥2 and n is an integer. In the fixed interval sampling strategy of most existing models, if n video sequence segments of length T are sampled at intervals of length b frames, then Among them F i d With F i r are the i-th frame of the distorted video sequence and the reference video sequence respectively. The adaptive sampling strategy changes the starting point of the sequence segment according to the degree of image distortion, focusing on sampling video frames that have a significant impact on human perception. For a video sequence with a total length of N frames, the PMD index of each frame needs to be calculated first. The position of the maximum PMD in the i-th subsequence is P i max , select the video frame based on this position

[0091] 2. Multi-scale spatiotemporal feature extraction network

[0092] Most existing FR-VQA tasks transfer knowledge from image quality assessment to VQA, using a feature-based transfer learning framework to predict quality frame by frame. While this approach extracts a large number of spatial features, it ignores the rich temporal information present in the video. With this in mind, this paper proposes a Multi-scale Temporal Information Aggregation Network (MTN), which perceives temporal information at different time scales and extracts rich high-level semantic features.

[0093] The network is mainly composed of two modules, namely the spatial feature extraction module (SFE, Spatial Feature Extraction) and the multi-scale temporal feature extraction module (MTFE, Multi-Scale Temporal Feature Extraction).

[0094] First, the spatial feature extraction module. Considering that the dataset in FR-VQA is small, using only neural network to extract spatial features is easily affected by uneven data distribution, while traditional manual features have the advantages of low complexity and strong robustness. Therefore, the present invention not only extracts the spatial features of the sampled video segment V i d With V i r Extracting deep and abstract semantic features using the pre-trained VGG16 model vgg , and also extract statistically significant saturation features γ by traditional methods sat , block effect characteristics γ blo , sharpness feature γ sha , contrast feature γ con and noise characteristics γ noi Describe the spatial information of the representative frame, and obtain n spatial feature maps S by serial splicing t =[γ vgg ,γ sat ,γ blo ,γ sha ,γ con ,γ noi ].

[0095] Then, the temporal feature extraction module is introduced, which uses a stacked LSTM network to increase the network's receptive field and extract spatiotemporal dependent features at multiple time scales. The large-scale sampling network has a larger receptive field and a strong ability to represent semantic information, but has low feature resolution and lacks feature details, while the small-scale sampling network has the opposite. The TFE module combines the advantages and disadvantages of large-scale sampling and small-scale sampling, and perceives the video at three scales {α1, α2, α3} to obtain the α at time t. i Scaled temporal perception feature map

[0096] 3. Quality decision module based on multi-channel self-attention mechanism

[0097] During subjective video quality assessment, subjects will recall previously perceived distortions and tend to focus more on the worst perceived moments. This paper proposes a quality decision method based on multi-channel self-attention, which considers the perceived quality of all video segments and adaptively adjusts their importance. The semantic features of a complete subsequence can be expressed as The semantic features extracted at the worst perception moment are composed of characters Representation. Comparison by scaling dot product and The similarity of the feature distribution of each video in α is used to adaptively complete the weight distribution. i The quality of all test sequence fragments under the scale in is the quality of the i-th sequence at the corresponding scale. Since the multi-scale temporal information aggregation network proposed in the present invention can perceive videos at different time scales, the quality decision based on the self-attention mechanism is made for the content of the video at each sampling granularity, and the final video score Q is expressed as the mean of the predicted scores at different time scales. The quality of the test video can be judged according to the value of the video score Q: if Q = 0, it means that the test video is not contaminated by noise; if 0 < Q ≤ 0.2, it means that the test video is slightly contaminated by noise; if 0.2 < Q ≤ 0.5, it means that the test video is moderately contaminated by noise; if Q > 0.5, it means that the test video is heavily contaminated by noise.

[0098] Combining the HVS perception characteristics, this paper proposes a full-reference video quality evaluation method based on adaptive sampling and multi-scale timing in two stages. It is suitable for the selection of video codec frameworks in industry and helps enterprises achieve a balance between video perception quality and video codec rate. Figure 1 It is a flow chart of the implementation of the present invention. Figure 2 The following is a detailed description of the present invention in conjunction with the accompanying drawings and specific embodiments. Figure 1 , the present invention comprises the following steps:

[0099] Step 1) Obtain the training sample set Xtrain and the test sample set Xtest:

[0100] We obtain a publicly available dataset of natural videos. For each of the V distorted video frames, we randomly select 80% of the frames and their corresponding true quality scores to form the training sample set Xtrain. We combine the remaining 20% ​​of the frames and their corresponding true quality scores to form the test sample set Xtest. Once a reference video is assigned to either the training or test set, all distorted videos generated from it are placed into the corresponding dataset to ensure no overlap between the training and test sets. In this example, we conduct experiments on the LIVE database, with V = 150, Xtrain size of 120, and Xtest size of 30.

[0101] Step 2) For each video V in the training set d With V r Perform adaptive sampling:

[0102] Step 2a) Determine the number of segments and the length of each sequence:

[0103] For each training video V d With V r The length of the test video is usually between 300 and 600 frames. In this example, the test video is divided into n sequence segments according to the length of T = 18 frames: V d={V1 d ...V i d ...V n d}, V r ={V1 r ...V i r ...V n r} Where V i d With V i r represents the i-th test distorted segment and the reference segment, i=1, 2, ..., n, n≥2 and n is an integer, in this example n=10.

[0104] Step 2b) Determine the starting point of each subsequence:

[0105] For each test segment in Indicates the frame with the largest PMD index in the i-th test sequence, V i d Indicated as this frame is the sequence segment from the starting point to T frames later, V i r Similarly, calculating the PMD index of each frame in the video requires comprehensive consideration of spatial distortion and temporal masking.

[0106] Step 2c) Calculate the spatial distortion:

[0107] The spatial distortion is presented in the form of an image of each frame. Static image quality evaluation methods include multi-scale structural similarity MS-SSIM, feature similarity FSIM and gradient similarity deviation GMSD. This example uses gradient similarity deviation for static quality estimation, and character FQ S (t) indicates.

[0108] Step 2c1) To calculate the gradient magnitude, two 3x3 Prewitt operators in orthogonal directions are used to convolve the reference image r and the distorted image d, respectively, to obtain the horizontal and vertical gradients of r and d. The root mean square of the gradients in the two orthogonal directions represents the image gradient magnitude. x , h y is defined as:

[0109]

[0110] m r (i) and m d (i) represents the gradient magnitude at position i, Represents the convolution operation:

[0111]

[0112]

[0113] Step 2c2) Based on the calculation results of (2c1), calculate the gradient amplitude similarity of each region i to measure its distortion level GMS:

[0114]

[0115] Step 2c3) Perform GMS average pooling on a total of K regions to obtain the average gradient magnitude similarity GMSM to measure the quality of the entire image:

[0116] Step 2c4) Perform standard deviation pooling on each region to obtain the image quality GMSD(t) at time t:

[0117]

[0118] Step 2d) To fully characterize the impact of temporal masking, this example describes both the motion intensity and the motion direction:

[0119] Step 2d1) The video frame is divided into N blo The blocks do not overlap the 48×48 macroblock size, and the mean value of the macroblock-level motion vector amplitude is used to represent the motion intensity of the current frame:

[0120]

[0121] where m i,j (t) refers to the motion vector amplitude of the macroblock at position j in frame i at time t.

[0122] Step 2d2) Calculate the optical flow histogram HOF(t,k) of the video frame to obtain the optical flow amplitude distributed over the k-angle range at time t. The dominant motion direction θ of the motion vector within the frame is determined by the optical flow with the largest amplitude. Summarize all directional optical flows to obtain the dominant motion direction of frame t:

[0123]

[0124] In this example, 2π is divided into 8 angle ranges.

[0125] Step 2d3) Define the temporal masking level FM T (t) is the product of the frame-level motion intensity and the dominant motion direction:

[0126]

[0127] Represents the normalization operation, the degree of motion FM T (t) is linearly normalized between 0 and 1, and the closer its value is to 1, the more obvious the motion is.

[0128] Step 2d4) Calculate the PMD for each frame:

[0129]

[0130] To ensure that the denominator is non-zero, γ is taken as a constant with a value of 1.

[0131] Step 3) Construct a multi-scale spatiotemporal feature extraction network MTN:

[0132] A multi-scale spatiotemporal feature extraction network (MTN) was constructed, which serially connected the spatial feature extraction module (SFE) and the multi-scale temporal feature extraction module (MTFE). The spatial feature extraction module (SFE) used a VGG-16 network structure with the last fully connected layer removed. The multi-scale temporal feature extraction module (MTFE) consisted of three parallel stacked LSTM network layers, two downsampling modules consisting of multiple average pooling layers, and an upsampling module consisting of two adjacent interpolation layers.

[0133] Step 3a) Construct the spatial feature extraction module SFE:

[0134] The spatial feature extraction module SFE includes 5 convolution modules and 2 fully connected layers. Its specific structure is: convolution module 1 → convolution module 2 → convolution module 3 → convolution module 4 → convolution module 5 → fully connected layer 1 → fully connected layer 2.

[0135] Convolution module 1 and convolution module 2 are both composed of 2 convolution layers and 1 maximum pooling layer. The specific structure is: convolution layer 1 → convolution layer 2 → maximum pooling layer 1; the convolution kernel size of the convolution layer is 3×3, the stride is 1, the number of convolution kernels in the two convolution layers of convolution module 1 is 64, and the number of convolution kernels in the two convolution layers of convolution module 2 is 128.

[0136] Convolution modules 3, 4, and 5 are all composed of 3 convolution layers and 1 maximum pooling layer. The specific structure of the convolution module is: convolution layer 1 → convolution layer 2 → convolution layer 3 → the first maximum pooling layer; the convolution kernel size of the convolution layer is 3×3, the step size is 1, the number of convolution kernels of the three convolution layers of convolution module 3 is set to 256, and the number of convolution kernels of the three convolution layers of convolution modules 4 and 5 is set to 512.

[0137] Step 3b) Construct a multi-scale temporal feature extraction module MTFE:

[0138] The multi-scale temporal module (MTFE) consists of three parallel stacked LSTM layers, two downsampling modules consisting of average pooling layers, and an upsampling module consisting of two adjacent interpolation layers. In this example, the sampling time steps {α1, α2, α3} are {0, 2, 4}, so the length of the LSTM layers decreases accordingly. To ensure the consistency of feature dimensionality between each LSTM layer, a downsampling module is added to reduce the number of data channels when aggregating from fine-grained features to coarse-grained features; while an upsampling module is required to increase the data dimension when aggregating from coarse-grained features to fine-grained features. This stack of three LSTM layers extracts local spatiotemporal distortion features of the frame sequence at different spatiotemporal scales, enriching the distortion information of the frame sequence and obtaining a more comprehensive quality representation than the original feature map, effectively improving the accuracy of quality assessment.

[0139] Step 4) Construct a quality decision module based on the multi-channel self-attention mechanism:

[0140] The attention analyzer included in the quality decision module consists of 3 fully connected layers and an average pooling layer.

[0141] Step 5) Iteratively train the full-reference video quality assessment model MTN:

[0142] Step 5a) In this example, the initial learning rate is set to 0.0008, the Adam optimizer is used, and the loss function is the L1 norm loss function. Let the number of iterations be t d , the maximum number of iterations is T d , tth d The video quality evaluation model of the iteration is M t In this example, T d =200, and let t d =1,M t =M.

[0143] Step 5b) Use the training sample set Xtrain as the full reference video quality evaluation model M t The spatial feature extraction module SFE extracts spatial features from each sequence in Xtrain frame by frame to obtain the spatial distortion feature map S t =[γ sat ,γ blo ,γ sha ,γ con ,γ noi ].

[0144] Step 5c) The two downsampling layers and the downsampling module in the time domain multi-scale module MTFE respectively perform the input spatial feature map S t Downsampling: In the first downsampling layer, S is pooled by time average with a scale of 2. tPerform temporal downsampling; in the second downsampling layer, S is pooled by temporal average pooling with a scale of 4. t Perform time downsampling; in the sampling module, the time average pooling of scales 2 and 4 is performed to obtain the local time domain distortion feature matrices of three scales. The three local spatiotemporal distortion feature matrices are defined as The three matrices are concatenated using the concat method to obtain the local distortion information of each frame sequence at different spatiotemporal scales. The spatiotemporal feature map is perceived using stacked LSTM layers, and the LSTM layers of different scales are represented as Therefore, the perceptual feature map is

[0145] Step 5d) Aggregate multi-scale perceptual feature maps:

[0146] This example simulates the complex feedforward and feedback processes in visual perception to fuse spatiotemporal features.

[0147]

[0148]

[0149]

[0150] In the following formula, h′ is the initial state of the network hidden layer, and the default setting is 0.1. δ(·), Represents the fully connected layer, ReLu activation layer, average pooling layer, q′ t Represents the predicted score of the t-th sub-video.

[0151] Step 5e) Quality decision process based on multi-channel self-attention mechanism:

[0152] The semantic features of a complete subsequence can be expressed as The semantic features extracted at the worst perception moment are composed of characters Representation. Comparison by scaling dot product and The similarity of the feature distribution of each video in the video is used to adaptively complete the weight distribution, where d h is the size of the hidden layer.

[0153]

[0154] From this we get i The quality of all test sequence fragments under the scale in is the quality of the i-th sequence at the corresponding scale. Adaptive weight The final video score Q is expressed as the mean of the predicted scores at different time scales:

[0155]

[0156] Step 6) Obtain the quality evaluation results of the full reference video:

[0157] The test sample set Xtest is used as the full reference video quality evaluation model M t The input is forward propagated to obtain the predicted quality score set of Xtest in Represents the video sequence V i The predicted quality score. Since the calculated quality value Q for each test sample ranges from 0 to 1, with larger Q values ​​indicating more severe noise pollution, the quality of the test video can be judged based on the value of Q: if Q = 0, it indicates that the test video is not contaminated by noise; if 0 < Q ≤ 0.2, it indicates that the test video is slightly contaminated by noise; if 0.2 < Q ≤ 0.5, it indicates that the test video is moderately contaminated by noise; and if Q > 0.5, it indicates that the test video is heavily contaminated by noise.

[0158] The effect of the present invention can be further illustrated by the following simulation experiments:

[0159] Simulation 1: The quality assessment accuracy test of the present invention is performed on the public video quality assessment datasets LIVE, CSIQ, and IVPL. The experimental environment is an Inter Core I7-8700 (3.2GHz), 11G NVIDIA 1080Ti, 16GB memory, and the software environment is PyTorch 1.1.0, Ubuntu 2020, and Python 3.6.5.

[0160] The evaluation results are shown in the table below. The Spearman correlation coefficient (SROCC) on each database is above 0.93, which is better than the results of the existing full-reference video quality evaluation algorithm, indicating that the evaluation results of the present invention are more consistent with human perception results.

[0161] Table 1 Performance comparison of various full-reference video quality evaluation algorithms on various databases (the best and second-best results in each column are marked bold and underlined respectively)

[0162]

[0163]

[0164] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A full-reference video quality assessment method based on two stages of adaptive sampling and multi-scale temporal sequence, characterized in that: The steps include: Step 1: Get the training sample set Xtrain and the test sample set Xtest: Step 2: For each video V in the training set d With V r Perform adaptive sampling: Step 2a: Determine the number of segments and the length of each sequence: For each training video V d With V r The test video is usually between 300 and 600 frames, and the length is T = 18 frames. The test video is divided into n sequence segments: in and represents the i-th test distorted segment and the reference segment, i = 1, 2, ..., n, n ≥ 2 and n is an integer; Step 2b: Determine the starting point of each subsequence: For each test segment in Indicates the frame with the largest PMD index in the i-th test sequence, Indicated as this frame The sequence segment from the starting point to T frames later, The same applies; Step 2c: Calculate spatial distortion: The spatial distortion is presented in the form of an image of each frame. The static image quality evaluation methods include multi-scale structural similarity MS-SSIM, feature similarity FSIM and gradient similarity deviation GMSD. Gradient similarity deviation is used for static quality estimation, and character FQ is used for static quality evaluation. S (t) indicates; Step 2c1: The calculation of the gradient magnitude requires the use of two 3x3 Prewitt operators in orthogonal directions to convolve with the reference image r and the distorted image d respectively to obtain the horizontal and vertical gradients of r and d. The root mean square of the gradients in the two orthogonal directions represents the image gradient magnitude. The 3x3 Prewitt operators h in the horizontal and vertical directions are x , h y is defined as: m r (i) and m d (i) represents the gradient magnitude at position i, Represents the convolution operation: Step 2c2: Based on the calculation results of step 2c1, calculate the gradient amplitude similarity of each region i to measure its distortion level GMS: Step 2c3: GMS average pooling of K regions is performed to obtain the average value of gradient magnitude similarity GMSM to measure the quality of the entire image: Step 2c4: Perform standard deviation pooling on each region to obtain the image quality GMSD(t) at time t: Step 2d: To fully characterize the impact of temporal masking, describe both the intensity and direction of motion: Step 2d1: The video frame is divided into N blo The blocks do not overlap the 48×48 macroblock size, and the mean value of the macroblock-level motion vector amplitude is used to represent the motion intensity of the current frame: where m i,j (t) refers to the motion vector amplitude of the macroblock at position j in frame i at time t; Step 2d2: By calculating the optical flow histogram HOF(t,k) of the video frame, the optical flow amplitude distributed in the k-angle range at time t is obtained. The dominant motion direction θ of the motion vector in the frame is determined by the optical flow with the largest amplitude. The dominant motion direction of the motion vector in the t frame is obtained by counting all the directional optical flows: Divide 2π into 8 angle ranges; Step 2d3: Define the temporal masking level FM T (t) is the product of the frame-level motion intensity and the dominant motion direction: Represents the normalization operation, the degree of motion FM T (t) is linearly normalized between 0 and 1, and the closer its value is to 1, the more obvious the movement; Step 2d4: Calculate the PMD for each frame: To ensure that the denominator is non-zero, the value of γ is taken as a constant of 1; Step 3: Construct a multi-scale spatiotemporal feature extraction network MTN: A multi-scale spatiotemporal feature extraction network (MTN) was constructed, which serially connected the spatial feature extraction module (SFE) and the multi-scale temporal feature extraction module (MTFE). The spatial feature extraction module (SFE) adopted the VGG-16 network structure with the last fully connected layer removed. The temporal multi-scale module (MTFE) consisted of three parallel stacked LSTM network layers, two downsampling modules consisting of multiple average pooling layers, and an upsampling module consisting of two adjacent interpolation layers. Step 3a: Construct the spatial feature extraction module SFE: The spatial feature extraction module SFE includes 5 convolution modules and 2 fully connected layers. Its specific structure is: convolution module 1 → convolution module 2 → convolution module 3 → convolution module 4 → convolution module 5 → fully connected layer 1 → fully connected layer 2; Convolutional module 1 and convolutional module 2 are both composed of two convolutional layers and one maximum pooling layer. The specific structure is: convolutional layer 1 → convolutional layer 2 → maximum pooling layer 1. The convolution kernel size of the convolution layer is 3×3, and the stride is 1. The number of convolution kernels in the two convolutional layers of convolutional module 1 is 64, and the number of convolution kernels in the two convolutional layers of convolutional module 2 is 128. Convolution modules 3, 4, and 5 are composed of three convolutional layers and one maximum pooling layer. The specific structure of the convolution module is: convolution layer 1 → convolution layer 2 → convolution layer 3 → the first maximum pooling layer. The convolution kernel size of the convolution layer is 3×3, and the stride is 1. The number of convolution kernels in the three convolution layers of convolution module 3 is set to 256, and the number of convolution kernels in the three convolution layers of convolution modules 4 and 5 is set to 512. Step 3b: Construct the multi-scale temporal feature extraction module MTFE: The time-domain multi-scale module MTFE includes three parallel stacked LSTM network layers, two downsampling modules consisting of average pooling layers, and an upsampling module consisting of two adjacent interpolation layers. The sampling time steps {α1, α2, α3} are {0, 2, 4}, so the length of the LSTM network layer also decreases in sequence. To ensure that the feature dimensions between every two LSTM layers are the same, when aggregating from fine-grained features to coarse-grained features, a downsampling module is added to reduce the number of data channels; when aggregating from coarse-grained features to fine-grained features, the upsampling module increases the dimension of the data. Step 4: Construct a quality decision module based on the multi-channel self-attention mechanism: Step 5: Iteratively train the full-reference video quality assessment model MTN: Step 6: Obtain the quality evaluation results of the full reference video.

2. The full-reference video quality assessment method based on two stages of adaptive sampling and multi-scale temporal sequence according to claim 1 is characterized in that: The step 1 includes the following specific steps: A public dataset of natural videos is obtained. For each of the V distorted video frame sequences, a portion of the frame sequences and their corresponding true quality scores are randomly selected to form the training sample set Xtrain. The remaining frame sequences and their corresponding true quality scores are combined into the test sample set Xtest. Once a reference video is assigned to the training set or test set, all distorted videos generated by it will be placed in the corresponding dataset to ensure that there is no intersection between the training set and the test set.

3. The full-reference video quality assessment method based on two stages of adaptive sampling and multi-scale temporal sequence according to claim 1 is characterized in that: The step 4 includes the following specific steps: The attention analyzer included in the quality decision module consists of 3 fully connected layers and an average pooling layer.

4. The full-reference video quality assessment method based on two stages of adaptive sampling and multi-scale temporal sequence according to claim 1 is characterized in that: The step 5 includes the following specific steps: Step 5a: The initial learning rate is set to 0.0008, the Adam optimizer is used, the loss function is the L1 norm loss function, and the number of iterations is set to t d , the maximum number of iterations is T d , t d The video quality evaluation model of the iteration is M t , T d =200, and let t d =1,M t =M; Step 5b: The training sample set Xtrain is used as the input of the full reference video quality assessment model Mt. The spatial feature extraction module SFE extracts spatial features from each sequence in Xtrain frame by frame to obtain the spatial distortion feature map S t =[Υ sat ,Υ blo ,Υ sha ,Υ con ,Υ noi ]; Step 5c: The two downsampling layers and downsampling modules in the time domain multi-scale module MTFE downsample the input spatial feature map St respectively: in the first downsampling layer, St is temporally downsampled by temporal average pooling with a scale of 2; in the second downsampling layer, St is temporally downsampled by temporal average pooling with a scale of 4; in the sampling module, the local temporal distortion feature matrices of three scales are obtained by temporal average pooling with scales of 2 and 4 respectively. The three local spatiotemporal distortion feature matrices are defined as The three matrices are concatenated using the concat method to obtain the local distortion information of each frame sequence at different spatiotemporal scales. The spatiotemporal feature map is perceived using stacked LSTM layers. LSTM layers of different scales are represented as Therefore, the perceptual feature map is Step 5d: Aggregate multi-scale perceptual feature maps: Simulate the complex feedforward and feedback process in visual perception to fuse spatiotemporal features. In the following formula, h′ is the initial state of the network hidden layer, and the default settings are 0.

1. δ(·), Represents the fully connected layer, ReLu activation layer, average pooling layer, q′ t represents the predicted score of the t-th sub-video; Step 5e: Quality decision process based on multi-channel self-attention mechanism: The semantic features of a complete subsequence can be expressed as The semantic features extracted at the worst perception moment are composed of characters Indicates that the scaling dot product method is used to compare and The similarity of the feature distribution of each video in the video is used to adaptively complete the weight distribution, where d h is the size of the hidden layer, From this we get i The quality of all test sequence fragments under the scale in is the quality of the i-th sequence at the corresponding scale, and the adaptive weight The final video score Q is expressed as the mean of the predicted scores at different time scales:

5. The full-reference video quality assessment method based on two stages of adaptive sampling and multi-scale temporal sequence according to claim 1, characterized in that: The step 6 includes the following specific steps: The test sample set Xtest is used as the input of the full reference video quality evaluation model Mt for forward propagation to obtain the predicted quality score set of Xtest where Q Vi Represents the video sequence V i The predicted quality score of the video is: since the calculated quality value Q of each test sample ranges from 0 to 1, the larger the value of Q, the more serious the pollution of the test sample. Therefore, the quality of the test video can be judged according to the size of the quality value Q: if Q = 0, it means that the test video is not polluted by noise; if 0 < Q ≤ 0.2, it means that the test video is slightly polluted by noise; if 0.2 < Q ≤ 0.5, it means that the test video is moderately polluted by noise; if Q > 0.5, it means that the test video is heavily polluted by noise.

Images