Transform-based no-reference video quality evaluation method

Abstract

The application discloses a kind of based on Transformer's no reference video quality evaluation method, first download the public data set KoNViD-1k of video quality evaluation, including video data and corresponding label file, then construct the network model of based on Transformer's no reference video quality evaluation method, video frame image is sent to the layered visual transformer Swin-Transformer module of multiple cascaded use mobile window to extract video feature and corresponding quality score file;Data set is divided into training set, test set and verification set, and the corresponding label file is also divided into training set, test set and verification set in the same way;Finally load video feature file and quality score file are sent to the pyramid aggregation module based on feature fusion, train model, finally obtain the quality score of the video to be evaluated.The application can complete the quality evaluation of video.

Description

Technical Field

[0001] This invention belongs to the field of video quality assessment technology, specifically relating to a no-reference video quality assessment method based on Transformer. Background Technology

[0002] Video quality assessment is the process of evaluating video quality based on subjective human observation. It is of great significance for video surveillance and improving the user's visual quality experience.

[0003] In the evaluation method using natural image statistics, it is necessary to construct a statistical model of the transform domain coefficients and extract relevant feature information from the model for evaluation. However, obtaining the coefficient distribution increases the computational complexity of the model, and the diversity of noise and the complexity of image content limit the performance of the model.

[0004] In deep learning-based quality assessment methods, most of the models used are based on convolutional neural networks. 2D-CNN mainly uses one frame of video as input, which easily ignores the correlation information between frames, makes it difficult to capture temporal information, has high time complexity, and still has problems such as a limited number of distorted training videos and label imbalance, which limit the performance, robustness and generalization ability of the algorithm.

[0005] Three-dimensional convolutional neural networks (3D-CNNs) take multiple consecutive frames as input, adding a temporal dimension and enabling the extraction of more expressive features. While the introduction of 3D convolution improves model performance, it requires wider application and more comprehensive experiments to validate its performance, rather than specific application use. Another challenge of VQA models is that they only achieve good results on artificially distorted videos, struggling to perform well on naturally distorted video databases. Videos are composed of multiple consecutive still images, and these still images exhibit sequence correlation. Based on digital image processing and computer vision techniques, video frames can be extracted from the video stream at a certain frequency to obtain the video to be evaluated. Summary of the Invention

[0006] The purpose of this invention is to provide a Transformer-based no-reference video quality assessment method that can perform video quality assessment.

[0007] The technical solution adopted in this invention is a no-reference video quality assessment method based on Transformer, which is implemented according to the following steps:

[0008] Step 1: Download the public dataset KoNViD-1k for video quality assessment, which includes video data and corresponding label files. Process the video data in the public dataset into RGB format video frames and save them in .jpg format.

[0009] Step 2: Construct a network model for a Transformer-based no-reference video quality assessment method, which includes two modules: one is a multi-layered, cascaded, hierarchical visual transformer using a moving window, the other is a pyramid feature aggregation module.

[0010] Step 3: Send the video frame images obtained in Step 1 to the multi-layered cascaded layered visual transformer Swin-Transformer module using a moving window to extract video features and corresponding quality score files;

[0011] Step 4: Divide the dataset downloaded in Step 1 into training set, test set and validation set. The corresponding label files are also divided into training set, test set and validation set according to the division method of the dataset.

[0012] Step 5: Load the video feature file and quality score file extracted in Step 3 into the feature fusion-based pyramid aggregation module to train the model and finally obtain the quality score of the video to be evaluated.

[0013] The invention is further characterized in that,

[0014] Step 1 is implemented in the following steps:

[0015] Step 1.1: Download the publicly available dataset used for video quality assessment, which includes video data and corresponding label files;

[0016] Step 1.2: Extract video V from the video dataset obtained in Step 1.1. i , will video V i Decomposed into n frames, as shown in formula (1);

[0017] T = {f1, f2, f3... f i ..,f n} (1)

[0018] Where T represents the set of video frames obtained from video decomposition, f1 is the first frame of the video, f2 is the second frame of the video, f3 is the third frame of the video, and f... i For the i-th video frame, f n It is the nth frame of the video;

[0019] Step 1.2: Use the resize function provided in the cross-platform computer vision and machine learning software library OpenCV to resize f.i Change the size to 768*432, format it as .jpg, and save it to a folder named after the video.

[0020] Step 2 is implemented in the following steps:

[0021] The network model built in step 2 is as follows:

[0022] A multi-layered, cascaded, layered visual transformer Swin-Transformer module using a moving window is constructed. This module consists of a Patch Partition layer and four Stages, namely Stage 1 to Stage 4. Each Stage consists of a linear embedding layer and a Swin Transformer Block. The pyramid assembly module consists of a GRU network and fully connected layers FC4 and FC5.

[0023] Step 2.1: Read the video data file and corresponding tag file obtained in Step 1.1;

[0024] Step 2.2: Read the data from Step 2.1 and the video frame images generated in Step 1.2 to obtain the corresponding image file name, video file name, and corresponding quality score;

[0025] Step 2.3: Send the video frame images generated in Step 1.2 into the multi-layered, cascaded, layered visual transformer Swin-Transformer module, which uses a moving window, in batches of 64 frames.

[0026] Step 2.4: Load the pre-trained ImageNet model using the Swin-Transformer module of the hierarchical visual transformer with a moving window.

[0027] Step 2.5: Cut the video frame image read in Step 2.2 into image blocks of size 224*224 and send them to the PatchPartition layer in the multi-layer cascaded layered visual transformer module Swin-Transformer built in Step 2 using a moving window to obtain the feature vector of the video; Step 2.6: Put the feature vector obtained in Step 2.5 into the linearembedding layer and set the dimension of the feature vector obtained in Step 2.5 to 96, so the feature vector size becomes 56*56*96, and then after straightening, it becomes 3136*96;

[0028] Step 2.7: Input the feature vector obtained in Step 2.6 into the Swin Transformer Block. Input the feature vector of the video into the stage2 module. This module consists of two parts: one part is self-attention calculation based on a fixed window, and the other part is self-attention calculation based on a moving window.

[0029] Step 2.8: The feature vector obtained after passing through the stage2 module in step 2.7 is processed by the patch merging layer, which changes the size of the entire feature vector from 56*56*96 to 28*28*192.

[0030] Step 2.9: Send the feature vector obtained in Step 2.8 into the Swing Transformer Block to obtain the processed feature vector;

[0031] Step 2.10: Since the operations of stage2-stage4 are the same, repeat steps 2.8 and 2.9 twice to obtain the final video vector;

[0032] Step 2.11: The features M obtained from each stage... l Cascading yields a feature set F represents the feature obtained by cascading 4 stages, and l represents the l-th stage block, with l taking values ​​of 0, 1, 2, or 3.

[0033] Step 2.12: The feature set F obtained in step 2.12 is spatially converged using the global average pooling layer GAP and the global maximum pooling layer GMP, as shown in formulas (3) and (4).

[0034]

[0035]

[0036] Feature mean The feature vector F is represented by the feature vector after global average pooling. max The feature vector representing feature vector F after global max pooling;

[0037] Step 2.12: Obtain the Feature from Step 2.12 mean and Feature max The splicing is shown in formula (5).

[0038] F q =concat(Feature) mean Feature max (5)

[0039] in, It is a feature set of video frames. concat means concatenating special frames from different block levels. Multi-level feature extraction ensures that low-level features containing distortion-related information and high-level semantics are included, and then the captured feature information is sent to the next module.

[0040] Step 2.13: Save the feature vectors obtained in step 2.12 to generate the corresponding feature file and quality score file.

[0041] Step 3 is implemented in the following steps:

[0042] Step 3.1: Set the parameters of the pyramid feature aggregation module, with a learning rate of 0.0001, an optimizer of Adam, and a loss function of L1.

[0043] Step 3.2: Divide the dataset obtained in Step 1.1 into a training set, a test set, and a validation set in a ratio of 8:1:1;

[0044] Step 3.3: Load the video feature file and quality score file obtained in step 2.13.

[0045] Step 5 is implemented in the following steps:

[0046] Step 5.1: Use a single fully connected layer (FC) for dimensionality reduction, as shown in formula (6):

[0047] X t =W fx F q +b fx (6)

[0048] X t It is the feature after dimensionality reduction, W fx and b fx It is a parameter in a single fully connected layer (FC), F q It is the feature vector obtained in step 2.12;

[0049] Step 5.2: Convert the dimensionality-reduced features X t The data is fed into a GRU network, where t = 1, ..., T. The hidden states of the GRU are considered as ensemble features. The initial value of the GRU network is h0, and the current feature vector is F. GRU Based on the current input X t and the previous hidden state h t-1 The formula is as shown in (7).

[0050] F GRU =GRU(X) t ,h t-1 (7)

[0051] W GRU It is the feature vector obtained through the GRU network, X t The feature obtained in step 4.1 is the dimensionality-reduced feature, h. t-1 It is the previous hidden state;

[0052] Step 5.3: Convert the features F obtained through the GRU network GRU The weight W is calculated by merging with the surrounding K-frames. GRU The formula is as shown in (8).

[0053] W GRU =Tahn(conv2(Relu(conv1)(F GRU (8)

[0054] Where conv1 and conv2 are two 1D convolutional neural networks (1DCNN), and Than and ReLU are activation functions, with Than defined as in (9):

[0055]

[0056] Step 5.4: Calculate the weighted feature F W The formula is as follows (10):

[0057] F W =W GRU ⊙F GRU (10)

[0058] ⊙ represents multiplication by corresponding positions, F W It is the feature vector obtained by weighting the feature vectors obtained in step 5.3;

[0059] Step 5.5: Weight the time-dimension frame-level features F W Aggregate in a pyramid shape, setting the number of layers to 7;

[0060] Step 5.6: Use the fully connected layers FC4 and FC5 of the network model to process the features F obtained in step 5.4. W The pyramid aggregation feature is performed, as shown in formula (11).

[0061] Q P =FC5(FC4(F w (11)

[0062] Among them, Q P It is the predicted quality score of the final video.

[0063] The beneficial effect of this invention is that it provides a Transformer-based no-reference video quality assessment method that takes into account the temporal relationship of the video and the human visual perception mechanism. It uses the swin-transformer as the backbone to extract the spatiotemporal features of the distorted video and input them into the pyramid time aggregation module to predict the final video quality score. Attached Figure Description

[0064] Figure 1 This is a network framework diagram of a transformer-based no-reference video quality assessment model according to the present invention;

[0065] Figure 2 This invention relates to a spatiotemporal distortion special frame extraction module based on an improved swin-transformer. Detailed Implementation

[0066] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0067] This invention presents a no-reference video quality assessment method based on Transformer, with the following workflow framework: Figure 1 As shown, please follow these steps:

[0068] Step 1: Download the public dataset KoNViD-1k for video quality assessment, which includes video data and corresponding label files. Process the video data in the public dataset into RGB format video frames and save them in .jpg format.

[0069] Step 1 is implemented in the following steps:

[0070] Step 1.1: Download the publicly available dataset used for video quality assessment, which includes video data and corresponding label files;

[0071] Step 1.2: Extract video V from the video dataset obtained in Step 1.1. i , will video V i Decomposed into n frames, as shown in formula (1);

[0072] T = {f1, f2, f3... f i ..,f n} (1)

[0073] Where T represents the set of video frames obtained from video decomposition, f1 is the first frame of the video, f2 is the second frame of the video, f3 is the third frame of the video, and f is the i-th video frame. n It is the nth frame of the video;

[0074] Step 1.2: Use the resize function provided in the cross-platform computer vision and machine learning software library OpenCV to resize f. i Change the size to 768*432, format it as .jpg, and save it to a folder named after the video.

[0075] Step 2: Construct a network model for a Transformer-based no-reference video quality assessment method, comprising two modules: one is a multi-layered, cascaded, hierarchical visual transformer called Swin-Transformer using moving windows, such as... Figure 2 As shown, another module is the pyramid feature aggregation module;

[0076] Step 2 is implemented in the following steps:

[0077] The network model built in step 2 is as follows:

[0078] A multi-layered, cascaded, layered visual transformer Swin-Transformer module using a moving window is constructed. This module consists of a Patch Partition layer and four Stages, namely Stage 1 to Stage 4. Each Stage consists of a linear embedding layer and a Swin Transformer Block. The pyramid assembly module consists of a GRU network and fully connected layers FC4 and FC5.

[0079] Step 2.1: Read the video data file and corresponding tag file obtained in Step 1.1;

[0080] Step 2.2: Read the data from Step 2.1 and the video frame images generated in Step 1.2 to obtain the corresponding image file name, video file name, and corresponding quality score;

[0081] Step 2.3: Send the video frame images generated in Step 1.2 into the multi-layered, cascaded, layered visual transformer Swin-Transformer module, which uses a moving window, in batches of 64 frames.

[0082] Step 2.4: Load the pre-trained ImageNet model using the Swin-Transformer module of the hierarchical visual transformer with a moving window.

[0083] Step 2.5: Cut the video frame image read in Step 2.2 into image blocks of size 224*224 and send them to the PatchPartition layer in the multi-layer cascaded layered visual transformer module Swin-Transformer built in Step 2 using a moving window to obtain the feature vector of the video; Step 2.6: Put the feature vector obtained in Step 2.5 into the linearembedding layer and set the dimension of the feature vector obtained in Step 2.5 to 96, so the feature vector size becomes 56*56*96, and then after straightening, it becomes 3136*96;

[0084] Step 2.7: Input the feature vector obtained in Step 2.6 into the Swin Transformer Block. Input the feature vector of the video into the stage2 module. This module consists of two parts: one part is self-attention calculation based on a fixed window, and the other part is self-attention calculation based on a moving window.

[0085] Step 2.8: The feature vector obtained after passing through the stage2 module in step 2.7 is processed by the patch merging layer, which changes the size of the entire feature vector from 56*56*96 to 28*28*192.

[0086] Step 2.9: Send the feature vector obtained in Step 2.8 into the Swing Transformer Block to obtain the processed feature vector;

[0087] Step 2.10: Since the operations of stage2-stage4 are the same, repeat steps 2.8 and 2.9 twice to obtain the final video vector;

[0088] Step 2.11: The features M obtained from each stage... l Cascading yields a feature set F represents the feature obtained by cascading 4 stages, and l represents the l-th stage block, with l taking values ​​of 0, 1, 2, or 3.

[0089] Step 2.12: The feature set F obtained in step 2.12 is spatially converged using the global average pooling layer GAP and the global maximum pooling layer GMP, as shown in formulas (3) and (4).

[0090]

[0091]

[0092] Feature mean The feature vector F is represented by the feature vector after global average pooling.max The feature vector representing feature vector F after global max pooling;

[0093] Step 2.12: Obtain the Feature from Step 2.12 mean and Feature max The splicing is shown in formula (5).

[0094] F q =concat(Feature) mean Feature max (5)

[0095] in, It is a feature set of video frames. concat means concatenating special frames from different block levels. Multi-level feature extraction ensures that low-level features containing distortion-related information and high-level semantics are included, and then the captured feature information is sent to the next module.

[0096] Step 2.13: Save the feature vectors obtained in step 2.12 to generate the corresponding feature file and quality score file.

[0097] Step 3: Send the video frame images obtained in Step 1 to the multi-layered cascaded layered visual transformer Swin-Transformer module using a moving window to extract video features and corresponding quality score files;

[0098] Step 3 is implemented in the following steps:

[0099] Step 3.1: Set the parameters of the pyramid feature aggregation module, with a learning rate of 0.0001, an optimizer of Adam, and a loss function of L1.

[0100] Step 3.2: Divide the dataset obtained in Step 1.1 into a training set, a test set, and a validation set in a ratio of 8:1:1;

[0101] Step 3.3: Load the video feature file and quality score file obtained in step 2.13.

[0102] Step 4: Divide the dataset downloaded in Step 1 into training set, test set and validation set. The corresponding label files are also divided into training set, test set and validation set according to the division method of the dataset.

[0103] Step 5: Load the video feature file and quality score file extracted in Step 3 into the feature fusion-based pyramid aggregation module to train the model and finally obtain the quality score of the video to be evaluated.

[0104] Step 5 is implemented in the following steps:

[0105] Step 5.1: The extracted content-aware features have high dimensionality, making them unsuitable for training the GRU. Therefore, dimensionality reduction is performed before feeding the feature vectors into the GEU. Dimensionality reduction is performed using a single fully connected layer (FC), as shown in formula (6):

[0106] X t =W fx F q +b fx (6)

[0107] X t It is the feature after dimensionality reduction, W fx and b fx It is a parameter in a single fully connected layer (FC), F q It is the feature vector obtained in step 2.12;

[0108] Step 5.2: Convert the dimensionality-reduced features X t The data is fed into a GRU network, where t = 1, ..., T. The hidden states of the GRU are considered as ensemble features. The initial value of the GRU network is h0, and the current feature vector is F. GRU Based on the current input X t and the previous hidden state h t-1 The formula is as shown in (7).

[0109] F GRU =GRU(X) t ,h t-1 (7)

[0110] F GRU It is the feature vector obtained through the GRU network, X t The feature obtained in step 5.1 is the dimensionality-reduced feature, h. t-1 It is the previous hidden state;

[0111] Step 5.3: Convert the features F obtained through the GRU network GRU The weight W is calculated by merging with the surrounding K-frames. GRU The formula is as shown in (8).

[0112] W GRU =Tahn(conv2(Relu(conv1)(F GRU (8)

[0113] Where conv1 and conv2 are two 1D convolutional neural networks (1DCNN), and Than and ReLU are activation functions, with Than defined as in (9):

[0114]

[0115] Step 5.4: Calculate the weighted feature F W The formula is as follows (10):

[0116] F W =W GRU ⊙F GRU (10)

[0117] ⊙ represents multiplication by corresponding positions, F W It is the feature vector obtained by weighting the feature vectors obtained in step 5.3;

[0118] Step 5.5: Weight the time-dimension frame-level features F W Aggregated in a pyramidal fashion, the sampling density along the time dimension is generally controlled by the number of layers. Let's set the number of layers to 7;

[0119] Step 5.6: Use the fully connected layers FC4 and FC5 of the network model to process the features F obtained in step 5.4. W The pyramid aggregation feature is performed, as shown in formula (11).

[0120] Q P =FC5(FC4(F w (11)

[0121] Among them, Q P It is the predicted quality score of the final video.

[0122] This strategy offers greater flexibility than single-layer aggregation by incorporating changes over time.

Claims

1. A no-reference video quality assessment method based on Transformer, characterized in that, The specific steps are as follows: Step 1: Download the public dataset KoNViD-1k for video quality assessment, which includes video data and corresponding label files. Process the video data in the public dataset into RGB format video frames and save them in .jpg format. Step 1 is implemented in the following steps: Step 1.1: Download the publicly available dataset used for video quality assessment, which includes video data and corresponding label files; Step 1.2: Extract video V from the video dataset obtained in Step 1.

1. i , will video V i Decomposed into n frames, as shown in formula (1); （1） Where T represents the set of video frames obtained from video decomposition. It is the first frame of the video. It's two frames from the video. It's the 3rd frame of the video. For the i-th video frame, It is the nth frame of the video; Step 1.2: Use the resize function provided in the cross-platform computer vision and machine learning software library OpenCV to... Change the size to 768*432, format it as .jpg, and save it to a folder named after the video. Step 2: Construct a network model for a Transformer-based no-reference video quality assessment method, which includes two modules: one is a multi-layered, cascaded, hierarchical visual transformer using a moving window, the other is a pyramid feature aggregation module. Step 2 is implemented in the following steps: The network model built in step 2 is as follows: A multi-layered, cascaded, layered visual transformer Swin-Transformer module using a moving window is constructed. This module consists of a Patch Partition layer and four Stages, namely Stage1 to Stage4. Each Stage consists of a linear embedding layer and a Swin Transformer Block. The pyramid assembly module consists of a GRU network and fully connected layers FC4 and FC5. Step 2.1: Read the video data file and corresponding tag file obtained in Step 1.1; Step 2.2: Read the data from Step 2.1 and the video frame images generated in Step 1.2 to obtain the corresponding image file name, video file name, and corresponding quality score; Step 2.3: Send the video frame images generated in Step 1.2 into the multi-layered, cascaded, layered visual transformer Swin-Transformer module, which uses a moving window, in batches of 64 frames. Step 2.4: Load the pre-trained ImageNet model using the Swin-Transformer module of the hierarchical visual transformer with a moving window. Step 2.5: Cut the video frame image read in Step 2.2 into image blocks of size 224*224 and send them to the PatchPartition layer in the multi-layer cascaded layered visual transformer module Swin-Transformer built in Step 2 using a moving window to obtain the feature vector of the video; Step 2.6: Put the feature vector obtained in Step 2.5 into the linearembedding layer and set the dimension of the feature vector obtained in Step 2.5 to 96, so the feature vector size becomes 56*56*96, and then after straightening, it becomes 3136*96; Step 2.7: Input the feature vector obtained in Step 2.6 into the Swin Transformer Block. Input the feature vector of the video into the stage2 module. This module consists of two parts: one part is self-attention calculation based on a fixed window, and the other part is self-attention calculation based on a moving window. Step 2.8: The feature vector obtained after passing through the stage2 module in step 2.7 is processed by the patch merging layer, which changes the size of the entire feature vector from 56*56*96 to 28*28*192. Step 2.9: Send the feature vector obtained in Step 2.8 into the Swing Transformer Block to obtain the processed feature vector; Step 2.10: Since the operations of stage2-stage4 are the same, repeat steps 2.8 and 2.9 twice to obtain the final video vector; Step 2.11: Combine the features obtained from each stage. Cascading yields a feature set F represents the feature obtained by cascading 4 stages, and l represents the l-th stage block, with values ​​of 0, 1, 2, and 3. Step 2.12: Combine the feature set obtained in Step 2.11 Spatially, the global average pooling layer (GAP) and the global maximum pooling layer (GMP) are used for convergence, as shown in formulas (3) and (4). (3) (4) This represents the feature vector F after global average pooling. The feature vector representing feature vector F after global max pooling; Step 2.13, obtain from step 2.12 and The splicing is shown in formula (5). (5) in, It is a feature set of video frames. concat means concatenating special frames from different block levels. Multi-level feature extraction ensures that low-level features containing distortion-related information and high-level semantics are included, and then the captured feature information is sent to the next module. Step 2.14: Save the feature vectors obtained in Step 2.13 to generate the corresponding feature file and quality score file; Step 3: Send the video frame images obtained in Step 1 to the multi-layered cascaded layered visual transformer Swin-Transformer module using a moving window to extract video features and corresponding quality score files; Step 4: Divide the dataset downloaded in Step 1 into training set, test set and validation set. The corresponding label files are also divided into training set, test set and validation set according to the division method of the dataset. Step 5: Load the video feature file and quality score file extracted in Step 3 into the feature fusion-based pyramid aggregation module to train the model and finally obtain the quality score of the video to be evaluated.

2. The Transformer-based no-reference video quality assessment method according to claim 1, characterized in that, Step 4 is implemented in the following steps: Step 3.1: Set the parameters of the pyramid feature aggregation module, with a learning rate of 0.0001, an optimizer of Adam, and a loss function of L1. Step 3.2: Divide the dataset obtained in Step 1.1 into a training set, a test set, and a validation set in a ratio of 8:1:1; Step 3.3: Load the video feature file and quality score file obtained in step 2.

13.

3. The Transformer-based no-reference video quality assessment method according to claim 2, characterized in that, Step 5 is implemented in the following steps: Step 5.1: Use a single fully connected layer (FC) for dimensionality reduction, as shown in formula (6): (6) These are features after dimensionality reduction. and These are parameters in a single fully connected layer (FC). It is the feature vector obtained in step 2.12; Step 5.2: Convert the dimensionality-reduced features Sent to the GRU network, The hidden states of GRU are considered as integrated features, and the initial values ​​of the GRU network are... The current feature vector is Based on the current input and the previous hidden state The formula is as shown in (7). (7) These are feature vectors obtained through the GRU network. These are the dimensionality-reduced features obtained in step 4.

1. It is the previous hidden state; Step 5.3: The features obtained through the GRU network... Calculate weights by merging with surrounding K-frames The formula is as shown in (8). (8) Where conv1 and conv2 are two 1D convolutional neural networks (1DCNN), and Than and ReLU are activation functions, with Than defined as in (9): (9) Step 5.4: Calculate the weighted features The formula is as follows (10): （10） This represents multiplying corresponding positions. It is the feature vector obtained by weighting the feature vectors obtained in step 5.3; Step 5.5: Weight the time-dimension frame-level features Aggregate in a pyramid shape, setting the number of layers to 7; Step 5.6: Use the fully connected layers FC4 and FC5 of the network model to process the features obtained in Step 5.

4. The pyramid aggregation feature is performed, as shown in formula (11). (11) in, It is the predicted quality score of the final video.

Images