A method for evaluating the compression damage level of ultra-high-definition video based on a deep learning network

Abstract

The application discloses a kind of based on deep learning network's ultra-high-definition video compression damage grade evaluation method, comprising the following steps;Step 1, establish multi-resolution compression damage training set, verification set, test set: step 2, construct the compression damage grade evaluation network of time-frequency domain combination: step 3, train the compression damage grade evaluation network of time-frequency domain combination: step 4, test the accuracy of multi-resolution compression damage test set;Step 5, evaluate the compression damage grade evaluation network of time-frequency domain combination.The application is evaluated to ultra-high-definition video compression damage, solves the problem that compression damage data set production cost is high, not easy to expand, video compression damage evaluation network does not adopt end-to-end design, and generalization ability is insufficient and the problem that ultra-high-definition video evaluation is time-consuming.

Description

Technical Field

[0001] This invention belongs to the technical field of ultra-high-definition video compression impairment level assessment, specifically relating to a method for assessing ultra-high-definition video compression impairment level based on deep learning networks. Background Technology

[0002] In deep learning-based video quality assessment research, existing datasets generally contain various distortion types and corresponding MOS scores. Commonly used video databases include LIVE, CSIQ, IVP, and KoNViD-1k, but these databases have limited data on compression impairments, and the scoring standards differ across databases. Deep learning models are data-driven, and deep learning networks trained on large amounts of data have greater advantages in prediction accuracy and generalization ability. Networks trained on existing datasets have limited generalization ability and cannot accurately evaluate most scenarios.

[0003] The paper "Compressed Video Quality Index Based on Saliency-Aware Artifact Detection" published by Lin L et al. proposes an objective evaluation method for compressed video quality. While this method uses salient regions of the image as input, addressing the influence of non-damaging regions on the network and making the predictions more reasonable, it still has shortcomings: First, the selection of salient regions relies on ACLNet, and the accuracy of ACLNet's region extraction is strongly positively correlated with the accuracy of subsequent evaluation results, indicating poor anti-interference capabilities of the network. Second, this method cannot achieve end-to-end compression impairment quality evaluation; the training process is complex, requiring separate network training for four common compression impairments and integration of the outputs of the four networks to obtain the final score.

[0004] The paper "No-Reference Deep Compressed-Based Video Quality Assessment" by M. Alizadeh et al. proposes a novel no-reference video quality assessment method based on an efficient video codec using convolutional neural networks. While this method utilizes the coding characteristics of HEVC encoding to assess compression impairment quality, enabling deep learning networks to more easily learn the deep features of the dataset, which greatly benefits the improvement of prediction accuracy and simplification of network training, it is not applicable to scenarios where this coding information is unavailable. Furthermore, for ultra-high-definition video images, this compression impairment assessment network is slow when processing high-resolution images, making it unsuitable for systems requiring real-time assessment of compression impairment in ultra-high-definition videos. Summary of the Invention

[0005] To overcome the shortcomings of the existing technology, the present invention aims to provide a method for evaluating the compression impairment level of ultra-high-definition video based on deep learning networks. This method evaluates compression impairment in ultra-high-definition video and solves the problems of high cost of compressibility impairment dataset production, difficulty in expansion, lack of end-to-end design in video compression impairment evaluation network, insufficient generalization ability, and long evaluation time for ultra-high-definition video.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for evaluating the compression impairment level of ultra-high-definition video based on deep learning networks includes the following steps;

[0008] Step 1: Establish multi-resolution compression impairment training, validation, and test sets:

[0009] Step 2: Construct a time-frequency domain combined compression damage level assessment network:

[0010] Step 3: Train the compression damage level assessment network that combines time and frequency domains:

[0011] Step 4, test the accuracy of the multi-resolution compression impairment test set: traverse and crop the video frame into 448×448 blocks, and input each block into the trained time-frequency domain combined compression impairment level evaluation network. Use the average value predicted by the image block as the compression impairment evaluation category of the frame.

[0012] Step 5: Evaluate the time-frequency domain combined compression damage assessment network.

[0013] Step 1 specifically includes:

[0014] Step 1.1: Initial data collection for training and validation sets. Collect lossless video stream data from the internet. The collected data includes various resolutions, including 1080p, 2K, and 4K resolutions, and the scene content covers people, animals, natural landscapes, animations, architecture, and movies. The collected data is filtered, and finally includes 60 lossless videos with a duration of 6-8 seconds, which are used as the lossless video sequence set for the training set.

[0015] The selection criteria follow the method for quantifying the spatial and temporal information of a scene as proposed in ITU-T P.910, ensuring that the selected sequences have appropriate coverage in the spatiotemporal plane.

[0016] Step 1.2: Cut the lossless video sequence set. Before creating the frame-level dataset, the long sequence is trimmed by discarding a portion of the data in 10-frame increments. The remaining data is divided into 10-frame short sequences. HEVC encoding is used to generate different levels of compression impairment by setting different constant rate factors, and finally, a distortion sequence with level labels is obtained.

[0017] Step 1.3: Play the distorted sequence generated in Step 1.2 in a well-lit environment using PotPlayer video viewer. Each annotator filters the video and corresponding grade labels displayed on the screen and removes abnormal video data.

[0018] Step 1.4: For each video data, corresponding to the network input, take the frame pair as the dataset to obtain multiple pairs of images. The level label of each pair of images corresponds to the label generated in the compression impairment video stream generation process in Step 1.2, which are level 0, 1, 2, 3, 4 respectively. The higher the level, the more severe the compression impairment contained in the image.

[0019] Step 1.5: The training set and validation set contain 80% and 20% of the data generated in step 1.4, respectively.

[0020] Step 1.6, test set creation, includes 1080p and 2k videos from HEVC standard sequences, as well as lossless videos collected from the Internet. The dataset is prepared using the methods described in Steps 1.2 and 1.3 to obtain videos with grade labels. The short video sequences are spliced ​​together. Finally, a total of 2 to 12 seconds of videos are provided in the validation database.

[0021] Step 2 specifically includes:

[0022] Step 2.1: Construct a convolutional module and an input convolutional layer. The module includes three convolutional layers and one summation layer. The three convolutional layers are concatenated in sequence and then fed into the summation layer along with the feature vector of the input first convolutional layer.

[0023] The forward propagation process of this convolutional module is as follows: the feature vector passes through the first convolutional layer, the second convolutional layer, and the third convolutional layer in sequence, and outputs a vector with the same shape as the feature vector. These two feature vectors are then added together by the addition layer to output a single feature vector.

[0024] Using PyTorch, the parameters for each layer are set as follows: the kernel size of the input convolutional layer is set to 7×7, the stride is set to 2, and the activation function is implemented using ReLU. The kernel sizes of the first, second, and third convolutional layers in the convolutional module are set to 1×1, 3×3, and 1×1, respectively, and the stride is set to 1. The activation function is implemented using ReLU. The summation layer requires that the two input vectors have the same shape.

[0025] Step 2.2: Generate temporal and frequency domain information. Temporal information is obtained using the variance between adjacent frames to reflect the motion of adjacent frames. Frequency domain information is obtained using discrete cosine transform (DCT). The input RGB image is converted into a YUV image, and a DCT is performed on the Y channel to obtain frequency domain information. Step 2.3: Construct a temporal feature extraction network, whose structure is as follows: input convolutional layer, first max pooling layer, first convolutional module, second convolutional module, third convolutional module, fourth convolutional module, fifth convolutional module, sixth convolutional module, seventh convolutional module, eighth convolutional module, ninth convolutional module, tenth convolutional module, eleventh convolutional module, twelfth convolutional module, thirteenth convolutional module, fourteenth convolutional module, fifteenth convolutional module, and sixteenth convolutional module.

[0026] The forward propagation process for temporal information feature extraction is as follows: After the temporal information obtained in step 2.2 is input into the input convolutional layer to extract features, it first passes through the first max pooling layer and sixteen concatenated convolutional modules to obtain the feature vector of the temporal information.

[0027] The parameters for each layer of the network in step 2.3 are set as follows:

[0028] Set the kernel size of the input convolutional layer to 7×7, the stride to 2, the number of input channels to 2, and the number of output channels to 64.

[0029] The pooling kernel size of the first max pooling layer is set to 3×3, the step size is set to 2, the fill size is set to 2, and the expansion coefficient is set to 1.

[0030] The structures of the first to sixteenth convolutional modules are the same as those of the convolutional module in step 2.1. The number of input channels in the first convolutional layer within the first to sixteenth convolutional modules is set sequentially as follows: 64, 256, 256, 256, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 2048, 2048; the number of output channels is set sequentially as follows: 64, 64, 64, 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 256, 512, 512, 512. The number of input channels in the second convolutional layer within the first to sixteenth convolutional modules is set sequentially as follows: 64, 64, 64, 128, 128, 128, 128, 256, 256, 256. The number of output channels is set sequentially as follows: 64, 64, 64, 128, 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 512, 512, 512; the number of input channels in the third convolutional layer within the first to sixteenth convolutional modules is set sequentially as follows: 64, 64, 64. , 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 512, 512, 512; The number of output channels is set sequentially as follows: 256, 256, 256, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048.

[0031] Step 2.4: Construct the frequency domain feature extraction network, whose structure is as follows: first convolutional module, second convolutional module, third convolutional module, fourth convolutional module, fifth convolutional module, sixth convolutional module, seventh convolutional module, eighth convolutional module, ninth convolutional module, tenth convolutional module, eleventh convolutional module, twelfth convolutional module, thirteenth convolutional module, fourteenth convolutional module, fifteenth convolutional module, and sixteenth convolutional module;

[0032] The forward propagation process for frequency domain information feature extraction is as follows: the frequency domain information obtained in step 2.2 is passed through sixteen cascaded convolutional modules to obtain the feature vector of the frequency domain information.

[0033] The parameters for each layer of the network in step 2.4 are set as follows:

[0034] The structures of the first to sixteenth convolutional modules are the same as those of the convolutional module in step 2.1. The number of input channels in the first convolutional layer within the first to sixteenth convolutional modules is set sequentially as follows: 64, 256, 256, 256, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 2048, 2048; the number of output channels is set sequentially as follows: 64, 64, 64, 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 256, 512, 512, 512. The number of input channels in the second convolutional layer within the first to sixteenth convolutional modules is set sequentially as follows: 64, 64, 64, 128, 128, 128, 128, 256, 256, 256. The number of output channels is set sequentially as follows: 64, 64, 64, 128, 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 512, 512, 512; the number of input channels in the third convolutional layer within the first to sixteenth convolutional modules is set sequentially as follows: 64, 64, 64. , 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 512, 512, 512; The number of output channels is set sequentially as follows: 256, 256, 256, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048.

[0035] Step 2.5: Construct a time-frequency domain fusion module and a compression damage level assessment module. The structure consists of a concat layer, an adaptive average pooling layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer.

[0036] The forward propagation process is as follows: the outputs of steps 2.3 and 2.4 are used as the inputs to the concat layer. The outputs of the concat layer are sequentially input into the adaptive average pooling layer, the first fully connected layer, the second fully connected layer, and the third fully connected layer, and the output is a five-class vector.

[0037] The parameter settings for each layer of the network in step 2.5 are as follows:

[0038] Set the concatenation dimension of the concat layer to one dimension;

[0039] Set the pooling parameter of the adaptive average pooling layer to 1×1;

[0040] The number of input neurons in the first fully connected layer is set to 4096, and the number of output neurons is set to 2048; the number of input neurons in the second fully connected layer is set to 2048, and the number of output neurons is set to 2048; the number of input neurons in the third fully connected layer is set to 2048, and the number of output neurons is set to 5.

[0041] Step 3 specifically includes:

[0042] Step 3.1: Configure the training environment and install the Python libraries required for network training;

[0043] Step 3.2: Set the training hyperparameters. Set the batch size to 16, the initial learning rate (lr) to 0.001, the weight decay rule to 0.1 times the learning rate every 10 training epochs, the solver to SGD, and the training epochs to 30.

[0044] Step 3.3: Train the time-frequency domain combined compression damage assessment network using stochastic gradient descent. First, process the input image data: transform the first frame of the input image pair to the frequency domain to obtain frequency information, and use an optical flow network to calculate the motion information of the image pair. Set the batch size to 448×448. Then, input the data into the compression damage classification network for forward propagation. Calculate the loss function L between the output and target values. Perform backpropagation using the SGD solver combined with the learning rate lr to update the network weights. After each epoch, assign the updated weights to the classification network. Input the validation set into the network to calculate the validation set compression damage test accuracy, assisting network training to avoid overfitting or underfitting. After 30 epochs of training, the loss function L converges. Assign the last updated weights to the time-frequency domain combined compression damage assessment network to obtain the trained time-frequency domain combined compression damage assessment network. During training, the loss function L is the multi-class cross-entropy loss function.

[0045]

[0046] Where N represents the training sample size, N is set to 12960; M represents the classification categories, M is set to 5; y ij The sign function represents 0 or 1; if the true class of sample i is equal to j, then y ij If y is 1, otherwise y ij p is 0; ij This represents the probability that the observed sample i in the network output belongs to category j.

[0047] Step 5 specifically includes:

[0048] Step 5.1: A video compression impairment test set was established. The test set was prepared according to step 1.6 and the compression impairment category labels were obtained frame by frame.

[0049] Step 5.2: Input each video in the test set into the trained time-frequency domain combined compression damage level evaluation network, output the final objective evaluation category of compression damage level, and calculate the average compression damage test accuracy of the test set.

[0050] The aforementioned rating evaluation method is used to help measure the encoding effect of compression encoders, evaluate the merits of image enhancement and image reconstruction algorithms, classify video compression damage, assign different push strengths, optimize video push rules, detect distortion in industrial or security video footage as anomaly detection, and assist video platforms in determining whether the current video needs image enhancement to repair videos that affect user viewing experience.

[0051] The beneficial effects of this invention are:

[0052] The training dataset established in this invention covers multiple resolutions, content scenes, and motion scenes, enabling the neural network to learn richer compression damage features, thereby improving the accuracy of compression damage assessment.

[0053] This invention designs the compression damage assessment network as a classification network, which simplifies the preparation of the dataset and the design and training of the network compared to the regression method.

[0054] This invention performs image processing operations on video frames when extracting compression impairment features. Compared with the method of directly inputting video frames into the network, this accelerates the convergence speed of the network and improves the accuracy of the classification results. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings.

[0057] like Figure 1 As shown: Step 1, establish multi-resolution compression impairment training set, validation set, and test set:

[0058] Step 1.1: Initial data collection for the training and validation sets. Distortionless video stream data was collected from the internet. The collected data included various resolutions, including 1080p, 2K, and 4K, and the scene content covered people, animals, natural landscapes, animation, architecture, movies, etc. The collected data was then filtered, ultimately resulting in 60 distortionless videos, each 6-8 seconds long.

[0059] Step 1.2: For the initial training data cropping, in order to reduce redundant information in long video sequences, before creating the frame-level dataset, the long sequences are cropped, and a portion of the data is discarded in 10-frame increments. The remaining data is divided into 10-frame short sequences, which are encoded using HEVC. Different levels of compression impairment are generated by controlling the constant rate factor. Using the constant rate factor can maximize the generation of video frames with consistent subjective perception of compression impairment at the same level, ultimately resulting in 5400 distortion sequences with level labels.

[0060] Step 1.3: In this embodiment of the invention, 8 annotators were organized to process the 5400 videos generated in step 1.2. The embodiment was conducted in a well-lit environment using PotPlayer video viewer with a display resolution of 1920×1080. Each evaluator screened the videos and corresponding tags displayed on the screen and removed abnormal video data.

[0061] Step 1.4: For each video data set, corresponding frame pairs are taken as the dataset, resulting in a total of 16,200 image pairs. The label for each image pair corresponds to the label generated during the compression impairment video stream generation process in Step 1.2, with levels 0, 1, 2, 3, and 4. A higher level indicates more severe compression impairment in the image.

[0062] Step 1.5: The training set and validation set contain 80% and 20% of the data generated in step 1.4, respectively.

[0063] Step 1.6: Test set creation, which includes 1080p and 2k videos from HEVC standard sequences, as well as lossless videos collected from the internet. These videos have different motion conditions and different content scenes. The dataset is prepared using the methods described in Steps 1.2 and 1.3, resulting in 541 videos with grade labels. For ease of display, the short video sequences are spliced ​​together. Finally, a total of 147 videos ranging from 2 to 12 seconds are provided in the validation database.

[0064] Step 2: Construct a time-frequency domain combined compression damage level assessment network:

[0065] Step 2.1: Construct a convolutional module and an input convolutional layer. The module includes three convolutional layers and one summation layer. The three convolutional layers are concatenated in sequence and then fed into the summation layer along with the input feature vector.

[0066] The forward propagation process of this convolutional module is as follows: the feature vector passes through the first convolutional layer, the second convolutional layer, and the third convolutional layer in sequence, and outputs a vector with the same shape as the feature vector. These two feature vectors are then added together by the addition layer to output a single feature vector.

[0067] Using PyTorch, the parameters for each layer are set as follows: the kernel size of the input convolutional layer is set to 7×7, the stride is set to 2, and the activation function is implemented using ReLU. The kernel sizes of the first, second, and third convolutional layers in the convolutional module are set to 1×1, 3×3, and 1×1, respectively, and the stride is set to 1. The activation function is implemented using ReLU. The summation layer requires that the two input vectors have the same shape.

[0068] Step 2.2: Generate time-domain information and frequency-domain information. The time-domain information is obtained by using the variance between adjacent frames to reflect the motion of adjacent frames. The frequency-domain information is obtained by using discrete cosine transform. The read-in RGB image is converted into a YUV image, and discrete cosine transform is performed on the Y channel to obtain the frequency-domain information.

[0069] Step 2.3: Construct a temporal feature extraction network, the structure of which is as follows: input convolutional layer, first max pooling layer, first convolutional module, second convolutional module, third convolutional module, fourth convolutional module, fifth convolutional module, sixth convolutional module, seventh convolutional module, eighth convolutional module, ninth convolutional module, tenth convolutional module, eleventh convolutional module, twelfth convolutional module, thirteenth convolutional module, fourteenth convolutional module, fifteenth convolutional module, and sixteenth convolutional module.

[0070] The forward propagation process for temporal information feature extraction is as follows: After the input temporal information enters the input convolutional layer to extract features, it first passes through the first max pooling layer and sixteen concatenated convolutional modules to obtain the feature vector of the temporal information.

[0071] The parameters for each layer of the network are set as follows:

[0072] Set the kernel size of the input convolutional layer to 7×7, the stride to 2, the number of input channels to 2, and the number of output channels to 64.

[0073] The pooling kernel size of the first max pooling layer is set to 3×3, the step size is set to 2, the fill size is set to 2, and the expansion coefficient is set to 1.

[0074] The structures of the first to sixteenth convolutional modules are the same as those of the convolutional module in step 2.1. The number of input channels in the first convolutional layer within the first to sixteenth convolutional modules is set sequentially as follows: 64, 256, 256, 256, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 2048, 2048; the number of output channels is set sequentially as follows: 64, 64, 64, 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 256, 512, 512, 512. The number of input channels in the second convolutional layer within the first to sixteenth convolutional modules is set sequentially as follows: 64, 64, 64, 128, 128, 128, 128, 256, 256, 256. The number of output channels is set sequentially as follows: 64, 64, 64, 128, 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 512, 512, 512; the number of input channels in the third convolutional layer within the first to sixteenth convolutional modules is set sequentially as follows: 64, 64, 64. , 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 512, 512, 512; The number of output channels is set sequentially as follows: 256, 256, 256, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048.

[0075] Step 2.4: Construct a frequency domain feature extraction network, whose structure is as follows: first convolutional module, second convolutional module, third convolutional module, fourth convolutional module, fifth convolutional module, sixth convolutional module, seventh convolutional module, eighth convolutional module, ninth convolutional module, tenth convolutional module, eleventh convolutional module, twelfth convolutional module, thirteenth convolutional module, fourteenth convolutional module, fifteenth convolutional module, and sixteenth convolutional module.

[0076] The forward propagation process for frequency domain information feature extraction is as follows: the input frequency domain information is passed through sixteen cascaded convolutional modules to obtain the feature vector of the frequency domain information.

[0077] The parameters for each layer of the network are set as follows:

[0078] The structures of the first to sixteenth convolutional modules are the same as those of the convolutional module in step 2.1. The number of input channels in the first convolutional layer within the first to sixteenth convolutional modules is set sequentially as follows: 64, 256, 256, 256, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 2048, 2048; the number of output channels is set sequentially as follows: 64, 64, 64, 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 256, 512, 512, 512. The number of input channels in the second convolutional layer within the first to sixteenth convolutional modules is set sequentially as follows: 64, 64, 64, 128, 128, 128, 128, 256, 256, 256. The number of output channels is set sequentially as follows: 64, 64, 64, 128, 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 512, 512, 512; the number of input channels in the third convolutional layer within the first to sixteenth convolutional modules is set sequentially as follows: 64, 64, 64. , 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 512, 512, 512; The number of output channels is set sequentially as follows: 256, 256, 256, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048.

[0079] Step 2.5: Construct a time-frequency domain fusion module and a compression damage level assessment module. The structure consists of a concat layer, an adaptive average pooling layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer.

[0080] The forward propagation process is as follows: the outputs of steps 2.2 and 2.3 are used as the inputs to the concat layer. The outputs of the concat layer are sequentially input into the adaptive average pooling layer, the first fully connected layer, the second fully connected layer, and the third fully connected layer, and the output is a five-class vector.

[0081] The parameters for each layer of the network are set as follows:

[0082] Set the concatenation dimension of the concat layer to one dimension;

[0083] Set the pooling parameter of the adaptive average pooling layer to 1×1;

[0084] The number of input neurons in the first fully connected layer is set to 4096, and the number of output neurons is set to 2048; the number of input neurons in the second fully connected layer is set to 2048, and the number of output neurons is set to 2048; the number of input neurons in the third fully connected layer is set to 2048, and the number of output neurons is set to 5.

[0085] Step 3: Train the compression damage level assessment network that combines time and frequency domains:

[0086] Step 3.1: Configure the training environment and install the Python libraries required for network training;

[0087] Step 3.2: Set the training hyperparameters. Set the batch size to 16, the initial learning rate (lr) to 0.001, the weight decay rule to 0.1 times the learning rate every 10 training epochs, the solver to SGD, and the training epochs to 30.

[0088] Step 3.3: Train the time-frequency domain combined compression damage assessment network using stochastic gradient descent. First, process the input image data: transform the first frame of the input image pair to the frequency domain to obtain frequency information, and use an optical flow network to calculate the motion information of the image pair. In this embodiment, the batch size is set to 448×448. Then, input the data into the compression damage classification network for forward propagation. Calculate the loss function L between the output and target values. The solver SGD combined with the learning rate lr performs backpropagation to update the network weights. After each epoch, assign the updated weights to the classification network. Input the validation set into the network to calculate the validation set compression damage test accuracy, assisting network training to avoid overfitting or underfitting. After 30 epochs of training, the loss function L converges. Assign the last updated weights to the time-frequency domain combined compression damage assessment network to obtain the trained time-frequency domain combined compression damage assessment network. During training, the loss function L is the multi-class cross-entropy loss function.

[0089]

[0090] Where N represents the training sample size, which is set to 12960 in this embodiment; M represents the classification categories, which is set to 5 in this embodiment; y ij The sign function represents 0 or 1; if the true class of sample i is equal to j, then y ij If y is 1, otherwise y ij p is 0; ij This represents the probability that the observed sample i in the network output belongs to category j.

[0091] Step 4, test the accuracy of the multi-resolution compression impairment test set: traverse and crop the video frame into 448×448 blocks, and input each block into the trained time-frequency domain combined compression impairment level evaluation network. Use the average value predicted by the image block as the compression impairment evaluation category of the frame.

[0092] Step 5: Evaluate the time-frequency domain combined compression damage assessment network:

[0093] Step 5.1: In this embodiment, a video compression impairment test set is established. The test set is prepared according to step 1.6 and frame-by-frame compression impairment category labels are obtained.

[0094] Step 5.2: Input each video in the test set into the trained time-frequency domain combined compression damage level evaluation network, output the final objective evaluation category of compression damage level, and calculate the average compression damage test accuracy of the test set.

[0095] The specific idea behind this invention is:

[0096] 1) Collect a large number of video sequences without any distortion, degrade the video sequences using the HEVC coding standard to obtain datasets with different levels of compression distortion. During the data production process, a constant rate factor is selected to control the degree of compression impairment in order to avoid inconsistencies in the subjective perception of data of the same level due to inconsistent motion in the video sequences.

[0097] The training set is composed of multi-resolution video datasets with different levels of compression impairment. This allows the neural network to learn compression impairment features at different resolutions and levels, which can improve the accuracy of video frame compression impairment assessment and can be used at different resolutions, thus expanding the network's applicable scenarios.

[0098] 2) Divide the video sequences into five categories: excellent, good, average, poor, and terrible, and evaluate the subjective perception of the compressed videos relative to the human eye.

[0099] 3) This invention achieves compression impairment level assessment by designing an end-to-end deep learning network that combines time and frequency domains. Frequency domain information is extracted by performing DCT transformation on video frame images, and inter-frame MV information is extracted by calculating optical flow in the time domain. These two types of information are fed into a feature extraction network, then fused and input into a fully connected layer, which outputs the compression impairment classification category. 4) In the evaluation and testing phase, this invention collects distortion-free time-frequency sequences different from the training set, while maintaining the same dataset preparation method as the training dataset. By randomly concatenating frames from the time-frequency sequences, a video sequence set containing multiple distortion levels is obtained. The evaluation accuracy and processing speed of this method on the test set are used as performance indicators.

[0100] Application prospects of this invention:

[0101] Today, video has become a major component of digital network traffic. Video data traffic accounts for approximately 80% of total network data traffic, and video coding technology has significantly reduced storage capacity and transmission bandwidth. However, lossy compression and transmission through variable channels inevitably cause various distortions, among which compression impairment has a significant impact on video perceptual quality. Visual content perception understanding has become a cutting-edge research direction in the fields of visual computing, computer vision, computational photography, and their interdisciplinary areas. Image quality assessment is a particularly hot research topic.

[0102] The compression impairment level assessment network based on time-frequency domain combination of the present invention can accurately and quickly assess the degree of compression impairment of ultra-high-definition video, and has a wide range of applications: (1) It can help measure the encoding effect of the compression encoder and provide users with better picture quality; (2) It can evaluate the merits of image enhancement and image reconstruction algorithms; (3) In short video recommendation, it can classify video compression impairment, give different push intensity, optimize video push rules, and maximize benefits; (4) In industry or security, it can use video image distortion detection as anomaly detection to provide corresponding alarms; (5) It can assist video platforms in determining whether the current video needs image enhancement to repair videos that affect the user's viewing experience.

Claims

1. A method for evaluating a compression impairment level of an ultra-high definition video based on a deep learning network, characterized in that, Comprise the following steps; Step 1, establish a multi-resolution compression damage training set, verification set, test set: Step 2, construct the compression damage level evaluation network combined with time-frequency domain: Step 3, train the compression damage level evaluation network combined with time-frequency domain: Step 4, test the accuracy of the multi-resolution compression damage test set: traverse the video frame to cut into 448x448 blocks, input into the trained compression damage level evaluation network combined with time-frequency domain block by block, and the average value predicted by the image block is taken as the compression damage evaluation category of the frame; Step 5, evaluate the compression damage level evaluation network combined with time-frequency domain: Said step 1 specifically comprises: Step 1.1, initial data collection of training set and verification set, collect non-distorted video stream data from the Internet, the collected data contains multiple resolutions, including 1080p, 2k, 4k resolution, and scene content covers people, animals, natural landscape, animation, building, movie; screening the collected data, finally containing 60 non-distorted videos of 6-8s length as the non-distorted video sequence set for making training set; Step 1.2, cut the non-distorted video sequence set, before making frame-level data set, cut the long sequence by step distance 10 frames, discard a part of data, and use HEVC encoding method to produce different levels of compression damage by setting different constant rate factors, finally get distorted sequence with level label; Step 1.3, play the distorted sequence produced in step 1.2 in a well-lit environment using PotPlayer video viewer, each annotator screens the video displayed on the display and the corresponding level label, and removes abnormal video data; Step 1.4, for each video data, take frame pairs as data set corresponding to network input, get multiple pairs of images, and the level label of each pair of images corresponds to the label generated in the compression damage video stream generation process in step 1.2, which is level 0, 1, 2, 3, 4, and the higher the level, the more serious the compression damage contained in the image; Step 1.5, the training set and the verification set respectively contain 80% and 20% of the data produced in step 1.4; Step 1.6, test set making, including 1080p, 2k video in HEVC standard sequence, and non-distorted video collected from the Internet, using the method introduced in steps 1.2 and 1.3 to prepare the data set, get the video containing the level label, splice the short video sequence, finally, provide 2-12s videos in the verification database; Said step 2 specifically comprises: Step 2.1, a convolution module and an input convolution layer are constructed, the module includes three convolution layers and an addition layer, wherein the three convolution layers are connected in turn and the feature vector input into the first convolution layer is sent to the addition layer together; The forward propagation process of the convolution module is: the feature vector passes through the first convolution layer, the second convolution layer and the third convolution layer in turn, and outputs a vector consistent with the shape of the feature vector, and the two feature vectors are added after the addition layer to output a feature vector; The parameters of each layer are set using pytorch: the convolution kernel size of the input convolution layer is set to 7x7, the step is set to 2, and the activation function is implemented using relu, the convolution kernel size of the first, second and third convolution layers in the convolution module is set to 1x1, 3x3 and 1x1 respectively, the step is set to 1, and the activation function is implemented using relu, the addition layer requires that the shapes of the two input vectors are consistent; Step 2.2, generate time domain information and frequency domain information, time domain information uses the variance between adjacent frames to obtain, used to reflect the motion of adjacent frames; Frequency domain information is obtained using discrete cosine transform, the read-in RGB image is converted into YUV image, and discrete cosine transform is performed on Y channel to obtain frequency domain information; Step 2.3, construct the time domain feature extraction network, the structure is in turn: input convolution layer, first maximum pooling layer, first convolution module, second convolution module, third convolution module, fourth convolution module, fifth convolution module, sixth convolution module, seventh convolution module, eighth convolution module, ninth convolution module, tenth convolution module, eleventh convolution module, twelfth convolution module, thirteenth convolution, fourteenth convolution module, fifteenth convolution module, sixteenth convolution module; The forward propagation process of time domain information feature extraction is: after the input step 2.2 obtains the time domain information into the input convolution layer to extract the features, first through the first maximum pooling layer and sixteen convolution modules in series, the feature vector of the time domain information is obtained; The parameters of each layer of the network in step 2.3 are set as follows: The convolution kernel size of the input convolution layer is set to 7x7, the step is set to 2, the input channel number is set to 2, and the output channel number is set to 64; The pooling kernel size of the first maximum pooling layer is set to 3x3, the step is set to 2, the padding is set to 2, and the expansion coefficient is set to 1; The structures of the first to sixteenth convolution modules are the same as the convolution module of step 2.1, the input channel numbers of the first convolution layer in the first to sixteenth convolution modules are set to be 64, 256, 256, 256, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 2048, and 2048 in sequence, and the output channel numbers are set to be 64, 64, 64, 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 512, 512, and 512 in sequence; the input channel numbers of the second convolution layer in the first to sixteenth convolution modules are set to be 64, 64, 64, 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 512, 512, and 512 in sequence, and the output channel numbers are set to be 64, 64, 64, 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 512, 512, and 512 in sequence; and the input channel numbers of the third convolution layer in the first to sixteenth convolution modules are set to be 64, 64, 64, 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 512, 512, and 512 in sequence, and the output channel numbers are set to be 256, 256, 256, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, and 2048 in sequence. Step 2.4, constructing a frequency domain feature extraction network, the structure of which is in sequence: a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a sixth convolution module, a seventh convolution module, an eighth convolution module, a ninth convolution module, a tenth convolution module, an eleventh convolution module, a twelfth convolution module, a thirteenth convolution module, a fourteenth convolution module, a fifteenth convolution module, and a sixteenth convolution module; The forward propagation process of the frequency domain information feature extraction is that the frequency domain information obtained in step 2.2 is input through the sixteen convolution modules connected in series to obtain a feature vector of the frequency domain information; The parameter settings of each layer of the network in step 2.4 are as follows: The structures of the first to sixteenth convolutional modules are the same as the convolutional module of step 2.1, the input channel numbers of the first convolutional layer in the first to sixteenth convolutional modules are set to be 64, 256, 256, 256, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 2048, and 2048 in sequence, and the output channel numbers are set to be 64, 64, 64, 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 512, 512, and 512 in sequence; the input channel numbers of the second convolutional layer in the first to sixteenth convolutional modules are set to be 64, 64, 64, 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 512, 512, and 512 in sequence, and the output channel numbers are set to be 64, 64, 64, 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 512, 512, and 512 in sequence; and the input channel numbers of the third convolutional layer in the first to sixteenth convolutional modules are set to be 64, 64, 64, 128, 128, 128, 128, 256, 256, 256, 256, 256, 256, 512, 512, and 512 in sequence, and the output channel numbers are set to be 256, 256, 256, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, and 2048 in sequence. Step 2.5, constructing a time-frequency domain fusion module and a compression damage level evaluation module, which has a structure of a concat layer, an adaptive average pooling layer, a first full connection layer, a second full connection layer, and a third full connection layer; The forward propagation process is as follows: the outputs of steps 2.2 and 2.3 are taken as inputs of the concat layer, the outputs of the concat layer are sequentially input into the adaptive average pooling layer, the first full connection layer, the second full connection layer, and the third full connection layer, and a five-classification vector is output; The parameter settings of each layer of the network of step 2.5 are as follows: The concatenation dimension of the concat layer is set to one dimension; The pooling parameter of the adaptive average pooling layer is set to 1x1; The number of input neurons of the first full connection layer is set to 4096, the number of output neurons is set to 2048; the number of input neurons of the second full connection layer is set to 2048, the number of output neurons is set to 2048; and the number of input neurons of the third full connection layer is set to 2048, and the number of output neurons is set to 5. 2.The method of claim 1, wherein, The step 3 specifically includes: Step 3.1, configuring a training environment and installing python libraries required for network training; Step 3.2, setting training hyperparameters, batch size is set to 16, initial learning rate lr is set to 0.001, weight decay rule is set to learning rate x 0.1 every 10 training periods, solver is selected as SGD, and training period epochs is set to 30; Step 3.3, training the time-frequency domain combined compression damage level evaluation network using the stochastic gradient descent method; first, processing the input image data, that is, transforming the first frame of the input image pair to the frequency domain to obtain frequency domain information, and using an optical flow network to calculate the motion information of the image pair, and setting batch to 448x448; then inputting the data into the compression damage classification network for forward propagation, calculating the loss function L with the output value and the target value, and updating the network weights through back propagation with the solver SGD and the learning rate lr; after each epoch, the updated weights are assigned to the classification network, the validation set is input into the network to calculate the compression damage test accuracy of the validation set, and the auxiliary network training is used to avoid overfitting or underfitting; after 30 epochs of training, the loss function L converges, the last updated weights are assigned to the time-frequency domain combined compression damage level evaluation network, and the trained time-frequency domain combined compression damage level evaluation network is obtained; during training, the loss function L is a multi-class cross-entropy loss function: where N denotes the training sample size, N is set to 12960; M denotes the classification category, M is set to 5; y ij denotes the symbol function 0 or 1, if the real category of sample i is equal to j, y ij is 1, otherwise y ij is 0; p ij denotes the probability that the observation sample i belongs to the category j output by the network. 3.The method of claim 1, wherein, The step 5 specifically comprises: Step 5.1, a video compression damage test set is established, and the test set is prepared according to step 1.6 to obtain frame-by-frame compression damage class label; Step 5.2, input each video in the test set into the trained time-frequency domain combined compression damage level evaluation network, output the final compression damage degree objective evaluation class, and calculate the average compression damage test accuracy of the test set.

4. The method of claim 1-3, wherein, The level evaluation method is applied to help measure the encoding effect of the compression encoder, evaluate the pros and cons of image enhancement and image reconstruction algorithms, classify video compression damage, give different pushing intensity, optimize video pushing rules, detect distortion of video pictures in industry or security as abnormal detection, assist video platform to judge whether the current video needs image enhancement, and repair the video affecting user perception.

Images