A no-reference video quality assessment method integrating dual deep learning networks

Abstract

The present invention discloses a reference-free video quality evaluation method integrating a dual-depth learning network. The video image of each frame in the video is input into the reference-free video quality evaluation network. The deep features of the video image are first extracted through the dual-depth learning network, and then input into the bidirectional gated recurrent network after feature dimension reduction processing through the ANN network. The connection between the video frame features is constructed through the bidirectional gated recurrent network to obtain the video frame quality score. Then, the video frame quality score is subjected to an improved time memory model to obtain a video quality score prediction component. At the same time, the video frame quality score is subjected to a Gaussian distribution regression prediction model to obtain a video quality score prediction component, and then the final video quality score is obtained by optimizing and adjusting the adjustment factor. The present invention can conveniently give a fast prediction evaluation score for the video image quality.

Description

Technical Field

[0001] The present invention relates to the field of computer image recognition, and in particular to a reference-free video quality evaluation method integrating dual deep learning networks. Background Art

[0002] Due to the limitations of transmission conditions, the acquired videos usually need to be compressed. Lossy compression will cause video information loss, resulting in distortion such as blur or blockiness. Packet loss may occur during the transmission process, and the impact of random noise cannot be ignored. Whether the photographer's skills are professional also affects the quality of the video. Distorted videos will bring users a bad viewing experience, so it is very necessary to develop and explore a reliable video quality evaluation method.

[0003] The subjective video quality evaluation method is a method of obtaining an evaluation model by statistically analyzing the collected subjective judgment scores. It has high reliability, but it consumes a lot of time and energy when collecting everyone's scores. Obviously, subjective video quality evaluation is not suitable for this fast-paced development era. Objective video quality evaluation can be divided into full reference video quality evaluation (FR-VQA), semi-reference video quality evaluation (RR-VQA) and no-reference video quality evaluation (NR-VQA) according to whether the source video needs to be involved. Compared with the full reference video quality evaluation and semi-reference video quality evaluation methods, no-reference video quality evaluation is not restricted by the source video, that is, the no-reference video quality evaluation model does not need to access the "perfect version" of the video during the entire evaluation process, but the no-reference video quality evaluation will be slightly inferior in the final evaluation index of the model. Although video quality evaluation has achieved many results, driven by scientific and technological progress and the era of big data, the content of videos on the Internet is increasing, and the integration of various distortion situations increases the complexity of the video. Obviously, video quality evaluation is also facing various difficulties.

[0004] Among the proposed video quality evaluation methods, some evaluate videos from the perspective of factors that affect the quality of user experience; some methods evaluate videos with multiple synthetic distortions by systematic modeling; but some solutions directly process unprocessed natural wild videos to build video quality evaluation models. Existing video quality evaluation methods have their own characteristics, but for unprocessed real videos directly generated by users, which are complex and changeable and affected by multiple distortions, existing quality evaluation methods cannot effectively predict them. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a reference-free video quality evaluation method integrating dual deep learning networks, so as to conveniently give a fast prediction evaluation score for the video image quality.

[0006] In order to solve the above technical problems, the present invention provides a reference-free video quality evaluation method integrating dual deep learning networks, the process comprising: t Input to the no-reference video quality assessment network, the no-reference video quality assessment network first extracts the video image I through the dual deep learning network t The deep feature F t , deep feature F t After the feature dimension reduction is processed by the ANN network, it is input into the bidirectional gated recurrent network, and the connection between the video frame features is constructed through the bidirectional gated recurrent network to obtain the video frame quality score q t , then the video frame quality score q t The improved temporal memory model is used to obtain the video quality score prediction component Q1, and the video frame quality score q t The video quality score prediction component Q2 is obtained through the Gaussian distribution regression prediction model, and the video quality score prediction component Q1 and the video quality score prediction component are then optimized and adjusted through the adjustment factor ρ to obtain the final video quality score Q.

[0007] As an improvement of the no-reference video quality assessment method integrating dual deep learning networks of the present invention:

[0008] The dual deep learning network includes a parallel InceptionV3 network and a ResNet50 network. The deep feature F t for:

[0009]

[0010] in, represents the convolution operation, g mean is the global average pooling g mean , g std is the global standard error pooling, Represents the deep feature map under the InceptionV3 network, Represents the deep feature map under the ResNet50 network:

[0011]

[0012]

[0013] As a further improvement of the no-reference video quality assessment method integrating dual deep learning networks of the present invention:

[0014] The video frame quality score q t for:

[0015]

[0016] Among them, ω t is the weight of the video frame quality information, b t is the deviation parameter of the video frame quality information, Represents each frame image I in the video t The features obtained after passing through the bidirectional gated recurrent network are:

[0017]

[0018] Among them, BiGRU(·) represents the bidirectional gated recurrent network operation, and ANN(·) represents the dimensionality reduction operation of the ANN network;

[0019] The ANN network includes two fully connected layers and one dropout layer.

[0020] As a further improvement of the no-reference video quality assessment method integrating dual deep learning networks of the present invention:

[0021] The improved time memory model includes adding proportional function fitting and exponential function fitting to obtain memory influencing factors in the fitting process of the time memory model. And add the parameter ε to the differentiable softmin function of the softmin-weighted module to obtain the visual retention influencing factor The parameter δ is introduced to balance the memory influencing factors Factors affecting visual retention In this way, we can obtain the quality score Q′ of the t-th frame image. t , and then obtain the prediction component Q1 of the improved time memory model, specifically:

[0022] In t m ∈(t-τ,t-1) frame’s memory impact on frame t for:

[0023]

[0024]

[0025] Among them, q m represents the frame quality score obtained after the minimum pooling operation, l t1 and l t2 They represent the results obtained by fitting using the direct proportional function and the exponential function, respectively. p ={t-τ,...,t-2,t-1};

[0026] In t c The factors affecting the visual retention of the video image within ∈(t,t+τ) on the t-th frame image for:

[0027]

[0028]

[0029] in, Represents the weight of each frame image, V c ={t,t+1,...,t+τ}, ε is a parameter, q n Indicates q t Video frame quality score at time t∈(t,t+τ);

[0030] The video quality score prediction component Q1 is:

[0031]

[0032]

[0033] As a further improvement of the no-reference video quality assessment method integrating dual deep learning networks of the present invention:

[0034] The video quality score prediction component Q2 is:

[0035]

[0036]

[0037] Among them, q c is the feature obtained after the average pooling operation, L is the feature dimension after bidirectional GRU processing, T is the total number of images in the video; μ c and σ c They are respectively the mean and variance of the features obtained by average pooling of all frames in the current video.

[0038] As a further improvement of the no-reference video quality assessment method integrating dual deep learning networks of the present invention:

[0039] The video quality score Q is:

[0040]

[0041] Among them, ρ is the adjustment factor, which is limited to 0 to 1.

[0042] As a further improvement of the no-reference video quality assessment method integrating dual deep learning networks of the present invention:

[0043] The training process of the no-reference video quality assessment network is as follows:

[0044] Videos from the KoNViD-1K database and the LIVE-VQC database were obtained and randomly divided into 60%, 20% and 20% as training set, validation set and test set respectively; the training set and validation set were input into the no-reference video quality assessment network for training, the loss function adopted the L1 loss function, and the optimization method adopted the Adam optimization method, in which the learning rate and the step size during training were set to 0.00001 and 16, and 1000 iterative optimizations were performed on the validation set with the goal of obtaining the maximum Spear-Norman rank coefficient to obtain the trained no-reference video quality assessment network; then the test sets were randomly divided and input into the trained no-reference video quality assessment network, and the Spear-Norman rank coefficient and the Pearson coefficient were used as video quality prediction evaluation indicators, and the average and standard deviation of 10 tests were taken as the final test results, thereby verifying the no-reference video quality assessment network used online.

[0045] The beneficial effects of the present invention are mainly reflected in:

[0046] 1. This paper proposes a new deep learning no-reference video quality assessment model.

[0047] 2. The present invention combines the advantages of dual deep learning networks to improve the effectiveness of video perception feature extraction.

[0048] 3. The improved time memory model of the present invention adds a Gaussian distribution regression model to improve the model evaluation index. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0050] Figure 1 It is the overall block diagram of the model of the present invention;

[0051] Figure 2 For the perceptual feature extraction structure diagram, take the t-th frame image as an example;

[0052] Figure 3 It is the structure diagram of the ANN model in the model;

[0053] Figure 4 The graph obtained by training with different values ​​of τ when δ = 0.5 and ε = 2 using the Spear-Norman rank coefficient (SROCC) as the indicator;

[0054] Figure 5 The training results are shown in Figure 1. The training results are shown in Figure 2. The training results are shown in Figure 3. The training results are shown in Figure 4. The training results are shown in Figure 5. The training results are shown in Figure 6. The training results are shown in Figure 7. The training results are shown in Figure 8. The training results are shown in Figure 9.

[0055] Figure 6 The training results are shown in Figure 1, using the Spear-Norman rank coefficient (SROCC) as the indicator when τ = 12, δ = 0.5 and different values ​​of ε;

[0056] Figure 7 The training results are shown in Figure 1, using the Spear-Norman rank coefficient (SROCC) as the indicator when τ = 12, δ = 0.5, ε = 2, and different values ​​of ρ;

[0057] Figure 8 Loss curves of the invented model trained on two databases.

[0058] Figure 8 In the figure, from top to bottom they are: LIVEVQC val_loss, koNVID-1k val_loss, UVEVQCtrain_loss, koNVID-1k train_loss. DETAILED DESCRIPTION

[0059] The present invention is further described below in conjunction with specific embodiments, but the protection scope of the present invention is not limited thereto:

[0060] Embodiment 1: A reference-free video quality assessment method integrating dual deep learning networks, such as Figure 1 As shown, the specific process is:

[0061] Step 1: Build a no-reference video quality assessment network

[0062] The no-reference video quality assessment network includes dual deep learning networks, ANN networks, bidirectional gated recurrent networks, improved temporal memory models and Gaussian distribution regression models, such as Figure 1 As shown, firstly, the extraction of video perception features is realized through a dual deep learning network: based on the dual deep learning network of InceptionV3 and ResNet50, the diversity of video information is ensured through global average pooling and global standard deviation pooling, so that the basic features and change information of the obtained video frames are more complete; then the connection between the video frame features is constructed through the bidirectional gated recurrent network (BiGRU) to obtain the video frame quality score, and finally, by increasing the exponential function fitting scale and introducing the improved time memory model with hyperparameters, the weight characteristics in the softmin-weighted average pooling module are improved, and the video quality prediction obtained by fusing the improved time memory model and the Gaussian distribution regression model is optimized and adjusted through the adjustment factor, so as to achieve more reliable video quality prediction. The specific process includes:

[0063] Step 1.1: Extraction of perceptual features

[0064] InceptionV3 network and ResNet50 network have wide applications in the field of machine learning, so these two deep learning networks are selected to extract video perception features.

[0065] First, let I t , t∈(1,2,3,...,T) represents each image in the video, T is the total number of images in each video, and the video image I t Two sets of deep feature maps are obtained from the top input of the deep network:

[0066]

[0067]

[0068] in, Represents the deep feature map under the InceptionV3 network, Represents the deep feature map under the ResNet50 network.

[0069] The obtained deep feature map cannot be directly used for the next step of processing. It is necessary to perform global average pooling on the obtained deep feature map according to the traditional algorithm. mean and global standard error pooling g std To save the extracted information. Because this model involves two sets of deep network features, it is necessary to fuse the obtained video information to obtain the deep feature F of each image in the video t :

[0070]

[0071] in, Represents the convolution operation, the purpose of which is to connect the extracted feature information to ensure the integrity of the video information. The specific perceptual feature extraction structure is as follows: Figure 2 shown.

[0072] Step 1.2: Prediction of video frame quality score

[0073] Video is composed of continuous images. When evaluating quality, we need to consider the impact of video continuity on the evaluation results. In this model, the prediction of video frame quality score uses a bidirectional gated recurrent network (BiGRU) network. Compared with the unidirectional gated recurrent network (GRU) network, the bidirectional gated recurrent network (BiGRU) adds a back propagation layer, which strengthens the connection between video content and improves the prediction effect:

[0074]

[0075] in, represents the features of each image after passing through the BiGRU, BiGRU(·) represents the BiGRU operation, and ANN(·) represents the dimensionality reduction operation (including two fully connected layers and one dropout layer, and its structure is as follows Figure 3The purpose of ANN(·) is to reduce the dimension of the video information features retained in the InceptionV3 and ResNet50 networks, making the extracted video content information more condensed.

[0076] After being processed by a bidirectional gated recurrent network (BiGRU), a fully connected layer is added to obtain the video frame quality score:

[0077]

[0078] Among them, q t is the predicted video frame quality score, ω t is the weight of the video frame quality information, b t is the deviation parameter of the video frame quality information.

[0079] Step 1.3: Video quality score prediction

[0080] (1) Prediction component of the improved temporal memory model

[0081] To obtain the prediction component of the improved temporal memory model, it is necessary to consider the m ∈(t-τ,t-1) and t c ∈(t,t+τ) The impact of the images in the two periods on the quality score of the t-th frame image. m ∈(t-τ,t-1) image affects the quality score of the t-th frame image. The present invention obtains important influencing factors by using the method of direct proportional function + exponential function fitting. t c The effect of the segment ∈(t,t+τ) is improved by adding the parameter ε to the softmin-weighted module composed of differentiable softmin functions. The weight characteristics are used to obtain important influencing factors Finally, the parameter δ is introduced to balance and In this way, we can obtain the quality score Q of the t-th frame image. t ', and finally obtain the prediction component Q1 of the improved time memory model. The specific steps are as follows.

[0082] Taking the prediction of the quality score of the t-th frame of the video as an example, the period t'∈(t-τ,t+τ) represents the period from the t-τth frame of the video to the t+τth frame of the video, which contains 2τ+1 pictures in total. Assume that the quality score Q of the t-th frame needs to be predicted. t ', introduce parameters Indicates that at t m ∈(t-τ,t-1) frame’s memory influence on the t-th frame, Indicates tc ∈(t,t+τ) is the factor affecting visual retention in the tth frame.

[0083] In t m ∈(t-τ,t-1), the exponential function fitting method is added to the time memory model. The specific steps of proportional function fitting and exponential function fitting are as follows:

[0084]

[0085]

[0086] Among them, q m represents the frame quality score obtained after the minimum pooling operation, l t1 and l t2 They represent the results obtained by fitting using the direct proportional function and the exponential function, respectively. p ={t-τ,...,t-2,t-1}, qt is the video frame quality prediction score.

[0087] The above part takes into account t m The impact of the segment on the t-th frame image, next we need to consider the c In-segment t c ∈(t,t+τ) on the tth frame. Using a differentiable softmin function to fit the planning weights can achieve better results. In order to further optimize the structure of the entire model and improve the performance of the system, the weights The parameter ε is introduced during definition to improve the characteristics of the weight.

[0088]

[0089]

[0090] in, Represents the weight of each frame image, For c The influence of the video image within ∈(t,t+τ) on the t-th frame image, V c ={t,t+1,...,t+τ}.

[0091] q n Indicates q t Video frame quality score at time t∈(t,t+t).

[0092] In getting t m ∈(t-τ,t-1) and t c ∈(t,t+τ) Memory influencing factors in each segment Factors affecting visual retention After that, the image quality score of the tth frame of the video can be derived, and then the global average pooling can be used to obtain the prediction score of the entire video quality:

[0093]

[0094]

[0095] Among them, δ is the introduced equilibrium and The parameters of the two influencing factors, Q t ' is the quality score of the t-th frame image, and Q1 is the video quality score prediction component obtained by the improved temporal memory model.

[0096] (2) Prediction component of Gaussian distribution regression model

[0097] In the Gaussian distribution regression model, μ and σ change dynamically with model training. The dynamic change of parameters can increase the stability and reliability of the model results.

[0098]

[0099]

[0100] Among them, q c is the feature obtained after the average pooling operation, L is the feature dimension after bidirectional GRU processing; Q2 is the prediction component of the Gaussian distribution regression model, and T is the total number of images in the video. c and σ c They are respectively the mean and variance of the features obtained by average pooling of all frames in the current video, and their values ​​vary depending on the video.

[0101] (3) Video Quality Score Prediction

[0102] Q1 is the video prediction component obtained by the improved temporal memory model, and Q2 is the prediction component obtained by the Gaussian distribution regression model. The principles of the two quality prediction models are different. In order to fully combine the advantages of the two quality prediction components, a hyperparameter ρ is introduced as an adjustment factor to obtain a more accurate video quality evaluation. In order to limit the range of the adjustment factor to 0 to 1, the form of the adjustment factor is taken as

[0103]

[0104] Among them, Q is the final video quality score.

[0105] Step 2: Train the no-reference video quality assessment network

[0106] Step 2.1: Training data

[0107] (1) KoNViD-1K database: This database uses YFCC100 as the baseline and consists of 793,436 creative commons video sequences, forming a total of 1,200 video data. Each video contains 240 frames of images with a resolution of 960×540. MOS is used as the subjective quality score, and the MOS value range is 1-5. The highest score is 4.64 and the lowest score is 1.22.

[0108] (2) LIVE-VQC database: This database was constructed by the Imaging and Video Engineering Laboratory of the University of Texas at Austin. The entire video database contains 585 videos and 80 different types of complex video distortion. 4776 subjects participated in the subjective evaluation. The subjective quality score ranged from 0 to 100, and the resolution was 320×240-1920×1080. The highest score was 94.2865 and the lowest score was 6.2237.

[0109] The videos in the KoNViD-1K database and the LIVE-VQC database are randomly divided into 60%, 20%, and 20% as training sets, validation sets, and test sets, respectively;

[0110] Step 2.1: Train a no-reference video quality assessment network

[0111] The training set obtained after random division in step 2.1 is used for training. The InceptionV3 network and ResNet50 network trained on ImageNet are used as the backbone feature extraction network to extract the perceptual features in the video. The features extracted from the two networks are 1536 dimensions and 4096 dimensions respectively, and the fusion feature is 5632 dimensions, that is, F t It is a 5632-dimensional video content information feature. The two feature extraction network structures selected by the present invention are quite different, so the limitation of extracting features by a single network can be greatly improved, making the present invention have a great advantage in the preliminary information extraction stage.

[0112] The obtained video content information features need to be processed by the ANN network for feature dimensionality reduction, and then the features input into the bidirectional gated recurrent network (BiGRU) are 256-dimensional features. After feature dimensionality reduction, the obtained video information features are more condensed. The hidden size of the bidirectional gated recurrent network is set to 32; in the fitting process of the improved temporal memory model, the exponential function scale is increased and the parameter ε is added to the differentiable softmin function in the softmin-weighted module to improve its weight characteristics. In the improved temporal memory model, the parameters τ, δ and ε need to be determined. The specific steps are as follows.

[0113] To determine the parameter τ, the data is randomly divided on the KoNViD-1k dataset. Under the premise of δ=0.5 and e=2, the SROCC parameter obtained by testing when the parameter τ changes between 3-24 with a step size of 3 is used as the evaluation index. It can be seen that the value of SROCC fluctuates around 0.785, and the video quality prediction result is optimal when τ=12. The specific change trend is as follows: Figure 4 .

[0114] To determine the parameter δ, the data is randomly divided on the KoNViD-1k dataset. Under the premise of τ=12 and ε=2, the SROCC parameter obtained by testing when the parameter δ changes from 0.1 to 0.9 with a step size of 0.1 is used as the evaluation index. As for the trend of the entire curve, it is generally high in the middle and low at both ends. When δ=0.5, the video quality prediction result is the best. The specific change trend is as follows Figure 5 shown.

[0115] To determine the parameter ε, the data is randomly divided on the KoNViD-1k dataset. Under the premise of τ = 12 and d = 0.5, the SROCC parameter obtained by testing the parameter ε from 1 to 5 with a step size of 1 is used as the evaluation index. It can be seen that when ε = 2, the video quality prediction result is the best. The specific change trend is as follows Figure 6 shown.

[0116] Finally, the τ value is set to 12, the δ value is set to 0.5, and the ε value is set to 2.

[0117] The video quality score prediction component Q1 is obtained from the improved and innovative time memory model. The Gaussian distribution model is introduced to make a second prediction of the video quality score. In the Gaussian distribution regression prediction model, the parameters μ and σ change dynamically with the model training to enhance the model stability. Then the video quality score prediction component Q2 is obtained.

[0118] A hyperparameter ρ is introduced as an adjustment factor to combine the advantages of the two prediction methods. To limit the range of the adjustment factor, the form of the adjustment factor is taken as The ρ value is set to 3, and the video quality score Q is finally obtained. For the determination of the parameter ρ, the data is randomly divided on the KoNViD-1k dataset. Under the premise of τ=12, δ=0.5 and ε=2, the SROCC parameter obtained by testing the ρ range between 2 and 50 is used as the evaluation index. When the ρ value is less than 3, the model performance is poor, but when ρ is greater than 3, as the ρ value increases, the system performance shows a downward trend, but its overall SROCC value is still greater than 0.77. When ρ=3, the best performance of the system is obtained, and the specific change trend is as follows Figure 7 shown.

[0119] The loss function in the model adopts L1 loss function, and the optimization method adopts Adam optimization method, in which the learning rate and the step size during training are set to 0.00001 and 16.

[0120] During the training process, the model was optimized for 1000 iterations with the goal of obtaining the model with the maximum Spear-Norman rank coefficient (SROCC) in the validation set to obtain the training model.

[0121] Figure 8 Specifically, the loss curves during training and validation on the KoNViD-1k and LIVEVQC datasets are shown.

[0122] Step 2.2: Test the No-reference Video Quality Assessment Network

[0123] The test uses the Spear-Norman rank coefficient (SROCC) and the Pearson coefficient (PLCC) as video quality prediction evaluation indicators. The closer the SROCC and PLCC are to 1, the better the prediction effect. The test set of each experiment is randomly divided, and the test results are displayed in terms of SROCC and PLCC values. The calculation methods of SROCC and PLCC are as follows;

[0124]

[0125] Where n is the total number of test videos, u i The video quality score obtained for the test is is the average video quality score obtained in the test, v i is the video quality score obtained from the subjective experiment (the video quality score given in the database), The average of the video quality scores obtained for the subjective experiments.

[0126]

[0127] Where X is the video quality score obtained by the test, and Y is the video quality score obtained by the subjective experiment (the video quality score given in the database). COV(X,Y) represents the covariance of the quality score obtained by the test and the video quality score obtained by the subjective test. X represents the standard deviation of the video quality scores obtained in the test, σ Y Indicates the standard deviation of the video quality scores obtained from subjective experiments

[0128] In the comparison test with other models, the average and standard deviation of SROCC and PLCC obtained from 10 experiments were used as the display results; in a random test, SROCC and PLCC were displayed as a single value. This verifies that the trained no-reference video quality assessment network can be used online.

[0129] Step 3: Use online

[0130] In the host computer, each frame of the video to be evaluated I t Input to the no-reference video quality assessment network, first extract each frame image I in the video through the dual deep learning network t The deep feature F t After the feature dimension reduction is processed by the ANN network, it is input into the bidirectional gated recurrent network, and the connection between the video frame features is constructed through the bidirectional gated recurrent network to obtain the video frame quality score q t , then the video frame quality score q t The improved temporal memory model is used to obtain the video quality score prediction component Q1, and the video frame quality score q t The video quality score prediction component Q2 is obtained through the Gaussian distribution regression prediction model, and the video quality score prediction component Q1 and the video quality score prediction component are then optimized and adjusted by the adjustment factor ρ to obtain the final video quality score Q, and the video quality score Q is presented in the host computer.

[0131] Experiment 1:

[0132] In order to verify the effectiveness of the present invention and reduce the impact of different data on the final analysis results, the training data set and data set used in step 2.1 of Example 1 are uniformly used.

[0133] The test environment is built with pytorch, torch-1.70, and Python 3.7. The experiments are conducted on a Lenovo desktop computer with an Intel(R) Xeon(R) Gold 6136 CPU@3.00GHz, a 2.99GHz dual-core processor, 256GB RAM, and an NVIDIA Quadro RTX 6000 graphics card.

[0134] We select VIIDEO[1], TLVQM[2], VSFA[3], V-BLIINDS[4], ChipQA[5], BRISQUE[6], Tang[7] and other models for comparison. The selected comparison models are all from published papers and are all no-reference video quality assessment models. The specific model sources are as follows:

[0135] Comparison model source:

[0136] [1]Min,X.,Zhai,G.,Zhou,J.,Farias,M.C.,&Bovik,A.C.(2020).Study ofsubjective and objective quality assessment of audio-visual signals.IEEETransactions on Image Processing,29,6054-6068.

[0137] [2]Korhonen,J.(2019).Two-level approach for no-reference consumervideo quality assessment.IEEE Transactions on Image Processing,28(12),5923-5938.

[0138] [3]Li,D.,Jiang,T.,&Jiang,M.(2019,October).Quality assessment of in-the-wild videos.In Proceedings of the 27th ACM International Conference onMultimedia(pp.2351-2359).

[0139] [4]Saad,M.A.,Bovik,A.C.,&Charrier,C.(2014).Blind prediction ofnatural video quality.IEEE Transactions on Image Processing,23(3),1352-1365.

[0140] [5]Ebenezer,J.P.,Shang,Z.,Wu,Y.,Wei,H.,Sethuraman,S.,&Bovik,A.C.(2021).ChipQA:No-reference video quality prediction via space-time chips.IEEETransactions on Image Processing,30,8059-8074.

[0141] [6] Mittal, A., Moorthy, AK, & Bovik, AC (2012). No-reference image quality assessment in the spatial domain. IEEE Transactions on image processing, 21 (12), 4695-4708.

[0142] [7]Tang, J., Dong, Y., Xie, R., Gu,

[0143] [8]Sinno,Z.,&Bovik,AC(2018).Large-scale study of perceptual videoquality.IEEE Transactions on Image Processing,28(2),612-627..

[0144] The experiment was conducted 10 times, and the database was randomly divided each time. The average and standard deviation of the 10 experiments were used as the final experimental results. The results of the selected comparison models are from the corresponding literature. The comparison results are shown in Table 1:

[0145] The results in Table 1 are presented in the form of “mean (variance)”, and the best and second-best results in each column are indicated by bold and underline. “*” indicates that the test results are from the literature [8], and “-” indicates that the original author of the paper did not present them in the paper. The calculation method for Overall Performance is score*(1200+585)=KoNViD-1k*1200+LIVEVQC*585

[0146]

[0147] As can be seen from Table 1, on the KoNViD-1k dataset, the experimental results of the no-reference video quality assessment network proposed by the present invention are obviously better than other algorithms in SROCC and PLCC. In particular, compared with the VSFA model, the no-reference video quality assessment network has improved by 3.87% in SROCC and 4.73% in PLCC, with obvious improvement effects; in the LIVEVQC dataset, the no-reference video quality assessment network is at the best in SROCC, 2.25% higher than the suboptimal result, and is suboptimal in the PLCC test results, with a difference of 1.77% from the best result. In terms of overall performance, SROCC is the best performance, 2.87% higher than the suboptimal result, and PLCC is the best result, 0.52% higher than the suboptimal result.

[0148] Finally, it should be noted that the above examples are only some specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments, and there are many variations. All variations that can be directly derived or associated with the content disclosed by a person skilled in the art should be considered as the protection scope of the present invention.

Claims

1. A no-reference video quality assessment method integrating dual deep learning networks, characterized in that : The video image of each frame in the video t Input to the no-reference video quality assessment network, the no-reference video quality assessment network first extracts the video image I through the dual deep learning network t The deep feature F t , deep feature F t After the feature dimension reduction is processed by the ANN network, it is input into the bidirectional gated recurrent network, and the connection between the video frame features is constructed through the bidirectional gated recurrent network to obtain the video frame quality score q t , then the video frame quality score q t The improved temporal memory model is used to obtain the video quality score prediction component Q1, and the video frame quality score q t The video quality score prediction component Q2 is obtained through the Gaussian distribution regression prediction model, and the video quality score prediction component Q1 and the video quality score prediction component are then optimized and adjusted through the adjustment factor r to obtain the final video quality score Q; The improved time memory model includes adding proportional function fitting and exponential function fitting to obtain memory influencing factors in the fitting process of the time memory model. And add the parameter ε to the differentiable softmin function of the softmin-weighted module to obtain the visual retention influencing factor The parameter δ is introduced to balance the memory influencing factors Factors affecting visual retention In this way, we can obtain the quality score Q of the t-th frame image. t ', then the prediction component Q1 of the improved time memory model is obtained, specifically: In t m ∈(tt,t-1) frame’s memory impact on frame t for: Among them, q m represents the frame quality score obtained after the minimum pooling operation, l t1 and l t2 They represent the results obtained by fitting using the direct proportional function and the exponential function, respectively. p ={tt,...,t-2,t-1}; In t c Factors affecting the visual retention of the video image within ∈(t,t+t) on the t-th frame image for: in, Represents the weight of each frame image, V c ={t,t+1,...,t+t}, ε is a parameter, qn represents q t Video frame quality score at time t∈(t,t+t); The video quality score prediction component Q1 is:

2. The method for no-reference video quality assessment integrating dual deep learning networks according to claim 1, characterized in that: The dual deep learning network includes a parallel InceptionV3 network and a ResNet50 network. The deep feature F t for: in, represents the convolution operation, g mean is the global average pooling g mean , g std is the global standard error pooling, Represents the deep feature map under the InceptionV3 network, Represents the deep feature map under the ResNet50 network:

3. The method for no-reference video quality assessment integrating dual deep learning networks according to claim 2, characterized in that: The video frame quality score q t for: Among them, ω t is the weight of the video frame quality information, b t is the deviation parameter of the video frame quality information, Represents each frame image I in the video t The features obtained after passing through the bidirectional gated recurrent network are: Among them, BiGRU(×) represents the bidirectional gated recurrent network operation, and ANN(×) represents the dimensionality reduction operation of the ANN network; The ANN network includes two fully connected layers and one dropout layer.

4. The method for no-reference video quality assessment integrating dual deep learning networks according to claim 3, characterized in that: The video quality score prediction component Q2 is: Among them, q c is the feature obtained after the average pooling operation, L is the feature dimension after bidirectional GRU processing, T is the total number of images in the video; μc and sc are the mean and variance of the features obtained after average pooling of all frames in the current entire video, respectively.

5. The method for no-reference video quality assessment integrating dual deep learning networks according to claim 4, characterized in that: The video quality score Q is: Among them, r is the adjustment factor, which is limited to 0 to 1.

6. The method for no-reference video quality assessment integrating dual deep learning networks according to claim 5, characterized in that: The training process of the no-reference video quality assessment network is as follows: Videos from the KoNViD-1K database and the LIVE-VQC database were obtained and randomly divided into 60%, 20% and 20% as training set, validation set and test set respectively; the training set and validation set were input into the no-reference video quality assessment network for training, the loss function adopted the L1 loss function, and the optimization method adopted the Adam optimization method, in which the learning rate and the step size during training were set to 0.00001 and 16, and 1000 iterative optimizations were performed on the validation set with the goal of obtaining the maximum Spear-Norman rank coefficient to obtain the trained no-reference video quality assessment network; then the test sets were randomly divided and input into the trained no-reference video quality assessment network, and the Spear-Norman rank coefficient and the Pearson coefficient were used as video quality prediction evaluation indicators, and the average and standard deviation of 10 tests were taken as the final test results, thereby verifying the no-reference video quality assessment network used online.

Images