A no-reference video quality assessment method based on short-time spatiotemporal fusion network and long-time series fusion network
Through the combination method of short-time and space-time fusion network and long-time sequence fusion network, the problem of poor quality evaluation performance without reference is solved, and more accurate and efficient video quality evaluation is achieved. The video score is improved through inter-frame impact factor fusion. accuracy.
Patent Information
- Application Number
- CN202110992742.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-27
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-08-27
AI Technical Summary
The performance of the video quality evaluation without reference is poor in existing video quality evaluation, making it difficult to accurately evaluate video quality without using human resources and reference video.
The video quality is objectively evaluated without reference quality by using short-time and space-time fusion network and long-time sequence fusion network. By dividing video frames into independent frames, 64-dimensional feature vectors and preliminary predicted quality scores are extracted, and a priori knowledge is used to transform them into frames. The inter-influence factor finally obtains the overall quality score of the video through long-term sequence fusion network.
It improves the accuracy and efficiency of video quality evaluation, fully integrates space-time characteristics, enhances the content correlation between frames, and makes the video scores obtained more accurate.
Smart Images

Figure CN113784113B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a reference-free video quality evaluation method based on a short-time spatiotemporal fusion network and a long-time sequence fusion network, and belongs to the technical field of digital video processing. Background Art
[0002] As a complex source of visual information, video contains a large amount of valuable information. The quality of video directly affects people's subjective feelings and information acquisition, and can guide other video tasks such as related equipment development, system monitoring, and quality restoration. The research on video quality assessment (VQA) has also received widespread attention in recent years.
[0003] Video quality evaluation methods can be divided into subjective evaluation methods and objective evaluation methods. Subjective evaluation is a subjective evaluation of video quality by observers. Although the evaluation results are consistent with people's subjective feelings, they also have disadvantages such as large workload and long time consumption. Objective evaluation methods are calculated by computers according to certain algorithms to obtain video quality indicators. Depending on whether reference videos are needed during evaluation, they can be divided into three types of evaluation methods: Full Reference (FR), Semi-reference (RR) and No Reference (NR):
[0004] (1) Full reference video quality evaluation method. The FR algorithm is to compare the difference between the video to be evaluated and the reference video under the standard of the given ideal video as the reference video, analyze the degree of distortion of the video to be evaluated, and thus obtain the quality evaluation result of the video to be evaluated. Common FR methods include: video quality evaluation based on video pixel statistics (mainly peak signal-to-noise ratio and mean square error), video quality evaluation based on deep learning, and video quality evaluation based on structural information (mainly structural similarity). The FR algorithm is the most reliable method for objective video quality evaluation so far.
[0005] (2) Semi-reference video quality assessment method. The RR algorithm extracts some feature information of the reference video as a reference, and compares and analyzes the video to be evaluated to obtain the quality assessment result of the video. Common RR algorithms are mainly: the method based on the original video features and the method based on the Wavelet domain statistical model.
[0006] (3) Non-reference video quality assessment method. The NR algorithm refers to a method for evaluating the quality of the video to be evaluated when there is no ideal video. Commonly used NR algorithms are: methods based on natural scene statistics and methods based on deep learning.
[0007] In the process of video acquisition, processing, transmission and recording, due to the imperfections of imaging systems, processing methods, transmission media and recording equipment, as well as the distortion and degradation caused by object movement, noise interference, etc., the video quality often needs to be measured before it is officially used. The method of obtaining the quality of distorted video directly without using human resources and reference videos is called objective non-reference video quality evaluation.
[0008] Chinese patent CN201811071199.1 discloses a method for image quality assessment without reference based on hierarchical feature fusion network, which mainly solves the problems of low precision and slow speed of existing technologies. Its implementation scheme is: select reference images from the MSCOCO dataset and establish a polluted image database by adding noise; remove the mean and crop the training set images and test set images at the same time; design a hierarchical feature fusion network model for end-to-end joint optimization according to the hierarchical processing mechanism of the human visual system from local features to global semantics; train the hierarchical feature fusion network model using the training set and the test set; remove the mean and crop the image to be evaluated, and input the processed image into the trained hierarchical feature fusion network model to obtain the image quality prediction score; thereby improving the accuracy and speed of the quality assessment without reference, which can be used for image screening, compression, and video quality monitoring.
[0009] Chinese patent CN201810239888.2 discloses a full-reference virtual reality video quality evaluation method based on convolutional neural network, including: video preprocessing: using the left view video and the right view video of the VR video to obtain a VR differential video, evenly extracting frames from the differential video, giving each frame non-overlapping blocks, and the video blocks at the same position of each frame constitute a VR video patch; establishing two convolutional neural network models with the same configuration; training the convolutional neural network model: using the gradient descent method, taking the VR video patch as input, each patch is equipped with the original video quality score as a label, and inputting it into the network in batches. After multiple iterations, the weights of each layer of the network are fully optimized, and finally a convolutional neural network model that can be used to extract virtual reality video features is obtained; using the convolutional neural network to extract features; using the support vector machine to obtain local scores, and using the score fusion strategy to obtain the final score to improve the accuracy of the objective evaluation method.
[0010] The present invention intends to use a short-time spatiotemporal fusion network and a long-time series fusion network to perform objective reference-free quality evaluation on video quality. Summary of the invention
[0011] In view of the problem of poor performance of no-reference video quality evaluation in existing video quality evaluation, the present invention proposes an objective no-reference quality evaluation method. The present invention divides the video into video frames, and obtains a 64-dimensional feature vector and a preliminary predicted quality score of each video frame through a short-time spatiotemporal fusion network. The feature vectors are then combined into a feature sequence in chronological order, and the preliminary predicted quality scores are converted into influencing factors between frames under the guidance of prior knowledge. They are then used as inputs of a long-time series fusion network to obtain the overall quality score of the video, thereby completing the quality evaluation process.
[0012] The technical solution adopted by the present invention is a method based on a short-time spatiotemporal fusion network and a long-time series fusion network to perform non-reference objective quality evaluation on video quality, comprising the following steps:
[0013] Step 1: Get video frames from the video.
[0014] For a video, we need to divide it into frames, downsample it at equal intervals, and obtain a series of independent video frames, which are then used as the input of the VQA model:
[0015] Step 1.1, extract video frames, convert the complete video sequence from YUV and other formats into BMP format, and save them frame by frame;
[0016] Step 1.2, sampling video frames, selecting video frames at intervals of 4, and directly discarding other video frames due to redundancy.
[0017] Step 1.3, generate a brightness map, convert the three-channel color video frame into a brightness map with only the Y channel.
[0018] Step 2: Build and train the short-term spatiotemporal fusion network.
[0019] Build and train a short-term spatiotemporal fusion network based on convolutional neural networks (CNN), whose input is the video frame obtained in step 1.2, and the output is a 64-dimensional feature vector and a preliminary predicted quality score:
[0020] Step 2.1, use UNet as the network to extract spatial features, take the video frames generated in step 1.2 as the input of this network, the size is [Batch-Size×3×Width×Height], the label used to train the network is the gradient magnitude (GradientMagnitude, GM) similarity in the feature similarity index measure (FSIM); the output of the network is the predicted GM similarity, the size is [Batch-Size×1×Width×Height];
[0021] Step 2.2, use the pre-trained PWCNet as the network for extracting temporal features, and use the video frames generated in step 1.2 as the input of this network, with a size of [Batch-Size×3×Width×Height]; convert the predicted optical flow map into a brightness map as the output of the network, with a size of [Batch-Size×1×Width×Height];
[0022] Step 2.3, build a network to extract spatiotemporal feature vectors and preliminary quality scores. The network has two inputs, which are the video frame brightness map generated in step 1.3 and the combination of the spatial features generated in step 2.1 and the temporal features generated in step 2.2. The size of each is [Batch-Size×2×Width×Height]. The network consists of 5 convolutional structures and 4 fully connected layers. Each convolutional structure consists of a convolutional layer, a pooling layer, a regularization layer and an activation function. The number of convolution kernels is 32, 64, 64, 128, and 128 respectively. The size of the convolution kernel is 3×3, and the step size of the pooling layer is 2×2; the sizes of the fully connected layers are 1024, 256, 64 and 1 respectively. The outputs of the third and fourth fully connected layers are used as the outputs of the network. The output of the third layer is a 64-dimensional feature vector with a size of [Batch-Size×64×1], and the output of the fourth layer is a preliminary predicted quality score with a size of [Batch-Size×1×1].
[0023] Step 2.4: Build a short-time spatiotemporal fusion network based on steps 2.1 to 2.3. The input of the network is the video frame generated in step 1.2, with a size of [Batch-Size×3×Width×Height]. First, obtain spatial features and temporal features through steps 2.1 and 2.2 respectively. Then, combine the brightness map generated in step 1.3 with these two features respectively and obtain the output of the network through step 2.3: feature vector and preliminary predicted quality score, with sizes of [Batch-Size×64×1] and [Batch-Size×1×1] respectively.
[0024] Step 2.5, prepare training data, use the video frame generated in step 1.2 as the input of the network built in step 2.4, and its label is the subjective quality score of the current video;
[0025] Step 2.6, train the short-time spatiotemporal fusion network, the input is the video frame generated in step 1.2, the output is the feature vector and the preliminary predicted quality score, use MSE Loss to compare the label and the preliminary predicted quality score and train the network.
[0026] Step 3: obtain feature sequences of several video segments and mutual influence factors of frames in the video segments.
[0027] Through the trained short-term spatiotemporal fusion network, a 64-dimensional feature vector and a preliminary predicted quality score are generated for each video frame, and the feature vectors are combined into a feature sequence in chronological order. The preliminary predicted quality scores are converted into the influence factors between each frame under the guidance of prior knowledge:
[0028] Step 3.1, combine the feature vectors into a feature sequence. In the video sequence, randomly select N starting points and take the 64-dimensional feature vectors of T frames along the time direction. T defaults to 8, and the interval between adjacent frames defaults to 4, to obtain a T×64 feature sequence, which serves as the minimum unit of subsequent network input.
[0029] Step 3.2, calculate the influence factor, and transform the preliminary predicted quality scores of each frame obtained in step 2 into their influence on quality under the guidance of prior knowledge. The influence is normalized to [0,1], and the influence of the current frame on its own quality is fixed to 1.
[0030] Step 4: Build and train the long-term sequence fusion network.
[0031] Build and train a long time series fusion network based on Graph Convolutional Network (GCN). The input is the fixed-length feature sequence output in step 3 and the influence factor between each frame. The output is the quality score of the video:
[0032] Step 4.1: Build a long-term sequence fusion network based on GCN. The input of the network is a feature sequence composed of several 64-dimensional feature vectors and the influence factors between videos represented by each feature vector. The sizes are Batch-Size×T×64 and Batch-Size×T×T respectively. The network contains a 2-layer GCN structure, followed by a fully connected layer with an output size of T.
[0033] Step 4.2, arranging the training data, arranging the several feature sequences obtained in step 3 and the influence factors between each frame into the format input in step 4.1 as the input of the long time series fusion network;
[0034] Step 4.3, train the long time series fusion network, use the video quality score as the label, and use MSE Loss for training.
[0035] Step 5: Evaluate the quality of the video.
[0036] Perform frame cutting, sampling, feature extraction, and quality evaluation on a video.
[0037] Step 5.1, according to the steps in step 1, perform frame cutting and sampling of the video to be tested into video frames;
[0038] Step 5.2, using the short-time spatiotemporal fusion network trained in step 2 to extract features and perform preliminary fusion on the video frames cut in step 5.1;
[0039] Step 5.3, use step 3 to organize and calculate the feature vector and preliminary prediction score obtained in step 5.2;
[0040] Step 5.4, use the long time series fusion network trained in step 4 to further fuse the video sequence and influencing factors obtained in step 5.3, so that a video can finally obtain the quality scores of M video frames.
[0041] Step 5.5, averaging the quality scores of the M video frames in step 5.4 to obtain the overall quality score of the video.
[0042] Compared with the prior art, the present invention has the following advantages:
[0043] (1) Existing VQA technical methods based on deep learning often focus on how to extract temporal and spatial features, dividing them into two independent individuals, while ignoring how to fuse these two features. The present invention designs a short-term spatiotemporal fusion network specifically for fusing spatiotemporal features, which better integrates the extracted features at the frame level, thereby characterizing the quality of the current frame.
[0044] (2) Existing two-dimensional VQA technical methods usually assume that the quality of each frame is determined by itself, ignoring the impact of the interaction between frames. The present invention uses a long time series fusion network and uses continuous T frames centered on the current frame to jointly determine the quality of the current frame. Due to the design of feature fusion on the time axis, the present invention can more fully extract the correlation between the content of video frames, and the features obtained by the network can better represent the characteristics of each frame, so that the final predicted video score is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 It is a flow chart of a specific implementation mode of the present invention.
[0046] Figure 2 It is a structural diagram of the short-time spatiotemporal fusion network and the long-time series fusion network of the present invention. DETAILED DESCRIPTION
[0047] The method is described in detail below with reference to the accompanying drawings and embodiments.
[0048] Implementation method.
[0049] The flowchart of the implementation method is as follows Figure 1 As shown, the following steps are included:
[0050] Step S10, obtaining a video frame from the video;
[0051] Step S20, building and training a short-term spatiotemporal fusion network;
[0052] Step S30, obtaining feature sequences of several video segments and mutual influence factors of frames in the video segments;
[0053] Step S40, building and training a long time series fusion network;
[0054] Step S50, evaluating the quality of the video;
[0055] The step S10 of obtaining a video frame adjustment from a video in the implementation manner further includes the following steps:
[0056] Step S100, extracting video frames, converting the complete video sequence from YUV or other formats into BMP format, and saving frame by frame;
[0057] Step S110 , sampling video frames, selecting video frames at intervals of 4, and directly discarding other video frames due to redundancy.
[0058] Step S120, generating a brightness map, converting the three-channel color video frame into a brightness map with only the Y channel.
[0059] The step S20 of building and training the short-time spatiotemporal fusion network adjustment in the implementation mode also includes the following steps:
[0060] Step S200, using UNet as a network for extracting spatial features, taking the video frame generated in step S110 as the input of this network, the size of which is [Batch-Size×3×Width×Height], and the label used for training the network is the gradient magnitude (GM) similarity in the feature similarity index measure (FSIM); the output of the network is the predicted GM similarity, the size of which is [Batch-Size×1×Width×Height];
[0061] Step S210, using the pre-trained PWCNet as the network for extracting temporal features, and using the video frames generated in step S110 as the input of this network, with a size of [Batch-Size×3×Width×Height]; converting the predicted optical flow map into a brightness map as the output of the network, with a size of [Batch-Size×1×Width×Height];
[0062] Step S220, build a network to extract spatiotemporal feature vectors and preliminary quality scores. The network has two inputs, namely, the video frame brightness map generated in step S120 and the combination of the spatial features generated in step S200 and the temporal features generated in step S210, both of which are of size [Batch-Size×2×Width×Height]. The network consists of 5 convolutional structures and 4 fully connected layers. Each convolutional structure consists of a convolutional layer, a pooling layer, a regularization layer, and an activation function. The number of convolutional kernels is 32, 64, 64, 128, and 128, respectively. The convolutional kernel size is 3×3, and the step size of the pooling layer is 2×2; the sizes of the fully connected layers are 1024, 256, 64, and 1, respectively. The outputs of the third and fourth fully connected layers are both used as the outputs of the network. The output of the third layer is a 64-dimensional feature vector of size [Batch-Size×64×1], and the output of the fourth layer is a preliminary predicted quality score of size [Batch-Size×1×1];
[0063] Step S230, building a short-time spatiotemporal fusion network based on steps S200 to S220, the input of the network is the video frame generated in step S110, the size of which is [Batch-Size×3×Width×Height], firstly obtain spatial features and temporal features through steps S200 and S210 respectively, then combine the brightness map generated in step S120 with the two features respectively, and obtain the output of the network through step S220: feature vector and preliminary predicted quality score, the sizes of which are [Batch-Size×64×1] and [Batch-Size×1×1] respectively;
[0064] Step S240, preparing training data, using the video frame generated in step S110 as the input of the network built in step S230, and its label is the subjective quality score of the current video;
[0065] Step S250, training a short-time spatiotemporal fusion network, the input is the video frame generated in step S110, the output is a feature vector and a preliminary predicted quality score, MSE Loss is used to compare the label and the preliminary predicted quality score and train the network.
[0066] The step S30 of obtaining the feature sequences of the plurality of video segments and adjusting the mutual influence factors of the frames in the video segments also includes the following steps:
[0067] Step S300, combining feature vectors into feature sequences, randomly selecting N starting points in the video sequence, taking 64-dimensional feature vectors of T frames along the time direction, T is 8 by default, and the interval between adjacent frames is 4 by default, to obtain a T×64 feature sequence, which serves as the minimum unit of subsequent network input.
[0068] Step S310, calculate the influence factor, and convert the preliminary predicted quality scores of each frame obtained in step S20 into their influence on quality under the guidance of prior knowledge. The influence is normalized to [0,1], and the influence of the current frame on its own quality is fixed to 1.
[0069] The step S40 of setting up and training the long-term sequence fusion network adjustment in the implementation manner further includes the following steps:
[0070] Step S400, building a long-term sequence fusion network based on GCN. The input of the network is a feature sequence composed of several 64-dimensional feature vectors and the influence factors between videos represented by each feature vector. The sizes are Batch-Size×T×64 and Batch-Size×T×T respectively. The network contains a 2-layer GCN structure, followed by a fully connected layer with an output size of T.
[0071] Step S410, arranging the training data, arranging the several feature sequences obtained in step S30 and the influence factors between each frame into the format input in step S400 as the input of the long time series fusion network;
[0072] Step S420: training a long time series fusion network, using the video quality score as a label, and using MSE Loss for training.
[0073] The step S50 of adjusting the video quality evaluation in the implementation manner further includes the following steps:
[0074] Step S500, performing frame cutting and sampling of the video to be tested into video frames according to the steps of step S10;
[0075] Step S510, using the short-time spatiotemporal fusion network trained in step S20 to extract features and perform preliminary fusion on the video frames cut in step S500;
[0076] Step S520, using step S30 to sort and calculate the feature vector and preliminary prediction score obtained in step S510;
[0077] Step S530, using the long time series fusion network trained in step S40 to further perform feature fusion on the video sequence and influencing factors obtained in step S520, so that a video finally obtains quality scores of M video frames.
[0078] Step S540, averaging the quality scores of the M video frames in step S530 to obtain an overall quality score of the video.
[0079] The experimental results of applying the present invention are given below.
[0080] Table 1 shows the performance results of the present invention on various VQA databases. (without pre-training) Table 1 Test results of the present invention on various VQA databases
[0081] database LIVE CISQ KoNVid-1k SRCC 0.844 0.851 0.762 PLCC 0.859 0.869 0.784
Claims
1. A reference-free video quality assessment method based on a short-time spatiotemporal fusion network and a long-time series fusion network, characterized in that: The method comprises the following steps: Step 1, obtaining a video frame from a video; For a video, it is necessary to divide it into frames and perform equal-interval downsampling operations to obtain independent video frames, which are then used as the input of the short-term spatiotemporal fusion network based on the convolutional neural network (CNN). Step 2: Build and train a short-term spatiotemporal fusion network; Build and train a short-term spatiotemporal fusion network based on a convolutional neural network (CNN), whose input is the independent video frame obtained in step 1, and whose output is a 64-dimensional feature vector and a preliminary predicted quality score; Step 3, obtaining feature sequences of several video segments and mutual influence factors of frames in the video segments; Through the trained short-term spatiotemporal fusion network, a 64-dimensional feature vector and a preliminary predicted quality score are generated for each video frame, and the feature vectors are combined into a feature sequence in chronological order. The preliminary predicted quality scores are converted into influence factors between frames under the guidance of prior knowledge. The steps of obtaining the feature sequences of several video segments and the mutual influence factors of each frame in the video segment are as follows: Step 3.1, combine the feature vectors into a feature sequence. In the video sequence, randomly select N starting points and take the 64-dimensional feature vectors of T frames along the time direction. T defaults to 8 and the interval between adjacent frames defaults to 4, to obtain a T×64 feature sequence, which is used as the minimum unit of subsequent network input. Step 3.2, calculate the influence factor, and transform the preliminary predicted quality scores of each frame obtained in step 2 into their influence on quality under the guidance of prior knowledge. The influence is normalized to [0, 1], and the influence of the current frame on its own quality is fixed to 1; Step 4: Build and train the long-term sequence fusion network; Build and train a long time series fusion network based on graph convolutional network GCN. The input is the fixed-length feature sequence output in step 3 and the influence factor between each frame. The output is the quality score of the video. Step 5, evaluating the quality of the video; Perform frame cutting, sampling, feature extraction, and quality evaluation on a video.
2. The method for no-reference video quality assessment based on a short-time spatiotemporal fusion network and a long-time series fusion network according to claim 1, characterized in that: The steps to get the video frame from the video are as follows: Step 1.1, extract the video frames, convert the complete video sequence from YUV format to BMP format, and save it frame by frame; Step 1.2, sampling video frames, selecting video frames at intervals of 4, and directly discarding other video frames due to redundancy; Step 1.3, generate a brightness map, convert the three-channel color video frame into a brightness map with only the Y channel.
3. The method for no-reference video quality assessment based on a short-time spatiotemporal fusion network and a long-time series fusion network according to claim 2, characterized in that: The steps to build and train a short-term spatiotemporal fusion network are as follows: Step 2.1, use UNet as the network to extract spatial features, take the video frames generated in step 1.2 as the input of this network, the size is [Batch-Size×3×Width×Height], the label used to train the network is the gradient amplitude GM similarity in the feature similarity index method FSIM; the output of the network is the predicted GM similarity, the size is [Batch-Size×1×Width×Height]; Step 2.2, use the pre-trained PWCNet as the network for extracting temporal features, and use the video frames generated in step 1.2 as the input of this network, with a size of [Batch-Size×3×Width×Height]; convert the predicted optical flow map into a brightness map as the output of the network, with a size of [Batch-Size×1×Width×Height]; Step 2.3, build a network to extract spatiotemporal feature vectors and preliminary quality scores. The network has two inputs, namely the video frame brightness map generated in step 1.3 and the combination of the spatial features generated in step 2.1 and the temporal features generated in step 2.
2. The sizes are all [Batch-Size×2×Width×Height]. The network consists of 5 convolutional structures and 4 fully connected layers. Each convolutional structure consists of a convolutional layer, a pooling layer, a regularization layer and an activation function. The number of convolution kernels is 32, 64, 64, 128, and 128 respectively. The convolution kernel size is 3×3, and the step size of the pooling layer is 2×2; the sizes of the fully connected layers are 1024, 256, 64 and 1 respectively; the outputs of the third and fourth fully connected layers are used as the outputs of the network. The output of the third layer is a 64-dimensional feature vector with a size of [Batch-Size×64×1], and the output of the fourth layer is a preliminary predicted quality score with a size of [Batch-Size×1×1]. Step 2.4: Build a short-time spatiotemporal fusion network based on steps 2.1 to 2.
3. The input of the network is the video frame generated in step 1.2, with a size of [Batch-Size×3×Width×Height]. First, obtain spatial features and temporal features through steps 2.1 and 2.2 respectively. Then, combine the brightness map generated in step 1.3 with these two features respectively and obtain the output of the network through step 2.3: feature vector and preliminary predicted quality score, with sizes of [Batch-Size×64×1] and [Batch-Size×1×1] respectively. Step 2.5, prepare training data, use the video frame generated in step 1.2 as the input of the network built in step 2.4, and its label is the subjective quality score of the current video; Step 2.6, train the short-time spatiotemporal fusion network, the input is the video frame generated in step 1.2, the output is the feature vector and the preliminary predicted quality score, use MSE Loss to compare the label and the preliminary predicted quality score and train the network.
4. The method for no-reference video quality assessment based on a short-time spatiotemporal fusion network and a long-time series fusion network according to claim 1, characterized in that: The steps to build and train a long time series fusion network are as follows: Step 4.1, build a long-term series fusion network based on GCN. The input of the network is a feature sequence composed of several 64-dimensional feature vectors and the influence factors obtained by predicting the quality score. The sizes are Batch-Size×T×64 and Batch-Size×T×T respectively. The network contains a 2-layer GCN structure, followed by a fully connected layer with an output size of T; the influence factors between video frames represented by the predicted video frame quality score; Step 4.2, arranging the training data, arranging the several feature sequences obtained in step 3 and the influence factors between each frame into the format input in step 4.1 as the input of the long time series fusion network; Step 4.3, train the long time series fusion network, use the video quality score as the label, and use MSE Loss for training.
5. The method for no-reference video quality assessment based on a short-time spatiotemporal fusion network and a long-time series fusion network according to claim 1, characterized in that: The steps for video quality evaluation are as follows: Step 5.1, cut the test video into frames and sample it into video frames according to the steps in step 1; Step 5.2, using the short-time spatiotemporal fusion network trained in step 2 to extract features and perform preliminary fusion on the video frames cut in step 5.1; Step 5.3, use step 3 to organize and calculate the feature vector and preliminary prediction score obtained in step 5.2; Step 5.4, using the long time series fusion network trained in step 4 to further fuse the video sequence and influencing factors obtained in step 5.3, so that a video finally obtains the quality scores of M video frames; Step 5.5, averaging the quality scores of the M video frames in step 5.4 to obtain the overall quality score of the video.
