Diagnosis method and device for temporal feature extractor in video quality assessment model

Abstract

The present invention relates to the field of multimedia technology and discloses a diagnostic method for a temporal feature extractor in a video quality evaluation model, comprising: inputting an ordered video frame set and an unordered video frame set into a video quality evaluation model for training to obtain an ordered training evaluation model and an unordered training evaluation model; using the ordered video frame set and the unordered video frame set to test the ordered training evaluation model and the unordered training evaluation model to obtain an ordered training ordered test quality score, an ordered training unordered test quality score, an unordered training ordered test quality score, and an unordered training unordered test quality score; calculating the evaluation accuracy of the video quality evaluation model to obtain a diagnostic result. The invention also proposes a diagnostic device, an electronic device, and a computer-readable storage medium for a temporal feature extractor in a video quality evaluation model. The present invention can solve the problems of the current video quality evaluation model in that it lacks a diagnostic temporal feature extraction module and consumes a large amount of computing resources.

Description

Technical Field

[0001] The present invention relates to the field of multimedia technology, and in particular to a diagnostic method, device, electronic device and computer-readable storage medium for a temporal feature extractor in a video quality assessment model. Background Art

[0002] With the advent of the information age, watching videos has become the main way for people to obtain information and relax and entertain themselves. Currently, mainstream video quality evaluation models have integrated temporal feature extraction modules, which can capture the temporal change characteristics between video frames.

[0003] Currently, there is no corresponding diagnostic method to determine whether the timing feature extraction module plays a role in the video quality evaluation model, and extracting the timing change characteristics between video frames based on the timing feature extraction module requires a large amount of computing resources. Therefore, the current video quality evaluation model lacks the diagnosis of the timing feature extraction module and consumes a large amount of computing resources. Summary of the Invention

[0004] The present invention provides a diagnostic method, device and computer-readable storage medium for a temporal feature extractor in a video quality assessment model, the main purpose of which is to solve the problems of the current video quality assessment model lacking a temporal feature extraction module check and consuming a large amount of computing resources.

[0005] To achieve the above-mentioned object, the present invention provides a diagnostic method for a temporal feature extractor in a video quality assessment model, comprising:

[0006] Acquire a video quality data frame set, sort the video frames in the video quality data frame set in chronological order to obtain an ordered video frame set, and disrupt the order of the video frames in the ordered video frame set to obtain an unordered video frame set;

[0007] Inputting the video frames in the ordered video frame set into a pre-built video quality assessment model in sequence for training to obtain an ordered training assessment model, wherein the video quality assessment model includes a temporal feature extractor;

[0008] Inputting the video frames in the disordered video frame set into the video quality evaluation model in sequence for training to obtain a disordered training evaluation model;

[0009] The ordered training evaluation model is tested using the ordered video frame set and the unordered video frame set respectively to obtain an ordered training ordered test quality score and an ordered training unordered test quality score;

[0010] The disordered training evaluation model is tested using the ordered video frame set and the disordered video frame set respectively to obtain a disordered training ordered test quality score and a disordered training disordered test quality score;

[0011] Calculating the evaluation accuracy of the video quality evaluation model according to the ordered training ordered test quality score, the ordered training disordered test quality score, the disordered training ordered test quality score, and the disordered training disordered test quality score;

[0012] According to the evaluation accuracy, a diagnosis result of the temporal feature extractor in the video quality evaluation model is obtained.

[0013] Optionally, disrupting the order of the video frames in the ordered video frame set to obtain an unordered video frame set includes:

[0014] randomly extracting video frames from the ordered video frame set according to a pre-built shuffling algorithm;

[0015] The video frames are sorted according to the extraction order of the video frames to obtain the unordered video frame set.

[0016] Optionally, the testing of the ordered training evaluation model using the ordered video frame set and the unordered video frame set respectively to obtain an ordered training-ordered test quality score and an ordered training-unordered test quality score includes:

[0017] Inputting the video frames in the ordered video frame set into the ordered training and evaluation model in sequence, and extracting features of the video frames in the ordered video frame set using a temporal feature extractor in the ordered training and evaluation model to obtain an ordered feature map;

[0018] Performing global average pooling and global standard deviation pooling on the ordered feature map to obtain an ordered multidimensional feature vector;

[0019] Performing a dimensionality reduction on the ordered multidimensional feature vector using a pre-constructed first dimensionality reduction formula to obtain an ordered first dimensionality reduction feature vector;

[0020] Inputting the ordered first dimension-reduced feature vector into a pre-built gated recurrent unit to obtain an ordered gated recurrent feature vector;

[0021] Performing secondary dimensionality reduction on the ordered gated cyclic feature vector using a pre-constructed second dimensionality reduction formula to obtain an ordered second dimensionality reduction feature vector;

[0022] Pooling the ordered second dimensionality reduction feature vectors to obtain quality scores of the video frames in the ordered video frame set;

[0023] Integrating the quality scores of all video frames in the ordered video frame set to obtain the ordered training and ordered testing quality scores;

[0024] Extracting features of video frames in the disordered video frame set using a temporal feature extractor in the ordered training evaluation model to obtain a disordered feature map;

[0025] Performing global average pooling and global standard deviation pooling on the disordered feature map to obtain a disordered multidimensional feature vector;

[0026] Performing a dimensionality reduction on the disordered multidimensional feature vector using the first dimensionality reduction formula to obtain an disordered first dimensionality reduction feature vector;

[0027] Inputting the unordered first dimension-reduced feature vector into the gated recurrent unit to obtain an unordered gated recurrent feature vector;

[0028] Performing secondary dimensionality reduction on the disordered gated cyclic feature vector using the second dimensionality reduction formula to obtain an disordered second dimensionality reduction feature vector;

[0029] Pooling the unordered second dimensionality reduction feature vector to obtain quality scores of the video frames in the unordered video frame set;

[0030] The quality scores of all video frames in the unordered video frame set are integrated to obtain the ordered training and unordered testing quality scores.

[0031] Optionally, the first dimensionality reduction formula is as follows:

[0032] X t =W FX F t +b x

[0033] Among them, X t Represents the eigenvector after the first dimensionality reduction formula, W FX represents the fully connected layer parameters, F t represents the feature vector to be reduced in dimension after global average pooling and global standard deviation pooling, b x represents the bias of the fully connected layer.

[0034] Optionally, the second dimensionality reduction formula is as follows:

[0035] Q t =W HQ H t +b q

[0036] Among them, Q t Represents the eigenvector after the second dimensionality reduction formula, W HQ represents the fully connected layer parameters, Ht represents the feature vector after being processed by the gated recurrent unit, b q represents the bias of the fully connected layer.

[0037] Optionally, calculating the evaluation accuracy of the video quality evaluation model according to the ordered training ordered test quality score, the ordered training disordered test quality score, the disordered training ordered test quality score, and the disordered training disordered test quality score includes:

[0038] Obtaining a true quality score of the video quality data frame set;

[0039] Utilizing a pre-constructed correlation coefficient formula, respectively calculating the rank correlation coefficients between the true quality score and the ordered training ordered test quality score, the ordered training unordered test quality score, the unordered training ordered test quality score, and the unordered training unordered test quality score;

[0040] The evaluation accuracy of the video quality evaluation model is judged according to the rank correlation coefficient.

[0041] Optionally, the correlation coefficient formula is as follows:

[0042]

[0043] Among them, ρ s represents the rank correlation coefficient, n represents the number of frames in the video quality data frame set, d i represents the rank difference of the i-th video frame.

[0044] Optionally, after obtaining the diagnosis result of the temporal feature extractor in the video quality assessment model based on the evaluation accuracy, the method further includes:

[0045] Dividing the video frames in the video quality data frame set into blocks according to different predetermined video frame block standards to obtain a block video frame set;

[0046] Randomly selecting a video frame from each block of the video frame set to obtain a block-filtered frame set;

[0047] Using the video quality evaluation model for training to obtain a quality evaluation model to be tested;

[0048] Using the block-screened frame set to test the quality evaluation model to be tested, to obtain a block-test quality score;

[0049] Calculating the evaluation accuracy of the video quality evaluation model under each video frame segmentation standard according to the segmentation test quality score;

[0050] An optimal video segmentation standard is obtained according to the evaluation accuracy under each video frame segmentation standard.

[0051] Optionally, the using the block-screened frame set to test the quality assessment model to be tested to obtain a block-test quality score includes:

[0052] Inputting the video frames in the block-filtered frame set into the quality assessment model to be tested in sequence, and extracting features of the video frames in the block-filtered frame set using a temporal feature extractor in the quality assessment model to be tested to obtain a block feature map;

[0053] Performing global average pooling and global standard deviation pooling on the block feature map to obtain a block multidimensional feature vector;

[0054] Performing a dimensionality reduction on the block multidimensional feature vector using the first dimensionality reduction formula to obtain a block first dimensionality reduction feature vector;

[0055] Inputting the first dimension-reduced feature vector of the block into the gated recurrent unit to obtain a gated recurrent feature vector of the block;

[0056] Performing secondary dimensionality reduction on the block gated cyclic feature vector using the second dimensionality reduction formula to obtain a block second dimensionality reduction feature vector;

[0057] Pooling the second dimensionality reduction feature vectors of the blocks to obtain quality scores of the video frames in the block-filtered frame set;

[0058] The quality scores of all video frames in the block-filtered frame set are integrated to obtain the block-test quality score.

[0059] In order to solve the above problems, the present invention also provides a diagnostic device for a temporal feature extractor in a video quality assessment model, the device comprising:

[0060] An ordered video frame set and an unordered video frame set acquisition module is used to acquire a video quality data frame set, sort the video frames in the video quality data frame set in chronological order to obtain an ordered video frame set, and disrupt the order of the video frames in the ordered video frame set to obtain an unordered video frame set;

[0061] An ordered training evaluation model and a disordered training evaluation model training module are configured to sequentially input the video frames in the ordered video frame set into a pre-built video quality evaluation model for training to obtain an ordered training evaluation model, wherein the video quality evaluation model includes a temporal feature extractor; and sequentially input the video frames in the disordered video frame set into the video quality evaluation model for training to obtain a disordered training evaluation model;

[0062] An ordered training evaluation model testing module is used to test the ordered training evaluation model using the ordered video frame set and the unordered video frame set respectively to obtain an ordered training ordered test quality score and an ordered training unordered test quality score;

[0063] A disordered training evaluation model testing module is used to test the disordered training evaluation model using the ordered video frame set and the disordered video frame set respectively to obtain a disordered training ordered test quality score and a disordered training disordered test quality score;

[0064] The video quality evaluation model diagnosis module is used to calculate the evaluation accuracy of the video quality evaluation model based on the ordered training ordered test quality score, the ordered training disordered test quality score, the disordered training ordered test quality score and the disordered training disordered test quality score; and obtain the diagnosis result of the temporal feature extractor in the video quality evaluation model based on the evaluation accuracy.

[0065] In order to solve the above problem, the present invention further provides an electronic device, comprising:

[0066] at least one processor; and,

[0067] a memory communicatively connected to the at least one processor; wherein,

[0068] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to implement the diagnosis method of the temporal feature extractor in the video quality assessment model described above.

[0069] In order to solve the above problems, the present invention also provides a computer-readable storage medium, which stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the diagnostic method of the temporal feature extractor in the video quality assessment model described above.

[0070] Compared with the background technology, the current video quality evaluation model lacks the inspection of the temporal feature extraction module and consumes a large amount of computing resources. The embodiment of the present invention obtains an ordered video frame set and an unordered video frame set, and uses the ordered video frame set and the unordered video frame set to train the video quality evaluation model to obtain the ordered training evaluation model and the unordered training evaluation model, and then uses the ordered video frame set and the unordered video frame set to test the ordered training evaluation model to obtain the ordered training ordered test quality score and the ordered training unordered test quality score, and uses the ordered video frame set and the unordered video frame set to test the unordered training evaluation model to obtain the unordered training ordered test quality score and the unordered training unordered test quality score. Finally, the evaluation accuracy of the video quality evaluation model can be calculated according to the ordered training ordered test quality score, the ordered training unordered test quality score, the unordered training ordered test quality score and the unordered training unordered test quality score. The diagnosis result of the temporal feature extractor in the video quality evaluation model can be obtained through the evaluation accuracy of the video quality evaluation model. Therefore, the diagnostic method, device, electronic device and computer-readable storage medium of the temporal feature extractor in the video quality evaluation model proposed in the present invention can solve the problems of the current video quality evaluation model lacking inspection of the temporal feature extraction module and large consumption of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A schematic flow chart of a diagnostic method for a temporal feature extractor in a video quality assessment model provided by one embodiment of the present invention;

[0072] Figure 2 for Figure 1 A detailed implementation flow chart of one of the steps in the

[0073] Figure 3 for Figure 1 A detailed implementation flow diagram of another step in FIG;

[0074] Figure 4 A functional module diagram of a diagnostic device for a temporal feature extractor in a video quality assessment model provided by one embodiment of the present invention;

[0075] Figure 5 A schematic structural diagram of an electronic device for implementing a diagnostic method for a temporal feature extractor in a video quality assessment model provided by an embodiment of the present invention.

[0076] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0077] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0078] An embodiment of the present application provides a diagnostic method for a temporal feature extractor in a video quality assessment model. The execution subject of the diagnostic method for a temporal feature extractor in a video quality assessment model includes but is not limited to at least one of electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the diagnostic method for a temporal feature extractor in a video quality assessment model can be executed by software or hardware installed in a terminal device or a server device. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0079] Example 1:

[0080] Reference Figure 1 FIG. 1 is a flow chart of a method for diagnosing a temporal feature extractor in a video quality assessment model according to an embodiment of the present invention. In this embodiment, the method for diagnosing a temporal feature extractor in a video quality assessment model includes:

[0081] S1. Obtain a video quality data frame set, sort the video frames in the video quality data frame set in chronological order to obtain an ordered video frame set, and disrupt the order of the video frames in the ordered video frame set to obtain an unordered video frame set.

[0082] In detail, the video quality data frame set refers to video resources in a video quality evaluation library, such as KoNViD-1k, CVD2014, LIVE-Qualcomm, and LIVE-VQC.

[0083] For details, see Figure 2 As shown, the order of the video frames in the ordered video frame set is disrupted to obtain an unordered video frame set, including:

[0084] S11. randomly extracting video frames from the ordered video frame set according to a pre-built shuffling algorithm;

[0085] S12. Sort the video frames according to the extraction order of the video frames to obtain the unordered video frame set.

[0086] S2. Input the video frames in the ordered video frame set into a pre-built video quality assessment model in sequence for training to obtain an ordered training assessment model, wherein the video quality assessment model includes a temporal feature extractor.

[0087] In an embodiment of the present invention, the video quality assessment model refers to a VSFA video quality assessment model, and the temporal feature extractor may be a ResNet50 spatial feature extractor and a 3D ResNet101 spatiotemporal feature extractor.

[0088] It should be understood that the training process of the video quality assessment model includes training and verification stages, which are collectively referred to as the training process.

[0089] Explainably, the input order of the ordered video frame set is as follows:

[0090] Input1=I1(t=1,2,3,…,T)

[0091] Wherein, I1 represents the input order of the ordered video frame set, t=1, 2, 3, ..., T represents a video frame sequence sorted in chronological order.

[0092] S3. Input the video frames in the disordered video frame set into the video quality evaluation model in sequence for training to obtain a disordered training evaluation model.

[0093] It can be explained that the order of the video frames in the unordered video frame set is obtained based on the shuffling algorithm with equal probability:

[0094] I t2 =Shuffle t (t=1, 2, ..., T)

[0095] Input2=I2(t=T a , T b , T c ,…,T t )

[0096] Among them, Shuffle t represents the shuffling algorithm, t=1, 2, ..., T represents the video frame sequence input into the shuffling algorithm in time order, I2(t=T a , T b , T c ,…,T t ) and Input2 represent the video frame sequence obtained according to the shuffling algorithm.

[0097] S4. Test the ordered training evaluation model using the ordered video frame set and the unordered video frame set respectively to obtain an ordered training ordered test quality score and an ordered training unordered test quality score.

[0098] In an embodiment of the present invention, the method of testing the ordered training evaluation model using the ordered video frame set and the unordered video frame set to obtain an ordered training ordered test quality score and an ordered training unordered test quality score includes:

[0099] Inputting the video frames in the ordered video frame set into the ordered training and evaluation model in sequence, and extracting features of the video frames in the ordered video frame set using a temporal feature extractor in the ordered training and evaluation model to obtain an ordered feature map;

[0100] Performing global average pooling and global standard deviation pooling on the ordered feature map to obtain an ordered multidimensional feature vector;

[0101] Performing a dimensionality reduction on the ordered multidimensional feature vector using a pre-constructed first dimensionality reduction formula to obtain an ordered first dimensionality reduction feature vector;

[0102] Inputting the ordered first dimension-reduced feature vector into a pre-built gated recurrent unit to obtain an ordered gated recurrent feature vector;

[0103] Performing secondary dimensionality reduction on the ordered gated cyclic feature vector using a pre-constructed second dimensionality reduction formula to obtain an ordered second dimensionality reduction feature vector;

[0104] Pooling the ordered second dimensionality reduction feature vectors to obtain quality scores of the video frames in the ordered video frame set;

[0105] Integrating the quality scores of all video frames in the ordered video frame set to obtain the ordered training and ordered testing quality scores;

[0106] Extracting features of video frames in the disordered video frame set using a temporal feature extractor in the ordered training evaluation model to obtain a disordered feature map;

[0107] Performing global average pooling and global standard deviation pooling on the disordered feature map to obtain a disordered multidimensional feature vector;

[0108] Performing a dimensionality reduction on the disordered multidimensional feature vector using the first dimensionality reduction formula to obtain an disordered first dimensionality reduction feature vector;

[0109] Inputting the unordered first dimension-reduced feature vector into the gated recurrent unit to obtain an unordered gated recurrent feature vector;

[0110] Performing secondary dimensionality reduction on the disordered gated cyclic feature vector using the second dimensionality reduction formula to obtain an disordered second dimensionality reduction feature vector;

[0111] Pooling the unordered second dimensionality reduction feature vector to obtain quality scores of the video frames in the unordered video frame set;

[0112] The quality scores of all video frames in the unordered video frame set are integrated to obtain the ordered training and unordered testing quality scores.

[0113] Optionally, each video frame is input into the ResNet50 spatial feature extractor to obtain the ordered feature map M t , perform global average pooling and global standard deviation pooling on the ordered feature map to extract 4096-dimensional feature F t :

[0114] M t =ResNet50(I t ),(t=1,2,…,T)

[0115] F t =GP Meanstd (M t )

[0116] Among them, ResNet50 represents the ResNet50 spatial feature extractor, I t represents the video frame sequence input to the spatial feature extractor, GP Meanstd represents global average pooling and global standard deviation pooling, F t Represents a multidimensional feature vector.

[0117] In the embodiment of the present invention, the first dimensionality reduction formula is as follows:

[0118] X t =W FX F t +b x

[0119] Among them, X t Represents the eigenvector after the first dimensionality reduction formula, W FX represents the fully connected layer parameters, F t represents the feature vector to be reduced in dimension after global average pooling and global standard deviation pooling, b x represents the bias of the fully connected layer.

[0120] It can be explained that the gated recurrent unit can avoid the problem of vanishing feature vector gradients as follows:

[0121] H t =GRU(X t ,H t-1 )

[0122] Among them, GRU represents the gated cycle unit calculation function, H t-1 Represents the hidden state passed down by the previous node, H t Represents the feature vector after being processed by the gated recurrent unit.

[0123] In this embodiment of the present invention, the second dimensionality reduction formula is as follows:

[0124] Qt =W HQ H t +b q

[0125] Among them, Q t Represents the eigenvector after the second dimensionality reduction formula, W HQ represents the fully connected layer parameters, H t represents the feature vector after being processed by the gated recurrent unit, b q represents the bias of the fully connected layer.

[0126] It is understandable that the feature vector after secondary dimensionality reduction is pooled as follows:

[0127] Q Video =Pooling(Q1,Q2,…,Q w )

[0128] Among them, Pooling represents the pooling algorithm, Q1, Q2, ..., Q N Indicates the quality score of the video frame.

[0129] S5. Test the disordered training evaluation model using the ordered video frame set and the disordered video frame set respectively to obtain a disordered training ordered test quality score and a disordered training disordered test quality score.

[0130] In an embodiment of the present invention, the process of testing the disordered training evaluation model using the ordered video frame set and the disordered video frame set is the same as the process of testing the ordered training evaluation model using the ordered video frame set and the disordered video frame set, and will not be repeated here.

[0131] S6. Calculate the evaluation accuracy of the video quality evaluation model according to the ordered training and ordered test quality scores, the ordered training and unordered test quality scores, the unordered training and ordered test quality scores, and the unordered training and unordered test quality scores.

[0132] It is understandable that the evaluation accuracy can be determined by calculating the correlation between the ordered training ordered test quality score, the ordered training disordered test quality score, the disordered training ordered test quality score and the disordered training disordered test quality score and the true quality score of the video quality data frame set.

[0133] For details, see Figure 3 As shown, the evaluation accuracy of the video quality evaluation model is calculated based on the ordered training ordered test quality score, the ordered training disordered test quality score, the disordered training ordered test quality score, and the disordered training disordered test quality score, including:

[0134] S61, obtaining a true quality score of the video quality data frame set;

[0135] S62. Calculate the rank correlation coefficients of the true quality score with the ordered training ordered test quality score, the ordered training unordered test quality score, the unordered training ordered test quality score, and the unordered training unordered test quality score, respectively, using a pre-constructed correlation coefficient formula;

[0136] S63: Determine the evaluation accuracy of the video quality evaluation model according to the rank correlation coefficient.

[0137] Alternatively, the correlation coefficient formula may be the Spearman Rank-order Correlation Coefficient (SRCC). In addition, the Pearson linear correlation coefficient, the Kendall rank correlation coefficient, and the root mean square error may also be used for judgment.

[0138] In the embodiment of the present invention, the correlation coefficient formula is as follows:

[0139]

[0140] Among them, ρ s represents the rank correlation coefficient, n represents the number of frames in the video quality data frame set, d i represents the rank difference of the i-th video frame.

[0141] In the embodiment of the present invention, the Spearman rank correlation coefficient is a prior art and will not be described in detail here.

[0142] S7. Obtain a diagnosis result of the temporal feature extractor in the video quality evaluation model based on the evaluation accuracy.

[0143] It can be understood that the role of the temporal feature extractor in the video quality evaluation model can be obtained based on the evaluation accuracy.

[0144] In the embodiment of the present invention, after obtaining the diagnosis result of the temporal feature extractor in the video quality assessment model based on the evaluation accuracy, the method further includes:

[0145] Dividing the video frames in the video quality data frame set into blocks according to different predetermined video frame block standards to obtain a block video frame set;

[0146] Randomly selecting a video frame from each block of the video frame set to obtain a block-filtered frame set;

[0147] Using the video quality evaluation model for training to obtain a quality evaluation model to be tested;

[0148] Using the block-screened frame set to test the quality evaluation model to be tested, to obtain a block-test quality score;

[0149] Calculating the evaluation accuracy of the video quality evaluation model under each video frame segmentation standard according to the segmentation test quality score;

[0150] According to the evaluation accuracy under each video frame segmentation standard, the optimal video segmentation standard is obtained. It can be explained that the video frame segmentation standard includes four types, as shown below:

[0151] Input k =I t (t=1, 2, ..., T)

[0152] Sample-k=Shuffle(Input k ), (k=0, 4, 8, 16)

[0153] Among them, I t (t=1,2,…,T) represents the video frame input sequence, Input k Indicates that every k video frames are regarded as a block, Shuffle represents a random algorithm that randomly selects a frame with equal probability, and Sample-k represents the video frame sequence obtained according to different video frame block standards.

[0154] In the embodiment of the present invention, the step of using the block-screened frame set to test the quality assessment model to be tested to obtain a block-test quality score includes:

[0155] Inputting the video frames in the block-filtered frame set into the quality assessment model to be tested in sequence, and extracting features of the video frames in the block-filtered frame set using a temporal feature extractor in the quality assessment model to be tested to obtain a block feature map;

[0156] Performing global average pooling and global standard deviation pooling on the block feature map to obtain a block multidimensional feature vector;

[0157] Performing a dimensionality reduction on the block multidimensional feature vector using the first dimensionality reduction formula to obtain a block first dimensionality reduction feature vector;

[0158] Inputting the first dimension-reduced feature vector of the block into the gated recurrent unit to obtain a gated recurrent feature vector of the block;

[0159] Performing secondary dimensionality reduction on the block gated cyclic feature vector using the second dimensionality reduction formula to obtain a block second dimensionality reduction feature vector;

[0160] Pooling the second dimensionality reduction feature vectors of the blocks to obtain quality scores of the video frames in the block-filtered frame set;

[0161] The quality scores of all video frames in the block-filtered frame set are integrated to obtain the block-test quality score.

[0162] It is understandable that the video frame set in the video quality data frame set can be divided into blocks of 1 frame, 4 frames, 8 frames and 16 frames, and then the block test quality scores obtained by each block method are tested, and then the block standard with the highest evaluation accuracy is used as the optimal video block standard.

[0163] Compared with the background technology, the current video quality evaluation model lacks the inspection of the temporal feature extraction module and consumes a large amount of computing resources. The embodiment of the present invention obtains an ordered video frame set and an unordered video frame set, and uses the ordered video frame set and the unordered video frame set to train the video quality evaluation model to obtain the ordered training evaluation model and the unordered training evaluation model, and then uses the ordered video frame set and the unordered video frame set to test the ordered training evaluation model to obtain the ordered training ordered test quality score and the ordered training unordered test quality score, and uses the ordered video frame set and the unordered video frame set to test the unordered training evaluation model to obtain the unordered training ordered test quality score and the unordered training unordered test quality score. Finally, the evaluation accuracy of the video quality evaluation model can be calculated according to the ordered training ordered test quality score, the ordered training unordered test quality score, the unordered training ordered test quality score and the unordered training unordered test quality score. The diagnosis result of the temporal feature extractor in the video quality evaluation model can be obtained through the evaluation accuracy of the video quality evaluation model. Therefore, the diagnostic method, device, electronic device and computer-readable storage medium of the temporal feature extractor in the video quality evaluation model proposed in the present invention can solve the problems of the current video quality evaluation model lacking inspection of the temporal feature extraction module and large consumption of computing resources.

[0164] Example 2:

[0165] like Figure 4 , which is a functional module diagram of a diagnostic device for a temporal feature extractor in a video quality assessment model provided by an embodiment of the present invention.

[0166] The diagnostic device 100 for the temporal feature extractor in the video quality assessment model of the present invention can be installed in an electronic device. According to the functions to be implemented, the diagnostic device 100 for the temporal feature extractor in the video quality assessment model can include an ordered video frame set and an unordered video frame set acquisition module 101, an ordered training evaluation model and an unordered training evaluation model training module 102, an ordered training evaluation model testing module 103, an unordered training evaluation model testing module 104, and a video quality assessment model diagnostic module 105. The module of the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can perform fixed functions, and is stored in the memory of the electronic device.

[0167] The ordered video frame set and disordered video frame set acquisition module 101 is used to acquire a video quality data frame set, sort the video frames in the video quality data frame set in chronological order to obtain an ordered video frame set, and disrupt the order of the video frames in the ordered video frame set to obtain an disordered video frame set;

[0168] The ordered training evaluation model and disordered training evaluation model training module 102 is used to sequentially input the video frames in the ordered video frame set into a pre-built video quality evaluation model for training to obtain an ordered training evaluation model, wherein the video quality evaluation model includes a temporal feature extractor; and sequentially input the video frames in the disordered video frame set into the video quality evaluation model for training to obtain a disordered training evaluation model;

[0169] The ordered training evaluation model testing module 103 is used to test the ordered training evaluation model using the ordered video frame set and the unordered video frame set respectively to obtain an ordered training ordered test quality score and an ordered training unordered test quality score;

[0170] The disordered training evaluation model testing module 104 is used to test the disordered training evaluation model using the ordered video frame set and the disordered video frame set respectively to obtain a disordered training ordered test quality score and a disordered training disordered test quality score;

[0171] The video quality assessment model diagnosis module 105 is used to calculate the evaluation accuracy of the video quality assessment model based on the ordered training ordered test quality score, the ordered training disordered test quality score, the disordered training ordered test quality score and the disordered training disordered test quality score; and obtain the diagnosis result of the temporal feature extractor in the video quality assessment model based on the evaluation accuracy.

[0172] In detail, the modules in the diagnostic device 100 of the temporal feature extractor in the video quality assessment model in the embodiment of the present invention are used in the same manner as above. Figure 1The diagnostic method of the temporal feature extractor in the video quality evaluation model described in is the same technical means and can produce the same technical effect, so it will not be repeated here.

[0173] Example 3:

[0174] like Figure 5 , which is a structural diagram of an electronic device for implementing a diagnostic method for a temporal feature extractor in a video quality assessment model provided by an embodiment of the present invention.

[0175] The electronic device 1 may include a processor 10, a memory 11, a bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a diagnostic program for a temporal feature extractor in a video quality assessment model.

[0176] The memory 11 includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 1. Furthermore, the memory 11 may include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can be used not only to store application software and various types of data installed in the electronic device 1, such as the code of the diagnostic program of the temporal feature extractor in the video quality assessment model, but also to temporarily store data that has been output or is to be output.

[0177] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing the programs or modules stored in the memory 11 (such as the diagnostic program of the temporal feature extractor in the video quality assessment model) and calling the data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0178] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable connection and communication between the memory 11 and at least one processor 10, etc.

[0179] Figure 5 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 5 The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0180] For example, although not shown, the electronic device 1 may further include a power source (such as a battery) for powering the various components. Preferably, the power source may be logically connected to the at least one processor 10 via a power management device, thereby implementing functions such as charging management, discharging management, and power consumption management through the power management device. The power source may further include any components such as one or more DC or AC power sources, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device 1 may further include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0181] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0182] Optionally, the electronic device 1 may further include a user interface, which may be a display or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0183] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0184] The diagnostic program of the temporal feature extractor in the video quality assessment model stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve the following:

[0185] Acquire a video quality data frame set, sort the video frames in the video quality data frame set in chronological order to obtain an ordered video frame set, and disrupt the order of the video frames in the ordered video frame set to obtain an unordered video frame set;

[0186] Inputting the video frames in the ordered video frame set into a pre-built video quality assessment model in sequence for training to obtain an ordered training assessment model, wherein the video quality assessment model includes a temporal feature extractor;

[0187] Inputting the video frames in the disordered video frame set into the video quality evaluation model in sequence for training to obtain a disordered training evaluation model;

[0188] The ordered training evaluation model is tested using the ordered video frame set and the unordered video frame set respectively to obtain an ordered training ordered test quality score and an ordered training unordered test quality score;

[0189] The disordered training evaluation model is tested using the ordered video frame set and the disordered video frame set respectively to obtain a disordered training ordered test quality score and a disordered training disordered test quality score;

[0190] Calculating the evaluation accuracy of the video quality evaluation model according to the ordered training ordered test quality score, the ordered training disordered test quality score, the disordered training ordered test quality score, and the disordered training disordered test quality score;

[0191] According to the evaluation accuracy, a diagnosis result of the temporal feature extractor in the video quality evaluation model is obtained.

[0192] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to Figures 1 to 4 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0193] Furthermore, if the modules / units integrated into the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0194] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor of an electronic device, the computer program can implement:

[0195] Acquire a video quality data frame set, sort the video frames in the video quality data frame set in chronological order to obtain an ordered video frame set, and disrupt the order of the video frames in the ordered video frame set to obtain an unordered video frame set;

[0196] Inputting the video frames in the ordered video frame set into a pre-built video quality assessment model in sequence for training to obtain an ordered training assessment model, wherein the video quality assessment model includes a temporal feature extractor;

[0197] Inputting the video frames in the disordered video frame set into the video quality evaluation model in sequence for training to obtain a disordered training evaluation model;

[0198] The ordered training evaluation model is tested using the ordered video frame set and the unordered video frame set respectively to obtain an ordered training ordered test quality score and an ordered training unordered test quality score;

[0199] The disordered training evaluation model is tested using the ordered video frame set and the disordered video frame set respectively to obtain a disordered training ordered test quality score and a disordered training disordered test quality score;

[0200] Calculating the evaluation accuracy of the video quality evaluation model according to the ordered training ordered test quality score, the ordered training disordered test quality score, the disordered training ordered test quality score, and the disordered training disordered test quality score;

[0201] According to the evaluation accuracy, a diagnosis result of the temporal feature extractor in the video quality evaluation model is obtained.

[0202] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and other division methods may be used in actual implementation.

[0203] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0204] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0205] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0206] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A diagnostic method for a temporal feature extractor in a video quality assessment model, characterized in that: The method comprises: Acquire a video quality data frame set, sort the video frames in the video quality data frame set in chronological order to obtain an ordered video frame set, and disrupt the order of the video frames in the ordered video frame set to obtain an unordered video frame set; Inputting the video frames in the ordered video frame set into a pre-built video quality assessment model in sequence for training to obtain an ordered training assessment model, wherein the video quality assessment model includes a temporal feature extractor; Inputting the video frames in the disordered video frame set into the video quality evaluation model in sequence for training to obtain a disordered training evaluation model; The ordered training evaluation model is tested using the ordered video frame set and the unordered video frame set respectively to obtain an ordered training ordered test quality score and an ordered training unordered test quality score; The disordered training evaluation model is tested using the ordered video frame set and the disordered video frame set respectively to obtain a disordered training ordered test quality score and a disordered training disordered test quality score; Calculating the evaluation accuracy of the video quality evaluation model according to the ordered training ordered test quality score, the ordered training disordered test quality score, the disordered training ordered test quality score, and the disordered training disordered test quality score; According to the evaluation accuracy, a diagnosis result of the temporal feature extractor in the video quality evaluation model is obtained.

2. The diagnostic method of the temporal feature extractor in the video quality assessment model according to claim 1, wherein: The step of disrupting the order of the video frames in the ordered video frame set to obtain the unordered video frame set includes: randomly extracting video frames from the ordered video frame set according to a pre-built shuffling algorithm; The video frames are sorted according to the extraction order of the video frames to obtain the unordered video frame set.

3. The diagnostic method of the temporal feature extractor in the video quality assessment model according to claim 2, wherein: The method of testing the ordered training evaluation model using the ordered video frame set and the unordered video frame set to obtain an ordered training ordered test quality score and an ordered training unordered test quality score comprises: Inputting the video frames in the ordered video frame set into the ordered training and evaluation model in sequence, and extracting features of the video frames in the ordered video frame set using a temporal feature extractor in the ordered training and evaluation model to obtain an ordered feature map; Performing global average pooling and global standard deviation pooling on the ordered feature map to obtain an ordered multidimensional feature vector; Performing a dimensionality reduction on the ordered multidimensional feature vector using a pre-constructed first dimensionality reduction formula to obtain an ordered first dimensionality reduction feature vector; Inputting the ordered first dimension-reduced feature vector into a pre-built gated recurrent unit to obtain an ordered gated recurrent feature vector; Performing secondary dimensionality reduction on the ordered gated cyclic feature vector using a pre-constructed second dimensionality reduction formula to obtain an ordered second dimensionality reduction feature vector; Pooling the ordered second dimensionality reduction feature vectors to obtain quality scores of the video frames in the ordered video frame set; Integrating the quality scores of all video frames in the ordered video frame set to obtain the ordered training and ordered testing quality scores; Extracting features of video frames in the disordered video frame set using a temporal feature extractor in the ordered training evaluation model to obtain a disordered feature map; Performing global average pooling and global standard deviation pooling on the disordered feature map to obtain a disordered multidimensional feature vector; Performing a dimensionality reduction on the disordered multidimensional feature vector using the first dimensionality reduction formula to obtain an disordered first dimensionality reduction feature vector; Inputting the unordered first dimension-reduced feature vector into the gated recurrent unit to obtain an unordered gated recurrent feature vector; Performing secondary dimensionality reduction on the disordered gated cyclic feature vector using the second dimensionality reduction formula to obtain an disordered second dimensionality reduction feature vector; Pooling the unordered second dimensionality reduction feature vector to obtain quality scores of the video frames in the unordered video frame set; The quality scores of all video frames in the unordered video frame set are integrated to obtain the ordered training and unordered testing quality scores.

4. The diagnostic method of the temporal feature extractor in the video quality assessment model according to claim 3, wherein: The first dimensionality reduction formula is as follows: X t =W FX F t +b x Among them, X t Represents the eigenvector after the first dimensionality reduction formula, W FX represents the fully connected layer parameters, F t represents the feature vector to be reduced in dimension after global average pooling and global standard deviation pooling, b x represents the bias of the fully connected layer.

5. The diagnostic method of the temporal feature extractor in the video quality assessment model according to claim 3, wherein: The second dimensionality reduction formula is as follows: Q t =W HQ H t +b q Among them, Q t Represents the eigenvector after the second dimensionality reduction formula, W HQ represents the fully connected layer parameters, H t represents the feature vector after being processed by the gated recurrent unit, b q represents the bias of the fully connected layer.

6. The diagnostic method for a temporal feature extractor in a video quality assessment model according to claim 3, wherein: Calculating the evaluation accuracy of the video quality evaluation model according to the ordered training ordered test quality score, the ordered training disordered test quality score, the disordered training ordered test quality score, and the disordered training disordered test quality score includes: Obtaining a true quality score of the video quality data frame set; Utilizing a pre-constructed correlation coefficient formula, respectively calculating the rank correlation coefficients between the true quality score and the ordered training ordered test quality score, the ordered training unordered test quality score, the unordered training ordered test quality score, and the unordered training unordered test quality score; The evaluation accuracy of the video quality evaluation model is judged according to the rank correlation coefficient.

7. The diagnostic method of the temporal feature extractor in the video quality assessment model according to claim 6, characterized in that: The correlation coefficient formula is as follows: Among them, ρ s represents the rank correlation coefficient, n represents the number of frames in the video quality data frame set, d i represents the rank difference of the i-th video frame.

8. The diagnostic method for a temporal feature extractor in a video quality assessment model according to claim 1, wherein: After obtaining the diagnosis result of the temporal feature extractor in the video quality evaluation model according to the evaluation accuracy, the method further includes: Dividing the video frames in the video quality data frame set into blocks according to different predetermined video frame block standards to obtain a block video frame set; Randomly selecting a video frame from each block of the video frame set to obtain a block-filtered frame set; Using the video quality evaluation model for training to obtain a quality evaluation model to be tested; Using the block-screened frame set to test the quality evaluation model to be tested, to obtain a block-test quality score; Calculating the evaluation accuracy of the video quality evaluation model under each video frame segmentation standard according to the segmentation test quality score; An optimal video segmentation standard is obtained according to the evaluation accuracy under each video frame segmentation standard.

9. The diagnostic method for a temporal feature extractor in a video quality assessment model according to claim 8, wherein: The step of using the block-screened frame set to test the quality evaluation model to be tested to obtain a block-test quality score includes: Inputting the video frames in the block-filtered frame set into the quality assessment model to be tested in sequence, and extracting features of the video frames in the block-filtered frame set using a temporal feature extractor in the quality assessment model to be tested to obtain a block feature map; Performing global average pooling and global standard deviation pooling on the block feature map to obtain a block multidimensional feature vector; Performing a dimensionality reduction on the block multidimensional feature vector using a first dimensionality reduction formula to obtain a block first dimensionality reduction feature vector; Inputting the first dimension-reduced feature vector of the block into a gated recurrent unit to obtain a gated recurrent feature vector of the block; Performing secondary dimensionality reduction on the block gated cyclic feature vector using a second dimensionality reduction formula to obtain a block second dimensionality reduction feature vector; Pooling the second dimensionality reduction feature vectors of the blocks to obtain quality scores of the video frames in the block-filtered frame set; The quality scores of all video frames in the block-filtered frame set are integrated to obtain the block-test quality score.

10. A diagnostic device for a temporal feature extractor in a video quality assessment model, characterized in that: The device comprises: An ordered video frame set and an unordered video frame set acquisition module is used to acquire a video quality data frame set, sort the video frames in the video quality data frame set in chronological order to obtain an ordered video frame set, and disrupt the order of the video frames in the ordered video frame set to obtain an unordered video frame set; An ordered training evaluation model and a disordered training evaluation model training module are configured to sequentially input the video frames in the ordered video frame set into a pre-built video quality evaluation model for training to obtain an ordered training evaluation model, wherein the video quality evaluation model includes a temporal feature extractor; and sequentially input the video frames in the disordered video frame set into the video quality evaluation model for training to obtain a disordered training evaluation model; An ordered training evaluation model testing module is used to test the ordered training evaluation model using the ordered video frame set and the unordered video frame set respectively to obtain an ordered training ordered test quality score and an ordered training unordered test quality score; A disordered training evaluation model testing module is used to test the disordered training evaluation model using the ordered video frame set and the disordered video frame set respectively to obtain a disordered training ordered test quality score and a disordered training disordered test quality score; The video quality evaluation model diagnosis module is used to calculate the evaluation accuracy of the video quality evaluation model based on the ordered training ordered test quality score, the ordered training disordered test quality score, the disordered training ordered test quality score and the disordered training disordered test quality score; and obtain the diagnosis result of the temporal feature extractor in the video quality evaluation model based on the evaluation accuracy.

Images