A method and related device for deep quality evaluation of video spatio-temporal resolution guidance

Abstract

The application discloses a kind of video space-time resolution guide's deep quality evaluation method and related equipment, the method includes: reference video and distorted video are respectively input backbone network, the global mean and variance in feature map are calculated;According to the global mean and variance of reference video feature map, feature expression is obtained by the quality score of each video frame;The original video frame quality score is obtained by the quality score;The code stream information of distorted video is input to the first and second fully connected layers to obtain the first and second weights;Dimensionality reduction processing is performed on the feature expression according to the first weight;After the dimensionality reduction processing, the feature expression and the second weight set are calculated by GRU to obtain the quality score;The first and second quality scores are obtained according to the quality score and the weight of the quality score, and the final quality score of the distorted video is obtained.The video quality algorithm constructed in the application achieves the best performance on the existing ultra-high-definition video quality evaluation database, and provides effective decision for the optimization of down-sampling video coding technology.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, system, terminal, and computer-readable storage medium for depth quality evaluation guided by video spatiotemporal resolution. Background Technology

[0002] With the widespread adoption of video capture devices among consumers, the amount of video content on the internet has become incredibly rich, and ultra-high-definition video presents a significant challenge to video encoding technology. A common solution is to downsample and encode ultra-high-definition video. However, as the complexity of video scene content and motion increases, resolution and frame rate downsampling encoding techniques may not necessarily save bitrate during decoding; instead, they may worsen the quality of the decoded video. To adaptively determine whether resolution and frame rate downsampling encoding techniques are necessary for the current video, it is crucial to design compressed video quality evaluation algorithms to provide decision-making support for temporal variable resolution coding techniques, thereby achieving optimal encoding performance.

[0003] However, existing video quality assessment algorithms suffer from poor prediction accuracy for ultra-high-definition (high resolution, high bitrate) videos, high algorithm complexity, and long running time. Furthermore, there is currently no video quality assessment model based on deep learning.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a method, system, terminal, and computer-readable storage medium for video spatiotemporal resolution-guided depth quality evaluation, aiming to solve the problems of poor prediction accuracy, high algorithm complexity, and long running time of existing algorithms for ultra-high-definition video.

[0006] To achieve the above objectives, the present invention provides a video spatiotemporal resolution-guided depth quality evaluation method, which includes the following steps:

[0007] The reference video and the distorted video are respectively input into the backbone network. The first feature map of the video frame of the reference video in the backbone network is calculated, and the second feature map of the video frame of the distorted video in the backbone network is calculated. The global mean and variance of the first feature map and the second feature map are calculated respectively.

[0008] The first aggregated feature is obtained by concatenating the global mean and variance of the first feature map, and the second aggregated feature is obtained by concatenating the global mean and variance of the second feature map. The feature representation is then calculated based on the first aggregated feature and the second aggregated feature.

[0009] The bitstream information of the distorted video is input into the first fully connected layer to obtain the first weight. The feature representation is then dimensionality-reduced based on the first weight. The bitstream information of the distorted video is input into the second fully connected layer to obtain the second weight. The dimensionality-reduced feature representation and the second weight are then processed by GRU to obtain the quality score. A set of video frames preceding the current video frame is preset. The quality score of the video frame with the smallest quality score in the set is taken as the first quality score. The second quality score is then calculated based on the quality score and the weight of the quality score.

[0010] The original quality score is obtained based on the first quality score and the second quality score, and the average of the original quality scores of all video frames is calculated as the final quality score of the video.

[0011] Optionally, the video spatiotemporal resolution-guided depth quality assessment method, wherein calculating the first feature map of the video frame of the reference video in the backbone network and calculating the second feature map of the video frame of the distorted video in the backbone network specifically includes:

[0012] Calculate the j-th first feature map of the video frame of the reference video in the i-th stage of the backbone network;

[0013] Calculate the j-th second feature map of the video frame of the distorted video in the i-th stage of the backbone network;

[0014] Where i = {1, 2, 3, 4}, j = {256, 512, 1024, 2048}.

[0015] Optionally, the video spatiotemporal resolution-guided depth quality evaluation method, wherein calculating the feature representation based on the first aggregated feature and the second aggregated feature specifically includes:

[0016] Subtracting the first aggregated feature from the second aggregated feature yields the residual feature, and then aggregating the second aggregated feature with the residual feature yields the feature representation of the distorted video and the corresponding residual.

[0017] The residual feature represents the Euclidean distance of the object's feature domain.

[0018] Optionally, in the video spatiotemporal resolution-guided depth quality assessment method, the feature representation calculated based on the first aggregated feature and the second aggregated feature is expressed as:

[0019] in, Indicates feature expression, and Let represent the first aggregated feature and the second aggregated feature respectively, t represent a certain video frame, and i and j represent the j-th feature map of the i-th stage.

[0020] Optionally, the video spatiotemporal resolution-guided depth quality evaluation method, wherein inputting the bitstream information of the distorted video into a first fully connected layer to obtain a first weight, and performing dimensionality reduction processing on the feature representation based on the first weight, specifically includes:

[0021] A prior information model based on downsampling is constructed. The resolution information in the bitstream information of the distorted video is passed through the first fully connected layer of the downsampling prior information model to obtain the first weight. The first weight is fused with the feature expression to optimize the feature expression and obtain the dimensionality-reduced feature expression.

[0022] The feature representation is reduced in dimensionality as follows:

[0023] in, This represents the two fully connected layers that generate the first weight, and Resolution represents the resolution information in the bitstream of the distorted video. Indicates the first weight. This represents the feature representation after the dimensionality reduction process.

[0024] Optionally, the video spatiotemporal resolution-guided depth quality evaluation method, wherein inputting the bitstream information of the distorted video into a second fully connected layer to obtain the second weights, specifically includes:

[0025] A global video temporal information model is constructed using GRU. The frame rate information in the bitstream information of the distorted video is input into the second fully connected layer of the global video temporal information model to obtain the second weight.

[0026] The process of obtaining the second weight is expressed as follows: in, This represents the two fully connected layers that generate the second weight, and Framate represents the frame rate information in the bitstream information of the distorted video. This represents the second weight.

[0027] Optionally, in the video spatiotemporal resolution-guided depth quality evaluation method, the process of obtaining a quality score by performing GRU calculation on the dimensionality-reduced feature representation and the second weight set is expressed as:

[0028] Where, q (t) This represents the quality score of the t-th video frame after GRU calculation.

[0029] Optionally, the video spatiotemporal resolution-guided depth quality evaluation method, wherein the step of pre-setting a set of video frames preceding the current video frame, and using the quality score of the video frame with the smallest quality score in the set as the first quality score, specifically includes:

[0030] A set of video frames preceding the current video frame is preset, and the quality score of the video frame with the smallest quality score among the preceding τ frames in the set is taken as the first quality score.

[0031] The process of obtaining the first mass fraction is represented as follows: Among them, v prev This represents the set of video frames preceding the current video frame, τ = 12, l (t) This represents the first mass fraction.

[0032] Optionally, the video spatiotemporal resolution-guided depth quality evaluation method, wherein obtaining the second quality score based on the quality score and its weight specifically includes:

[0033] Based on the characteristics of short-term memory in the human visual perception system, and combining the quality score and its weight, the rate of increase in video score is limited to obtain a second quality score.

[0034] The process of obtaining the second mass fraction is represented as follows: Where τ=12, This represents the collection of video frames following the current video frame, where T represents the number of video frames.

[0035] The weight q represents the quality score. j q k Indicates in The index of the video frame in the collection, m (t) This represents the second mass fraction.

[0036] Optionally, the video spatiotemporal resolution-guided depth quality evaluation method, wherein the process of obtaining an original quality score based on the first quality score and the second quality score, and calculating the average of the original quality scores as the final quality score of the video, is expressed as follows:

[0037] o (t) =d (t) +(1-α)m (t) , Where α = 0.5, Q is the final quality score of the distorted video, and o (t) Let be the original quality score of the t-th video frame.

[0038] Furthermore, to achieve the above objectives, the present invention also provides a video spatiotemporal resolution-guided depth quality evaluation system, wherein the video spatiotemporal resolution-guided depth quality evaluation system comprises:

[0039] The feature extraction module is used to input the reference video and the distorted video into the backbone network respectively, calculate the first feature map of the video frame of the reference video in the backbone network, calculate the second feature map of the video frame of the distorted video in the backbone network, and calculate the global mean and variance of the first feature map and the second feature map respectively.

[0040] The first aggregated feature is obtained by concatenating the global mean and variance of the first feature map, and the second aggregated feature is obtained by concatenating the global mean and variance of the second feature map. The feature representation is then calculated based on the first aggregated feature and the second aggregated feature.

[0041] The feature fusion module inputs the bitstream information of the distorted video into a first fully connected layer to obtain a first weight, performs dimensionality reduction processing on the feature representation based on the first weight, inputs the bitstream information of the distorted video into a second fully connected layer to obtain a second weight, and calculates a quality score by performing GRU calculation on the dimensionality-reduced feature representation and the second weight. A set of video frames before the current video frame is preset, and the quality score of the video frame with the smallest quality score in the set is taken as the first quality score. A second quality score is calculated based on the quality score and the weight of the quality score.

[0042] The video quality calculation module obtains the original quality score based on the first quality score and the second quality score, and calculates the average of the original quality scores of all video frames as the final quality score of the video.

[0043] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a video spatiotemporal resolution-guided depth quality evaluation program stored in the memory and executable on the processor, wherein when the video spatiotemporal resolution-guided depth quality evaluation program is executed by the processor, it implements the steps of the video spatiotemporal resolution-guided depth quality evaluation method as described above.

[0044] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a video spatiotemporal resolution-guided depth quality evaluation program, which, when executed by a processor, implements the steps of the video spatiotemporal resolution-guided depth quality evaluation method as described above.

[0045] In this invention, a reference video and a distorted video are input into a backbone network, and the global mean and variance of the feature maps are calculated. Based on the global mean and variance of the reference video feature map, a feature representation is obtained. The quality scores of each video frame are then used to obtain the original video frame quality scores. The bitstream information of the distorted video is input into the first and second fully connected layers to obtain the first and second weights. The feature representation is then dimensionality-reduced based on the first weight. The dimensionality-reduced feature representation and the second weight set are processed by a GRU to obtain the quality score. The first and second quality scores are obtained based on the quality scores and their weights, resulting in the final quality score of the distorted video. The video quality algorithm constructed in this invention achieves optimal performance on existing ultra-high-definition video quality evaluation databases, providing effective decision-making for optimizing downsampling video coding techniques. Attached Figure Description

[0046] Figure 1 This is a flowchart of a preferred embodiment of the video spatiotemporal resolution-guided depth quality evaluation method of the present invention;

[0047] Figure 2 This is a model framework diagram of the video spatiotemporal resolution-guided depth quality assessment method of the present invention;

[0048] Figure 3 This is a flowchart of steps S100 and S200 in the video spatiotemporal resolution-guided depth quality evaluation method of the present invention;

[0049] Figure 4 This is a flowchart of step S300 in the video spatiotemporal resolution-guided depth quality evaluation method of the present invention;

[0050] Figure 5 This is a schematic diagram of a preferred embodiment of the video spatiotemporal resolution-guided depth quality evaluation system of the present invention;

[0051] Figure 6 This is a schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0053] The preferred embodiment of the video spatiotemporal resolution-guided depth quality evaluation method of the present invention, such as... Figure 1 and Figure 2 As shown, the video spatiotemporal resolution-guided depth quality evaluation method includes the following steps:

[0054] Step S100: Input the reference video and the distorted video into the backbone network respectively, calculate the first feature map of the video frame of the reference video in the backbone network, calculate the second feature map of the video frame of the distorted video in the backbone network, and calculate the global mean and variance of the first feature map and the second feature map respectively.

[0055] like Figure 3 As shown, step S100 includes:

[0056] S110. Input the reference video and the distorted video into the backbone network respectively, and calculate the first feature map of the video frame of the reference video in the backbone network.

[0057] A feature map is the result of an input image being convolved by a neural network. It represents a feature in the neural space; its resolution depends on the stride of the previous convolution kernel.

[0058] Specifically, the reference video and the distorted video are input into the backbone network respectively to extract content-aware and distortion-sensitive depth features. The content-aware depth features are obtained from the parameter video, and the distortion-sensitive depth features are obtained from the distorted video. In this invention, the content-aware depth features are represented as a first feature map, and the distortion-sensitive depth features are represented as a second feature map.

[0059] The distorted video is generated by adding distortions such as compression, transmission, and reconstruction to the reference video. The reference video and the distorted video need to have the same resolution and frame rate. The reference video is required to have no noticeable distortion perceptible to the human eye, and its high quality is guaranteed by being captured by a high-performance camera.

[0060] It should be noted that in this embodiment, the backbone network is preferably a ResNet50 backbone network, but it is not limited to the ResNet50 backbone network (the ResNet50 network contains 49 convolutional layers and one fully connected layer. The first part does not contain residual blocks and mainly performs convolution, regularization, activation function, and max pooling calculations on the input. The second, third, fourth, and fifth parts of the structure all contain residual blocks. In the ResNet50 network structure, each residual block has three convolutional layers. The network has a total of 49 convolutional layers, plus the final fully connected layer, for a total of 50 layers, which is the origin of the name ResNet50. The network input is 224×224×3. After the convolution calculations of the first five parts, the output is 7×7×2048. The pooling layer will transform it into a feature vector, and finally the classifier will calculate this feature vector and output the class probability). The feature extraction proposed in this invention can also be replaced by other deep backbone networks.

[0061] Calculate the first feature map of the video frames of the reference video (videos are composed of still images, which are called frames) in the ResNet50 backbone network, and calculate the second feature map of the video frames of the distorted video in the ResNet50 backbone network.

[0062] Calculate the j-th first feature map of the video frame of the reference video in the i-th stage of the ResNet50 backbone network; calculate the j-th second feature map of the video frame of the distorted video in the i-th stage of the ResNet50 backbone network; where i = {1, 2, 3, 4}, j = {256, 512, 1024, 2048}.

[0063] Specifically, the feature maps of the j-th reference image and the distorted image obtained by ResNet50 in the i-th stage can be represented as follows: and in, This represents the j-th first feature map of the t-th video frame at the i-th stage of the reference video. This represents the j-th second feature map of the t-th video frame at the i-th stage of the distorted video.

[0064] S120. Calculate the global mean and variance of the first feature map and the second feature map respectively.

[0065] Furthermore, this invention, taking into account the multi-scale characteristics of the human visual perception system, calculates the global mean and variance statistical characteristics of the first and second feature maps at each stage, respectively. This process is expressed as follows:

[0066]

[0067] Where GAP(.) and GSP(.) represent the global mean and variance operations, respectively. Let $\mathbf{i}$ represent the global mean of the $j$-th first feature map in the $i$-th stage of the $t$-th video frame of the reference video. Let $\mathbf{j}$ represent the global mean of the $j$-th second feature map in the $i$-th stage of the $t$-th video frame of the distorted video. This represents the variance of the j-th first feature map in the i-th stage of the t-th video frame of the reference video. Let represent the variance of the j-th second feature map in the i-th stage of the t-th video frame of the distorted video.

[0068] It should be noted that this invention can also extract other higher-order moment features besides the mean and variance.

[0069] S130. The global mean and variance of the first feature map are concatenated to obtain the first aggregated feature, and the global mean and variance of the second feature map are concatenated to obtain the second aggregated feature.

[0070] Specifically, the calculated global mean and variance are concatenated to obtain the first aggregated feature (see the aggregated feature in the reference video). The second aggregation feature (aggregation features of distorted videos) This process can be represented as:

[0071]

[0072] Here, [.] represents the feature concatenation operation.

[0073] S140. The feature expression is calculated based on the first aggregation feature and the second aggregation feature.

[0074] Specifically, the first aggregated feature is subtracted from the second aggregated feature to obtain the residual feature, and the second aggregated feature is aggregated with the residual feature to obtain the feature representation of the distorted video and the corresponding residual.

[0075] The feature representation of the distorted video and the corresponding residual obtained by aggregating the second aggregated feature and the residual feature is expressed as follows:

[0076] in, Indicates feature expression, and Let represent the aggregated features of the reference video and the distorted video, respectively. t represents a specific video frame, and i and j represent the j-th feature map of the i-th stage.

[0077] The residual feature represents the Euclidean distance of the object's feature domain.

[0078] Step S300: Input the bitstream information of the distorted video into a first fully connected layer to obtain a first weight. Perform dimensionality reduction processing on the feature representation based on the first weight. Input the bitstream information of the distorted video into a second fully connected layer to obtain a second weight. Calculate the quality score by performing GRU calculation on the dimensionality-reduced feature representation and the second weight set. Preset a set of video frames before the current video frame. Take the quality score of the video frame with the smallest quality score in the set as the first quality score. Calculate the second quality score based on the quality score and its weight.

[0079] like Figure 4 As shown, step S300 includes:

[0080] S310. Input the bitstream information of the distorted video into the first fully connected layer to obtain the first weight, and perform dimensionality reduction processing on the feature representation according to the first weight.

[0081] In obtaining the feature representation of the distorted video and the corresponding residual Subsequently, the present invention further fuses the aforementioned feature expressions.

[0082] Since the weight information is determined by the distorted video bitstream information when fusing the feature representations, and the resolution and frame rate information in the bitstream information largely determine the video quality, this invention utilizes bitstream priors to improve the performance of quality prediction.

[0083] First, Dimensionality reduction is achieved using fully connected layers, whose weights are determined by the resolution of the distorted video.

[0084] A fully connected layer is a layer where every node is connected to all nodes in the previous layer, combining the extracted features. Due to its fully connected nature, fully connected layers typically have the most parameters. Fully connected layers act as a "classifier" in the entire convolutional neural network. If convolutional layers, pooling layers, and activation functions map the raw data to the hidden feature space, then fully connected layers map the learned "distributed feature representation" to the sample label space. In practice, fully connected layers can be implemented using convolutional operations.

[0085] The resolution information in the bitstream information of the distorted video is used to predict the weight information of spatial distortion in the distorted video.

[0086] Specifically, a prior information model based on downsampling is constructed. The resolution information in the bitstream information of the distorted video is passed through the first fully connected layer (two fully connected layers) of the downsampling prior information model to obtain the first weight (weight information of spatial distortion of the distorted video). The first weight is then fused with the feature expression to optimize the feature expression and obtain the dimensionality-reduced feature expression.

[0087] The feature representation is reduced in dimensionality as follows: in, This represents the two fully connected layers that generate the first weight, and Resolution represents the resolution information in the bitstream of the distorted video. Indicates the first weight. This represents the feature representation after the dimensionality reduction process.

[0088] S320. Input the bitstream information of the distorted video into the second fully connected layer to obtain the second weight. Then, calculate the quality score by performing GRU calculation on the dimensionality-reduced feature representation and the second weight.

[0089] This invention uses GRU to construct global timing units between frames, which can better capture large-interval dependencies in timing data.

[0090] GRU is a commonly used gated recurrent neural network proposed to solve problems such as long-term memory and gradients in backpropagation. GRU has two gates: the reset gate and the update gate.

[0091] The frame rate information in the bitstream information of the distorted video is used as the weight information to predict the temporal distortion of the distorted video.

[0092] Specifically, a global video temporal information model is constructed using GRU, and the frame rate information in the bitstream information of the distorted video is input into the second fully connected layer of the global video temporal information model to obtain the second weight (weight information of temporal distortion of the distorted video).

[0093] The process of obtaining the second weight is as follows: in, This represents the two fully connected layers that generate the second weight, and Framerate represents the frame rate information in the bitstream information of the distorted video. This represents the second weight.

[0094] Furthermore, the quality score is obtained by calculating the dimensionality-reduced feature representation and the second weight using GRU.

[0095] The process of obtaining the quality score by applying the dimensionality-reduced feature representation and the second weight set to GRU is expressed as follows:

[0096] Where, q (t) This represents the quality score of the t-th video frame after GRU calculation.

[0097] It is evident that this invention is the first to use the resolution and frame rate information in the distorted video bitstream as prior information input to predict the weight information of spatial and temporal distortion in the distorted video, thereby improving the accuracy of weight prediction.

[0098] S330. Preset a set of video frames preceding the current video frame, and take the quality score of the video frame with the smallest quality score in the set as the first quality score.

[0099] This invention proposes a local temporal modeling unit that can adaptively weight based on the characteristics of human short-term memory to predict the final quality score of the distorted video.

[0100] The adaptive weighting strategy is based on the fact that when the human eye observes a poor-quality video frame, it will naturally give a lower score to video frames near that area. However, as the video plays, even if the video quality improves significantly, the score given by the human will not increase quickly.

[0101] Therefore, the present invention pre-defines a set of video frames preceding the current video frame, and takes the quality score of the video frame with the lowest quality score in the set as the first quality score.

[0102] Specifically, a set of video frames preceding the current video frame is preset, and the quality score of the video frame with the lowest quality score among the preceding τ frames in the set is taken as the first quality score; the process of obtaining the first quality score is expressed as follows: in, This represents the set of video frames preceding the current video frame, τ = 12, l (t) This represents the first mass fraction.

[0103] S340. Calculate the second quality score based on the quality score and the weight of the quality score.

[0104] Specifically, since subsequent videos are also affected by the poor-quality footage viewed earlier, this invention, based on the characteristics of short-term memory in the human visual perception system, combines the quality score and its weight to limit the rate at which the video score increases, thus obtaining a second quality score.

[0105] The process of obtaining the second mass fraction is represented as follows: Where τ=12, This represents the collection of video frames following the current video frame, where T represents the number of video frames. The weight q represents the quality score. j q k Indicates in The index of the video frame in the collection, m (t) This represents the second mass fraction.

[0106] Step S400: Obtain the original quality score based on the first quality score and the second quality score, and calculate the average of the original quality scores of all video frames as the final quality score of the video.

[0107] Specifically, the original quality score is obtained based on the first quality score and the second quality score. The original quality scores of each video frame are then linearly weighted, and the average of these weighted original quality scores is calculated as the final quality score of the video. The process of obtaining the final quality score for the video is expressed as follows:

[0108] o (t) =αl (t) +(1-α)m (t) , Where α = 0.5, Q is the final quality score of the distorted video, and o (t)Let be the original quality score of the t-th video frame.

[0109] The video spatiotemporal resolution-guided depth quality assessment technique proposed in this invention calculates the correlation between predicted scores and subjective scores on the ETRI-LIVE[1] database using Pearson linear correlation coefficient (PLCC) and Spearman rank correlation coefficient (SROCC). PLCC and SRCC are currently the most commonly used performance evaluation metrics for quality assessment models. The ETRI-LIVE database contains 15 4K ultra-high-definition reference videos, and 437 ultra-high-definition distortion videos are generated using downsampling-compression-upsampling. Each video has a corresponding subjective score.

[0110] algorithm SRCC PLCC PSNR 0.5092(0.1282) 0.4680(0.1466) SSIM[2] 0.5317(0.1391) 0.3106(0.2243) MS-SSIM[3] 0.5186(0.1299) 0.3773(0.1924) VIF[4] 0.5976(0.1350) 0.5399(0.1774) ST-RRED[5] 0.5181(0.1397) 0.2615(0.2008) SpEED[6] 0.4791(0.1229) 0.2079(0.1670) VMAF[7] 0.6552(0.1733) 0.6590(0.1770) VSTR[8] 0.7702(0.1137) 0.7767(0.1210) Ours 0.7760(0.1868) 0.7833(0.1855)

[0111] Table 1

[0112] This invention is the first depth full-reference quality assessment method that utilizes prior information on video spatiotemporal resolution. As shown in Table 1, the algorithm (Ours) proposed in this invention achieves optimal performance in the mixed distortion scenario of resolution and frame rate changes and compression.

[0113] In summary, the video quality algorithm constructed in this invention achieves optimal performance on existing ultra-high-definition video quality evaluation databases, providing effective decision-making for optimizing downsampling video coding techniques.

[0114] Furthermore, such as Figure 5 As shown, based on the above-mentioned video spatiotemporal resolution-guided depth quality evaluation method, the present invention also provides a video spatiotemporal resolution-guided depth quality evaluation system, wherein the video spatiotemporal resolution-guided depth quality evaluation system includes:

[0115] The feature extraction module 51 is used to input the reference video and the distorted video into the backbone network respectively, calculate the first feature map of the video frame of the reference video in the backbone network, calculate the second feature map of the video frame of the distorted video in the backbone network, and calculate the global mean and variance of the first feature map and the second feature map respectively.

[0116] The first aggregated feature is obtained by concatenating the global mean and variance of the first feature map, and the second aggregated feature is obtained by concatenating the global mean and variance of the second feature map. The feature representation is then calculated based on the first aggregated feature and the second aggregated feature.

[0117] The feature fusion module 52 inputs the bitstream information of the distorted video into a first fully connected layer to obtain a first weight, performs dimensionality reduction processing on the feature representation based on the first weight, inputs the bitstream information of the distorted video into a second fully connected layer to obtain a second weight, and calculates a quality score by performing GRU calculation on the dimensionality-reduced feature representation and the second weight. A set of video frames before the current video frame is preset, and the quality score of the video frame with the smallest quality score in the set is taken as the first quality score. A second quality score is calculated based on the quality score and the weight of the quality score.

[0118] The video quality calculation module 53 obtains the original quality score based on the first quality score and the second quality score, and calculates the average of the original quality scores of all video frames as the final quality score of the video.

[0119] Furthermore, such as Figure 6 As shown, based on the above-mentioned video spatiotemporal resolution-guided depth quality evaluation method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 6 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0120] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a video spatiotemporal resolution-guided depth quality evaluation program 40, which can be executed by the processor 10 to implement the video spatiotemporal resolution-guided depth quality evaluation method of this application.

[0121] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the video spatiotemporal resolution-guided depth quality evaluation method.

[0122] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0123] In one embodiment, when processor 10 executes the video spatiotemporal resolution-guided depth quality assessment program 40 in memory 20, the following steps are performed:

[0124] The reference video and the distorted video are respectively input into the backbone network. The first feature map of the video frame of the reference video in the backbone network is calculated, and the second feature map of the video frame of the distorted video in the backbone network is calculated. The global mean and variance of the first feature map and the second feature map are calculated respectively.

[0125] The first aggregated feature is obtained by concatenating the global mean and variance of the first feature map, and the second aggregated feature is obtained by concatenating the global mean and variance of the second feature map. The feature representation is then calculated based on the first aggregated feature and the second aggregated feature.

[0126] The bitstream information of the distorted video is input into the first fully connected layer to obtain the first weight. The feature representation is then dimensionality-reduced based on the first weight. The bitstream information of the distorted video is input into the second fully connected layer to obtain the second weight. The dimensionality-reduced feature representation and the second weight are then processed by GRU to obtain the quality score. A set of video frames preceding the current video frame is preset. The quality score of the video frame with the smallest quality score in the set is taken as the first quality score. The second quality score is then calculated based on the quality score and the weight of the quality score.

[0127] The original quality score is obtained based on the first quality score and the second quality score, and the average of the original quality scores of all video frames is calculated as the final quality score of the video.

[0128] Optionally, the video spatiotemporal resolution-guided depth quality assessment method, wherein calculating the first feature map of the video frame of the reference video in the backbone network and calculating the second feature map of the video frame of the distorted video in the backbone network specifically includes:

[0129] Calculate the j-th first feature map of the video frame of the reference video in the i-th stage of the backbone network;

[0130] Calculate the j-th second feature map of the video frame of the distorted video in the i-th stage of the backbone network;

[0131] Where i = {1, 2, 3, 4}, j = {256, 512, 1024, 2048}.

[0132] Optionally, the video spatiotemporal resolution-guided depth quality evaluation method, wherein calculating the feature representation based on the first aggregated feature and the second aggregated feature specifically includes:

[0133] Subtracting the first aggregated feature from the second aggregated feature yields the residual feature, and then aggregating the second aggregated feature with the residual feature yields the feature representation of the distorted video and the corresponding residual.

[0134] The residual feature represents the Euclidean distance of the object's feature domain.

[0135] Optionally, in the video spatiotemporal resolution-guided depth quality assessment method, the feature representation calculated based on the first aggregated feature and the second aggregated feature is expressed as:

[0136] in, Indicates feature expression, and Let represent the first aggregated feature and the second aggregated feature respectively, t represent a certain video frame, and i and j represent the j-th feature map of the i-th stage.

[0137] Optionally, the video spatiotemporal resolution-guided depth quality evaluation method, wherein inputting the bitstream information of the distorted video into a first fully connected layer to obtain a first weight, and performing dimensionality reduction processing on the feature representation based on the first weight, specifically includes:

[0138] A prior information model based on downsampling is constructed. The resolution information in the bitstream information of the distorted video is passed through the first fully connected layer of the downsampling prior information model to obtain the first weight. The first weight is fused with the feature expression to optimize the feature expression and obtain the dimensionality-reduced feature expression.

[0139] The feature representation is reduced in dimensionality as follows:

[0140] in, This represents the two fully connected layers that generate the first weight, and Resolution represents the resolution information in the bitstream of the distorted video. Indicates the first weight. This represents the feature representation after the dimensionality reduction process.

[0141] Optionally, the video spatiotemporal resolution-guided depth quality evaluation method, wherein inputting the bitstream information of the distorted video into a second fully connected layer to obtain the second weights, specifically includes:

[0142] A global video temporal information model is constructed using GRU. The frame rate information in the bitstream information of the distorted video is input into the second fully connected layer of the global video temporal information model to obtain the second weight.

[0143] The process of obtaining the second weight is expressed as follows: in, This represents the two fully connected layers that generate the second weight, where Framerate represents the frame rate information in the bitstream information of the distorted video. This represents the second weight.

[0144] Optionally, in the video spatiotemporal resolution-guided depth quality evaluation method, the process of obtaining a quality score by performing GRU calculation on the dimensionality-reduced feature representation and the second weight set is expressed as:

[0145] Where, q (t) This represents the quality score of the t-th video frame after GRU calculation.

[0146] Optionally, the video spatiotemporal resolution-guided depth quality evaluation method, wherein the step of pre-setting a set of video frames preceding the current video frame, and using the quality score of the video frame with the smallest quality score in the set as the first quality score, specifically includes:

[0147] A set of video frames preceding the current video frame is preset, and the quality score of the video frame with the smallest quality score among the preceding τ frames in the set is taken as the first quality score.

[0148] The process of obtaining the first mass fraction is represented as follows: in, This represents the set of video frames preceding the current video frame, τ = 12, l (t) This represents the first mass fraction.

[0149] Optionally, the video spatiotemporal resolution-guided depth quality evaluation method, wherein obtaining the second quality score based on the quality score and its weight specifically includes:

[0150] Based on the characteristics of short-term memory in the human visual perception system, and combining the quality score and its weight, the rate of increase in video score is limited to obtain a second quality score.

[0151] The process of obtaining the second mass fraction is represented as follows: Where τ=12, This represents the collection of video frames following the current video frame, where T represents the number of video frames.

[0152] The weight q represents the quality score. j q k Indicates in The index of the video frame in the collection, m (t) This represents the second mass fraction.

[0153] Optionally, the video spatiotemporal resolution-guided depth quality evaluation method, wherein the process of obtaining an original quality score based on the first quality score and the second quality score, and calculating the average of each original quality score as the final quality score of the video, is expressed as follows:

[0154] o (t) =αl (t) +(1-α)m (t) , Where α = 0.5, Q is the final quality score of the distorted video, and o (t) Let be the original quality score of the t-th video frame.

[0155] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a video spatiotemporal resolution-guided depth quality evaluation program, which, when executed by a processor, implements the steps of the video spatiotemporal resolution-guided depth quality evaluation method as described above.

[0156] In summary, this invention provides a video spatiotemporal resolution-guided depth quality assessment method and related equipment. The method includes: inputting a reference video and a distorted video into a backbone network, respectively, and calculating the global mean and variance in the feature maps; obtaining a feature representation based on the global mean and variance of the reference video feature map; obtaining the original video frame quality score from the quality scores of each video frame; inputting the bitstream information of the distorted video into first and second fully connected layers to obtain first and second weights; performing dimensionality reduction processing on the feature representation based on the first weights; calculating the quality score by applying a GRU to the dimensionality-reduced feature representation and the second weight set; obtaining the first and second quality scores based on the quality scores and their weights; and finally obtaining the final quality score of the distorted video. The video quality algorithm constructed by this invention achieves optimal performance on existing ultra-high-definition video quality assessment databases, providing effective decision-making for optimizing downsampling video coding techniques.

[0157] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0158] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0159] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A video spatiotemporal resolution-guided depth quality assessment method, characterized in that, The video spatiotemporal resolution-guided depth quality assessment method includes: The reference video and the distorted video are respectively input into the backbone network. The first feature map of the video frame of the reference video in the backbone network is calculated, and the second feature map of the video frame of the distorted video in the backbone network is calculated. The global mean and variance of the first feature map and the second feature map are calculated respectively. The first aggregated feature is obtained by concatenating the global mean and variance of the first feature map, and the second aggregated feature is obtained by concatenating the global mean and variance of the second feature map. The feature representation is then calculated based on the first aggregated feature and the second aggregated feature. The bitstream information of the distorted video is input into the first fully connected layer to obtain the first weight. The feature representation is then dimensionality-reduced based on the first weight. The bitstream information of the distorted video is input into the second fully connected layer to obtain the second weight. The dimensionality-reduced feature representation and the second weight are then processed by GRU to obtain the quality score. A set of video frames preceding the current video frame is preset. The quality score of the video frame with the smallest quality score in the set is taken as the first quality score. The second quality score is then calculated based on the quality score and the weight of the quality score. The original quality score is obtained based on the first quality score and the second quality score, and the average of the original quality scores of all video frames is calculated as the final quality score of the video.

2. The video spatiotemporal resolution-guided depth quality evaluation method according to claim 1, characterized in that, The calculation of the first feature map of the video frame of the reference video in the backbone network and the calculation of the second feature map of the video frame of the distorted video in the backbone network specifically include: Calculate the j-th first feature map of the video frame of the reference video in the i-th stage of the backbone network; Calculate the j-th second feature map of the video frame of the distorted video in the i-th stage of the backbone network; Where i = {1, 2, 3, 4}, j = {256, 512, 1024, 2048}.

3. The video spatiotemporal resolution-guided depth quality evaluation method according to claim 2, characterized in that, The step of calculating the feature representation based on the first aggregated feature and the second aggregated feature specifically includes: Subtracting the first aggregated feature from the second aggregated feature yields the residual feature, and then aggregating the second aggregated feature with the residual feature yields the feature representation of the distorted video and the corresponding residual. The residual feature represents the Euclidean distance of the object's feature domain.

4. The video spatiotemporal resolution-guided depth quality evaluation method according to claim 3, characterized in that, The feature expression calculated based on the first aggregated feature and the second aggregated feature is expressed as follows: ,in, Indicates feature expression, Let represent the first aggregated feature and the second aggregated feature respectively, t represent a certain video frame, and i and j represent the j-th feature map of the i-th stage.

5. The video spatiotemporal resolution-guided depth quality evaluation method according to claim 4, characterized in that, The step of inputting the bitstream information of the distorted video into a first fully connected layer to obtain a first weight, and then performing dimensionality reduction processing on the feature representation based on the first weight, specifically includes: A prior information model based on downsampling is constructed. The resolution information in the bitstream information of the distorted video is passed through the first fully connected layer of the downsampling prior information model to obtain the first weight. The first weight is fused with the feature expression to optimize the feature expression and obtain the dimensionality-reduced feature expression. The feature representation is reduced in dimensionality as follows: , ,in, , This indicates the generation of two fully connected layers with the first weight. This represents the resolution information within the bitstream of the distorted video. Indicates the first weight. This represents the feature representation after the dimensionality reduction process.

6. The video spatiotemporal resolution-guided depth quality evaluation method according to claim 5, characterized in that, The step of inputting the bitstream information of the distorted video into the second fully connected layer to obtain the second weight specifically includes: A global video temporal information model is constructed using GRU. The frame rate information in the bitstream information of the distorted video is input into the second fully connected layer of the global video temporal information model to obtain the second weight. The process of obtaining the second weight is expressed as follows: ,in, , This indicates the generation of two fully connected layers with the second weight. This refers to the frame rate information in the bitstream information of the distorted video. This represents the second weight.

7. The video spatiotemporal resolution-guided depth quality evaluation method according to claim 6, characterized in that, The process of obtaining the quality score by applying the dimensionality-reduced feature representation and the second weight set to GRU is expressed as follows: ,in, This represents the quality score of the t-th video frame after GRU calculation.

8. The video spatiotemporal resolution-guided depth quality evaluation method according to claim 7, characterized in that, The preset set of video frames preceding the current video frame, using the quality score of the video frame with the lowest quality score in the set as the first quality score, specifically includes: A set of video frames preceding the current video frame is predefined, and the set of frames preceding the current video frame is then... The quality score of the video frame with the lowest quality score in the frame is taken as the first quality score; The process of obtaining the first mass fraction is represented as follows: , in, This represents the collection of video frames preceding the current video frame. , Indicates the first mass fraction. This represents the quality score of each video frame at index k.

9. The video spatiotemporal resolution-guided depth quality evaluation method according to claim 8, characterized in that, The step of obtaining the second quality score based on the quality score and its weight specifically includes: Based on the characteristics of short-term memory in the human visual perception system, and combining the quality score and its weight, the rate of increase in video score is limited to obtain a second quality score. The process of obtaining the second mass fraction is represented as follows: , , in, , This represents the collection of video frames following the current video frame, where T represents the number of video frames. The weights representing the quality scores are... , Indicates in The index of the video frame in the collection. This represents the second mass fraction.

10. The video spatiotemporal resolution-guided depth quality evaluation method according to claim 9, characterized in that, The process of obtaining the original quality score based on the first quality score and the second quality score, and then calculating the average of the original quality scores as the final quality score of the video, is expressed as follows: , ,in, Q represents the final quality score of the distorted video. Let be the original quality score of the t-th video frame.

11. A video spatiotemporal resolution-guided depth quality assessment system, characterized in that, The video spatiotemporal resolution-guided depth quality assessment system includes: The feature extraction module is used to input the reference video and the distorted video into the backbone network respectively, calculate the first feature map of the video frame of the reference video in the backbone network, calculate the second feature map of the video frame of the distorted video in the backbone network, and calculate the global mean and variance of the first feature map and the second feature map respectively. The first aggregated feature is obtained by concatenating the global mean and variance of the first feature map, and the second aggregated feature is obtained by concatenating the global mean and variance of the second feature map. The feature representation is then calculated based on the first aggregated feature and the second aggregated feature. The feature fusion module inputs the bitstream information of the distorted video into a first fully connected layer to obtain a first weight, performs dimensionality reduction processing on the feature representation based on the first weight, inputs the bitstream information of the distorted video into a second fully connected layer to obtain a second weight, and calculates a quality score by performing GRU calculation on the dimensionality-reduced feature representation and the second weight. A set of video frames before the current video frame is preset, and the quality score of the video frame with the smallest quality score in the set is taken as the first quality score. A second quality score is calculated based on the quality score and the weight of the quality score. The video quality calculation module obtains the original quality score based on the first quality score and the second quality score, and calculates the average of the original quality scores of all video frames as the final quality score of the video.

12. A terminal, characterized in that, The terminal includes: a memory, a processor, and a video spatiotemporal resolution-guided depth quality evaluation program stored in the memory and executable on the processor. When the video spatiotemporal resolution-guided depth quality evaluation program is executed by the processor, it implements the steps of the video spatiotemporal resolution-guided depth quality evaluation method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a video spatiotemporal resolution-guided depth quality evaluation program, which, when executed by a processor, implements the steps of the video spatiotemporal resolution-guided depth quality evaluation method as described in any one of claims 1-10.

Images