A method and device for no-reference video quality assessment based on spatio-temporal edge feature extraction

Abstract

The application discloses a no-reference video quality evaluation method and device based on space-time edge feature extraction, and specifically comprises the following steps: step one, key frames of a video are extracted for a traditional feature extraction branch and a deep feature extraction branch respectively; for the deep feature extraction branch, a region with a specific size needs to be cropped from the key frames as an input of a lower layer network; step two, two groups of data are respectively sent into corresponding branches to extract features; step three, the features obtained from the two branches are spliced and sent into a quality regressor to obtain a final quality score; the application relates to the technical field of image and video processing. The no-reference video quality evaluation method based on space-time edge feature extraction extracts only a 256*256 region with the maximum edge feature intensity of a frame when extracting deep features, so that high-frequency features of the video frame can be captured as much as possible.

Description

Technical Field

[0001] This invention relates to the field of video and image processing technology, specifically to a no-reference video quality assessment method and device based on spatiotemporal edge feature extraction. Background Technology

[0002] In recent years, No-Reference Video Quality Assessment (NRVQA) methods have made significant progress driven by deep learning. These methods offer new insights into video quality assessment by combining Natural Scene Statistics (NSS) and deep learning techniques. For example, one approach utilizes optical flow features to simulate the human visual system's sensitivity to motion, extracting motion features by calculating optical flow between adjacent frames, thus more accurately assessing video quality. Furthermore, research has proposed hybrid models based on spatiotemporal visual sensitivity, combining spatial and temporal visual sensitivity features to assess video quality in a way that more closely resembles human visual perception.

[0003] However, these methods face challenges when processing cloud gaming videos because their feature distribution differs from that of regular videos. To address this issue, some researchers have proposed adjusting the feature distribution of cloud gaming videos by adding Gaussian white noise to align it with that of general videos. This is followed by statistical analysis of natural scene features, combined with feature extraction using a convolutional neural network (CNN), and finally, evaluation scores obtained through support vector regression (SVR). While this method improves evaluation performance to some extent, the way noise is added may interfere with edge features, preventing high-quality videos from achieving the high scores they deserve.

[0004] To address the shortcomings of the aforementioned studies, this paper proposes an improved no-reference video quality assessment method. The NSS feature extraction module based on added Gaussian white noise is replaced with an edge-based spatiotemporal feature extraction module, and the frame selection method for the deep feature extraction branch is optimized. Results show that this proposed method achieves outstanding results on four metrics: Spearman Rank Correlation Coefficient (SRCC), Kendall Rank Correlation Coefficient (KRCC), Pearson Linear Correlation Coefficient (PLCC), and Root Mean Square Error (RMSE). Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a no-reference video quality assessment method based on spatiotemporal edge feature extraction, which solves the problems mentioned in the background.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a no-reference video quality assessment method based on spatiotemporal edge feature extraction, specifically comprising the following steps:

[0007] Step 1: Obtain the original video and extract keyframes for both the traditional feature extraction branch and the deep feature extraction branch. For the deep feature extraction branch, a region of a specific size needs to be cropped from the keyframes as input to the lower-level network.

[0008] Step 2: Input the two sets of data into the corresponding branches to extract features;

[0009] Step 3: Concatenate the features obtained from the two branches and feed them into the quality regressor to obtain the final quality score;

[0010] The present invention further specifies that the keyframe selection method for the traditional feature extraction branch in step one includes:

[0011] For the input cloud gaming video, divide the video into several time slices F, per second. (t) For each time slice t∈[1,N], two frames are uniformly extracted as a frame pair to extract local features of the video; frame pair Pair t ={F (t,1) ,F (t,2) The selection formula for} is as follows:

[0012]

[0013] Where Video is the input video, and framerate is the number of frames displayed per second, i.e., the frame rate.

[0014] The present invention is further configured such that: the keyframe selection and cropping method of the depth feature extraction branch in step one includes:

[0015] To capture as much detail as possible in the video, the video is divided into several time slices per second, and then 8 frames are evenly extracted from each time slice as keyframes; finally, the region with the strongest edge intensity in each frame is calculated (the number of slices varies for videos with different resolutions).

[0016] The present invention is further configured such that: the keyframe cropping method of the depth feature extraction branch in step one includes:

[0017] On the one hand, this is to extract temporal feature information; on the other hand, it is to reduce computational load. The region selected from the first frame is then applied to the remaining seven frames extracted from the same time slice. For the first frame, the Canny edge detection algorithm is first applied to obtain an edge intensity feature map; secondly, integral image calculation is performed on a non-repeating 256×256 region to obtain the edge with the highest intensity value. (H×W is the frame resolution) Non-overlapping regions are identified, and their location information is obtained and then applied to subsequent keyframes.

[0018] The present invention is further configured such that the traditional feature extraction method in step two includes:

[0019] Frame-to-frame binary encoded texture extraction and frame-to-frame multi-channel edge feature fusion.

[0020] The present invention is further configured such that: the frame-pair binary encoded texture extraction method in step two includes:

[0021] For each frame pair, first convert each frame into a grayscale image. And calculate their gradient maps respectively. The Canny edge detection algorithm is used here, with the intensity value I for each frame. (t,k) It can be represented as:

[0022]

[0023] Secondly, I is calculated through local encoding. (t,k) Texture features of frame pairs are obtained based on statistical features of Local Binary Pattern (LBP). The calculation formula is as follows:

[0024]

[0025] Where W and H represent the width and height of the frame, respectively, δ is the Kronecker delta function; l∈[0,8] represents the pattern index in the LBP graph, which is set to 9 here; LBP(m,n) refers to calculating the LBP feature of the pixel at frame (m,n); G c and G e These represent the gradient values ​​of the center pixel and its surrounding pixels, respectively. G represents the LBP-based gradient domain feature of the k-th frame in the t-th frame pair; t It represents the overall gradient domain feature corresponding to the t-th frame pair, which here represents the texture feature of the current time slice.

[0026] The present invention is further configured such that: the frame-pair multi-channel edge feature fusion method in step two includes:

[0027] First, the color component images of the three RGB channels of each frame in the frame pair are extracted. Then, the edge intensity R of each channel is calculated using the Cauchy operator. (t,k) G (t,k) and B (t,k) Next, the edge maps of these three images are extracted to calculate the Weber contrast enhancement metric (EME). The calculation method is as follows:

[0028]

[0029] Where k1k2 is the number of blocks in the grayscale image, here we use a division method of k1 = W / 7, k2 = H / 7, I (col,a,b) It is a block at (a, b) in the current frame's color map. Ultimately, it will be composed of the colors from all three channels. Intensity weighting yields the final edge feature E (t,k) The calculation is as follows:

[0030]

[0031] Where λ1, λ2, and λ3 represent the weights of the three channels, which are set to 0.299, 0.587, and 0.114 respectively, resulting in E. t It is a one-dimensional feature vector of size 1.

[0032] The present invention is further configured such that the deep feature extraction method in step two includes:

[0033] The MobileViT_s model was used as the main framework of the deep network, and the classification head was replaced with a convolutional layer to obtain 1024 activation features as deep features.

[0034] The present invention is further configured such that the feature splicing and quality regressor in step three includes:

[0035] The traditional features (20N, num_of_videos) and the deep features (1024, num_of_videos) are concatenated in the first dimension to obtain the final features (20N+1024, num_of_videos), where N is the number of time slices into which the video is divided.

[0036] The quality regressor is a Support Vector Regressor (SVR) that randomly splits the dataset into training and test sets (80% / 20%), based on content, for over 1000 iterations. The training set is further split and subjected to five cross-validations.

[0037] This invention provides a no-reference video quality assessment method based on spatiotemporal edge feature extraction. It has the following beneficial effects:

[0038] This invention combines traditional video quality assessment and deep quality assessment. It applies frame-pair feature extraction to the traditional video quality assessment path and keyframe selection and cropping to the deep quality assessment path, effectively simplifying the complexity of feature extraction while ensuring the accuracy of the model. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0040] Figure 2 This is a schematic diagram of spatiotemporal edge feature extraction in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of patch extraction and depth feature extraction in an embodiment of the present invention;

[0042] Figure 4 This is a scatter plot of Mos vs Predicted scores on the GamingVideoSet dataset.

[0043] Figure 5 This is a scatter plot of Mos vs Predicted scores on the LIVE-Meta MCG dataset. Detailed Implementation

[0044] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0045] Please see Figures 1-3 The present invention provides the following technical solution: a no-reference video quality assessment method based on spatiotemporal edge feature extraction, comprising the following steps:

[0046] Step 1: Obtain the source video and extract keyframes from the video using both the traditional feature extraction branch and the deep feature extraction branch. For the deep feature extraction branch, a region of a specified size needs to be cropped from the keyframes as input to the lower-level network.

[0047] Step 2: Input the two sets of original video data into the corresponding branches to extract features;

[0048] Step 3: The features obtained from the two branches are concatenated and fed into the quality regressor to obtain the final quality score.

[0049] Specifically, this can be explained as:

[0050] Step 1: Data Preprocessing. For the input video, to facilitate processing, this invention divides the video into several time slices according to the frame rate, assuming the number of time slices is N. This invention includes a traditional feature extraction branch and a deep feature extraction branch. For the former, this invention extracts two keyframes from each time slice as the input of the branch; for the latter, this invention first extracts 8 keyframes from each time slice, then enlarges the keyframes to 1K resolution, and then calculates and extracts the top 25% of the 8 images with the highest edge intensity, each with a size of 256×256, as the input of the branch.

[0051] Step 2: Feature Extraction. Traditional feature extraction branches calculate the two input frames to obtain a one-dimensional feature vector of size 20N; deep feature extraction branches extract features from the extracted patches. This invention extracts the output of the intermediate layer and calculates a one-dimensional feature vector of size 1024 through convolution.

[0052] Step 3, Quality Regression. First, the feature vectors extracted from the two branches are concatenated to obtain a concatenated feature of size 1024 + 20N. Then, the concatenated feature is input into a support vector regression machine for 5-fold cross-validation, with 1000 iterations each time, and the median value is taken as the output.

[0053] The following is a detailed explanation:

[0054] Step 1, as follows Figure 1 As shown, the overall network is divided into a traditional feature extraction branch and a deep feature extraction branch. For the traditional feature extraction branch, keyframes of the video need to be extracted; for the deep feature extraction branch, a region of a specific size needs to be cropped from the keyframes as input to the lower-level network. For the input cloud gaming video, the video is divided into several time slices F, each second in length. (t) For each time slice t∈[1,N], two frames are uniformly extracted as a frame pair to extract local features of the video; frame pair Pair t ={F (t,1) ,F (t,2) The selection formula for} is as follows:

[0055]

[0056] Where Video is the input video and framerate is the frame rate.

[0057] To capture as much detail as possible in the video, it was divided into several time slices per second, and then 8 frames were evenly extracted from each time slice as keyframes, such as... Figure 3 As shown; finally, the region with the highest edge intensity in each frame is calculated (the number of cropped blocks varies for videos with different resolutions).

[0058] On the one hand, this is to extract temporal feature information; on the other hand, it is to reduce computational load. The region selected from the first frame is then applied to the remaining seven frames extracted from the same time slice. For the first frame, the Canny edge detection algorithm is first applied to obtain an edge intensity feature map; secondly, integral image calculation is performed on a non-repeating 256×256 region to obtain the edge with the highest intensity value. (H×W is the frame resolution) Non-overlapping regions are identified, and their location information is obtained and then applied to subsequent keyframes.

[0059] Step 2: Input the two sets of data into the corresponding branches to extract features; see the diagram of traditional feature extraction. Figure 2 For each frame pair, first convert each frame into a grayscale image. And calculate their gradient maps respectively. The Canny edge detection algorithm is used here, with the intensity value I for each frame. (t,k) It can be represented as:

[0060]

[0061] Secondly, I is calculated through local encoding. (t,k) Texture features of frame pairs are obtained based on statistical features of Local Binary Pattern (LBP). The calculation formula is as follows:

[0062]

[0063] Where W and H represent the width and height of the frame, respectively, δ is the Kronecker delta function; l∈[0,8] represents the pattern index in the LBP graph, which is set to 9 here; LBP(m,n) refers to calculating the LBP feature of the pixel at frame (m,n); G c and G e These represent the gradient values ​​of the center pixel and its surrounding pixels, respectively. G represents the LBP-based gradient domain feature of the k-th frame in the t-th frame pair; t It represents the overall gradient domain feature corresponding to the t-th frame pair, which here represents the texture feature of the current time slice.

[0064] First, the color component images of the three RGB channels of each frame in the frame pair are extracted. Then, the edge intensity R of each channel is calculated using the Cauchy operator. (t,k) G (t,k) and B (t,k) Next, the edge maps of these three images are extracted to calculate the Weber contrast enhancement metric (EME). The calculation method is as follows:

[0065]

[0066] Where k1k2 is the number of blocks in the grayscale image, here we use a division method of k1 = W / 7, k2 = H / 7, I (col,a,b) It is a block at (a, b) in the current frame's color map. Ultimately, it will be composed of the colors from all three channels. Intensity weighting yields the final edge feature E (t,k) The calculation is as follows:

[0067]

[0068]

[0069] Where λ1, λ2, and λ3 represent the weights of the three channels, which are set to 0.299, 0.587, and 0.114 respectively, resulting in E. t It is a one-dimensional feature vector of size 1.

[0070] For the deep feature extraction module, the MobileViT_s model was used as its main framework and the classification head module was replaced with a convolutional layer, and the output 1024-sized one-dimensional feature vector was used as the deep feature.

[0071] Step 3: Concatenate the features obtained from the two branches and feed them into the quality regressor to obtain the final quality score; concatenate the traditional features (20N, num_of_videos) and the deep features (1024, num_of_videos) in the first dimension to obtain the final features (20N+1024, num_of_videos), where N is the number of time slices into which the video is divided.

[0072] The quality regressor is a Support Vector Regressor (SVR) that randomly splits the dataset into training and test sets (80% / 20%), based on content, for over 1000 iterations. The training set is further split and subjected to five cross-validations.

[0073] This invention has achieved good results on multiple datasets. Next, we will demonstrate the specific metrics of this invention on the GamingVideoSet and LIVE-Meta-MCG datasets. The closer the SRCC, PLCC, and KRCC metrics are to 1, the better the performance. The lower the RMSE, the better the performance.

[0074] Table 1 Evaluation results of the present invention on the GamingVideoSet and LIVE-Meta MCG datasets.

[0075]

[0076] To illustrate the effectiveness of this invention, the following scatter plots of MOS vs. Predicted score for the two datasets are provided, as shown below. Figure 4 and Figure 5 As shown, the denser the distribution of the scatter plot and the closer it tends to the straight line y=x, the more accurate the prediction result.

[0077] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0078] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0079] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the methods described in the above embodiments.

[0080] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0081] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0082] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0083] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0084] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for no-reference video quality assessment based on spatio-temporal edge feature extraction, characterized in that: Specifically comprising the following steps: Step one, obtain the source video, and extract the key frames of the video through the traditional feature extraction branch and the deep feature extraction branch respectively; for the deep feature extraction branch, a region of a specified size needs to be cropped from the key frame as the input of the lower layer network; Step two, respectively send the two groups of original video data into the corresponding branch to extract features; Step three, splice the features obtained by the two branches and send them into the quality regressor to obtain the final quality score; The key frame selection and cropping method of the deep feature extraction branch in step one includes: Divide the video into several time slices according to seconds, and then evenly extract 8 frames of images from each time slice as key frames; finally, calculate the region with the maximum edge intensity of each frame, and the number of cropped blocks is different for videos with different resolutions; The key frame cropping method of the deep feature extraction branch in step one includes: The region selected from the first frame is applied to the remaining 7 frames of images extracted in the same time slice, for the first frame, first, the Canny edge detection algorithm is performed to obtain an edge intensity feature map; second, integral image calculation is performed on the non-repeated 256x256 regions to obtain the first H W is the resolution of the frame, the position information of the regions is obtained, and then applied to the subsequent key frames;​ The feature extraction method of the traditional feature extraction branch includes, Frame pair binary coding texture extraction method and frame pair multi-channel edge feature fusion method.

2. The spatio-temporal edge feature based no-reference video quality assessment method of claim 1, wherein: The key frame selection method of the traditional feature extraction branch in step one includes: For the input cloud game video Video, divide the video into several time slices by seconds , and extract two frames from each time slice as a frame pair Pair to extract local features of the video; the frame pair The selection formula of the frame pair is as follows: Wherein, Video is the input video, framerate is the number of frames displayed per second of the video, that is, the frame rate.

3. The spatio-temporal edge feature based no-reference video quality assessment method according to claim 1, wherein, The frame pair binary coding texture extraction method includes: For each frame pair, first convert each frame to a grayscale image and calculate its gradient image, here we use Canny edge detection algorithm, the intensity value of each frame is represented as: ​ Secondly, by local coding, calculate The statistical features based on local binary pattern (LBP) are used to obtain the texture features of the frame pair, and the calculation formula is as follows: where, and denote the width and height of the frame, respectively, is the Kronecker delta function; denotes the pattern index in the LBP map, which is set to 9 here; denotes the LBP feature of the pixel at ; and denote the gradient values of the center pixel and its surrounding pixels, respectively; denotes the LBP-based gradient domain feature of the k-th frame of the t-th frame pair; is the overall gradient domain feature corresponding to the t-th frame pair, which represents the texture feature of the current time slice.

4. The spatio-temporal edge feature based no-reference video quality assessment method of claim 1, wherein, The frame pair multi-channel edge feature fusion method includes: Firstly, the color component maps of each frame in the frame pair are extracted from the three channels of the RGB image, and the edge intensity of each channel is calculated by the Canny operator , and Secondly, the edge maps of the three maps are extracted to calculate the Weber contrast enhancement measure (EME), and the calculation method is as follows: where, is the number of blocks in the gray scale partitioning, here we use 7 partitioning, is a block in the col color map of the current frame; finally the intensities in the three channels are weighted to get the final edge feature , which is computed as follows: ​​ where , and represent the respective weights of the 3 channels, which are set to 0.299, 0.587 and 0.114, and the final is a one-dimensional feature vector of size 1.

5. The spatio-temporal edge feature based no-reference video quality assessment method of claim 1, wherein The deep feature extraction branch extraction method includes: The MobileViT_s model with an input size of 256x256 is used as the main framework of the deep network, and the classification head module of the original model is replaced with a convolution layer, so that its output is a one-dimensional feature vector with a size of 1024.

6. The spatio-temporal edge feature based no-reference video quality assessment method of claim 1, wherein The splicing of the features obtained by the two branches in step three includes: concatenate traditional features and deep features in the first dimension to get the final feature where N is the number of time slices the video is divided into. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The computer program is executed by the processor, so that the processor executes the steps of the method according to any one of claims 1 to 6. The computer program is executed by the processor, so that the processor executes the steps of the method according to any one of claims 1 to 6.

Images