No-reference video quality evaluation method based on space-time perception feature fusion

Through the method of fusion of space-time perception features, using frame difference diagrams and motion guidance attention modules, combined with bidirectional long and short-term memory networks, the problem of real distorted video quality evaluation in the existing technology is solved, and more accurate video quality evaluation is achieved.

CN118968267AInactive Publication Date: 2024-11-15SHAANXI WEIHE ECOLOGICAL GRP CO LTD +2

Patent Information

Application Number
CN202411094083.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art is difficult to effectively evaluate the quality of true distortion videos, especially in the case of uneven distribution of time and space domains, and it is impossible to fully capture the dynamic changes and timing relationships between video frames.

Method used

Using a reference-free video quality evaluation method based on space-time perception characteristics fusion, the time information between video frames is extracted through the frame difference graph, the motion guidance attention module is added to the backbone network, and the short-term memory network is used to capture the short-term between video frames. and long-term dependencies and improve network performance through pre-training and fine-tuning.

Benefits of technology

It improves the accuracy and performance of video quality evaluation, can better describe the dynamic changes and timing relationship between video frames, explains the temporary memory effect, alleviates the dilemma of lack of balanced labels for distorted videos, and has good practical value.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968267A_ABST
    Figure CN118968267A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of video quality evaluation, and relates to a no-reference video quality evaluation method based on space-time perception feature fusion, which comprises the following steps: 001, data preprocessing: processing an original video into a video frame and a frame difference map; 002, using a residual network as a backbone network, and then adding a motion guidance attention mechanism in the video frame branch; 003, taking a bidirectional long-short-term memory network as a sequence model to design a space pooling network, and capturing a short-term dependency relationship between the patches of the video frames; 004, capturing a long-term dependency relationship between the video frames through a time pooling network; and 005, performing full connection operation through the quality regression network to obtain a video quality score. According to the technical scheme, the influence of the object motion effect and non-uniform distortion on the video frame quality and the influence of past and future adjacent frames on the current frame are considered, the quality evaluation score of the video is finally obtained through the quality regression model and is used for public data set analysis, and the practical value is good.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of video quality assessment, and in particular relates to a reference-free video quality assessment method based on spatiotemporal perception feature fusion. Background Art

[0002] Most existing studies focus on synthetically distorted video datasets, but relatively few studies focus on real distorted datasets. The uniqueness of this dataset is that the distortion types are diverse, the distortion range is wide, and the distortion is unevenly distributed in the spatial and temporal domains, which leads to fluctuations in video quality.

[0003] The existing processing method is to use the mean as a pooling strategy to solve this non-uniform distortion, but this method cannot well represent the distribution of spatiotemporal quality. In addition, unlike images, videos are dynamic data with a time dimension, in which motion information is expressed by recording the position changes between adjacent video frames. These relative position relationships carry rich dynamic information and are an indispensable part of video sequences. At the same time, when the video is distorted, not only will the pixel values ​​in the video change suddenly, but the temporal relationship between frames will also be destroyed due to distortion, resulting in video distortion.

[0004] Based on this, a reference-free video quality assessment method based on spatiotemporal perception feature fusion is studied and developed. Summary of the invention

[0005] The present invention provides a reference-free video quality evaluation method based on spatiotemporal perception feature fusion, which uses a frame difference map to extract time information between video frames, achieves a good description of dynamic changes between video frames, comprehensively considers the impact of moving objects on video quality, adds a motion guidance attention module to the backbone network, and improves model performance and accuracy; finally, two bidirectional long short-term memory networks are used to capture the dependencies between video frames, wherein the first one is used to capture the short-term dependencies between image patches, and the second one captures the long-term dependencies between video frames to explain the temporary memory effect, and at the same time, by migrating samples from distorted images, the dilemma of distorted videos lacking rich and balanced labels is alleviated, and the pre-training and fine-tuning steps improve network performance. Thus, technical problems such as sudden changes in pixel values ​​in the video and the destruction of the temporal relationship between video frames due to distortion, which lead to video distortion, are solved.

[0006] A no-reference video quality assessment method based on spatiotemporal perception feature fusion includes the following steps: Step 001, processing the original video into video frames and frame difference images, and detecting the movement and change of objects in the original video; Step 002, using a residual network as a backbone network, and then adding a motion-guided attention mechanism to the branch of the video frame, dividing the original video into two branches and sending them to the network to extract the spatiotemporal features of the original video, one branch sends the frame difference map to the residual network to extract motion features, and the other branch interacts with each other in the video sequence, and adds a motion-guided attention module to the residual network; Step 003, designing a spatial pooling network by using a bidirectional long short-term memory network as a sequence model to capture short-term dependencies between patches of the video frame; Step 004, capturing the long-term dependency between the video frames through a temporal pooling network; Step 005, perform a fully connected layer operation through a quality regression network to obtain a final video quality score.

[0007] Optionally, in step 001, the original video is processed into video frames and frame difference maps, and the specific operation is: using the frame difference method to extract the frame difference map, and the calculation formula of the frame difference map is as follows: (1) Among them, FD i+1 is the obtained frame difference map, F i is the video frame at the current time i, F i+1 is the video frame at the next moment i+1, and i is the i-th video frame.

[0008] Optionally, step 002 includes the following steps: Step 00201, dividing the original video into two branches and sending them into the network to extract the spatiotemporal features of the video, wherein one branch sends the frame difference map into the residual network to extract the motion features of the object in the video frame; Step 00202, the other branch adds a motion-guided attention module to the residual network to improve the ability to capture global information and improve the modeling capabilities of spatial and temporal features.

[0009] The operation of the motion guidance attention module in step 00202 is as follows: (2) In formula (2), f d The salient map of motion features is obtained by convolution and Sigmoid activation function, which is consistent with f v After multiplication, we get f' v , f d、 f v are the outputs of the network at the corresponding stage; (3) In formula (3), the global average pooling (GAP) f' is used v A one-dimensional vector is obtained, which represents the appearance characteristics of the significant motion area, and then a weighted C channel vector is obtained by convolution; secondly, Normalize the output, and finally f v ’ ⊗[·] Multiply the spatial feature with the processed normalized attention vector; Among them, f v , f d Represent the features of the distorted video frame and the features of the corresponding frame difference map, respectively, v 、f v ' and f v The shape of '' is C×H×W, f d The shape is C'×H×W.

[0010] In the motion-guided attention module, the motion information is first used to calculate the attention weight, and then the spatial features and the temporal features are weighted and converged by the attention weight, so as to concentrate the attention weight on the surrounding area of ​​the moving target.

[0011] Optionally, in step 003, a bidirectional long short-term memory network is used as a sequence model to design a spatial pooling network to capture short-term dependencies between patches of the video frame, specifically comprising the following steps: Step 00301, the architecture of the spatial pooling network consists of two bidirectional long short-term memory network layers, each with k units, followed by a 256-node fully connected layer that pools the feature vector (y i 1 ,y i 2 ,…y i N ) T Input into the spatial pooling network, as shown in the following formula: (4) Among them, g∮ s The training parameters are s The parameter function of the spatial pooling network, N is the dimension of the feature vector, and T is the number of video frames that are simultaneously input into the spatial pooling network; Step 00302, using the KonIQ-10k dataset for pre-training to obtain a pre-trained model; Step 00303, training the pre-trained model on the video quality assessment dataset.

[0012] Optionally, in step 00302, the KonIQ-10k dataset is a large-scale image quality assessment dataset of real distorted scenes, and the dataset includes 10,073 images of different resolutions.

[0013] Step 004, capturing the long-term dependencies between video frames through a temporal pooling network, specifically: using a bidirectional long short-term memory network to aggregate video frame-level features and convert them into a global feature vector for the entire video sequence, as shown in the following formula: (5) Among them, g∮ t The training parameters are t is the parameter function of the temporal pooling network, T is the number of video frames input to the spatial pooling network, is the global eigenvector.

[0014] Step 005, performing a fully connected layer operation through a quality regression network to obtain a final video quality score, specifically: after aggregating the extracted quality perception features into a single vector, the quality perception features are mapped to the final video quality score, as shown below: (6) Among them, ξ is the fully connected layer and q is the video quality score.

[0015] Optionally, a fully connected layer is used as a regression model with a linear activation function, and after the extracted quality-aware features are aggregated into a single vector, the quality-aware feature map is used to obtain a final video quality score.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention is a reference-free video quality evaluation method based on the fusion of spatiotemporal perception features. It mainly extracts the time information between video frames by using the frame difference map, which can better describe the dynamic changes between video frames. Then, the motion-guided attention mechanism is added to the backbone network to improve the performance and accuracy of the model and avoid or reduce the impact of moving objects on the video quality. From the perspective that time information will change the human visual system's perception of spatial distortion, a bidirectional long short-term memory network is used to capture the temporal relationship between video frames and explain the temporary memory effect. On the other hand, the dilemma of the lack of rich balanced labels for distorted videos is alleviated by migrating samples from distorted images. The pre-training and fine-tuning steps improve the network performance and have good practical value.

[0017] In the technical solution of the present application, a quality assessment score of a video is obtained through a quality regression model or a quality regression network, and is applied to a public data set for experimental analysis, which has strong practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is the overall architecture diagram described in this application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments. The illustrative implementation modes of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0020] Example: like Figure 1 As shown, a reference-free video quality assessment method based on spatiotemporal perception feature fusion includes the following steps: Step 001, processing the original video into video frames and frame difference images, and detecting the movement and change of objects in the original video; In step 001, the original video is processed into video frames and frame difference images, which are mainly used to detect the movement and changes of objects in the video. The frame difference image is obtained by performing a difference operation on the current frame and the previous frame, and needs to be binarized. In the binarization process, points with pixel values ​​greater than the threshold are marked as moving object points, and points with pixel values ​​less than or equal to the threshold are marked as background points. In order to facilitate the observation of the frame difference image, the frame difference image is visualized.

[0021] Specifically, the frame difference method is used to extract the frame difference map, and the calculation formula of the frame difference map is as follows: (1) Among them, FD i+1 is the obtained frame difference image, F i is the video frame at the current time i, F i+1 is the video frame at the next moment i+1, and i is the i-th video frame.

[0022] Step 002, using a residual network as a backbone network, and then adding a motion-guided attention mechanism to the branch of the video frame, dividing the original video into two branches and sending them to the network to extract the spatiotemporal features of the original video, one branch sends the frame difference map to the residual network to extract motion features, and the other branch interacts with each other in the video sequence, and adds a motion-guided attention module to the residual network; In step 002, the following steps are included: Step 00201, dividing the original video into two branches and sending them into the network to extract the spatiotemporal features of the video, wherein one branch sends the frame difference map into the residual network to extract the motion features of the object in the video frame; Step 00202, the other branch adds a motion-guided attention module to the residual network to improve the ability to capture global information and improve the modeling capabilities of spatial and temporal features.

[0023] The operation of the motion guidance attention module in step 00202 is as follows: (2) In formula (2), f d The salient map of motion features is obtained by convolution and Sigmoid activation function, which is consistent with f v After multiplication, we get f' v , f d、 f v are the outputs of the network at the corresponding stage; (3) In formula (3), the global average pooling (GAP) f' is used v A one-dimensional vector is obtained, which represents the appearance characteristics of the significant motion area, and then a weighted C channel vector is obtained by convolution; secondly, Normalize the output, and finally f v ’ ⊗[·] Multiply the spatial feature with the processed normalized attention vector; Among them, f v , f d Represent the features of the distorted video frame and the features of the corresponding frame difference map, respectively, v 、f v ' and f v The shape of '' is C×H×W, f d The shape is C'×H×W.

[0024] In the motion-guided attention module, the motion information is first used to calculate the attention weight, and then the spatial features and the temporal features are weighted and converged through the attention weight, focusing the attention weight on the surrounding area of ​​the moving target, thereby enhancing the temporal feature representation of the target.

[0025] In step 003, a spatial pooling network is designed by using a bidirectional long short-term memory network as a sequence model to capture the short-term dependencies between patches of the video frame, which specifically includes the following steps: Step 00301, the architecture of the spatial pooling network consists of two bidirectional long short-term memory network layers, each with k units, followed by a 256-node fully connected layer that pools the feature vector (y i 1 ,y i 2 ,…y i N ) T Input into the spatial pooling network, as shown in the following formula: (4) Among them, g∮ s The training parameters are s The parameter function of the spatial pooling subnetwork, N is the dimension of the feature vector, and T is the number of video frames input to the network at the same time; The spatial pooling network is used to capture short-term dependencies between image patches to process the patch sequence of each video frame instead of processing each patch independently. The spatial pooling module is designed using a bidirectional long short-term memory network as a sequence model, which provides the ability to process cross-patch dependencies and can effectively capture the dependencies between patches of video frames. The bidirectional long short-term memory network is composed of the stacking of two independent long short-term memory networks, which can take into account forward and backward information.

[0026] Step 00302, using the KonIQ-10k dataset for pre-training to obtain a pre-trained model; Pre-training is done using the KonIQ-10k dataset. This is a large-scale image quality assessment dataset of real distorted scenes, which includes 10,073 images of different resolutions. Pre-training models can reduce the cost of model training and achieve faster convergence speed when combined with downstream tasks, while effectively improving the performance of the model. Transfer the knowledge gained by the model after training on the KonIQ-10k dataset to the target task. Show the model more diverse content by using a larger real image quality assessment dataset.

[0027] Step 00303, training the pre-trained model on the video quality assessment dataset.

[0028] Step 004, capture the long-term dependencies between video frames through a temporal pooling network to explain the temporal memory effect. Considering the temporal changes of video distortion and the temporal memory effect of human perception, a temporal pooling method based on a long short-term memory network is proposed. It is able to consider both backward and forward information at the same time, which makes it possible to capture the long-term dependencies between video frames.

[0029] The video frame-level features are aggregated using a bidirectional long short-term memory network and converted into a global feature vector for the entire video sequence, as shown in formula (5). Similar to the spatial pooling network, this module consists of two bidirectional long short-term memory network layers, each with 64 units, followed by a fully connected layer with 256 nodes. The long short-term memory network is able to consider both backward and forward information, which makes it possible to capture long-term dependencies between video frames.

[0030] Specifically, use a bidirectional long short-term memory network to aggregate video frame-level features and convert them into a global feature vector for the entire video sequence, as shown in the following formula: (5) Among them, g∮ t The training parameters are t The parameter function of the temporal pooling subnetwork.

[0031] Step 005, performing a fully connected operation through a quality regression network to obtain a final video quality score.

[0032] Specifically, a full connection operation is performed through a quality regression network to obtain the final video quality score. Specifically, after the extracted quality perception features are aggregated into a single vector, the quality perception features are mapped to the final video quality score, as shown below: (6) Among them, ξ is the fully connected layer and q is the video quality score.

[0033] Specifically, the fully connected layer is used as a regression model with a linear activation function to aggregate the extracted quality-aware features into a single vector, which is then mapped to the final video quality score.

[0034] The present embodiment describes a method scheme for no-reference video quality evaluation based on the fusion of spatiotemporal perception features. On the one hand, it mainly uses the frame difference map to extract the time information between video frames, and achieves a good description of the dynamic changes between video frames. And taking into account the impact of moving objects on video quality, the motion guidance attention module is added to the backbone network to improve the performance and accuracy of the model. From the perspective that time information will change the human visual system's perception of spatial distortion, a bidirectional long short-term memory network is used to capture the temporal relationship between video frames and explain the temporary memory effect. On the other hand, by migrating samples from distorted images, the dilemma of the lack of rich balanced labels for distorted videos is alleviated. The pre-training and fine-tuning steps improve the network performance and have good practical value. For real distorted videos, it can do a good quality evaluation and is applied to public data sets for experimental analysis.

[0035] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A no-reference video quality assessment method based on spatiotemporal perception feature fusion, characterized by: The following steps are included: Step 001, processing the original video into video frames and frame difference images, and detecting the movement and change of objects in the original video; Step 002, using a residual network as a backbone network, and then adding a motion-guided attention mechanism to the branch of the video frame, dividing the original video into two branches and sending them to the network to extract the spatiotemporal features of the original video, one branch sends the frame difference map to the residual network to extract motion features, and the other branch interacts with each other in the video sequence, and adds a motion-guided attention module to the residual network; Step 003, designing a spatial pooling network by using a bidirectional long short-term memory network as a sequence model to capture short-term dependencies between patches of the video frame; Step 004, capturing the long-term dependency between the video frames through a temporal pooling network; Step 005, perform a fully connected layer operation through a quality regression network to obtain a final video quality score.

2. The method for no-reference video quality assessment based on spatiotemporal perception feature fusion according to claim 1, characterized in that: In step 001, the original video is processed into video frames and frame difference images. The specific operation is: the frame difference method is used to extract the frame difference image. The calculation formula of the frame difference image is as follows: (1) Among them, FD i+1 is the obtained frame difference image, F i is the video frame at the current time i, F i+1 is the video frame at the next moment i+1, and i is the i-th video frame.

3. The method for no-reference video quality assessment based on spatiotemporal perception feature fusion according to claim 1, characterized in that: In step 002, the following steps are included: Step 00201, dividing the original video into two branches and sending them into the network to extract the spatiotemporal features of the video, wherein one branch sends the frame difference map into the residual network to extract the motion features of the object in the video frame; Step 00202, the other branch adds a motion-guided attention module to the residual network to improve the ability to capture global information and improve the modeling capabilities of spatial and temporal features.

4. The method for no-reference video quality assessment based on spatiotemporal perception feature fusion according to claim 2, characterized in that: The operation of the motion guidance attention module in step 00202 is as follows: (2) In formula (2), f d The salient map of motion features is obtained by convolution and Sigmoid activation function, which is consistent with f v After multiplication, we get f' v , f d、 f v are the outputs of the network at the corresponding stage; (3) In formula (3), the global average pooling (GAP) f' is used v A one-dimensional vector is obtained, which represents the appearance characteristics of the significant motion area, and then a weighted C channel vector is obtained by convolution; secondly, Normalize the output, and finally f v ’ ⊗[·] Multiply the spatial feature with the processed normalized attention vector; Among them, f v , f d Represent the features of the distorted video frame and the features of the corresponding frame difference map, respectively, v 、f v ' and f v The shape of '' is C×H×W, f d The shape is C'×H×W.

5. The method for no-reference video quality assessment based on spatiotemporal perception feature fusion according to claim 4, characterized in that: In the motion-guided attention module, the motion information is first used to calculate the attention weight, and then the spatial features and the temporal features are weighted and converged by the attention weight, so as to concentrate the attention weight on the surrounding area of ​​the moving target.

6. The method for no-reference video quality assessment based on spatiotemporal perception feature fusion according to claim 4, characterized in that: In step 003, a spatial pooling network is designed by using a bidirectional long short-term memory network as a sequence model to capture the short-term dependencies between patches of the video frame, which specifically includes the following steps: Step 00301, the architecture of the spatial pooling network consists of two bidirectional long short-term memory network layers, each with k units, followed by a 256-node fully connected layer that pools the feature vector (y i 1 ,y i 2 ,…y i N ) T Input into the spatial pooling network, as shown in the following formula: (4) Among them, g∮ s The training parameters are s The parameter function of the spatial pooling network, N is the dimension of the feature vector, and T is the number of video frames that are simultaneously input into the spatial pooling network; Step 00302, using the KonIQ-10k dataset for pre-training to obtain a pre-trained model; Step 00303, training the pre-trained model on the video quality assessment dataset.

7. The method for no-reference video quality assessment based on spatiotemporal perception feature fusion according to claim 6, characterized in that: In step 00302, the KonIQ-10k dataset is a large-scale image quality assessment dataset of real distorted scenes, which includes 10,073 images of different resolutions.

8. The method for no-reference video quality assessment based on spatiotemporal perception feature fusion according to claim 6, characterized in that: Step 004, capturing the long-term dependencies between video frames through a temporal pooling network, specifically: using a bidirectional long short-term memory network to aggregate video frame-level features and convert them into a global feature vector for the entire video sequence, as shown in the following formula: (5) Among them, g∮ t The training parameters are t is the parameter function of the temporal pooling network, T is the number of video frames input to the spatial pooling network, is the global eigenvector.

9. The method for no-reference video quality assessment based on spatiotemporal perception feature fusion according to claim 8, characterized in that: Step 005, performing a fully connected layer operation through a quality regression network to obtain a final video quality score, specifically: after aggregating the extracted quality perception features into a single vector, the quality perception features are mapped to the final video quality score, as shown below: (6) Among them, ξ is the fully connected layer and q is the video quality score.

10. The method for no-reference video quality assessment based on spatiotemporal perception feature fusion according to claim 9, characterized in that: After aggregating the extracted quality-aware features into a single vector using a fully connected layer as a regression model with a linear activation function, the quality-aware features are mapped to the final video quality score.

Citation Information

Patent Citations

  • No-reference video quality evaluation method fusing spatio-temporal features

    CN112954312A

  • No-reference video quality evaluation method and system based on space-time perception feature fusion

    CN117876851A

Cited By

  • Video quality evaluation method and system based on space-frequency combination and time sequence interaction

    CN121397207A

  • Video quality evaluation method and system based on space-frequency joint and time sequence interaction

    CN121397207B