A method for evaluating video action quality based on space-time domain perception

Abstract

The application discloses an online carbon semantic map construction method based on sparse fusion, which comprises the following steps: 1) performing spatio-temporal segmentation on an input video to divide the video into a video segment set p and a key frame segment set x; 2) extracting multi-rate motion features of the video segment set p, and then obtaining time domain motion features through an interpolation alignment module of fast features and slow features; 3) extracting multi-scale spatial features of the key frame segment set x, and then obtaining aligned space domain features through a multi-scale feature alignment module; 4) fusing the aligned time domain motion features and the space domain features to obtain a video motion quality feature with spatio-temporal perception characteristics, and then entering a quality regression module to obtain a video motion quality evaluation score. The application uses very sparse picture frames to extract spatial features and dense video frames with very low spatial resolution to extract motion features, thereby improving the precision while reducing the calculation complexity, and being more suitable for deployment and application of edge devices.

Description

Technical Field

[0001] This invention relates to the field of video quality assessment, and more particularly to a video motion quality assessment method based on spatiotemporal domain perception. Background Technology

[0002] In recent years, user-generated content (UGC) videos have exploded on the internet. Browsing, creating, and sharing videos has become a daily routine for many, through social media applications such as YouTube, TikTok, and Twitter. Given the rich variety of actions and movements in these videos, service providers need to analyze and monitor video quality to provide users with a better experience. With millions of UGC videos generated daily, relying solely on human visual systems for timely quality assessment is labor-intensive and inefficient. Therefore, developing reliable video motion quality assessment models to ensure high-quality video services is crucial.

[0003] Video action quality assessment is an important research area in computer vision. Its goal is to automate the evaluation of actions in videos using computer algorithms. Many methods have been proposed in this field, most of which are based on machine learning and use labeled data to train quality prediction models. Early traditional video action quality assessment employed handcrafted features, utilizing a set of general quality-aware features and combining them with popular image quality methods to predict quality through regression. Because the spatiotemporal domains are key attributes of video action quality assessment, these methods focused primarily on temporal and spatial features. Recent research has proposed convolutional neural network models, which can better extract temporal and spatial information for video action quality assessment, thus achieving superior performance.

[0004] Many of the methods mentioned above have taken into account the importance of temporal and spatial information for video motion quality assessment models, but they still have the following drawbacks:

[0005] 1) With the current trend of videos exhibiting multiple frame rates and resolutions, most related studies have not fully considered the impact of multi-rate temporal features and multi-scale spatial features on the evaluation of video motion quality.

[0006] 2) Most works do not make sufficient use of the feature relationship between temporal motion features and spatial semantic features. The training process ignores the cross relationship between spatiotemporal features, resulting in problems such as feature mismatch and semantic confusion. Summary of the Invention

[0007] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a video motion quality evaluation method based on spatiotemporal domain perception.

[0008] A video motion quality assessment method based on spatiotemporal domain awareness includes the following steps:

[0009] 1) Perform spatiotemporal segmentation on the input video, dividing it into a set of video segments p and a set of keyframe segments x;

[0010] 2) Extract multi-rate motion features from the video clip set p, and then obtain temporal motion features through the interpolation and alignment module of fast and slow features;

[0011] 3) Extract multi-scale spatial features from the keyframe fragment set x, and then obtain aligned spatial features through the multi-scale feature alignment module;

[0012] 4) The aligned temporal motion features and spatial features are fused to obtain a video motion quality feature with spatiotemporal awareness characteristics, which is then entered into the quality regression module to obtain the video motion quality evaluation score.

[0013] Furthermore, in step 1), the input video undergoes spatiotemporal segmentation, dividing it into a set of video segments p and a set of keyframe segments x. The input video V is segmented into N... k The vectors contained in The video consists of consecutive segments. Each video segment is p. i Including N f Frame: p i ={f i，j Select a keyframe f in each video clip. i，1 To extract spatial features, use p i Motion features are extracted from all frames.

[0014] Furthermore, in step 2), the video segment set p is extracted. i The method of this invention obtains temporal motion features by analyzing multi-rate motion features and then aligning them through an interpolation module that matches fast and slow features. The method uses a pre-trained slow-fast motion recognition model, SLOW-FAST, to acquire motion features for each video segment. The SLOW-FAST model extracts slow-rate and block-rate features through Slow and Fast branches respectively, enabling the feature representation of the motion recognition network to effectively reflect the motion information of the subject in the video.

[0015] Therefore, given a video segment p i (i∈{1, 2, 3, ..., N) c Using an action recognition network, slow-rate and fast-rate features were obtained respectively. and By concatenating these features, we obtain the slow-rate feature set and the fast-rate feature set:

[0016]

[0017] Subsequently, the attention mechanism was applied to different motion rates X. slow X fast The input features are used to learn attention weights and perform a weighted summation of multiple motion features to obtain multiple rate-adaptively aligned motion features F. s F t :

[0018] W att =Softmax(ReLu(Conv1(X) slow )+Conv2(X fast (2)

[0019]

[0020] Where Conv1(·) and Conv2(·) are two two-dimensional convolutional kernels with a single kernel size, ReLU(·) and Softmax(·) are activation functions, and W att These are attention weights. To obtain data with the same size and sampling rate across the time dimension, we use a multi-rate interpolation method for interpolation and alignment. For each timestamp t, we perform interpolation and alignment on the processed features F. s F t Perform interpolation and alignment to obtain the alignment characteristics of this timestamp:

[0021]

[0022] in These are the features extracted by the SLOW-FAST network at video frame t, where s and f represent the slow path and fast path, respectively, and r i It is the temporal sampling rate of the i-th frame in the path. This represents the timestamp closest to t in the i-th frame. i (t) are the interpolation coefficients obtained through linear or nearest neighbor interpolation, as follows:

[0023]

[0024] Finally, the features aligned to the slow-rate path and the fast-rate path are fused according to a certain ratio α to obtain the final video motion feature representation:

[0025]

[0026] Furthermore, in step 3), multi-scale spatial features of the keyframe fragment set are extracted, and then aligned spatial features are obtained through a multi-scale feature alignment module. The method of this invention uses a pre-trained residual neural network model, ResNet, to extract spatial features, obtaining feature representations with robust generalization ability to single-frame spatial resolution. Spatial domain convolutions of different sizes and depths can capture different types of semantic information. Previous research has shown that, compared to high-level features, low-level features are characterized by higher resolution, wider location, and more detailed information, but lower semantic content and higher noise levels. In contrast, high-level features provide strong semantic information with low resolution and poor detail awareness; they prioritize content-aware information and reflect global distortions in the video. Consider input frame p... i and stage characteristics X s We define X s As the multi-scale output of the convolutional neural network model CNN at stage s (s∈1,2,3,4):

[0027] X s =CNN s (X s-1 (7)

[0028] To effectively align features representing different semantics and scales, this invention proposes a multi-scale channel attention module. This module uses channel attention at multiple scales obtained by varying the spatial pool size to align features at different scales. To improve computational efficiency, we choose pointwise convolution to achieve local context aggregation, which only utilizes pointwise channel interactions at each spatial location. Given multi-scale features X = {X1, X2, X3, X4}, the output of the multi-scale channel attention module is defined as follows:

[0029]

[0030] Where L(X) and G(X) represent the local and global channel contexts used by the multi-scale channel attention module, respectively. Indicates broadcast addition. This represents element-wise multiplication, where σ is a sigmoid() activation function. Note that the weights L(X) and G(X) are defined as follows:

[0031]

[0032]

[0033] Among them W (1) W (2) W (3) and W (4) It consists of four convolutional layers with learnable weights, where δ represents the ReLU activation function. This is represented as batch normalization. Indicates global average pooling:

[0034]

[0035] Furthermore, in step 4), the aligned temporal motion features and spatial features are fused to obtain a video motion quality feature with spatiotemporal awareness characteristics, which is then fed into the quality regression module to obtain a video motion quality evaluation score. Using the feature extraction and feature alignment model in the above steps of the method of this invention, a spatiotemporal awareness feature representation is obtained. Subsequently, a multilayer perceptron is used to map these features to a quality regression model to calculate the corresponding quality score.

[0036] Q final =MLP(Concat(F motion ,X′)) (12)

[0037] Q final The predicted score is represented by the input of the final quality regression layer of the quality regression model, which combines temporal motion and spatial information. The input includes two main information streams that characterize human perception of video content.

[0038] During the training phase, the method of this invention uses two loss functions to optimize the performance of the proposed video action quality assessment model, namely the mean absolute error loss (MAE) and the RANK loss, defined as follows:

[0039]

[0040]

[0041] Where i and j are the indices that identify the videos in the batch, and N is the number of videos in the batch. The definition is as follows:

[0042]

[0043] Subsequently, L RANK The calculation is as follows:

[0044]

[0045] Finally, summing the results yields the loss function:

[0046] L = L MAE +λ·L RANK (17)

[0047] Where λ is a hyperparameter that balances MAE and RANK loss.

[0048] This invention provides a video motion quality assessment method based on spatiotemporal domain awareness, which performs combined multi-rate motion feature and multi-scale spatial feature alignment. By integrating an alignment mechanism based on Lagrange multi-rate interpolation and multi-scale attention fusion into the network architecture of this invention, the method can effectively process multi-frame-rate, multi-resolution, and multi-attribute video data without information loss or conflict issues. Furthermore, the addition of an attention-based feature alignment module in this invention can correlate temporal and spatial information, thereby enhancing the assessment of low-quality regions around the subject and ultimately providing superior results. The proposed model has been evaluated on various benchmark datasets, demonstrating superior performance compared to existing methods, proving the effectiveness of our combined approach.

[0049] The advantages of this invention are as follows: This invention develops a spatiotemporally aware alignment network for multi-attribute feature alignment in video motion quality assessment, addressing the information bias problem in the fusion process of spatiotemporal features. This invention implements an effective attention-based feature alignment mechanism that can correlate spatial and temporal series information, particularly emphasizing low-quality regions around the main object. Compared to similar methods, this invention utilizes very sparse image frames to extract spatial features and dense video frames with very low spatial resolution to extract motion features, resulting in lower computational complexity. Comparative experiments show that this invention achieves higher accuracy than similar methods while reducing parameter computation by 60.7% and increasing computation speed to 6.929 seconds, making it more suitable for deployment on edge devices and practical applications. Attached Figure Description

[0050] Figure 1 This is a flowchart of the present invention.

[0051] Figure 2 This is a flowchart of the video motion evaluation network in this invention.

[0052] Figure 3 This is the attention feature map of the moving subject in this invention. Detailed Implementation

[0053] The specific implementation process of the present invention will be described below with reference to the accompanying drawings.

[0054] Invention flowchart as follows Figure 1 As shown.

[0055] A video motion quality assessment method based on spatiotemporal domain awareness includes the following steps:

[0056] 1) Spatiotemporal segmentation is performed on the input video, dividing it into a set of video segments p and a set of keyframe segments x. The input video V is segmented into Nk segments contained in vectors The video consists of consecutive segments. Each video segment is p. i Including N f Frame: p i ={f i，j Select a keyframe f in each video clip. i，1 To extract spatial features, use p i Motion features are extracted from all frames.

[0057] 2) Extract the video clip set p i The method of this invention obtains temporal motion features by analyzing multi-rate motion features and then aligning them through an interpolation module that matches fast and slow features. The method uses a pre-trained slow-fast motion recognition model, SLOW-FAST, to acquire motion features for each video segment. The SLOW-FAST model extracts slow-rate and block-rate features through Slow and Fast branches respectively, enabling the feature representation of the motion recognition network to effectively reflect the motion information of the subject in the video.

[0058] Therefore, given a video segment p i (i∈{1, 2, 3, ..., N) c Using an action recognition network, slow-rate and fast-rate features were obtained respectively. and By concatenating these features, we obtain the slow-rate feature set and the fast-rate feature set:

[0059]

[0060] Subsequently, the attention mechanism was applied to different motion rates X. slow X fast The input features are used to learn attention weights and perform a weighted summation of multiple motion features to obtain multiple rate-adaptively aligned motion features F. s F t :

[0061] W att =Softmax(ReLu(Conv1(X) slow )+Conv2(X fast (2)

[0062]

[0063] Where Conv1(·) and Conv2(·) are two two-dimensional convolutional kernels with a single kernel size, ReLU(·) and Softmax(·) are activation functions, and W att These are attention weights. To obtain data with the same size and sampling rate across the time dimension, we use a multi-rate interpolation method for interpolation and alignment. For each timestamp t, we perform interpolation and alignment on the processed features F.s F t Perform interpolation and alignment to obtain the alignment characteristics of this timestamp:

[0064]

[0065] in These are the features extracted by the SLOW-FAST network at video frame t, where s and f represent the slow path and fast path, respectively, and r i It is the temporal sampling rate of the i-th frame in the path. This represents the timestamp closest to t in the i-th frame. i (t) are the interpolation coefficients obtained through linear or nearest neighbor interpolation, as follows:

[0066]

[0067] Finally, the features aligned to the slow-rate path and the fast-rate path are fused according to a certain ratio α to obtain the final video motion feature representation:

[0068]

[0069] 3) Extract multi-scale spatial features from the keyframe fragment set, and then obtain aligned spatial features through a multi-scale feature alignment module. This invention uses a pre-trained ResNet residual neural network model to extract spatial features, obtaining feature representations with robust generalization ability to single-frame spatial resolution. Spatial domain convolutions of different sizes and depths can capture different types of semantic information. Previous research has shown that, compared to high-level features, low-level features are characterized by higher resolution, wider location, and more detailed information, but lower semantic content and higher noise levels. In contrast, high-level features provide strong semantic information with low resolution and poor detail awareness; they prioritize content-aware information and reflect global distortions in the video. Consider input frame p... i and stage characteristics X s We define X s As the multi-scale output of the convolutional neural network model CNN at stage s (s∈1,2,3,4):

[0070] X s =CNN s (X s-1 (7)

[0071] To effectively align features representing different semantics and scales, this invention proposes a multi-scale channel attention module. This module uses channel attention at multiple scales obtained by varying the spatial pool size to align features at different scales. To improve computational efficiency, we choose pointwise convolution to achieve local context aggregation, which only utilizes pointwise channel interactions at each spatial location. Given multi-scale features X = {X1, X2, X3, X4}, the output of the multi-scale channel attention module is defined as follows:

[0072]

[0073] Where L(X) and G(X) represent the local and global channel contexts used by the multi-scale channel attention module, respectively. Indicates broadcast addition. This represents element-wise multiplication, where σ is a sigmoid() activation function. Note that the weights L(X) and G(X) are defined as follows:

[0074]

[0075]

[0076] Among them W (1) W (2) W (3) and W (4) It consists of four convolutional layers with learnable weights, where δ represents the ReLU activation function. This is represented as batch normalization. Indicates global average pooling:

[0077]

[0078] 4) The aligned temporal motion features and spatial features are fused to obtain a video motion quality feature with spatiotemporal awareness, which is then fed into the quality regression module to obtain the video motion quality evaluation score. Using the feature extraction and feature alignment model in the above steps of this invention, a spatiotemporal awareness feature representation is obtained. Subsequently, a multilayer perceptron is used to map these features to the quality regression model to calculate the corresponding quality score.

[0079] Q final =MLP(Concat(F motion ,X′)) (12)

[0080] Q final The predicted score is represented by the input of the final quality regression layer of the quality regression model, which combines temporal motion and spatial information. The input includes two main information streams that characterize human perception of video content.

[0081] During the training phase, the method of this invention uses two loss functions to optimize the performance of the proposed model, namely the mean absolute error loss (MAE) and the RANK loss, defined as follows:

[0082]

[0083]

[0084] Where i and j are the indices that identify the videos in the batch, and N is the number of videos in the batch. The definition is as follows:

[0085]

[0086] Subsequently, L RANK The calculation is as follows:

[0087]

[0088] Finally, summing the results yields the loss function:

[0089] L = L MAE +λ·L RANK (17)

[0090] Where λ is a hyperparameter that balances MAE and RANK loss.

[0091] The final test results of the technical solution in this application on four open-source datasets are shown in Table 1:

[0092] Table 1 shows the median SRCC and PLCC values ​​tested on the CVD2014, KoNViD-1k, LIVE-VQC, and YouTube-UGC datasets under a single training setting. The best-performing model for each metric is highlighted in bold.

[0093]

[0094] The method of this invention uses two commonly used metrics to test the performance of the model: the Spearman correlation coefficient (SRCC) to evaluate the monotonicity of the experimental results, and the Pearson linear correlation coefficient (PLCC) to evaluate the predictive accuracy of the experimental results. Figure 3 This application presents an attention feature map of a moving subject using the method of this invention, as shown in Table 1 and... Figure 3It can be seen that the method of this invention is superior to other existing methods, including VSFA (Reference: Li, D., Jiang, T., Jiang, M.: Quality assessment of in-the-wild videos. In: Proceedings of the 27th ACM International Conference on Multimedia.pp.2351–235, 2019), VIDEVAL (Reference: Tu, Z., Wang, Y., Birkbeck, N., Adsumilli, B., Bovik, AC: Ugc-vqa: Benchmarking blind video quality assessment for user-generated content. IEEE Transactions on Image Processing 30, 4449–4464, 2021), and GSTVQA (Reference: Chen, B., Zhu, L., Li, G., Lu, F., Fan, H., Wang, S.: Learning generalized spatial temporal deep feature representation for no-reference video quality assessment. IEEE Transactions on Circuits and Systems for Video Technology). 32(4),1903–1916,2021), BVQA (Reference Li, B., Zhang, W., Tian, ​​M., Zhai, G., Wang,

Claims

1. A spatio-temporal domain-aware video motion quality assessment method, comprising the following steps: 1) performing spatio-temporal segmentation on the input video to divide it into a video segment set p and a key frame segment set x; 2) extracting multi-rate motion features of the video segment set p, and then obtaining time domain motion features through an interpolation alignment module of fast features and slow features; 3) extracting multi-scale spatial features of the key frame segment set x, and then obtaining aligned spatial features through a multi-scale feature alignment module; 4) fusing the aligned time domain motion features and spatial features to obtain a video motion quality feature with spatio-temporal perception characteristics, and then entering a quality regression module to obtain a video motion quality assessment score; Step 1) specifically includes: performing spatiotemporal segmentation on the input video, dividing it into a set of video segments p and a set of keyframe segments x; the input video V is segmented into N... k The vectors contained in The continuous segments in the video; each video segment p i Including N f Frame: p i ={f i,j }; Select a keyframe f in each video segment i,1 To extract spatial features, use p i Motion features are extracted from all frames. The step 2) specifically comprises: extracting a multi-rate motion feature of the video clip set p i , and then obtaining a time-domain motion feature through an interpolation alignment module of fast features and slow features; using a pre-trained slow-fast action recognition model SLOW-FAST to obtain an action feature of each video clip; the SLOW-FAST model extracts slow-rate feature and block-rate feature information through Slow and Fast branches respectively, so that the feature representation of the action recognition network can effectively reflect the motion information of the subject in the video; Thus, given a video segment p i (i e {1,2,3,...,N c}), slow and fast rate features are obtained using action recognition networks, respectively and By concatenating these features, slow and fast rate feature sets are obtained: Subsequently, the attention mechanism is applied to different motion rates X slow , X fast input features, by learning attention weights and performing weighted summation of multiple motion features, to obtain a variety of rate adaptive alignment motion features F s , F t : W att = Softmax(ReLu(Conl(X slow )+Conv2(X fast ))) (2) where Conv1(·) and Conv2(·) are two two-dimensional convolution kernels with a single kernel size, ReLu(·) and Softmax(·) are activation functions, W att are attention weights; in order to obtain data with the same size and sampling rate in the time dimension, interpolation and alignment are performed using a multi-rate interpolation method; for each timestamp t, interpolation and alignment are performed on the processed feature F s , F t to obtain the aligned feature of the timestamp: where is the feature extracted by the SLOW-FAST network at video frame t, s and f represent the slow and fast paths, respectively, r i is the temporal sampling rate of the path at the ith frame, denotes the timestamp in the ith frame closest to t; v i (t) is the interpolation coefficient obtained by linear or nearest-neighbor interpolation as follows: Finally, the features aligned by the slow rate path and the fast rate path are fused according to a certain proportion α to obtain the final video motion feature representation: The step 3) specifically comprises: extracting multi-scale spatial features of the key frame segment set, and then obtaining aligned spatial features through a multi-scale feature alignment module; using a pre-trained residual neural network model ResNet to extract spatial features, so as to obtain a feature representation which has robust generalization capability to single-frame spatial resolution; spatial domain convolution of different sizes and depths can capture different types of semantic information, and the input frame p i and stage feature X s , define X s as the multi-scale output of the convolutional neural network model CNN in the stage s (s e 1, 2, 3, 4): X s = CNN s (X s-1 ) (7) The multi-scale channel attention module uses multiple scales of channel attention obtained by changing the size of the spatial pool to align features of different scales; in order to improve the calculation efficiency, pointwise convolution is selected to realize local context aggregation, which only uses pointwise channel interaction at each spatial position; given a multi-scale feature X = {X1, X2, X3, X4}, the output of the multi-scale channel attention module is defined as follows: where L(X) and G(X) represent the local and global channel contexts used by the multi-scale channel attention module, respectively, denotes a broadcast addition, denotes an element-wise multiplication, and σ is a sigmoid() activation function; note that the weights L(X) and G(X) are defined as follows: where W (1) , W (2) , W (3) , and W (4) are four convolutional layers of learnable weights, and δ denotes a ReLU activation function, denotes batch normalization, denotes global average pooling:

2. The video motion quality evaluation method based on spatiotemporal domain awareness as described in claim 1, characterized in that: Step 4) specifically includes: fusing the aligned time domain motion features and spatial features to obtain a video motion quality feature with spatio-temporal perception characteristics, and then entering a quality regression module to obtain a video motion quality assessment score; using a feature extraction and feature alignment model to obtain a spatio-temporal perception feature representation, and then using a multi-layer perceptron to map these features to a quality regression model to calculate the corresponding quality score: Q final = MLP(Concat(F motion ,X′)) (12) where Q final denotes the prediction score, by combining temporal motion and spatial information, the input of the final quality regression layer of the quality regression model includes two main information streams characterizing human perception of video content; In the training stage, two loss functions are used to optimize the performance of the proposed model, i.e. the mean absolute error loss MAE and the RANK loss defined as follows: where i and j are indices identifying videos in a batch, N is the number of videos in the batch, are defined as follows: Subsequently, L RANK The calculation is as follows: Finally, the loss function is summed as follows: L = L MAE + λ · L RANK (17) Where λ is a hyperparameter balancing the MAE and RANK losses.

Images