Video quality evaluation method and system based on space-frequency joint and time sequence interaction

Abstract

The application discloses a video quality evaluation method and system based on space-frequency joint and time sequence interaction, relates to the technical field of computer vision, and the method comprises the following steps: standardizing and data enhancing key frames of an immersive video; then constructing space and frequency domain double-path features through learnable Gabor convolution and two-dimensional Fourier change and logarithmic amplitude spectrum respectively; then realizing space-frequency dynamic fusion through bidirectional Cross-Attention to generate joint representation; then inputting a Transformer architecture to build inter-frame time sequence dynamics and global dependence to obtain time sequence correlation features; finally outputting frame quality scores through learnable weighted MLP regression and aggregating the frame quality scores into overall quality scores. Through space-frequency joint modeling and time sequence interaction learning, the application realizes high-fidelity no-reference quality evaluation of distortion of an immersive video.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a video quality evaluation method and system based on spatial-frequency joint and temporal interaction. Background Technology

[0002] With the widespread application of virtual reality and immersive video technology, people have placed higher demands on subjective visual experience during the viewing of videos in multiple scenes and from multiple viewpoints. Immersive videos have the characteristics of high resolution, wide viewing angle and complex spatiotemporal structure compared with traditional videos. However, they are easily affected by a variety of degradation factors during the acquisition, encoding, transmission and rendering process, such as compression artifacts, noise pollution, blurring and defocusing, color drift, motion inconsistency and frame rate fluctuations. These will all lead to a significant decline in the subjective perceived quality of users. Existing video quality evaluation methods are mainly divided into: (1) Full Reference (FR) method: relies on the original video as a reference, but it is difficult to obtain in actual scenes; (2) Simplified Reference (RR) method: only uses some feature information, still requires some reference signals; (3) No Reference (NR) method: completely relies on the distorted video itself, which is the focus and difficulty of immersive video quality perception research.

[0003] Most existing no-reference models are based on spatial domain convolutional networks, focusing only on the structural and textural information of images while ignoring the energy distribution in the frequency domain and the dynamic relationships between frames. However, the degradation of immersive videos often manifests simultaneously in spatial texture degradation, spectral energy perturbations, and temporal continuity disruption. Therefore, constructing a unified modeling framework that integrates spatial, frequency, and temporal information has become a key issue in improving the accuracy of immersive video quality assessment. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a video quality assessment method and system based on spatial-frequency joint and temporal interaction. Through spatial-frequency dual-path feature extraction, bidirectional Cross-Attention fusion, Transformer temporal modeling, and learnable weighted regression, it achieves high-precision, highly robust, and human-perceptual-compliant immersive video quality assessment under no-reference conditions.

[0005] On the one hand, video quality assessment methods based on space-frequency joint and temporal interaction include:

[0006] S1, acquire the immersive video sequence, perform keyframe extraction, size normalization and color normalization on the immersive video sequence to obtain standardized keyframes, and perform data augmentation on the standardized keyframes to obtain enhanced keyframes.

[0007] S2, the enhanced keyframe is input into the spatial feature extraction branch, and multi-directional texture and edge structure information in the enhanced keyframe is extracted through learnable Gabor convolution kernels to obtain the spatial features of the keyframe. Multi-stage residual network is used to model the spatial features of the keyframe at multiple scales to obtain a spatial feature representation fused from deep and shallow layers.

[0008] S3 inputs the enhanced keyframe into the frequency domain feature extraction branch, performs a two-dimensional fast Fourier transform on the enhanced keyframe to obtain the logarithmic amplitude spectrum, captures the energy distribution and periodicity features of the keyframe through the logarithmic amplitude spectrum, and then inputs the logarithmic amplitude spectrum into the convolutional network to extract the spectral energy and noise structure features of the keyframe to obtain the frequency domain features of the keyframe.

[0009] S4. Input the spatial feature representation and frequency domain features fused from the deep and shallow layers into the space-frequency interaction module. The dynamic fusion of space-frequency features is achieved through the bidirectional Cross-Attention mechanism to obtain the joint space-frequency feature representation.

[0010] S5. The joint spatial frequency features are input into a multi-head self-attention network based on the Transformer architecture in chronological order. By modeling the dynamic changes and global temporal dependencies of the joint spatial frequency features, inter-frame temporal correlation features are obtained.

[0011] S6. Input the inter-frame temporal correlation features into the quality regression module. After predicting the quality score of each frame of video through the multilayer perceptron (MLP), the quality scores of all frames are weighted and averaged based on the learnable weights to obtain the overall video quality score.

[0012] Furthermore, in S2, the spatial feature extraction branch includes learnable Gabor convolution kernels;

[0013] The learnable Gabor convolution kernel is defined as follows:

[0014] ;

[0015] in, and Represents pixel coordinates in the spatial domain; exp() represents the exponential function; cos() represents the cosine function. , Represents the coordinate components after Gabor coordinate rotation; Indicates wavelength; Indicates the direction parameter; Indicates phase shift; Indicates the width of the Gaussian envelope; Indicates the aspect ratio of a space.

[0016] Furthermore, in S3, a two-dimensional fast Fourier transform is performed on the enhanced keyframes to obtain the logarithmic amplitude spectrum, calculated as follows:

[0017] ;

[0018] ;

[0019] in, and This represents the row and column index of the input frame in the spatial domain; Indicates the input frame; Indicates the height of the frame; Indicates the width of the frame; Represents frequency domain coordinates; Represents the spectrum in complex form; This represents the logarithmic amplitude spectrum, and log(.) represents the logarithmic function.

[0020] Furthermore, in S4, a bidirectional Cross-Attention mechanism is used to dynamically fuse spatial-frequency features, resulting in a joint spatial-frequency feature representation. The calculation process is as follows:

[0021] ;

[0022] ;

[0023] Where Softmax(.) represents the standardized exponential function; Presentation layer normalization operation; and This represents the bias term used in the attention calculation between the source and target locations. and This represents the scaling factor between the source and target locations; This represents the noise term from the spatial domain to the frequency domain; This represents the noise term from the frequency domain to the spatial domain; This indicates a discard operation. For discard rate; and A query mapping matrix representing spatial and frequency domain features; and A key mapping matrix representing spatial and frequency domain features; and The value mapping matrix representing spatial domain features and frequency domain features; and These represent the bidirectional attention weight matrix, with subscripts... Indicates airspace, subscript Represents the frequency domain. Indicates the embedding dimension; This represents the joint space-frequency characteristic representation; These are learnable fusion weights.

[0024] Furthermore, in S5, the joint spatial-frequency features are input sequentially into a multi-head self-attention network based on the Transformer architecture. By modeling the dynamic changes and global temporal dependencies of the joint spatial-frequency features, inter-frame temporal correlation features are obtained, and the calculation formula is as follows:

[0025] ;

[0026] in, Indicates the first Layer Self-attention calculation for each head; MHA(.) represents multi-head self-attention operation; Concat(.) represents concatenation operation; Presentation layer normalization operation; This indicates a discard operation. For discard rate; MHA ( ) represents the inter-frame temporal correlation feature; This represents the feature matrix input into the multi-head self-attention network; Indicates the first Layer Learnable bias terms for each head; Indicates the first Layer Scaling factor for each head; Indicates the first Layer Noise items for each head; This represents a globally learnable bias term; , and Indicates the first The query, key, and value matrix of each attention head; Indicates the number of heads of attention; Represents a linear output mapping matrix; This represents the dimension of the key vector.

[0027] Furthermore, in S6, after predicting the quality score of each video frame using a multilayer perceptron (MLP), the quality scores are weighted and averaged based on learnable weights to obtain the overall video quality score. The calculation formula is as follows:

[0028] ;

[0029] ;

[0030] in, This represents the fusion feature of frame t; Indicates frame-level score; , , and Indicates trainable parameters; Represents a non-linear activation function; Represents the learnable weights; The overall quality score for the video; Calculate the weight vector; This represents the total number of video frames.

[0031] On the other hand, a video quality assessment system based on space-frequency joint and temporal interaction includes:

[0032] The keyframe enhancement module is used to acquire immersive video sequences, extract keyframes, normalize their size, and normalize their color to obtain standardized keyframes. The standardized keyframes are then subjected to brightness adjustment, random cropping, and horizontal flipping to obtain enhanced keyframes.

[0033] The spatial feature representation acquisition module is used to input the enhanced keyframe into the spatial feature extraction branch, extract multi-directional texture and edge structure information in the enhanced keyframe through learnable Gabor convolution kernels, obtain the spatial features of the keyframe, and use a multi-stage residual network to perform multi-scale modeling of the spatial features of the keyframe to obtain a deep and shallow layer fused spatial feature representation.

[0034] The frequency domain feature acquisition module is used to input the enhanced keyframe into the frequency domain feature extraction branch, perform a two-dimensional fast Fourier transform on the enhanced keyframe to obtain the logarithmic amplitude spectrum, capture the energy distribution and periodicity features of the keyframe through the logarithmic amplitude spectrum, and then input the logarithmic amplitude spectrum into the convolutional network to extract the spectral energy and noise structure features of the keyframe to obtain the frequency domain features of the keyframe.

[0035] The spatial-frequency feature fusion module is used to input the spatial feature representations fused from the deep and shallow layers and the frequency domain features into the spatial-frequency interaction module. The dynamic fusion of spatial-frequency features is achieved through a bidirectional Cross-Attention mechanism to obtain a joint spatial-frequency feature representation.

[0036] The inter-frame temporal correlation feature acquisition module is used to input the joint spatial frequency features into a multi-head self-attention network based on the Transformer architecture in chronological order. By modeling the dynamic changes and global temporal dependencies of the joint spatial frequency features, the inter-frame temporal correlation features are obtained.

[0037] The quality scoring module is used to input the inter-frame temporal correlation features into the quality regression module. After predicting the quality score of each frame of video through a multilayer perceptron (MLP), the quality scores are weighted and averaged based on learnable weights to obtain the overall video quality score.

[0038] The present invention adopts the above technical solution and has the following beneficial effects:

[0039] (1) This invention extracts spatial texture edge features by learning Gabor convolution and multi-stage residual network, extracts frequency energy and noise features by combining two-dimensional FFT logarithmic amplitude spectrum and spectral convolution, and uses bidirectional Cross-Attention to realize spatial-frequency dynamic interaction fusion, thereby improving the comprehensiveness and fine-grained representation ability of quality assessment.

[0040] (2) This invention uses a multi-head self-attention mechanism based on the Transformer architecture to perform temporal modeling of joint spatial frequency features, capture long-range inter-frame dependencies and dynamic change patterns, and achieve effective discrimination of motion blur, temporal distortion and cross-frame artifacts;

[0041] (3) This invention directly maps the spatial-frequency joint features and temporal-related features to the quality score and introduces a learnable weighted averaging strategy to achieve adaptive quality prediction that does not depend on the reference video. It takes into account high accuracy, strong robustness and real-time applicability, and significantly improves the consistency with human subjective perception. Attached Figure Description

[0042] Figure 1 This is a flowchart of a video quality evaluation method based on space-frequency joint and temporal interaction according to an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the overall principle of the video quality evaluation method according to an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the space-frequency interaction attention module according to an embodiment of the present invention;

[0045] Figure 4 This is a diagram of a video quality evaluation system based on space-frequency joint and temporal interaction, according to an embodiment of the present invention. Detailed Implementation

[0046] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0047] like Figure 1 As shown, the present invention provides a video quality assessment method based on spatial-frequency joint and temporal interaction, comprising:

[0048] S1. Obtain the immersive video sequence, perform keyframe extraction, size normalization, and color normalization on the immersive video sequence to obtain standardized keyframes, and perform brightness adjustment, random cropping, and horizontal flipping on the standardized keyframes to obtain enhanced keyframes.

[0049] Specifically, in this embodiment, the original video sequence is obtained from the immersive video source:

[0050] ;

[0051] in, This represents the image in frame t. Given the total number of video frames, a keyframe extraction algorithm is used to select representative frames from each video segment, covering the main content and scene changes. Each frame image is subjected to size normalization, color standardization, and data augmentation processing, including brightness adjustment, random cropping, and horizontal flipping, to unify the input format and enhance the robustness of the model.

[0052] S2 inputs the enhanced keyframe into the spatial feature extraction branch, extracts multi-directional texture and edge structure information in the enhanced keyframe through learnable Gabor convolution kernels, obtains the spatial features of the keyframe, and uses a multi-stage residual network to perform multi-scale modeling of the spatial features of the keyframe to obtain a spatial feature representation fused from deep and shallow layers.

[0053] Specifically, the spatial feature extraction branch includes learnable Gabor convolution kernels;

[0054] The learnable Gabor convolution kernel is defined as follows:

[0055] ;

[0056] in, and Represents pixel coordinates in the spatial domain; exp() represents the exponential function; cos() represents the cosine function. , Represents the coordinate components after Gabor coordinate rotation; Indicates wavelength; Indicates the direction parameter; Indicates phase shift; Indicates the width of the Gaussian envelope; Indicates the aspect ratio of a space.

[0057] S3 inputs the enhanced keyframe into the frequency domain feature extraction branch, performs a two-dimensional fast Fourier transform on the enhanced keyframe to obtain the logarithmic amplitude spectrum, captures the energy distribution and periodicity features of the keyframe through the logarithmic amplitude spectrum, and then inputs the logarithmic amplitude spectrum into the convolutional network to extract the spectral energy and noise structure features of the keyframe to obtain the frequency domain features of the keyframe.

[0058] Specifically, a two-dimensional fast Fourier transform is performed on the enhanced keyframes to obtain the logarithmic amplitude spectrum, calculated as follows:

[0059] ;

[0060] ;

[0061] in, and This represents the row and column index of the input frame in the spatial domain; Indicates the input frame; Indicates the height of the frame; Indicates the width of the frame; Represents frequency domain coordinates; Represents the spectrum in complex form; This represents the logarithmic amplitude spectrum, and log(.) represents the logarithmic function.

[0062] Specifically, in this embodiment, high-dimensional semantic features are extracted through a multi-layer residual network to obtain a spatial feature set:

[0063] ;

[0064] in, Represents the set of spatial features. This represents the spatial features output by the spatial feature extraction function at the k-th stage. Each frame of the image is then input into the frequency domain feature extraction module, which performs a two-dimensional fast Fourier transform (FFT) on each frame of the image.

[0065] S4 inputs the spatial feature representation and frequency domain features fused from the deep and shallow layers into the space-frequency interaction module, and achieves dynamic fusion of space-frequency features through a bidirectional Cross-Attention mechanism to obtain a joint space-frequency feature representation.

[0066] Specifically, a bidirectional Cross-Attention mechanism is used to dynamically fuse spatial-frequency features, resulting in a joint spatial-frequency feature representation. The calculation process is as follows:

[0067] ;

[0068] ;

[0069] Where Softmax(.) represents the standardized exponential function; Presentation layer normalization operation; and This represents the bias term used in the attention calculation between the source and target locations. and This represents the scaling factor between the source and target locations; This represents the noise term from the spatial domain to the frequency domain; This represents the noise term from the frequency domain to the spatial domain; This indicates a discard operation. For discard rate; and A query mapping matrix representing spatial and frequency domain features; and A key mapping matrix representing spatial and frequency domain features; and The value mapping matrix representing spatial domain features and frequency domain features; and These represent the bidirectional attention weight matrix, with subscripts... Indicates airspace, subscript Represents the frequency domain. Indicates the embedding dimension; This represents the joint space-frequency characteristic representation; These are learnable fusion weights.

[0070] Specifically, in this embodiment, high-frequency and low-frequency mask decomposition is used to extract high-frequency details and low-frequency structural information of the image, respectively:

[0071] ;

[0072] in, Indicates high-frequency details, This indicates a low-frequency structure.

[0073] Frequency domain feature maps are obtained through convolutional layers:

[0074] ;

[0075] in, A function representing a convolution operation.

[0076] Furthermore, spatial characteristics Frequency domain characteristics Input the spatial frequency interactive attention module, and construct a query, key, and value matrix through linear mapping:

[0077] ;

[0078] in, , A query mapping matrix representing spatial and frequency domain features. , Key mapping matrix representing spatial and frequency domain features. , Value mapping matrix representing spatial and frequency domain features. , , The weight matrix representing the query, key, and value of spatial domain features. , , A weight matrix representing the query, key, and value features in the frequency domain.

[0079] S5 inputs the joint spatial frequency features into a multi-head self-attention network based on the Transformer architecture in chronological order. By modeling the dynamic changes and global temporal dependencies of the joint spatial frequency features, inter-frame temporal correlation features are obtained.

[0080] Specifically, the joint spatial-frequency features are input sequentially into a multi-head self-attention network based on the Transformer architecture. By modeling the dynamic changes and global temporal dependencies of the joint spatial-frequency features, inter-frame temporal correlation features are obtained. The calculation formula is as follows:

[0081] ;

[0082] in, Indicates the first Layer Self-attention calculation for each head; MHA(.) represents multi-head self-attention operation; Concat(.) represents concatenation operation; Presentation layer normalization operation; This indicates a discard operation. For discard rate; MHA ( ) represents the inter-frame temporal correlation feature; This represents the feature matrix input into the multi-head self-attention network; Indicates the first Layer Learnable bias terms for each head; Indicates the first Layer Scaling factor for each head; Indicates the first Layer Noise items for each head; This represents a globally learnable bias term; , and Indicates the first The query, key, and value matrix of each attention head; Indicates the number of heads of attention; Represents a linear output mapping matrix; This represents the dimension of the key vector.

[0083] S6. Input the inter-frame temporal correlation features into the quality regression module. After predicting the quality score of each video frame through a multilayer perceptron (MLP), the quality scores are weighted and averaged based on learnable weights to obtain the overall video quality score.

[0084] Specifically, after predicting the quality score of each frame of the video using a multilayer perceptron (MLP), the quality scores are weighted and averaged based on learnable weights to obtain the overall video quality score. The calculation formula is as follows:

[0085] ;

[0086] ;

[0087] in, This represents the fusion feature of frame t; Indicates frame-level score; , , and Indicates trainable parameters; Represents a non-linear activation function; Represents the learnable weights; The overall quality score for the video; Calculate the weight vector; This represents the total number of video frames.

[0088] Specifically, Figure 2 This paper demonstrates the overall architecture of the immersive video quality assessment model proposed in this invention, covering the complete process from keyframe extraction, spatial-frequency dual-path feature extraction, spatial-frequency interactive fusion, to temporal modeling and quality regression. Specifically, the spatial branch uses learnable Gabor convolutions and a multi-stage residual network to extract texture and structural information, while the frequency branch obtains the logarithmic amplitude spectrum through two-dimensional FFT and combines it with a convolutional network to mine energy distribution and noise features. After dynamic fusion via bidirectional Cross-Attention, the two are input into a Transformer-based temporal modeling module, and finally, MLP regression outputs the overall video quality score, achieving a comprehensive, robust, and human-perceptually consistent quality assessment of multiple types of distortion. Figure 3 Detailed description Figure 2 The spatial-frequency interaction module focuses on the design and implementation of a bidirectional Cross-Attention mechanism. This module maps spatial and frequency domain features into query, key, and value vectors, respectively. Through attention calculations in both spatial-frequency and frequency-spatial directions, it achieves cross-domain guidance and complementary enhancement between features. Multi-head parallel processing further improves the modeling capability. The fused joint features retain the structural details of spatial domain while incorporating the statistical properties of frequency domain energy, significantly enhancing the model's accuracy and generalization ability in discriminating complex distortions such as blur, noise, and compression artifacts.

[0089] Specifically, the main objective of this invention is to provide an immersive video quality assessment method based on spatial-frequency joint and temporal interactive modeling. By organically fusing spatial (spatial structure), frequency (energy distribution), and temporal (dynamic dependence) information, an end-to-end no-reference video quality prediction model is constructed. This enables automated, high-precision, and interpretable evaluation of immersive video quality without the need for a reference video. Compared with existing technologies, this invention effectively overcomes the problems of insufficient spectral information, inadequate temporal perception, and decreased accuracy in multi-distortion scenarios, achieving consistency between perceived video quality and subjective human ratings.

[0090] Specifically, in this embodiment, the proposed immersive video quality assessment model based on joint space-frequency and temporal interaction modeling is built using PyTorch and tested with an NVIDIA RTX A6000 GPU. The minimum resolution size of keyframes is adjusted to 520 while maintaining the original aspect ratio. During training, keyframes are randomly cropped to 448*448. For video blocks, the resolution is adjusted to 224*224 during training, the batch size is 64, the number of training epochs is 100, the initial learning rate is set to 0.00001, and the Adam optimizer is used for training. The experimental datasets are public datasets such as LIVE-360, CVIQ, and VQA-ODV. 80% of the datasets are used for training, and the remaining 20% ​​are used for testing. Spearman Rank Correlation Coefficient (SROCC), Pearson Linear Correlation Coefficient (PLCC), and Root Mean Square Error (RMSE) are selected to evaluate the performance of the above model.

[0091] like Figure 4 As shown, this embodiment also discloses a video quality evaluation system based on space-frequency joint and temporal interaction, including:

[0092] The keyframe enhancement module 41 is used to acquire an immersive video sequence, perform keyframe extraction, size normalization and color normalization on the immersive video sequence to obtain standardized keyframes, and perform brightness adjustment, random cropping and horizontal flipping on the standardized keyframes to obtain enhanced keyframes.

[0093] The spatial feature representation acquisition module 42 is used to input the enhanced key frame into the spatial feature extraction branch, extract multi-directional texture and edge structure information in the enhanced key frame through learnable Gabor convolution kernels, obtain the spatial features of the key frame, and use a multi-stage residual network to perform multi-scale modeling of the spatial features of the key frame to obtain a deep and shallow layer fused spatial feature representation.

[0094] The frequency domain feature acquisition module 43 is used to input the enhanced key frame into the frequency domain feature extraction branch, perform a two-dimensional fast Fourier transform on the enhanced key frame to obtain the logarithmic amplitude spectrum, capture the energy distribution and periodicity features of the key frame through the logarithmic amplitude spectrum, and then input the logarithmic amplitude spectrum into the convolutional network to extract the spectral energy and noise structure features of the key frame to obtain the frequency domain features of the key frame.

[0095] The spatial-frequency feature fusion module 44 is used to input the spatial feature representations and frequency domain features fused from the deep and shallow layers into the spatial-frequency interaction module, and to achieve dynamic fusion of spatial-frequency features through a bidirectional Cross-Attention mechanism to obtain a joint spatial-frequency feature representation.

[0096] The inter-frame temporal correlation feature acquisition module 45 is used to input the joint spatial frequency features into the multi-head self-attention network based on the Transformer architecture in chronological order, and to obtain the inter-frame temporal correlation features by modeling the dynamic changes and global temporal dependencies of the joint spatial frequency features.

[0097] The quality scoring module 46 is used to input the inter-frame temporal correlation features into the quality regression module. After predicting the quality score of each video frame through the multilayer perceptron (MLP), the quality scores are weighted and averaged based on learnable weights to obtain the overall video quality score.

[0098] The specific implementation of the video quality evaluation system based on space-frequency joint and temporal interaction is the same as the video quality evaluation method based on space-frequency joint and temporal interaction, and will not be described again in this embodiment.

[0099] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A video quality assessment method based on spatial-frequency joint and temporal interaction, characterized in that, Includes the following steps: S1, acquire the immersive video sequence, perform keyframe extraction, size normalization and color normalization on the immersive video sequence to obtain standardized keyframes, and perform data augmentation on the standardized keyframes to obtain enhanced keyframes. S2, the enhanced keyframe is input into the spatial feature extraction branch, and multi-directional texture and edge structure information in the enhanced keyframe is extracted through learnable Gabor convolution kernels to obtain the spatial features of the keyframe. Multi-stage residual network is used to model the spatial features of the keyframe at multiple scales to obtain a spatial feature representation fused from deep and shallow layers. S3 inputs the enhanced keyframe into the frequency domain feature extraction branch, performs a two-dimensional fast Fourier transform on the enhanced keyframe to obtain the logarithmic amplitude spectrum, captures the energy distribution and periodicity features of the keyframe through the logarithmic amplitude spectrum, and then inputs the logarithmic amplitude spectrum into the convolutional network to extract the spectral energy and noise structure features of the keyframe to obtain the frequency domain features of the keyframe. S4. Input the spatial feature representation and frequency domain feature fused from the deep and shallow layers into the space-frequency interaction module. The dynamic fusion of space-frequency features is achieved through the bidirectional Cross-Attention mechanism to obtain the joint space-frequency feature representation. S5. The joint spatial frequency features are input into a multi-head self-attention network based on the Transformer architecture in chronological order. By modeling the dynamic changes and global temporal dependencies of the joint spatial frequency features, inter-frame temporal correlation features are obtained. S6. Input the inter-frame temporal correlation features into the quality regression module. After predicting the quality score of each frame of video through the multilayer perceptron (MLP), the quality scores of all frames are weighted and averaged based on the learnable weights to obtain the overall video quality score. The joint spatial-frequency feature representation is obtained by dynamically fusing spatial-frequency features through a bidirectional Cross-Attention mechanism. The calculation process is as follows: ； ； Where Softmax(.) represents the standardized exponential function; Presentation layer normalization operation; and This represents the bias term used in the attention calculation between the source and target locations. and This represents the scaling factor between the source and target locations; This represents the noise term from the spatial domain to the frequency domain; This represents the noise term from the frequency domain to the spatial domain. This indicates a discard operation. Discard rate; and A query mapping matrix representing spatial and frequency domain features; and A key mapping matrix representing spatial and frequency domain features; and The value mapping matrix represents the spatial domain features and the frequency domain features; and These represent the bidirectional attention weight matrix, with subscripts indicating their respective values. Indicates airspace, subscript Represents the frequency domain. Indicates the embedding dimension; This represents the joint space-frequency characteristic representation; These are learnable fusion weights; The joint spatial-frequency features are input sequentially into a multi-head self-attention network based on the Transformer architecture. By modeling the dynamic changes and global temporal dependencies of the joint spatial-frequency features, inter-frame temporal correlation features are obtained. The calculation formula is as follows: ； in, Indicates the first Layer Self-attention calculation for each head; MHA(.) represents multi-head self-attention operation; Concat(.) represents concatenation operation; Presentation layer normalization operation; This indicates a discard operation. For discard rate; MHA ( ) represents the inter-frame temporal correlation feature; This represents the feature matrix input into the multi-head self-attention network; Indicates the first Layer Learnable bias terms for each head; Indicates the first Layer Scaling factor for each head; Indicates the first Layer Noise items for each head; This represents a globally learnable bias term; , and Indicates the first The query, key, and value matrix of each header; Indicates the number of heads of attention; This represents a linear output mapping matrix; This represents the dimension of the key vector.

2. The video quality evaluation method based on spatial-frequency joint and temporal interaction according to claim 1, characterized in that, In S2, the spatial feature extraction branch includes learnable Gabor convolution kernels; The learnable Gabor convolution kernel is defined as follows: ； in, and Represents pixel coordinates in the spatial domain; exp() represents the exponential function; cos() represents the cosine function. , Represents the coordinate components after Gabor coordinate rotation; Indicates wavelength; Indicates the direction parameter; Indicates phase shift; Indicates the width of the Gaussian envelope; Indicates the aspect ratio of a space.

3. The video quality evaluation method based on space-frequency joint and temporal interaction according to claim 1, characterized in that, In S3, a two-dimensional fast Fourier transform is performed on the enhanced keyframes to obtain the logarithmic amplitude spectrum, calculated as follows: ； ； in, and This represents the row and column index of the input frame in the spatial domain; Indicates the input frame; Indicates the height of the frame; Indicates the width of the frame; Represents frequency domain coordinates; Represents the spectrum in complex form; This represents the logarithmic magnitude spectrum, and log(.) represents the logarithmic function. It represents the imaginary unit.

4. The video quality evaluation method based on spatial-frequency joint and temporal interaction according to claim 1, characterized in that, In S6, after predicting the quality score of each frame of video using a multilayer perceptron (MLP), the quality scores are weighted and averaged based on learnable weights to obtain the overall video quality score. The calculation formula is as follows: ； ； in, This represents the fusion feature of frame t; Indicates frame-level score; , , and Indicates trainable parameters; Represents a non-linear activation function; Represents the learnable weights; The overall quality score for the video; Calculate the weight vector; This represents the total number of video frames.

5. A video quality evaluation system based on space-frequency joint and temporal interaction, characterized in that, include: The keyframe enhancement module is used to acquire immersive video sequences, extract keyframes, normalize their size, and normalize their color to obtain standardized keyframes. The standardized keyframes are then subjected to brightness adjustment, random cropping, and horizontal flipping to obtain enhanced keyframes. The spatial feature representation acquisition module is used to input the enhanced keyframe into the spatial feature extraction branch, extract multi-directional texture and edge structure information in the enhanced keyframe through learnable Gabor convolution kernels, obtain the spatial features of the keyframe, and use a multi-stage residual network to perform multi-scale modeling of the spatial features of the keyframe to obtain a deep and shallow layer fused spatial feature representation. The frequency domain feature acquisition module is used to input the enhanced keyframe into the frequency domain feature extraction branch, perform a two-dimensional fast Fourier transform on the enhanced keyframe to obtain the logarithmic amplitude spectrum, capture the energy distribution and periodicity features of the keyframe through the logarithmic amplitude spectrum, and then input the logarithmic amplitude spectrum into the convolutional network to extract the spectral energy and noise structure features of the keyframe to obtain the frequency domain features of the keyframe. The spatial-frequency feature fusion module is used to input the spatial feature representations fused from the deep and shallow layers and the frequency domain features into the spatial-frequency interaction module. The dynamic fusion of spatial-frequency features is achieved through a bidirectional Cross-Attention mechanism to obtain a joint spatial-frequency feature representation. The inter-frame temporal correlation feature acquisition module is used to input the joint spatial frequency features into a multi-head self-attention network based on the Transformer architecture in chronological order. By modeling the dynamic changes and global temporal dependencies of the joint spatial frequency features, the inter-frame temporal correlation features are obtained. The quality scoring module is used to input the inter-frame temporal correlation features into the quality regression module. After predicting the quality score of each frame of video through a multilayer perceptron (MLP), the quality scores are weighted and averaged based on learnable weights to obtain the overall video quality score. The joint spatial-frequency feature representation is obtained by dynamically fusing spatial-frequency features through a bidirectional Cross-Attention mechanism. The calculation process is as follows: ； ； Where Softmax(.) represents the standardized exponential function; Presentation layer normalization operation; and This represents the bias term used in the attention calculation between the source and target locations. and This represents the scaling factor between the source and target locations; This represents the noise term from the spatial domain to the frequency domain; This represents the noise term from the frequency domain to the spatial domain. This indicates a discard operation. Discard rate; and A query mapping matrix representing spatial and frequency domain features; and A key mapping matrix representing spatial and frequency domain features; and The value mapping matrix represents the spatial domain features and the frequency domain features; and These represent the bidirectional attention weight matrix, with subscripts indicating their respective values. Indicates airspace, subscript Represents the frequency domain. Indicates the embedding dimension; This represents the joint space-frequency characteristic representation; These are learnable fusion weights; The joint spatial-frequency features are input sequentially into a multi-head self-attention network based on the Transformer architecture. By modeling the dynamic changes and global temporal dependencies of the joint spatial-frequency features, inter-frame temporal correlation features are obtained. The calculation formula is as follows: ； in, Indicates the first Layer Self-attention calculation for each head; MHA(.) represents multi-head self-attention operation; Concat(.) represents concatenation operation; Presentation layer normalization operation; This indicates a discard operation. For discard rate; MHA ( ) represents the inter-frame temporal correlation feature; This represents the feature matrix input into the multi-head self-attention network; Indicates the first Layer Learnable bias terms for each head; Indicates the first Layer Scaling factor for each head; Indicates the first Layer Noise items for each head; This represents a globally learnable bias term; , and Indicates the first The query, key, and value matrix of each header; Indicates the number of heads of attention; This represents a linear output mapping matrix; This represents the dimension of the key vector.

Images