Screen content video quality evaluation method and device based on frequency space complementarity and semantics

Abstract

The application discloses a screen content video quality evaluation method and device based on frequency-space complementation and semantics, and relates to the field of computer vision, and the method comprises the following steps: S1, extracting a video block and a key frame of a screen content video, inputting the key frame into a high-frequency structure texture information extraction branch to obtain high-frequency structure texture information; S2, inputting the key frame into a noise perception module to obtain noise perception features; S3, inputting the noise perception features into an adaptive time sequence embedding module to obtain noise semantic information; S4, splicing the high-frequency structure texture information and the noise semantic information, performing quality regression to obtain a quality score of a single key frame, and performing summation and averaging to obtain a spatial domain video quality score; S5, inputting the video block into a quality evaluation branch based on Fast-VQA to obtain a time sequence distortion perception degradation score; and S6, dynamically fusing the spatial domain video quality score and the time sequence distortion perception degradation score to obtain a final video quality score. The method realizes effective evaluation of the quality of a screen content video.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus for evaluating the quality of screen content video based on frequency-space complementarity and semantics. Background Technology

[0002] With the rapid development of artificial intelligence and digital multimedia technologies, and the widespread adoption of various portable devices, screen content video is increasingly used in digital multimedia applications (such as online education, video conferencing, webcasting, and game videos), providing users with a more flexible and free multimedia experience. Unlike natural video, screen content video is a mixture of images and computer-generated text / image areas. However, at different processing stages of screen content video (such as acquisition, transmission, and display), it is inevitably subject to various noise interferences, leading to varying degrees of degradation in perceived quality and severely impacting the user experience. Therefore, there is an urgent need to develop an effective video quality assessment method to optimize the visual quality of screen content video and improve the performance of related tasks. Summary of the Invention

[0003] To address the above issues, this invention proposes a method and apparatus for evaluating the quality of screen content video based on frequency-space complementarity and semantics. By introducing an adaptive temporal embedding module, it dynamically fuses temporal information with distortion semantic information. By introducing a quality evaluation branch based on Fast-VQA, it jointly utilizes high-frequency structural texture information, noise semantic information, and temporal distortion perception degradation scores to improve robustness and generalization ability under different scenarios and various distortion conditions. This effectively enhances the perception and evaluation accuracy of complex distortion situations in screen content video. While ensuring evaluation accuracy, it significantly improves computational efficiency and real-time performance, making it suitable for practical application scenarios.

[0004] On the one hand, the screen content video quality evaluation method based on frequency-space complementarity and semantics has the following specific steps:

[0005] S1, acquire screen content video, extract several video blocks and several key frames from the screen content video, input the key frames into the high-frequency structure texture information extraction branch, and obtain high-frequency structure texture information.

[0006] The high-frequency structural texture information extraction branch includes a pre-trained ResNet50 network, a multi-scale progressive channel attention module, and a frequency-space complementary attention module. The pre-trained ResNet50 network performs multi-stage feature extraction on keyframes, obtaining feature maps in four stages. The multi-scale progressive channel attention module uses a channel attention mechanism to perform cross-stage feature enhancement on the feature maps of each stage, outputting the enhanced features of each stage. The frequency-space complementary attention module uses a learnable filter to adaptively filter the enhanced features of each stage to obtain frequency domain features. Spatial features are obtained by enhancing the enhanced features through spatial attention. The frequency domain features, spatial features, and enhanced features of each stage are added element-wise to obtain the corresponding frequency-space attention-enhanced feature maps. The frequency-space attention-enhanced feature maps of each stage are pooled and concatenated to obtain the high-frequency structural texture information.

[0007] S2, input the keyframe into the noise perception module, match the distortion type and quality level in the keyframe to obtain the noise perception features;

[0008] S3, input the noise-perceived features into the adaptive temporal embedding module, match the temporal information of the noise-perceived features, and obtain the noise semantic information;

[0009] S4 concatenates high-frequency structural texture information and noise semantic information, and inputs it into the quality regression module to obtain the quality score of a single key frame. The quality scores of all key frames are summed and averaged to obtain the spatial video quality score.

[0010] S5, input the video block into the Fast-VQA-based quality evaluation branch to perform video quality evaluation and obtain the temporal distortion perceived degradation score;

[0011] S6 dynamically fuses the spatial video quality score and the temporal distortion perception degradation score to obtain the final video quality score, thus completing the screen content video quality evaluation based on frequency-space complementarity and semantics.

[0012] Preferably, the multi-scale progressive channel attention module is as follows:

[0013] For the feature map at each stage, the feature map output at the current stage is... and the feature map of the output of the next stage The dimensionality reduction and feature extraction are achieved through two independent global average pooling layers and 1×1 convolutional layers, respectively. and ; indicates as:

[0014] ;

[0015] ;

[0016] in, This represents a 1×1 convolution operation; Indicates global average pooling; express Features after global average pooling and 1×1 convolution; express Features after global average pooling and 1×1 convolution; This represents the feature map of the current stage i. The feature map represents the next stage;

[0017] Will and The concatenated feature maps are then subjected to 1×1 convolution and Sigmoid operations to obtain a multi-scale channel attention map A.

[0018] The multi-scale channel attention map A is arranged according to... and The dimensions are split into and and Dimensionally consistent attention graph and ;

[0019] feature map and Multiplying them together yields a weighted feature map. ; feature map and Multiplying them together yields a weighted feature map. ;

[0020] Use transposed convolutional layers on feature maps Upsample by 2x and adjust the number of channels to make its dimensions and sum Consistent, then with Element-by-element addition ;

[0021] right A Haar wavelet transform is performed to generate low-frequency components and detail components in the horizontal, vertical, and diagonal directions. The low-frequency components are preserved, and the number of channels in the feature map is adjusted by performing a 1×1 convolution on them. Batch normalization and ReLU activation are then applied to obtain enhanced features. .

[0022] Preferably, the frequency-space complementary attention module adaptively filters the enhanced features of each stage using a learnable filter to obtain frequency domain features; it then enhances the enhanced features using spatial attention to obtain spatial features; finally, it adds the frequency domain features, spatial features, and enhanced features of each stage element-by-element to obtain the corresponding frequency-space attention-adjusted feature map, as detailed below:

[0023] Enhanced features output by the multi-scale progressive channel attention module are obtained using a two-dimensional discrete Fourier transform. Transform to the frequency domain to obtain the frequency domain feature map;

[0024] Set mask and The mask A small square area is set at the center of the mask and assigned a value of 0, while the remaining areas are assigned a value of 1; the mask Set a square region with a side length of n at the center of the mask and assign it a value of 1, and assign a value of 0 to the rest of the region;

[0025] frequency domain feature map and Element-wise multiplication yields the low-frequency components. ; frequency domain feature map and Element-wise multiplication yields high-frequency components. ;

[0026] Using adaptive learnable global filters Perform filtering, then extract the high-frequency features. With the filtered Reintegration yields the spectrum after global learning and feature fusion. ; indicates as:

[0027] ;

[0028] Where N represents an adaptive, learnable global filter. This indicates element-wise addition; Indicates multiplication;

[0029] right The frequency domain features are obtained using the two-dimensional inverse discrete Fourier transform. ;

[0030] right Spatial attention is used to obtain spatial features. ; indicates as:

[0031] ;

[0032] in, Indicates global average pooling. This indicates max pooling. Represents the sigmoid function; This indicates a splicing operation. This represents a 3×3 convolution. Indicates the multiplication operation;

[0033] Will , and Element-wise addition yields the feature map after frequency-spatial attention. .

[0034] Preferably, the step of inputting keyframes into the noise perception module, matching the distortion type and quality level in the keyframes, and obtaining noise perception features is as follows:

[0035] Set a text template and encode the text template into text features using a text editor; the text template describes the distortion type and quality level of the video frame;

[0036] Use an image encoder to encode keyframes into image features;

[0037] The cosine similarity between text features and image features is calculated to obtain noise-perceived features.

[0038] Preferably, the step of inputting the noise-perceived features into the adaptive temporal embedding module, matching the temporal information of the noise-perceived features, and obtaining noise semantic information is as follows:

[0039] Set a text template and encode the text template into a text feature using a text editor; the text template describes the ordinal position of the video frame in the sequence of all video frames.

[0040] Use an image encoder to encode keyframes into image features;

[0041] Calculate the cosine similarity between text features and image features to obtain temporal information;

[0042] By fusing temporal information and noise perception features, noise semantic information is obtained.

[0043] Preferably, the step of fusing temporal information and noise-perceived features to obtain noise semantic information is expressed as follows:

[0044] ;

[0045] in, Represents semantic information about noise; This indicates the extracted time-series information; Indicates noise perception characteristics; This indicates element-wise addition. This represents a 1×1 convolution operation; Represents the sigmoid function; Indicates a fully connected layer; This indicates a splicing operation.

[0046] Preferably, the high-frequency structural texture information and noise semantic information are concatenated and input into the quality regression module to obtain the quality score of a single keyframe. The quality scores of all keyframes are then summed and averaged to obtain the spatial domain video quality score; expressed as:

[0047] ;

[0048] ;

[0049] in, This represents the quality score of a single keyframe. Indicates a fully connected layer. This indicates a splicing operation. Represents high-frequency structural texture information. Represents semantic information about noise; Indicates the number of keyframes. This represents the video quality score in the spatial domain.

[0050] Preferably, the step of dynamically fusing the spatial video quality score and the temporal distortion-perceived degradation score to obtain the final video quality score is as follows:

[0051] Video quality score in the spatial domain Perceived degradation score of temporal distortion Adaptive dynamic fusion is performed to obtain the final video quality score, denoted as . The calculation is as follows:

[0052] ;

[0053] in, This indicates the final video quality score. Indicates hyperparameters, This represents the video quality score in the spatial domain. This represents the temporal distortion perception degradation score.

[0054] On the other hand, the screen content video quality evaluation device based on frequency-space complementarity and semantics includes the following:

[0055] The high-frequency structural texture information acquisition module is used to acquire screen content video, extract several video blocks and several key frames from the screen content video, input the key frames into the high-frequency structural texture information extraction branch, and obtain high-frequency structural texture information.

[0056] The high-frequency structural texture information extraction branch includes a pre-trained ResNet50 network, a multi-scale progressive channel attention module, and a frequency-space complementary attention module. The pre-trained ResNet50 network performs multi-stage feature extraction on keyframes, obtaining feature maps in four stages. The multi-scale progressive channel attention module uses a channel attention mechanism to perform cross-stage feature enhancement on the feature maps of each stage, outputting the enhanced features of each stage. The frequency-space complementary attention module uses a learnable filter to adaptively filter the enhanced features of each stage to obtain frequency domain features. Spatial features are obtained by enhancing the enhanced features through spatial attention. The frequency domain features, spatial features, and enhanced features of each stage are added element-wise to obtain the corresponding frequency-space attention-enhanced feature maps. The frequency-space attention-enhanced feature maps of each stage are pooled and concatenated to obtain the high-frequency structural texture information.

[0057] The noise perception feature acquisition module is used to input keyframes into the noise perception module, match the distortion type and quality level in the keyframes, and obtain noise perception features.

[0058] The noise semantic information acquisition module is used to input noise perception features into the adaptive temporal embedding module, match the temporal information of the noise perception features, and obtain noise semantic information.

[0059] The spatial video quality score acquisition module is used to splice high-frequency structural texture information and noise semantic information, and input them into the quality regression module to obtain the quality score of a single key frame. The quality scores of all key frames are summed and averaged to obtain the spatial video quality score.

[0060] The temporal distortion perceived degradation score acquisition module is used to input video blocks into the Fast-VQA-based quality evaluation branch for video quality evaluation and obtain temporal distortion perceived degradation scores.

[0061] The video quality score acquisition module is used to dynamically fuse the spatial video quality score and the temporal distortion perception degradation score to obtain the final video quality score, thus completing the screen content video quality evaluation based on frequency-space complementarity and semantics.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] (1) This method combines high-frequency structural texture information, temporal information, and semantic information of distortion type and quality level. Through information fusion, it effectively improves the perception and evaluation accuracy of complex distortion of screen content video.

[0064] (2) The adaptive temporal embedding module of the present invention can dynamically fuse temporal information with distorted semantic information, thereby improving the robustness and generalization ability of the model in different scenarios and under multiple distortion conditions.

[0065] (3) This invention introduces a quality evaluation branch based on Fast-VQA on the basis of traditional quality regression, and obtains the final quality score through a dynamic fusion mechanism. While ensuring the accuracy of the evaluation, it significantly improves the computational efficiency and real-time performance, and is suitable for practical application scenarios. Attached Figure Description

[0066] The present invention will now be described in further detail with reference to the accompanying drawings;

[0067] Figure 1 This is a flowchart of a screen content video quality evaluation method based on frequency-space complementarity and semantics, according to an embodiment of the present invention.

[0068] Figure 2 This is a schematic diagram of the screen content video quality evaluation method based on frequency-space complementarity and semantics according to an embodiment of the present invention.

[0069] Figure 3 This is a schematic diagram of the multi-scale progressive channel attention module of the screen content video quality evaluation method based on frequency-space complementarity and semantics in an embodiment of the present invention.

[0070] Figure 4 This is a schematic diagram of the frequency-space complementary attention module in the screen content video quality evaluation method based on frequency-space complementarity and semantics according to an embodiment of the present invention.

[0071] Figure 5 This is a structural block diagram of a screen content video quality evaluation device based on frequency-space complementarity and semantics, according to an embodiment of the present invention. Detailed Implementation

[0072] The present invention will be further described below through specific embodiments.

[0073] like Figure 1 As shown, a screen content video quality assessment method based on frequency-space complementarity and semantics is presented. Figure 2 This is a schematic diagram of the method in this embodiment. The specific steps of this method are as follows:

[0074] S1, acquire screen content video, extract several video blocks and several key frames from the screen content video, input the key frames into the high-frequency structure texture information extraction branch, and obtain high-frequency structure texture information.

[0075] S2, input the keyframe into the noise perception module, match the distortion type and quality level in the keyframe to obtain the noise perception features;

[0076] S3, input the noise-perceived features into the adaptive temporal embedding module, match the temporal information of the noise-perceived features, and obtain the noise semantic information;

[0077] S4 concatenates high-frequency structural texture information and noise semantic information, and inputs it into the quality regression module to obtain the quality score of a single key frame. The quality scores of all key frames are summed and averaged to obtain the spatial video quality score.

[0078] S5, input the video block into the Fast-VQA-based quality evaluation branch to perform video quality evaluation and obtain the temporal distortion perceived degradation score;

[0079] S6 dynamically fuses the spatial video quality score and the temporal distortion perception degradation score to obtain the final video quality score, thus completing the screen content video quality evaluation based on frequency-space complementarity and semantics.

[0080] In a specific embodiment, S1 specifically includes:

[0081] Several key frames are input into the high-frequency structural texture information extraction branch, and ResNet50 is used to extract features from the key frames. The calculation formula is as follows:

[0082] ;

[0083] Where I represents a keyframe. This indicates that a pre-trained ResNet50 is used to extract features from keyframes. This represents the feature maps of ResNet50 at each stage of keyframe feature extraction. ,

[0084] The feature maps output from the four stages of the ResNet50 network are input into the multi-scale progressive channel attention module. The schematic diagram of the multi-scale progressive channel attention module is shown below. Figure 3 As shown, the specific operation is as follows: the currently output feature map... and the feature map of the output of the next stage After two independent global average pooling and 1×1 convolutional layers for dimensionality reduction and feature extraction, the following is obtained: and .

[0085] ;

[0086] ;

[0087] in, This represents a 1×1 convolution operation. Indicates global average pooling. express Features after global average pooling and 1×1 convolution. express Features after global average pooling and 1×1 convolution. The feature map representing the current stage. This represents the feature map for the next stage.

[0088] Will and The concatenated feature maps are then subjected to 1×1 convolution and Sigmoid operations to obtain a multi-scale channel attention map A.

[0089] ;

[0090] in, The feature map represents the next stage. express Feature map after global average pooling and 1×1 convolution This indicates a splicing operation. This represents the sigmoid function.

[0091] The multi-scale channel attention map A is arranged according to... and The dimensions are split into and and Dimensionally consistent attention graph and .

[0092] ;

[0093] Where A represents the multi-scale channel attention map. , They represent the sum and the sum derived from A. , A multi-scale channel attention map with consistent dimensions, where Split represents the splitting operation.

[0094] feature map and Each with the corresponding and Multiplying them together yields a weighted feature map. and .

[0095] ;

[0096] ;

[0097] in, This indicates a multiplication operation. and They represent and , and The feature map is obtained by multiplying the corresponding parts.

[0098] Use transposed convolutional layers on feature maps Upsample by 2x and adjust the number of channels to make its dimensions and sum Consistent, then with Element-by-element addition , represented as:

[0099] ;

[0100] Where UP represents the transposed convolutional layer. This indicates element-wise addition. express and The result of upsampling by 2 and then adding each element together.

[0101] Then on A Haar wavelet transform is performed to generate low-frequency components and detail components in the horizontal, vertical, and diagonal directions. The low-frequency components are preserved. The feature map channel number is adjusted by a 1×1 convolution, and batch normalization and ReLU activation are performed to obtain the output feature map. The calculation formula is as follows:

[0102] ;

[0103] ;

[0104] in, , , , , These represent the low-frequency components and the high-frequency components in the vertical, horizontal, and diagonal directions, respectively. `norm` represents batch normalization, and `ReLU` represents the ReLU function activation. The feature map representing the i-th stage The feature map obtained after passing through multi-scale progressive channel attention blocks. This indicates a convolution operation.

[0105] For feature maps Input the frequency-space complementary attention module. The schematic diagram of the frequency-space complementary attention module is shown below. Figure 4 As shown, the specific operation is as follows: use the two-dimensional discrete Fourier transform to convert to the frequency domain, and obtain... .

[0106] ;

[0107] Where H represents the height of the feature map, W represents the width of the feature map, U and V represent the frequency components in the horizontal and vertical directions in the frequency domain, respectively, and x and y represent the position coordinates of the feature map in the horizontal and vertical directions, respectively. express The feature map is transformed into the frequency domain by the two-dimensional discrete Fourier transform.

[0108] Set mask and Regarding the mask A small square area is set in the center of the mask and assigned a value of 0, while the rest of the area is assigned a value of 1. For the mask... A square region with side length n is set at the center of the mask and assigned a value of 1, while the rest of the region is assigned a value of 0. Then... and Element-wise multiplication yields the low-frequency components. , and Element-wise multiplication yields high-frequency components. , represented as:

[0109] ;

[0110] ;

[0111] in, Indicates a low-frequency mask. Indicates a high-frequency mask. This indicates a multiplication operation. Indicates low-frequency characteristics. Indicates high-frequency characteristics.

[0112] against An adaptive, learnable global filter is applied for filtering. Then, the original high-frequency features are... With the filtered Reintegration yields the spectrum after global learning and feature fusion. This is represented as:

[0113] ;

[0114] Where N represents an adaptive, learnable global filter. This indicates element-wise addition. Represents the original high-frequency features With the filtered The re-integrated spectrum.

[0115] Subsequently, the frequency domain features were obtained using the two-dimensional inverse discrete Fourier transform. This is represented as:

[0116] ;

[0117] in, Indicates the spectrum Frequency domain features are obtained using the two-dimensional inverse discrete Fourier transform.

[0118] At the same time, Using spatial attention, Max pooling and global average pooling are performed separately. The features resulting from max pooling and global average pooling are concatenated, followed by a 3×3 convolution and a sigmoid operation. The results are then combined with... Multiply to obtain the characteristics .

[0119] ;

[0120] in, Indicates global average pooling. This indicates max pooling. This indicates a splicing operation. This represents a 3×3 convolution. This indicates a multiplication operation.

[0121] Will , , Adding each element together, we obtain the feature map after frequency-spatial attention, denoted as . .

[0122] ;

[0123] in, This indicates an element-wise addition operation. Indicates to The results of using spatial attention express and and The result of adding elements one by one.

[0124] Next, the feature maps of each stage will be generated. Pooling and splicing are performed to obtain high-frequency structural texture information, denoted as... .

[0125] In a specific embodiment, S2 specifically includes:

[0126] The keyframes are input into the noise perception module. The designed text template is {"A video frame has {n} distortion and has a quality of {c}"}, where n represents the distortion type, including "blur," "noise," and "color distortion," and c represents the distortion level, including five levels: "poor," "below average," "medium," "above average," and "good." The Clip text encoder encodes the text template into text features, and the Clip image encoder encodes the keyframes into image features. The cosine similarity between the text features and the image features is calculated as follows:

[0127] ;

[0128] Where I represents a keyframe. Representing text features, Representing image features, express Norm, This indicates noise perception characteristics.

[0129] In a specific embodiment, S3 specifically includes:

[0130] The noise-aware features are input into the adaptive temporal embedding module. The text template designed by the adaptive temporal embedding module is {“This is the {}th frame of the video.”}. The text template is encoded into text features using the Clip's text encoder, and the keyframes are encoded into image features using the Clip's image encoder. The cosine similarity between the text features and the image features is calculated to obtain the temporal information. The calculation is as follows:

[0131] ;

[0132] in, This indicates the extracted time-series information. Representing text features, Representing image features, express Norm.

[0133] After concatenating the temporal information and noise-aware features, the data is passed through a fully connected layer, followed by a sigmoid operation. The input is then a 1×1 convolution, and the output is element-wise added to the original temporal information and noise-aware features to obtain the noise semantic information, denoted as [missing information]. The calculation is as follows:

[0134] ;

[0135] in, This indicates element-wise addition. This represents a 1×1 convolution operation. This represents the sigmoid function. Indicates a fully connected layer. This indicates a splicing operation. This indicates the extracted time-series information. Indicates noise perception characteristics. Represents semantic information about noise.

[0136] In a specific embodiment, S4 specifically includes:

[0137] The high-frequency structural texture information and the noise semantic information are concatenated and input into the quality regression module to obtain the quality score of a single keyframe, denoted as . Specifically, the high-frequency structural texture information and noise semantic information of the stitched data are regressed into preliminary quality scores for single-frame keyframes through the fully connected layer of the quality regression module. The preliminary quality scores of the T keyframes are then summed and averaged to obtain the video quality score in the spatial domain, calculated as follows:

[0138] ;

[0139] ;

[0140] in, This represents the quality score of a single keyframe. Indicates a fully connected layer. This indicates a splicing operation. Represents high-frequency structural texture information. Represents semantic information about noise. Indicates the number of keyframes. This represents the video quality score in the spatial domain.

[0141] In a specific embodiment, S5 specifically includes:

[0142] The video block is input into the quality assessment branch based on Fast-VQA (Fast Video Quality Assessment) to obtain the temporal distortion perceived degradation score, denoted as... The calculation is as follows:

[0143] ;

[0144] Where FastVQA indicates the use of the FastVQA method. V represents the temporal distortion perceived degradation score, and V represents the video block.

[0145] In a specific embodiment, S6 specifically includes:

[0146] Video quality score in the spatial domain Perceived degradation score of temporal distortion Adaptive dynamic fusion is performed to obtain the final video quality score, denoted as . The calculation is as follows:

[0147] ;

[0148] in, This indicates the final video quality score. Indicates hyperparameters, This represents the video quality score in the spatial domain. This represents the temporal distortion perception degradation score.

[0149] like Figure 5 As shown, the present invention also discloses a screen content video quality evaluation device based on frequency-space complementarity and semantics, comprising:

[0150] The high-frequency structural texture information acquisition module 501 is used to acquire screen content video, extract several video blocks and several key frames from the screen content video, input the key frames into the high-frequency structural texture information extraction branch, and obtain high-frequency structural texture information.

[0151] The high-frequency structural texture information extraction branch includes a pre-trained ResNet50 network, a multi-scale progressive channel attention module, and a frequency-space complementary attention module. The pre-trained ResNet50 network performs multi-stage feature extraction on keyframes, obtaining feature maps in four stages. The multi-scale progressive channel attention module uses a channel attention mechanism to perform cross-stage feature enhancement on the feature maps of each stage, outputting the enhanced features of each stage. The frequency-space complementary attention module uses a learnable filter to adaptively filter the enhanced features of each stage to obtain frequency domain features. Spatial features are obtained by enhancing the enhanced features through spatial attention. The frequency domain features, spatial features, and enhanced features of each stage are added element-wise to obtain the corresponding frequency-space attention-enhanced feature maps. The frequency-space attention-enhanced feature maps of each stage are pooled and concatenated to obtain the high-frequency structural texture information.

[0152] The noise perception feature acquisition module 502 is used to input key frames into the noise perception module, match the distortion type and quality level in the key frames, and obtain noise perception features.

[0153] The noise semantic information acquisition module 503 is used to input the noise perception features into the adaptive temporal embedding module, match the temporal information of the noise perception features, and obtain the noise semantic information.

[0154] The spatial video quality score acquisition module 504 is used to splice high-frequency structural texture information and noise semantic information, and input them into the quality regression module to obtain the quality score of a single key frame. The quality scores of all key frames are summed and averaged to obtain the spatial video quality score.

[0155] The temporal distortion perceived degradation score acquisition module 505 is used to input video blocks into the Fast-VQA-based quality evaluation branch for video quality evaluation and obtain temporal distortion perceived degradation scores.

[0156] The video quality score acquisition module 506 is used to dynamically fuse the spatial domain video quality score and the temporal distortion perception degradation score to obtain the final video quality score, thus completing the screen content video quality evaluation based on frequency-space complementarity and semantics.

[0157] The specific implementation of the screen content video quality evaluation device based on frequency-space complementarity and semantics is the same as the screen content video quality evaluation method based on frequency-space complementarity and semantics, and will not be described again in this embodiment.

[0158] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A method for evaluating the video quality of screen content based on frequency-space complementarity and semantics, characterized in that, Includes the following steps: S1, acquire screen content video, extract several video blocks and several key frames from the screen content video, input the key frames into the high-frequency structure texture information extraction branch, and obtain high-frequency structure texture information. The high-frequency structural texture information extraction branch includes a pre-trained ResNet50 network, a multi-scale progressive channel attention module, and a frequency-space complementary attention module. The pre-trained ResNet50 network performs multi-stage feature extraction on keyframes, obtaining feature maps in four stages. The multi-scale progressive channel attention module uses a channel attention mechanism to perform cross-stage feature enhancement on the feature maps of each stage, outputting the enhanced features of each stage. The frequency-space complementary attention module uses a learnable filter to adaptively filter the enhanced features of each stage to obtain frequency domain features. Spatial features are obtained by enhancing the enhanced features through spatial attention. The frequency domain features, spatial features, and enhanced features of each stage are added element-wise to obtain the corresponding frequency-space attention-enhanced feature maps. The frequency-space attention-enhanced feature maps of each stage are pooled and concatenated to obtain the high-frequency structural texture information. S2, input the keyframe into the noise perception module, match the distortion type and quality level in the keyframe to obtain the noise perception features; S3, input the noise-perceived features into the adaptive temporal embedding module, match the temporal information of the noise-perceived features, and obtain the noise semantic information; S4 concatenates high-frequency structural texture information and noise semantic information, and inputs it into the quality regression module to obtain the quality score of a single key frame. The quality scores of all key frames are summed and averaged to obtain the spatial video quality score. S5, input the video block into the Fast-VQA-based quality evaluation branch to perform video quality evaluation and obtain the temporal distortion perceived degradation score; S6 dynamically fuses the spatial video quality score and the temporal distortion perception degradation score to obtain the final video quality score, thus completing the screen content video quality evaluation based on frequency-space complementarity and semantics. S2 and S3 are specifically as follows: Set a first text template, and use a text editor to encode the text template into a first text feature; the first text template describes the distortion type and quality level of the video frame; Set a second text template, and use a text editor to encode the second text template as a second text feature; the second text template describes the ordinal position of the video frame in the sequence of all video frames. Use an image encoder to encode keyframes into image features; Calculate the cosine similarity between the first text feature and the image feature to obtain the noise-perceived feature; Calculate the cosine similarity between the second text feature and the image feature to obtain temporal information; By fusing temporal information and noise perception features, noise semantic information is obtained. The process of fusing temporal information and noise perception features to obtain noise semantic information is as follows: ； in, Represents semantic information about noise; This indicates the extracted time-series information; Indicates noise perception characteristics; This indicates element-wise addition. This represents a 1×1 convolution operation; Represents the sigmoid function; Indicates a fully connected layer; This indicates a splicing operation.

2. The screen content video quality evaluation method based on frequency-space complementarity and semantics according to claim 1, characterized in that, The multi-scale progressive channel attention module is as follows: For the feature map of each stage, the feature map output of the current stage is... and the feature map of the output of the next stage The dimensions are reduced and features are extracted by performing two independent global average pooling and 1×1 convolutional layers respectively. and ; indicates as: ； ； in, This represents a 1×1 convolution operation; Indicates global average pooling; express Features after global average pooling and 1×1 convolution; express Features after global average pooling and 1×1 convolution; This represents the feature map of the current stage i. The feature map represents the next stage; Will and The concatenated feature maps are then subjected to 1×1 convolution and Sigmoid operations to obtain a multi-scale channel attention map A. The multi-scale channel attention map A is arranged according to... and The dimensions are split into and and Dimensionally consistent attention graph and ; feature map and Multiplying them together yields a weighted feature map. ; feature map and Multiplying them together yields a weighted feature map. ; Use transposed convolutional layers on feature maps Upsample by 2x and adjust the number of channels to make its dimensions and sum Consistent, then with Element-by-element addition ; right A Haar wavelet transform is performed to generate low-frequency components and detail components in the horizontal, vertical, and diagonal directions. The low-frequency components are preserved, and the number of channels in the feature map is adjusted by performing a 1×1 convolution on them. Batch normalization and ReLU activation are then applied to obtain enhanced features. .

3. The screen content video quality evaluation method based on frequency-space complementarity and semantics according to claim 1, characterized in that, The frequency-space complementary attention module adaptively filters the enhanced features of each stage using a learnable filter to obtain frequency domain features; it then enhances the enhanced features using spatial attention to obtain spatial features. Finally, it adds the frequency domain features, spatial features, and enhanced features of each stage element-by-element to obtain the corresponding frequency-space attention-adjusted feature map, as detailed below: Enhanced features output by the multi-scale progressive channel attention module are obtained using a two-dimensional discrete Fourier transform. Transform to the frequency domain to obtain the frequency domain feature map; Set mask and The mask A small square area is set at the center of the mask and assigned a value of 0, while the remaining areas are assigned a value of 1; the mask Set a square region with a side length of n at the center of the mask and assign it a value of 1, and assign a value of 0 to the rest of the region; frequency domain feature map and Element-wise multiplication yields the low-frequency components. ; frequency domain feature map and Element-wise multiplication yields high-frequency components. ; Using adaptive learnable global filters Perform filtering, then extract the high-frequency features. With the filtered Reintegration yields the spectrum after global learning and feature fusion. ; indicates as: ； Where N represents an adaptive, learnable global filter. This indicates element-wise addition; Indicates multiplication; right The frequency domain features are obtained using the two-dimensional inverse discrete Fourier transform. ; right Spatial attention is used to obtain spatial features. ; indicates as: ； in, Indicates global average pooling. This indicates max pooling. Represents the sigmoid function; This indicates a splicing operation. This represents a 3×3 convolution. Indicates the multiplication operation; Will , and Element-wise addition yields the feature map after frequency-spatial attention. .

4. The screen content video quality evaluation method based on frequency-space complementarity and semantics according to claim 1, characterized in that, The process involves concatenating high-frequency structural texture information and noise semantic information, inputting the result into a quality regression module to obtain the quality score of a single keyframe, and then summing and averaging the quality scores of all keyframes to obtain the spatial domain video quality score; expressed as: ； ； in, This represents the quality score of a single keyframe. Indicates a fully connected layer. This indicates a splicing operation. Represents high-frequency structural texture information. Represents semantic information about noise; Indicates the number of keyframes. This represents the video quality score in the spatial domain.

5. The screen content video quality evaluation method based on frequency-space complementarity and semantics according to claim 1, characterized in that, The spatial video quality score and the temporal distortion-perceived degradation score are dynamically fused to obtain the final video quality score, as detailed below: Video quality score in the spatial domain Perceived degradation score of temporal distortion Adaptive dynamic fusion is performed to obtain the final video quality score, denoted as . The calculation is as follows: ； in, This indicates the final video quality score. Indicates hyperparameters, This represents the video quality score in the spatial domain. This represents the temporal distortion perception degradation score.

6. A screen content video quality evaluation device based on frequency-space complementarity and semantics, characterized in that, The screen content video quality evaluation method based on frequency-space complementarity and semantics as described in any one of claims 1-5 includes the following: The high-frequency structural texture information acquisition module is used to acquire screen content video, extract several video blocks and several key frames from the screen content video, input the key frames into the high-frequency structural texture information extraction branch, and obtain high-frequency structural texture information. The high-frequency structural texture information extraction branch includes a pre-trained ResNet50 network, a multi-scale progressive channel attention module, and a frequency-space complementary attention module. The pre-trained ResNet50 network performs multi-stage feature extraction on keyframes, obtaining feature maps in four stages. The multi-scale progressive channel attention module uses a channel attention mechanism to perform cross-stage feature enhancement on the feature maps of each stage, outputting the enhanced features of each stage. The frequency-space complementary attention module uses a learnable filter to adaptively filter the enhanced features of each stage to obtain frequency domain features. Spatial features are obtained by enhancing the enhanced features through spatial attention. The frequency domain features, spatial features, and enhanced features of each stage are added element-wise to obtain the corresponding frequency-space attention-enhanced feature maps. The frequency-space attention-enhanced feature maps of each stage are pooled and concatenated to obtain the high-frequency structural texture information. The noise perception feature acquisition module is used to input keyframes into the noise perception module, match the distortion type and quality level in the keyframes, and obtain noise perception features. The noise semantic information acquisition module is used to input noise perception features into the adaptive temporal embedding module, match the temporal information of the noise perception features, and obtain noise semantic information. The spatial video quality score acquisition module is used to splice high-frequency structural texture information and noise semantic information, and input them into the quality regression module to obtain the quality score of a single key frame. The quality scores of all key frames are summed and averaged to obtain the spatial video quality score. The temporal distortion perceived degradation score acquisition module is used to input video blocks into the Fast-VQA-based quality evaluation branch for video quality evaluation and obtain temporal distortion perceived degradation scores. The video quality score acquisition module is used to dynamically fuse the spatial video quality score and the temporal distortion perception degradation score to obtain the final video quality score, thus completing the screen content video quality evaluation based on frequency-space complementarity and semantics.

Images