Video quality evaluation method and system

Abstract

The invention discloses a video quality evaluation method and system, and relates to the technical field of video quality evaluation, and the method comprises the steps: obtaining a UGC video and a text description corresponding to the UGC video, inputting the UGC video into a Conv2Former model and a SlowFast model which are established in advance, outputting to obtain a spatial feature and a time feature, fusing the spatial feature and the time feature, and determining the quality of the UGC video according to the fused spatial feature and time feature. Obtaining a final output feature; the text description corresponding to the UGC video is input into a pre-established Transform model, text features are output and obtained, and the text features comprise semantic information of video content and evaluation of video quality; and fusing the final output feature with the text feature to obtain a fused feature, inputting the fused feature into a pre-established MLP network model, and outputting to obtain a video quality score.

Description

Technical Field

[0001] The present invention relates to the technical field of video quality evaluation, and in particular to a video quality evaluation method and system. Background Art

[0002] With the rapid development of the Internet and multimedia technology, user-generated content (UGC) videos have exploded on social media platforms. Assessing the quality of these videos has become a crucial issue. In recent years, deep learning technology has made significant progress in the fields of computer vision and natural language processing, especially the development of convolutional neural networks (CNNs) and Transformer architectures, which have greatly enhanced the capabilities of feature extraction and information fusion. However, these traditional deep learning-based methods usually rely only on the visual features of the video, ignoring the multimodal information of the video content, resulting in limited performance in complex scenes. Currently, existing video quality assessment methods rarely use both visual and textual information for comprehensive assessment. However, in practical applications, the quality of a video depends not only on its visual features, but also on the semantic information of its content, such as the user's description and evaluation of the video content. These textual information provides subjective judgments on the quality of the video and content context. In addition, the dynamic changes and complexity of video content also put higher requirements on video quality assessment. Although existing deep learning methods can capture the static features of videos, they still have certain limitations in extracting temporal features and motion information. Summary of the invention

[0003] In order to address the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a video quality evaluation method and system, which utilizes deep learning technology to fuse visual encoders and text encoders to extract and analyze the features of videos and their corresponding text information, thereby achieving more accurate and comprehensive video quality evaluation.

[0004] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a video quality evaluation method, the method comprising the following steps: Obtain UGC videos and the text descriptions corresponding to the UGC videos, input the UGC videos into the pre-established Conv2Former model and SlowFast model respectively, output spatial features and temporal features, fuse the spatial features and temporal features to obtain the final output features; Input the text description corresponding to the UGC video into a pre-established Transformer model, and output text features, wherein the text features include semantic information of the video content and evaluation of the video quality; The final output features are fused with the text features to obtain fused features, which are input into the pre-established MLP network model to output the video quality score.

[0005] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of collecting the UGC video: Several videos are collected from the YouTube-8M database. Then, Gaussian blur, Gaussian noise, contrast adjustment, and HEVC compression distortion processing are applied to each video.

[0006] In combination with the first aspect, in some implementations of the first aspect, the method further includes: inputting the UGC video into a pre-established Conv2Former model and a SlowFast model respectively, and outputting a process of obtaining spatial features and temporal features: The Conv2Former model is used as a spatial feature extractor, and the SlowFast model is used as a temporal feature extractor; Given a UGC video , N represents the number of frames, the Conv2Former model is used as the spatial branch, and its input is the key frames of the video. The SlowFast model is used as the temporal branch, and its input is the video sequence. The key frame of the video is extracted by sampling the first frame of the video frame sequence:

[0007] r represents the frame sampling rate of the video, represents the obtained video sequence, which serves as the input of the time branch, The first frame in the image will be extracted as the key frame. , as the input of the spatial branch; Among them, the key frame Input into Conv2Former model to get spatial features :

[0008] The video sequence obtained Input into the SlowFast model and calculate the time characteristics :

[0009] in Indicates SlowFast without classification header, Represents the global average pooling method.

[0010] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of fusing the spatial features and the temporal features to obtain the final output features:

[0011] Among them, F fusion is the final output feature, is the weight of feature fusion.

[0012] In combination with the first aspect, in some implementations of the first aspect, the method further includes: using the pre-trained Transformer model architecture as a text encoder, and then encoding the text to obtain an output of the text encoder. , as text features.

[0013] In combination with the first aspect, in some implementations of the first aspect, the method further includes: when the final output feature is fused with the text feature, mapping the text feature to obtain a mapped text feature , fusion feature F final as follows:

[0014] in Represents the concatenation operation of features.

[0015] In combination with the first aspect, in some implementations of the first aspect, the method further includes: inputting the fusion features into a pre-established MLP network model, and outputting a video quality score The calculation process:

[0016] In the formula, FC1 and FC2 are two fully connected layers, and GELU is the activation function used.

[0017] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of evaluating the video quality score: The model is trained by minimizing the PLCC loss. The quality scores predicted by the model and the actual quality scores are expressed as and q; The PLCC loss calculation loss is as follows:

[0018] in, for The covariance between and q, and Respectively expressed as and the variance of q.

[0019] In a second aspect, in order to achieve the above-mentioned object, the present invention discloses a video quality evaluation system, comprising: The visual encoding module is used to obtain UGC videos and the text descriptions corresponding to the UGC videos, input the UGC videos into the pre-established Conv2Former model and SlowFast model respectively, output spatial features and temporal features, and fuse the spatial features and temporal features to obtain the final output features; A text encoding module, used to input the text description corresponding to the UGC video into a pre-established Transformer model, and output text features, wherein the text features include semantic information of the video content and evaluation of the video quality; The feature regression module is used to fuse the final output features with the text features to obtain fused features, input the fused features into the pre-established MLP network model, and output the video quality score.

[0020] In conjunction with the second aspect, in some implementations of the second aspect, the system further includes: a process of collecting UGC video in the visual coding module: Collect several videos from the YouTube-8M database, then apply Gaussian blur, Gaussian noise, contrast adjustment, and HEVC compression distortion to each video; In the visual coding module, the UGC video is input into the pre-established Conv2Former model and SlowFast model respectively, and the spatial and temporal features are output: The Conv2Former model is used as a spatial feature extractor, and the SlowFast model is used as a temporal feature extractor; Given a UGC video , N represents the number of frames, the Conv2Former model is used as the spatial branch, and its input is the key frames of the video. The SlowFast model is used as the temporal branch, and its input is the video sequence. The key frame of the video is extracted by sampling the first frame of the video frame sequence:

[0021] r represents the frame sampling rate of the video, represents the obtained video sequence, which serves as the input of the time branch, The first frame in the image will be extracted as the key frame. , as the input of the spatial branch; Among them, the key frame Input into Conv2Former model to get spatial features :

[0022] The video sequence obtained Input into the SlowFast model and calculate the time characteristics :

[0023] in Indicates SlowFast without classification header, represents the global average pooling method; The process of fusing spatial features and temporal features in the visual encoding module to obtain the final output features:

[0024] Among them, F fusion is the final output feature, is the weight of feature fusion.

[0025] In the text encoding module, the pre-trained Transformer model architecture is used as a text encoder, and then the text is encoded to obtain the output of the text encoder. , as text features; When the final output feature is integrated with the text feature in the feature regression module, the text feature is mapped to obtain the mapped text feature. , fusion feature F final as follows:

[0026] in Represents the concatenation operation of features; In the feature regression module, the fusion features are input into the pre-established MLP network model, and the output is the video quality score. The calculation process:

[0027] In the formula, FC1 and FC2 are two fully connected layers, and GELU is the activation function used.

[0028] The process of video quality score evaluation in the feature regression module: The model is trained by minimizing the PLCC loss. The quality scores predicted by the model and the actual quality scores are expressed as and q; PLCC loss calculation as follows:

[0029] in, for The covariance between and q, and Respectively expressed as and the variance of q.

[0030] Beneficial effects of the present invention: The present invention achieves a more comprehensive and accurate evaluation by integrating the features extracted by the visual encoder and the text encoder, and comprehensively utilizing the visual information and text information of the video, and has significant advantages in the field of video quality assessment. The present invention uses a text encoder to extract text features related to the video, including the semantic background of the video content and the user's subjective evaluation of the video quality. These text information make up for the shortcomings of traditional methods that rely solely on visual features. Secondly, the present invention combines the Conv2Former for spatial feature extraction and the fast path of the SlowFast model for temporal feature extraction, which effectively solves the problem of insufficient capture of video dynamic changes and complexity in the prior art. In summary, the present invention not only improves the accuracy and comprehensiveness of video quality assessment through the method of multimodal feature fusion, but also performs well in video content in complex scenes, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work. Figure 1 It is a schematic flow chart of the method of the present invention; Figure 2 It is a schematic diagram of the workflow of the present invention; Figure 3 It is a schematic diagram of the framework of the video quality evaluation method of the present invention; Figure 4 It is a schematic diagram of the system structure of the present invention; Figure 5 This is a visualization result diagram of the KoNViD-1k data set in this embodiment; Figure 6 This is a visualization result diagram of the LIVE-VQC data set in this embodiment. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Embodiment 1: The following is an introduction to the relevant terms involved in the embodiments of the present application: The Transformer model is a neural network architecture based on the self-attention mechanism. It has been widely used in the field of natural language processing by using a multi-head attention mechanism and a fully connected layer stack.

[0034] The advantages of the Transformer model include: Efficient computing: Through parallel computing, the Transformer model can significantly improve computing efficiency, especially when processing large amounts of data. This is especially important when processing long texts and performing tasks such as machine translation.

[0035] Accurately learn long-range dependencies: Compared with traditional recurrent neural networks (RNNs), the Transformer model is better able to capture long-range dependencies in sentences because it can simultaneously focus on all positions in the input sequence, not just the previous one or a few time steps.

[0036] Powerful representation ability: The Transformer model learns the internal representation of input data through the self-attention mechanism. This representation method can capture the key information in the input data, making the model perform well in various natural language processing tasks.

[0037] Easy to expand and optimize: The structure of the Transformer model is relatively simple and easy to implement, which makes it easy to expand to larger models and further improve performance through various optimization techniques.

[0038] Wide application: Transformer models have been widely used in various natural language processing tasks, such as machine translation, text generation, question answering systems, sentiment analysis, etc., and have achieved remarkable success.

[0039] like Figure 1 and Figure 2 As shown, a video quality evaluation method comprises the following steps: S101: Obtain a UGC video and a text description corresponding to the UGC video, input the UGC video into a pre-established Conv2Former model and a SlowFast model respectively, output spatial features and temporal features, fuse the spatial features and temporal features, and obtain a final output feature; First, 310 videos are collected from the YouTube-8M database. Then, distortion processing such as Gaussian blur, Gaussian noise, contrast adjustment, and HEVC compression is applied to each video. At the same time, in order to extract text features, it is necessary to add relevant text descriptions to each video. The description content can be a brief description of the video content and a quality description. It is worth noting that the quality of the video after adding distortion should be lower than the quality of the video without any artificial distortion, so the description of the video quality in the text should reflect this difference.

[0040] like Figure 3 As shown, the Conv2Former model is used as a spatial feature extractor, and the SlowFast model is used as a temporal feature extractor; The visual encoder is divided into two branches, one for extracting spatial features and the other for extracting temporal features to supplement the information of the temporal dimension of the video. Therefore, we first need to process the video V in step 1 to satisfy the input of the two branches. Given a video , N represents the number of frames, the input of the spatial branch is the key frame of the video, and the input of the temporal branch is the video sequence. We sample the first frame of the video frame sequence to obtain the key frame.

[0042] r represents the frame sampling rate of the video, represents the video sequence we obtained, which serves as the input of the time branch, The first frame in the image will be extracted as the key frame. , as the input of the spatial branch.

[0043] Get the input of two branches and After that, we start to extract features. First, we use Conv2Former as a spatial feature extractor to process the keyframes to extract high-quality spatial features. These spatial features can capture the details and complex scene information of the video content.

[0044]

[0045] We then use the fast path of the SlowFast model to extract motion features for each video block. To obtain the temporal features, we remove the classification head of the SlowFast model and compute the temporal features by global average pooling of the feature maps at the last stage of the fast path. .

[0046]

[0047] in Indicates SlowFast without classification header. Represents the global average pooling method. This design enables the visual encoder to take into account both the spatial and temporal features of the video, providing rich feature representation for subsequent feature fusion and video quality evaluation.

[0048] The process of fusing spatial features and temporal features to obtain the final output features: The fusion process can be achieved through weighted averaging, and the weights can be learned according to actual conditions to balance the contributions of spatial features and temporal features:

[0049] Among them, F fusion is the final output feature, is the weight of feature fusion.

[0050] S102: Inputting the text description corresponding to the UGC video into a pre-established Transformer model, and outputting text features, wherein the text features include semantic information of the video content and an evaluation of the video quality; In order to extract text features, we use the pre-trained Transformer architecture to encode the text and get the output of the text encoder Transformer is a powerful deep learning model that is particularly suitable for processing sequence data. In this scenario, the text is a description of the video content and an evaluation of the video quality. This information contains the semantic information of the video and the user's subjective evaluation. Specifically, Transformer is used to map the text sequence into a high-dimensional feature vector representation.

[0051] S103: The final output feature is fused with the text feature to obtain a fused feature, and the fused feature is input into a pre-established MLP network model to output a video quality score.

[0052] Since the dimensions of visual features and text features may be different, we choose to use feature mapping to solve the dimension mismatch problem so that the two features can be aligned in dimension. Specifically, we can use a fully connected layer to map the text features to the same dimension as the visual features. Assume that the mapped text features are , then the fused feature F final It can be expressed as:

[0053] in The splicing operation of representing features, so that the fused features can more comprehensively reflect the visual and textual information of the video, and help improve the accuracy and comprehensiveness of video quality evaluation.

[0054] Using a two-layer MLP network Regression to quality score .

[0055]

[0056] Among them, FC1 and FC2 are two fully connected layers, and GELU is the activation function we use.

[0057] In the video quality assessment task, we hope that the quality score predicted by the model has a high correlation with the actual score to improve the accuracy of the assessment. Therefore, we use PLCC (Pearson Linear Correlation Coefficient) loss to optimize the proposed VQA model. By minimizing the PLCC loss to train the model, the model can better learn the feature representation of video quality. The quality score predicted by the model and the actual quality score are expressed as and q, specifically, the PLCC loss can be expressed as:

[0058] in, for The covariance between and q, and Respectively expressed as By minimizing the PLCC loss, we can enable the model to better learn the relevant features of video quality, thereby improving the accuracy and comprehensiveness of the evaluation.

[0059] Embodiment 2: In the second aspect, as Figure 4 As shown, in order to achieve the above-mentioned purpose, the present invention discloses a video quality evaluation system, comprising: The visual encoding module 11 is used to obtain the UGC video and the text description corresponding to the UGC video, input the UGC video into the pre-established Conv2Former model and SlowFast model respectively, output the spatial features and the temporal features, and fuse the spatial features and the temporal features to obtain the final output features; A text encoding module 12, used to input the text description corresponding to the UGC video into a pre-established Transformer model, and output text features, wherein the text features include semantic information of the video content and evaluation of the video quality; The feature regression module 13 is used to fuse the final output feature with the text feature to obtain a fused feature, input the fused feature into a pre-established MLP network model, and output a video quality score.

[0060] In combination with the second aspect, in some implementations of the second aspect, the system further includes: a process of collecting UGC videos in the visual coding module 11: Collect several videos from the YouTube-8M database, then apply Gaussian blur, Gaussian noise, contrast adjustment, and HEVC compression distortion to each video; In the visual coding module 11, the UGC video is input into the pre-established Conv2Former model and SlowFast model respectively, and the spatial features and temporal features are outputted: The Conv2Former model is used as a spatial feature extractor, and the SlowFast model is used as a temporal feature extractor; Given a UGC video , N represents the number of frames, the Conv2Former model is used as the spatial branch, and its input is the key frames of the video. The SlowFast model is used as the temporal branch, and its input is the video sequence. The key frame of the video is extracted by sampling the first frame of the video frame sequence:

[0061] r represents the frame sampling rate of the video, represents the video sequence we obtained, which serves as the input of the time branch, The first frame in the image will be extracted as the key frame. , as the input of the spatial branch.

[0062] Among them, the key frame Input into Conv2Former model to get spatial features :

[0063] The video sequence obtained Input into the SlowFast model and calculate the time characteristics :

[0064] in Indicates SlowFast without classification header, represents the global average pooling method; The process of fusing spatial features and temporal features in the visual encoding module 11 to obtain the final output features:

[0065] Among them, F fusion is the final output feature, is the weight of feature fusion.

[0066] The pre-trained Transformer model architecture in the text encoding module 12 acts as a text encoder to encode the text to obtain the output of the text encoder , as text features; When the final output feature is integrated with the text feature in the feature regression module 13, the text feature is mapped to obtain the mapped text feature. , fusion feature F final as follows:

[0067] in Represents the concatenation operation of features; The feature regression module 13 inputs the fusion features into the pre-established MLP network model and outputs the video quality score. The calculation process:

[0068] In the formula, FC1 and FC2 are two fully connected layers, and GELU is the activation function we use.

[0069] The process of video quality score evaluation in the feature regression module 13: The model is trained by minimizing the PLCC loss. The quality scores predicted by the model and the actual quality scores are expressed as and q; The PLCC loss calculation loss is as follows:

[0070] in, for The covariance between and q, and Respectively expressed as and the variance of q.

[0071] In order to verify the effect of the present invention, we visualized the results of the trained model on the public datasets KoNViD-1k and LIVE-VQC. The visualization results are shown in Figure 2. Figure 5 , 6 As shown in the figure. These scatter plots more intuitively show the relationship between the quality score predicted by the model and the true quality value. The fitting curve depicts the trend between the predicted value and the true quality value. From the scatter plot, it can be seen that the predicted value is distributed around the fitting curve, which shows that the proposed method can maintain good consistency with the true quality value on the public data set, further illustrating the accuracy and reliability of the method.

[0072] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0073] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by the processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0074] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0075] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A video quality evaluation method, It is characterized in that The method comprises the following steps: Obtain UGC videos and the text descriptions corresponding to the UGC videos, input the UGC videos into the pre-established Conv2Former model and SlowFast model respectively, output spatial features and temporal features, fuse the spatial features and temporal features to obtain the final output features; Input the text description corresponding to the UGC video into a pre-established Transformer model, and output text features, wherein the text features include semantic information of the video content and evaluation of the video quality; The final output features are fused with the text features to obtain fused features, which are input into the pre-established MLP network model to output the video quality score.

2. A video quality assessment method according to claim 1, It is characterized in that The collection process of the UGC video: Several videos are collected from the YouTube-8M database. Then, Gaussian blur, Gaussian noise, contrast adjustment, and HEVC compression distortion processing are applied to each video.

3. A video quality assessment method according to claim 1, It is characterized in that The process of inputting the UGC video into the pre-established Conv2Former model and SlowFast model respectively, and outputting the spatial features and temporal features: The Conv2Former model is used as a spatial feature extractor, and the SlowFast model is used as a temporal feature extractor; Given a UGC video , N represents the number of frames, the Conv2Former model is used as the spatial branch, and the input is the key frames of the video. The SlowFast model is used as the temporal branch, and the input is the video sequence. The key frame of the video is extracted by sampling the first frame of the video frame sequence: r represents the frame sampling rate of the video, represents the obtained video sequence, as the input of the time branch, The first frame in the image will be extracted as the key frame. , as the input of the spatial branch; Among them, the key frame Input into Conv2Former model to get spatial features : The video sequence obtained Input into the SlowFast model and calculate the time characteristics : in Indicates SlowFast without classification header, Represents the global average pooling method.

4. A video quality evaluation method according to claim 3, It is characterized in that The process of fusing spatial features and temporal features to obtain the final output features is as follows: Among them, F fusion is the final output feature, is the weight of feature fusion.

5. A video quality assessment method according to claim 1, It is characterized in that Use the pre-trained Transformer model architecture as a text encoder, and then encode the text to get the output of the text encoder , as text features.

6. A video quality assessment method according to claim 1, It is characterized in that When the final output feature is merged with the text feature, the text feature is mapped to obtain the mapped text feature. , fusion feature F final as follows: in Represents the concatenation operation of features.

7. A video quality assessment method according to claim 1, It is characterized in that The fusion features are input into the pre-established MLP network model, and the video quality score is output. The calculation process: In the formula, FC 1 and FC 2 There are two fully connected layers, and GELU is the activation function used.

8. A video quality assessment method according to claim 7, It is characterized in that The process of video quality score evaluation: The model is trained by minimizing the PLCC loss. The quality scores predicted by the model and the actual quality scores are expressed as and q; The PLCC loss calculation loss is as follows: in, for The covariance between and q, and Respectively expressed as and the variance of q.

9. A video quality evaluation system, It is characterized in that include: The visual encoding module is used to obtain UGC videos and the text descriptions corresponding to the UGC videos, input the UGC videos into the pre-established Conv2Former model and SlowFast model respectively, output spatial features and temporal features, and fuse the spatial features and temporal features to obtain the final output features; A text encoding module, used to input the text description corresponding to the UGC video into a pre-established Transformer model, and output text features, wherein the text features include semantic information of the video content and evaluation of the video quality; The feature regression module is used to fuse the final output features with the text features to obtain fused features, input the fused features into the pre-established MLP network model, and output the video quality score.

10. A video quality evaluation system according to claim 9, It is characterized in that The acquisition process of UGC video in the visual coding module: Collect several videos from the YouTube-8M database, then apply Gaussian blur, Gaussian noise, contrast adjustment, and HEVC compression distortion to each video; In the visual coding module, the UGC video is input into the pre-established Conv2Former model and SlowFast model respectively, and the spatial and temporal features are output: The Conv2Former model is used as a spatial feature extractor, and the SlowFast model is used as a temporal feature extractor; Given a UGC video , N represents the number of frames, the Conv2Former model is used as the spatial branch, and the input is the key frames of the video. The SlowFast model is used as the temporal branch, and the input is the video sequence. The key frame of the video is extracted by sampling the first frame of the video frame sequence: r represents the frame sampling rate of the video, represents the obtained video sequence, which serves as the input of the time branch, The first frame in the image will be extracted as the key frame. , as the input of the spatial branch; Among them, the key frame Input into Conv2Former model to get spatial features : The video sequence obtained Input into the SlowFast model and calculate the time characteristics : in Indicates SlowFast without classification header, represents the global average pooling method; The process of fusing spatial features and temporal features in the visual encoding module to obtain the final output features: Among them, F fusion is the final output feature, is the weight of feature fusion; In the text encoding module, the pre-trained Transformer model architecture is used as a text encoder to encode the text to obtain the output of the text encoder. , as text features; When the final output feature is integrated with the text feature in the feature regression module, the text feature is mapped to obtain the mapped text feature. , fusion feature F final as follows: in Represents the concatenation operation of features; In the feature regression module, the fusion features are input into the pre-established MLP network model, and the output is the video quality score. The calculation process: In the formula, FC 1 and FC 2 There are two fully connected layers, and GELU is the activation function used; The process of video quality score evaluation in the feature regression module: The model is trained by minimizing the PLCC loss. The quality scores predicted by the model and the actual quality scores are expressed as and q; The PLCC loss calculation loss is as follows: in, for The covariance between and q, and Respectively expressed as and the variance of q.

Images