A video quality evaluation method and device based on Conv2Former and contrast learning

Abstract

The application discloses a video quality evaluation method and device based on Conv2Former and contrast learning, which uses Conv2Former as a spatial feature extraction module and combines a feature pyramid technology to perform spatial feature extraction on a video frame. Then, an Attention-LSTM model is used to process time sequence information of the video to capture time relations between the video frames. Finally, an MLP layer is used to map the time sequence features into a final video quality score. In order to solve the problem of insufficient data, contrast learning is used for pre-training, and unlabeled data is used to learn more discriminative feature representations, so as to improve the model performance and generalization ability. The deep learning model provided by the application can extract spatial features while maintaining high computing efficiency, and realizes accurate and automatic video quality evaluation. In particular, the pre-training through contrast learning provides a reliable solution to the problem of insufficient labeled data in video quality evaluation.

Description

Technical Field

[0001] This invention relates to a video quality evaluation method and apparatus based on Conv2Former and contrastive learning, belonging to the field of video quality evaluation technology. Background Technology

[0002] With the rapid growth of user-generated content (UGC) videos on social media platforms, assessing the quality of these videos has become crucial. To address the challenges of UGC video quality assessment, it is necessary to explore and develop novel deep learning models and assessment methods suitable for NR-VQA (No Reference Video Quality Assessment) to ensure accurate and reliable quality evaluation of non-professional-generated videos, thereby providing a better user experience and meeting the ever-increasing demand for UGC videos.

[0003] The Transformer, based on the self-attention mechanism, has seen rapid development in computer vision. Existing techniques have applied the Transformer to video quality assessment (VQA), demonstrating the excellent potential of attention in this field. An end-to-end no-reference VQA model was proposed, using a 3D convolutional network for feature extraction, followed by quality regression using a Transformer. Finally, a fully connected layer is used to predict the overall video quality score, yielding promising results. However, the Transformer is not a lightweight framework; it typically has a large number of parameters and complex computational structures, thus increasing computational costs.

[0004] Meanwhile, when performing NRVQA, in addition to designing a suitable deep learning model, a large amount of labeled training data is also needed to support the training of the model. However, since labeling data requires a lot of manpower and time investment, some researchers have begun to use self-supervised methods to handle VQA tasks. [3] They comprehensively utilize the self-supervised information of distortion degree, distortion type and frame rate, and use it as prior knowledge to build a novel self-supervised framework for VQA tasks, enabling the pre-trained VQA model to have the ability to represent richer and more distortion-sensitive features. However, designing effective auxiliary tasks requires domain expertise and experience, and requires a lot of time and computing resources to train and optimize multiple task models. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a video quality evaluation method and apparatus based on Conv2Former and contrastive learning. It utilizes Conv2Former and LSTM to extract the spatiotemporal features of the video, and introduces contrastive learning to pre-train the proposed model, thereby improving the model's quality evaluation performance and generalization ability.

[0006] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0007] In a first aspect, the present invention provides a video quality evaluation method based on Conv2Former and contrastive learning, comprising the following steps:

[0008] Get the videos to be evaluated;

[0009] The video to be evaluated is subjected to data augmentation and frame segmentation to obtain a data-enhanced video, which consists of a combination of consecutive frames;

[0010] The augmented video is input into the trained video quality assessment model to obtain the predicted video quality score to be evaluated.

[0011] Furthermore, the video quality evaluation model includes a feature extraction part and a quality score prediction module; the feature extraction part includes a spatial feature extraction module and a temporal information processing module;

[0012] The spatial feature extraction module is used to extract spatial features using feature pyramid technology;

[0013] The time-series information processing module is used to process time-series information based on the extracted spatial features. Through a self-attention mechanism, it performs weighted processing on video frames at different times to capture the temporal relationship between video frames and obtain spatiotemporal features.

[0014] The quality score prediction module includes a multilayer perceptron (MLP) layer, which is used to weight and combine the features output by the temporal information processing module to map the spatiotemporal feature representation into a quality score, which will reflect the perceptual quality and visual experience of the video.

[0015] Furthermore, the spatial feature extraction module is a Conv2Former backbone network;

[0016] The Conv2Former backbone network is used to perform convolution operations on each video frame and output a set of spatial features at each convolutional layer, which represent image information at different levels and levels of abstraction.

[0017] The feature pyramid technique is used to generate multi-scale features by using spatial features at different levels obtained from the backbone network.

[0018] Furthermore, the time-series information processing module is an Attention-LSTM model;

[0019] The Attention-LSTM model consists of an input layer, an LSTM layer, an Attention layer, and a fully connected layer. The LSTM layer is used to capture long-term dependencies in the video sequence, and the fully connected layer performs non-linear mapping and integration of the high-level features learned by the LSTM and Attention layers to obtain the final spatiotemporal features.

[0020] Furthermore, the method for obtaining the trained video quality assessment model includes:

[0021] Obtain the video set for model pre-training;

[0022] The video set used for model pre-training is input into the video quality assessment model, and the feature extraction part of the video quality assessment model is self-supervised pre-trained using contrastive learning.

[0023] After training, an additional MLP layer is added to the pre-trained model to serve as the final video quality evaluation model.

[0024] Furthermore, methods for obtaining video sets for model pre-training include:

[0025] Select 500 videos from the ImageNet VID dataset or other applicable datasets as the pre-input dataset.

[0026] The pre-input dataset is preprocessed, including parsing the video format, unifying the frame rate and resolution, and cropping or padding video frames to achieve a uniform size and format, resulting in preprocessed video data.

[0027] The preprocessed video data is augmented by random cropping, horizontal flipping, and color transformation to obtain a video set for model pre-training.

[0028] Furthermore, the video set used for model pre-training is input into the video quality assessment model, and contrastive learning is used to perform self-supervised pre-training on the feature extraction part of the video quality assessment model, including:

[0029] Self-supervised learning is performed on unlabeled video data, and the contrastive loss function used is:

[0030]

[0031] In the formula, τ is the temperature hyperparameter, q is a pre-coded feature, and k0, k1, k2, ... are a series of pre-coded samples in a dictionary, where each dictionary contains only one key, k. + If it matches q, then k +q and k are positive sample pairs, and the rest are positive sample pairs. i For q, it is a negative sample.

[0032] Furthermore, the method also includes:

[0033] The entire quality assessment model is then fine-tuned, using a simple mean squared error loss as the adjustment loss, as shown in the following formula:

[0034]

[0035] Where n represents the number of videos in the batch, Q′ i and Q i These are subjective quality labels and predicted quality scores.

[0036] The fine-tuned model is tested, and the spatiotemporal feature representations obtained from the feature extraction part are weighted and combined through a multilayer perceptron (MLP) layer to obtain the final video quality score.

[0037] Secondly, the present invention provides a video quality evaluation device based on Conv2Former and contrastive learning, the device comprising:

[0038] Input module: Used to obtain the video to be evaluated;

[0039] Enhancement module: used to perform data enhancement and frame segmentation on the video to be evaluated to obtain a data-enhanced video, the video comprising a combination of consecutive frames;

[0040] Prediction module: This module takes the augmented video data and inputs it into the trained video quality assessment model to obtain a predicted video quality score.

[0041] Thirdly, the present invention provides a video quality evaluation device based on Conv2Former and contrastive learning, including a processor and a storage medium;

[0042] The storage medium is used to store instructions;

[0043] The processor is configured to operate according to the instructions to perform the steps of the method described in the first aspect.

[0044] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0045] This invention proposes a video quality evaluation method and apparatus based on Conv2Former and contrastive learning. The method utilizes Conv2Former as a spatial feature extraction module and combines it with feature pyramid technology to extract spatial features from video frames. Simultaneously, through contrastive learning, useful feature representations can be effectively learned, thereby improving the accuracy of UGC video quality evaluation. Compared with traditional methods, this method has the following advantages: First, it does not rely on a large amount of labeled data, reducing training costs; second, by introducing Conv2Former and contrastive learning, it can better capture the spatial features and temporal relationships of the video, improving the accuracy of UGC video quality evaluation; finally, this method has high efficiency and scalability when processing large-scale UGC videos.

[0046] The video quality evaluation method based on Conv2Former and contrastive learning proposed in this invention can overcome the shortcomings of existing technologies, achieve accurate evaluation of UGC videos, provide users with a better viewing experience, and meet the ever-growing demand for UGC videos.

[0047] This invention employs contrastive learning for pre-training, utilizing unlabeled video data to learn more discriminative feature representations. This effectively addresses the problem of insufficient data and improves the model's generalization ability and performance. It provides an effective solution for evaluating the quality of user-generated content (UGC) videos—those produced in a non-professional manner. Attached Figure Description

[0048] Figure 1 This is a flowchart of a video quality evaluation method and apparatus based on Conv2Former and contrastive learning according to an embodiment of the present invention;

[0049] Figure 2 This is a framework diagram of a video quality evaluation method and apparatus based on Conv2Former and contrastive learning according to one embodiment of the present invention. Detailed Implementation

[0050] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0051] Example 1:

[0052] like Figure 1 As shown, the present invention provides a video quality assessment method based on Conv2Former and contrastive learning, including a video quality assessment model, which includes: a spatial feature extraction module, a temporal information processing module, and a quality score prediction module.

[0053] This process employs a common paradigm for video quality assessment, first processing the spatial information of the video and then processing the temporal information. Video frames are input into the Conv2Former spatial feature extraction network. During feature extraction, a feature pyramid technique is used to better capture the multi-scale expressive power of video frame features.

[0054] The extracted spatial features are input into the Attention-LSTM model. This model is specifically designed to handle temporal information between video frames. Through a self-attention mechanism, Attention-LSTM can weight video frames at different times, thereby capturing the temporal relationships between video frames and better understanding the dynamic characteristics of the video.

[0055] To address the issue of insufficient data, contrastive learning is employed to pre-train the model for the entire feature extraction component. Through self-supervised learning on unlabeled video data, contrastive learning enables the model to learn more discriminative feature representations. Pre-training allows the model to acquire superior feature representation capabilities, thereby improving the accuracy and generalization ability of video quality assessment.

[0056] Finally, the pre-trained feature extraction model is combined with an MLP and fine-tuned and tested on a public dataset. Weighted summaries are then performed using multilayer perceptron (MLP) layers to obtain an overall video quality evaluation. This score reflects the perceptual quality and visual experience of the video, providing a final video quality score.

[0057] As a preferred embodiment, the construction of the pre-trained data module includes:

[0058] Collect video data from the ImageNet VID dataset or other applicable datasets. Ensure that the selected video data covers videos of different scenes, content, and quality to guarantee the model's generalization ability and applicability. Preprocess the collected video data to ensure data consistency and usability.

[0059] A portion of the preprocessed video data is randomly sampled and used as the input dataset for pre-training. The quantity and quality of the sampled videos are ensured to meet the requirements of the pre-training process.

[0060] Data augmentation is performed on the sampled video data to increase its diversity and richness. Common data augmentation methods include random cropping, horizontal flipping, and color dithering to simulate changes in video quality and content under different conditions.

[0061] To mitigate the impact of video resolution on model performance, all videos were cropped into 224×224 blocks and used as input to the video quality evaluation model.

[0062] As a preferred embodiment, the video quality assessment model includes: a spatial feature extraction module, a temporal information processing module, and a quality score prediction module.

[0063] As a preferred embodiment, the spatial feature extraction module includes:

[0064] Video frames are fed as input into the Conv2Former backbone network. Conv2Former performs convolution operations on each video frame and outputs a set of spatial features at each convolutional layer. These spatial features represent image information at different levels and abstractions. In the feature pyramid module, spatial features from different levels obtained from the backbone network are used to generate multi-scale features.

[0065] As a preferred embodiment, the time-series information processing module includes:

[0066] The multi-scale features obtained from the feature pyramid module are fed into the Attention-LSTM model as input. This module consists of an input layer, an LSTM layer, an Attention layer, and a fully connected layer. The LSTM layer can capture long-term dependencies in video sequences, allowing the model to remember and understand important patterns and dynamics between distant time steps. The fully connected layer non-linearly maps and integrates the high-level features learned by the LSTM and Attention layers to obtain the final spatiotemporal feature representation.

[0067] As a preferred embodiment, the quality prediction module includes: weighting and combining the spatiotemporal feature representations obtained from the feature extraction part through multilayer perceptron (MLP) layers to map them into a quality score, which will reflect the perceptual quality and visual experience of the video.

[0068] This process employs a common paradigm for video quality assessment, first processing the spatial information of the video and then its temporal information. Video frames are input into a Conv2Former network, which serves as the spatial feature extraction module. During feature extraction, feature pyramid techniques are incorporated to further enhance the multi-scale representation of video frame features. The spatial features extracted by Conv2Former are then input into an Attention-LSTM model, which specifically handles the temporal information between video frames. Through a self-attention mechanism, Attention-LSTM weights video frames at different times, capturing the temporal relationships between them and thus providing a better understanding of the video's dynamic characteristics.

[0069] After processing by the Attention-LSTM model, the overall quality assessment result of the video is obtained through weighted summation and combination of multilayer perceptron (MLP) layers. This score will reflect the perceptual quality and visual experience of the video.

[0070] To address the issue of insufficient data, contrastive learning is employed to pre-train the model for the entire feature extraction component. Through self-supervised learning on unlabeled video data, contrastive learning enables the model to learn more discriminative feature representations. The pre-trained model exhibits better feature representation capabilities, contributing to improved accuracy and generalization ability in video quality assessment.

[0071] Finally, the pre-trained model is combined with a multilayer perceptron layer and fine-tuned and tested on a public dataset to obtain the final video quality score, thereby verifying the model's performance and generalization ability in various real-world scenarios.

[0072] Pre-training data includes:

[0073] Collect video data from the ImageNet VID dataset or other suitable datasets. Ensure the selected video data covers videos of different scenes, content, and quality to guarantee the model's generalization ability and applicability. Preprocess the collected video data to ensure data consistency and usability. Randomly sample a portion of the preprocessed video data as the input dataset for pre-training. Ensure the quantity and quality of the sampled videos meet the requirements of the pre-training process. Perform data augmentation on the sampled video data to increase its diversity and richness. Common data augmentation methods include random cropping, horizontal flipping, and color dithering to simulate changes in video quality and content under different conditions. To mitigate the impact of video resolution on model performance, crop all videos into 224×224 blocks as input to the video quality evaluation model.

[0074] like Figure 2 As shown, the video quality assessment model is trained by following these steps:

[0075] Step 1: Construct a dataset for model pre-training.

[0076] Furthermore, the method of step 1 is as follows:

[0077] Step 1.1: Select 500 videos from the ImageNet VID dataset or other applicable datasets as the input dataset for pre-training.

[0078] Step 1.2 Preprocesses the collected videos, including parsing the video format, unifying the frame rate and resolution, and cropping or padding video frames to achieve a uniform size and format.

[0079] Step 1.3 Perform data augmentation on the preprocessed video data, including random cropping, horizontal flipping, and color transformation, to increase the diversity and richness of the data and simulate changes in video quality and content under different conditions.

[0080] Step 2: Use the enhanced video as input to the model and use contrastive learning to perform self-supervised pre-training on the model for the entire feature extraction part.

[0081] Furthermore, the method for step 2 is as follows:

[0082] Step 2.1, the feature extraction part of the model, consists of two modules. The first is the spatial feature extraction module, which takes video frames as input and feeds them into the Conv2Former backbone network. Conv2Former combines the design concepts of ConvNet and Vision Transformer, simplifying self-attention through convolutional modulation operations. This improves efficiency when processing high-resolution images and reduces the computational cost caused by the self-attention mechanism. For an input labeled sequence X of length N, the traditional self-attention module first obtains the key, query, and value through a linear layer. The output is the attention matrix, which is the similarity score matrix.

[0083] Attention(X) = AV

[0084] The relationship between each pair of input tokens in the formula can be written as:

[0085] A = Softmax(QK) T )

[0086] The computational complexity of the self-attention module increases quadratically with the increase of sequence length N, resulting in higher computational costs.

[0087] The convolutional modulation module does not calculate the similarity score matrix using the above formula. Instead, it uses a k×k Depth-wise convolution and a Hadamard product to calculate the output Z:

[0088] Z = A⊙V,

[0089] A = DConv k×k (W1X)

[0090] V = W²X,

[0091] Where ⊙ is the Hadamard product, W1 and W2 are the weight matrices of the two linear layers, and DConv k×kThis represents a depthwise convolution with a kernel size of k×k. Conv2Former performs convolution operations on each video frame and outputs a set of spatial features at each convolutional layer. These spatial features represent image information at different levels and levels of abstraction. In the feature pyramid module, spatial features from different levels obtained from the backbone network are used to generate multi-scale features. Assume that the features output by each stage of Conv2Former are {C1, C2, C3, C4}. The formula for multi-scale feature pyramid fusion is as follows:

[0092] M4 = Conv(K,(1×1),1)(C4)

[0093] M i =Conv(K,(1×1),1)(C i )+Upsample(M i+1 ), i = 3, 2, 1

[0094] Subsequently, another convolutional layer with K kernels of size (3×3) is executed on the fused feature map {M1,M2,M3,M4} to generate the perceptual map {P1,P2,P3,P4}. {P1,P2,P3,P4} are then concatenated using adaptive average pooling to obtain multi-scale features, which serve as input to the temporal information processing module.

[0095] Step 2.2 inputs the multi-scale features obtained from the feature pyramid module into the temporal information processing module. This module consists of an input layer, an LSTM layer, an Attention layer, and a fully connected layer. The LSTM layer captures long-term dependencies in video sequences, allowing the model to memorize and understand important patterns and dynamics between distant time steps. The fully connected layer performs non-linear mapping and integration of the high-level features learned by the LSTM and Attention layers to obtain the final spatiotemporal feature representation.

[0096] Step 2.3 uses contrastive learning to pre-train the model for feature extraction. Through self-supervised learning on unlabeled video data, contrastive learning enables the model to learn more discriminative feature representations. The contrastive loss function used is:

[0097]

[0098] In the formula, τ is the temperature hyperparameter, q is a pre-coded feature, and k0, k1, k2, ... are a series of pre-coded samples, which can be viewed as part of a dictionary. Assume the dictionary contains only one key, k. + (called kpositive) is a match with q, then k + q and k are positive sample pairs, and the rest are positive sample pairs. iThese are negative samples of q. The pre-trained model has better feature representation capabilities, which helps improve the accuracy and generalization ability of video quality evaluation.

[0099] Step 3: After pre-training, the model is followed by an MLP layer. The entire quality assessment model is then fine-tuned, and its performance and generalization ability are tested. The loss for fine-tuning is a simple Mean Squared Error (MSE), as shown in the following formula:

[0100]

[0101] Where n represents the number of videos in the batch, Q′ i and Q i These are the subjective quality label and the predicted quality score. The fine-tuned model is then tested by weighting and combining the spatiotemporal feature representations obtained from feature extraction through a multilayer perceptron (MLP) layer to obtain the final video quality score. This score reflects the perceived quality and visual experience of the video.

[0102] This embodiment proposes a video quality assessment method and apparatus based on Conv2Former and contrastive learning. It can fully capture the spatial and temporal features of videos, thereby more accurately assessing video quality and making the evaluation results more reliable. Using Conv2Former as the backbone network, spatial features are extracted from video frames, reducing computation while maintaining model expressive power. Furthermore, feature pyramid technology is used to further enhance the multi-scale expressive power of video frame features. Simultaneously, contrastive learning is employed for pre-training, utilizing unlabeled video data to learn more discriminative feature representations, effectively addressing the problem of insufficient data and improving the model's generalization ability and performance. This provides an effective solution for evaluating the quality of user-generated content (UGC) videos—those generated in a non-professional manner.

[0103] The main feature extraction components of the video quality assessment model are spatial feature extraction and temporal information processing.

[0104] Self-attention is a key mechanism proposed in Transformer, but it suffers from a computational complexity of O(N^2), which imposes a significant computational burden. Therefore, Conv2Former proposes using a convolutional modulation module to simplify self-attention and reduce the computational load.

[0105] The key idea behind feature pyramids is to fuse features from different levels to obtain rich, multi-scale feature representations. This is achieved by extracting features from different layers of the network. Typically, Conv2Former models contain multiple convolutional layers or blocks, whose output features have different scales and semantic levels. Shallower features usually have higher resolution but lower semantic information, while deeper features have lower resolution but higher semantic information. The role of feature pyramids is to extract and utilize feature information at different scales to enhance the model's perceptual and expressive capabilities at multiple scales.

[0106] The self-attention module is only used to introduce the convolutional modulation module, and the feature pyramid is used to enrich the extracted features.

[0107] The final output of a video quality assessment is usually a numerical score representing the overall quality of the video. This score can be used to compare the quality differences between different videos or to determine whether a video meets the expected quality requirements.

[0108] This paper uses Conv2Former for spatial feature extraction. Conv2Former combines the design concepts of ConvNet and ViT and simplifies self-attention by using convolution modulation operation. This operation combines convolution operation with self-attention mechanism, which can reduce the amount of computation while maintaining the expressive power of the model.

[0109] The contrastive learning method presented in this paper is generally relatively simple and does not rely on domain expertise or complex task design. By comparing the similarities or differences between samples, useful feature representations can be learned effectively. Therefore, this paper uses contrastive learning as a self-supervised approach to handle VQA tasks.

[0110] Example 2:

[0111] This embodiment provides a video quality assessment device based on Conv2Former and contrastive learning, the device comprising:

[0112] Input module: Used to obtain the video to be evaluated;

[0113] Enhancement module: used to perform data enhancement and frame segmentation on the video to be evaluated to obtain a data-enhanced video, the video comprising a combination of consecutive frames;

[0114] Prediction module: This module takes the augmented video data and inputs it into the trained video quality assessment model to obtain a predicted video quality score.

[0115] The apparatus in this embodiment can be used to implement the method described in Embodiment 1.

[0116] Example 3:

[0117] This invention also provides a video quality evaluation device based on Conv2Former and contrastive learning, including a processor and a storage medium;

[0118] The storage medium is used to store instructions;

[0119] The processor is configured to operate according to the instructions to execute the steps of the method described in Embodiment 1.

[0120] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0122] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0123] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0124] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A video quality assessment method based on Conv2Former and contrastive learning, characterized in that, Includes the following steps: Get the videos to be evaluated; The video to be evaluated is subjected to data augmentation and frame segmentation to obtain a data-enhanced video, which consists of a combination of consecutive frames; The augmented video is input into the trained video quality assessment model to obtain the predicted video quality score to be evaluated. The video quality evaluation model includes a feature extraction part and a quality score prediction module; the feature extraction part includes a spatial feature extraction module and a temporal information processing module. The spatial feature extraction module is used to extract spatial features using feature pyramid technology; The time-series information processing module is used to process time-series information based on the extracted spatial features. Through a self-attention mechanism, it performs weighted processing on video frames at different times to capture the temporal relationship between video frames and obtain spatiotemporal features. The quality score prediction module includes a multi-layer perceptron layer, which is used to weight and combine the features output by the temporal information processing module, and map the spatiotemporal feature representation into a quality score, which reflects the perceptual quality and visual experience of the video. The spatial feature extraction module is a Conv2Former backbone network; The Conv2Former backbone network is used to perform convolution operations on each video frame and output a set of spatial features at each convolutional layer, which represent image information at different levels and levels of abstraction. The feature pyramid technique is used to generate multi-scale features by using spatial features at different levels obtained from the backbone network. The method for obtaining the trained video quality assessment model includes: Obtain the video set for model pre-training; The video set used for model pre-training is input into the video quality assessment model, and the feature extraction part of the video quality assessment model is self-supervised pre-trained using contrastive learning. After training, an additional MLP layer is added to the feature extraction part of the obtained video quality assessment model to form the final video quality assessment model.

2. The video quality assessment method based on Conv2Former and contrastive learning according to claim 1, characterized in that, The time-series information processing module is an Attention-LSTM model; The Attention-LSTM model consists of an input layer, an LSTM layer, an Attention layer, and a fully connected layer. The LSTM layer is used to capture long-term dependencies in the video sequence, and the fully connected layer performs non-linear mapping and integration of the high-level features learned by the LSTM layer and the Attention layer to obtain the final spatiotemporal features.

3. The video quality assessment method based on Conv2Former and contrastive learning according to claim 1, characterized in that, Methods for obtaining video sets for model pre-training include: Select 500 videos from the ImageNet VID dataset or other applicable datasets as the pre-input dataset; The pre-input dataset is preprocessed, including parsing the video format, unifying the frame rate and resolution, and cropping or padding video frames to achieve a uniform size and format, resulting in preprocessed video data. The preprocessed video data is augmented by random cropping, horizontal flipping, and color transformation to obtain a video set for model pre-training.

4. The video quality assessment method based on Conv2Former and contrastive learning according to claim 1, characterized in that, The video set used for model pre-training is input into the video quality assessment model. A contrastive learning approach is used to perform self-supervised pre-training on the feature extraction part of the video quality assessment model, including: Self-supervised learning is performed on unlabeled video data, and the contrastive loss function used is: ； In the formula, τ is the temperature hyperparameter, and q is a pre-coded feature. It is a series of coded samples from dictionaries, each dictionary containing only one key. Is it with If it matches, then and For pairs that are mutually positive samples, the remaining keys... for Negative samples.

5. The video quality assessment method based on Conv2Former and contrastive learning according to claim 1, characterized in that, The method further includes: The entire quality assessment model is then fine-tuned, using a simple mean squared error loss as the adjustment loss, as shown in the following formula: ； Where n represents the number of videos in the batch. and These are subjective quality labels and predicted quality scores; The fine-tuned model is tested, and the spatiotemporal feature representations obtained from the feature extraction part are weighted and combined through a multi-layer perceptron layer to obtain the final video quality score.

6. A video quality assessment apparatus based on Conv2Former and contrastive learning for performing the method as described in claim 1, characterized in that, The device includes: Input module: Used to obtain the video to be evaluated; Enhancement module: used to perform data enhancement and frame segmentation on the video to be evaluated to obtain a data-enhanced video, the video comprising a combination of consecutive frames; Prediction module: This module takes the augmented video data and inputs it into the trained video quality assessment model to obtain a predicted video quality score.

7. A video quality evaluation device based on Conv2Former and contrastive learning, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1-5.

Images