A CLIP-based video quality evaluation method and device

Abstract

The application relates to the technical field of video quality evaluation, and discloses a video quality evaluation method and device based on CLIP. The method extracts frame tokens in a video to be evaluated, uses a preset CLIPVQA model to concatenate the frame tokens and preset pseudo MOS tokens, generates frame token representation, pseudo MOS token representation containing quality information of video frames, and fusion token representation containing space-time information of the video frames; uses a space-time quality aggregation network to obtain quality representation at a video level, encodes language description of the video based on a CLIP-based encoder, extracts and analyzes space-time quality characteristics, and then generates video language quality representation through video content and language aggregation; and finally, the quality representation at the video level and the video language quality representation are fused to obtain an evaluation result of the video to be evaluated. The CLIPVQA model has the most advanced VQA performance, good generalization ability and robustness, and can improve video quality evaluation precision and video quality prediction accuracy.

Description

Technical Field

[0001] This invention relates to the field of video quality assessment technology, and in particular to a CLIP-based video quality assessment method and apparatus. Background Technology

[0002] With advancements in internet and video compression technologies, user-generated video content is growing exponentially across various social media platforms such as TikTok, YouTube, Meta, and Twitter. Simultaneously, the emergence of high-definition photography equipment has significantly increased the resolution of videos created by both professional and amateur users (e.g., 1080p, 4K, and even 8K videos). The ever-increasing volume and resolution of videos in real-world environments present significant challenges to computer vision systems, particularly in video quality assessment (VQA). Therefore, developing efficient and feasible VQA methods is crucial for these real-world videos to better meet users' demands for quality of experience (QoE).

[0003] Currently, based on the availability of the original reference video, Video Quality Assurance (VQA) can be divided into three categories: full-reference VQA, partial-reference VQA, and no-reference / blind VQA. The first two VQA paradigms heavily or partially rely on the reference for lossless video generation quality prediction, making them impractical in scenarios where the original video is unavailable or even impossible to obtain. Current no-reference VQA suffers from a significant gap between the source and target domains, and its performance is poor due to the inability to integrate network structure and video quality assessment. Summary of the Invention

[0004] This invention provides a CLIP-based video quality assessment method and apparatus, which uses the CLIPVQA model to assess video quality, thereby improving the accuracy of video quality assessment and expanding the application of CLIP in VQA scenarios.

[0005] To address the aforementioned technical problems, this invention provides a CLIP-based video quality assessment method, comprising:

[0006] Extract frame tokens from the video to be evaluated;

[0007] Using a preset CLIPVQA model, the frame token and the preset pseudo-MOS token are frame-aware to generate a frame token representation, a pseudo-MOS token representation, and a fused token representation. Spatiotemporal quality aggregation is then performed on the pseudo-MOS token representation and the fused token representation to generate a video-level quality representation. Subsequently, a video language quality representation is generated through the language aggregation module and the video content module within the model. The video-level quality representation and the video language quality representation are then fused to obtain the quality assessment result of the video to be evaluated.

[0008] This invention extracts frame tokens from the video to be evaluated, and uses a preset CLIP VQA model to perform frame awareness on the frame tokens and preset pseudo-MOS tokens to generate frame token representations, pseudo-MOS token representations, and fused token representations. It then uses a spatiotemporal quality aggregation network to obtain video-level quality representations; generates video language quality representations through video content and language aggregation; and finally fuses the video-level quality representations and video language quality representations to obtain the evaluation results for the video to be evaluated, thus achieving video quality assessment, improving the accuracy of video quality assessment, and expanding the application of CLIP in VQA scenarios.

[0009] Furthermore, the extraction of frame tokens from the video to be evaluated specifically involves:

[0010] Sample video frames from the video to be evaluated;

[0011] A two-dimensional convolutional network is used to embed the position information of the video frame into the video frame to generate a frame token.

[0012] Furthermore, the step of performing frame awareness on the frame token and the preset pseudo-MOS token to generate a frame token representation, a pseudo-MOS token representation, and a fused token representation specifically involves:

[0013] The frame token and the preset pseudo MOS token are concatenated and processed using the frame-aware Transformer in the CLIPVQA model to generate frame markers and pseudo MOS tokens.

[0014] The pseudo-MOS tokens are processed using multi-head self-attention to obtain the fusion token;

[0015] The frame-aware Transformer is used to perform deep processing on the frame marker, pseudo-MOS, and fusion marker to generate a frame token representation, a pseudo-MOS token representation, and a first fusion token representation, respectively; wherein, the pseudo-MOS token representation includes the quality information of the video frame; and the fusion token representation includes the spatiotemporal information of the video frame.

[0016] This invention utilizes a frame-aware Transformer to extract both the frame marker and pseudo-MOS of a video frame. The pseudo-MOS token is processed using a self-attention mechanism to obtain a fusion marker, thus acquiring the spatiotemporal information of the video frame. By using the frame-aware Transformer to deeply process the frame marker, pseudo-MOS, and fusion marker, frame-level spatiotemporal quality features can be generated, which can reduce the computational load of acquiring spatiotemporal features and improve computational efficiency.

[0017] Furthermore, the spatiotemporal quality aggregation of the pseudo-MOS token representation and the fused token representation to generate a video-level quality representation specifically involves:

[0018] The location information of the video frame is embedded into the pseudo MOS token representation;

[0019] The fusion token representation is weighted.

[0020] The pseudo-MOS token representation with embedded location information and the weighted fused token representation are aggregated to generate a video-level quality representation.

[0021] This invention processes the pseudo-MOS token representation containing quality information of video frames and the fused token representation containing spatiotemporal information of video frames, respectively, and aggregates the processed pseudo-MOS token representation and fused token representation to generate a video-level quality representation. This deeply integrates the quality information and spatiotemporal information of the video, improving the accuracy of video quality prediction.

[0022] Furthermore, the generation of video language quality representation through the language aggregation module and video content module within the model specifically involves:

[0023] Using the language aggregation module in the CLIPVQA model, the natural language in the video to be evaluated is coded to generate quality language codes;

[0024] The video content module in the CLIPVQA model is used to perform dimensionality reduction on the frame token representation to generate a video content representation;

[0025] The quality language encoding and the video content representation are aggregated to generate a video language quality representation.

[0026] Furthermore, the step of performing dimensionality reduction processing on the frame token representation to generate a video content representation specifically involves:

[0027] Using a two-dimensional convolutional network, the frame token representation is reduced in dimensionality to generate a video content representation, specifically expressed as follows:

[0028]

[0029] Among them, Y c For the representation of video content, It is represented by a frame token.

[0030] Furthermore, the process of fusing the video-level quality representation and the video language quality representation to obtain the quality assessment result of the video to be evaluated is as follows:

[0031] The video-level quality representation and the video language quality representation are fused together by element-wise multiplication to generate the prediction vector of the video to be evaluated.

[0032] The expression for the element-wise multiplication is:

[0033]

[0034] in, Let be the predicted vector of the video to be evaluated. For video language quality representation; y v represents the quality at the video level; Soft(·) is the Softmax activation function.

[0035] Furthermore, the CLIPVQA model is specifically as follows:

[0036] Acquire several training videos; wherein the training videos include a first training video and a second training video;

[0037] The first training video is input into the CLIPVQA model to obtain the prediction vector of the first training video;

[0038] Obtain the quality score of the second training video;

[0039] Convert the mass score numerical value into a mass score vector;

[0040] A loss function is generated by combining the quality score vector and the prediction vector of the first training video;

[0041] The CLIPVQA model is optimized using the loss function.

[0042] In optimizing the CLIPVQA model, this invention inputs a first training video into the model to obtain a prediction vector, and converts the actual quality score of the second training video into a quality score vector. Then, the prediction vector of the first training video and the quality score vector of the second training video are combined to generate a loss function. Using this loss function to optimize the CLIPVQA model can improve the accuracy of video quality assessment.

[0043] This invention provides a CLIP-based video quality assessment method. Frame tokens are extracted from the video to be evaluated. Using a pre-defined CLIP VQA model, the frame tokens and pre-defined pseudo-MOS tokens are concatenated to generate a frame token representation, a pseudo-MOS token representation containing quality information of the video frames, and a fused token representation containing spatiotemporal information of the video frames. A spatiotemporal quality aggregation network is used to obtain the video-level quality representation. The video's language description is encoded using a CLIP-based encoder to extract and analyze spatiotemporal quality features. Then, a video language quality representation is generated through video content and language aggregation. Finally, the video-level quality representation and the video language quality representation are fused to obtain the evaluation result of the video to be evaluated, thus achieving video quality assessment, improving the accuracy of video quality assessment, and expanding the application of CLIP in VQA scenarios.

[0044] Accordingly, the present invention provides a CLIP-based video quality assessment device, comprising: an extraction module and an assessment module;

[0045] The extraction module is used to extract frame tokens from the video to be evaluated;

[0046] The evaluation module is used to perform frame awareness on the frame token and the preset pseudo-MOS token using a preset CLIPVQA model, generating a frame token representation, a pseudo-MOS token representation, and a fused token representation. It then performs spatiotemporal quality aggregation on the pseudo-MOS token representation and the fused token representation to generate a video-level quality representation. Subsequently, it generates a video language quality representation through the language aggregation module and the video content module within the model, and fuses the video-level quality representation and the video language quality representation to obtain the quality evaluation result of the video to be evaluated.

[0047] Furthermore, the evaluation module includes: an embedding unit, a weighting unit, and an aggregation unit;

[0048] The embedding unit is used to embed the position information of the video frame into the pseudo MOS token representation;

[0049] The weighting unit is used to weight the fusion token representation;

[0050] The aggregation unit is used to aggregate the pseudo MOS token representation embedded with location information and the weighted fused token representation to generate a video-level quality representation.

[0051] This invention provides a CLIP-based video quality assessment device, which, based on the organic integration of modules, can improve the accuracy of video quality assessment and expand the application of CLIP in VQA scenarios. Attached Figure Description

[0052] Figure 1 A flowchart illustrating an embodiment of the CLIP-based video quality assessment method provided by the present invention;

[0053] Figure 2 This is a schematic diagram of the CLIPVQA model provided by the present invention;

[0054] Figure 3 This is a schematic diagram of one embodiment of the CLIP-based video quality assessment device provided by the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Example 1

[0057] See Figure 1 This is a flowchart illustrating an embodiment of the CLIP-based video quality assessment method provided by the present invention. The method includes steps 101 to 102, each of which is detailed below:

[0058] Step 101: Extract frame tokens from the video to be evaluated.

[0059] Furthermore, in the first embodiment of the present invention, the step of extracting frame tokens from the video to be evaluated specifically includes:

[0060] Sample video frames from the video to be evaluated;

[0061] A two-dimensional convolutional network is used to embed the position information of the video frame into the video frame to generate a frame token.

[0062] In the first embodiment of the present invention, a preset number of video frames are sampled from the video sequence to be evaluated, and then the video frames are input into a two-dimensional convolutional network. Embedding the position information of the video frames generates frame tokens. Specifically, to reduce redundancy between different frames in the video, an equal-interval sampling scheme can be used for frame selection. Each sampled frame is randomly cropped into an image of size h×w×3, where h and w represent the height and width of the cropped frame, respectively, and 3 represents the number of color channels. Through image cropping, most of the semantic information of the original frame can be covered within dozens of training cycles, while maintaining image quality. Subsequently, each cropped video frame is divided into non-overlapping patches of size s×s, thereby obtaining... Each patch is further expanded into a token of length d = s × s × 3. Therefore, assuming N video frames are sampled, x(n,p)∈R d Let represent the initial token of the p-th patch in the n-th selected frame, where n∈[1,N] and p∈[1,P]. In this way, a sampled video can contain K = N × P tokens. Furthermore, to capture long-range dependencies of spatial information through a self-attention mechanism, spatial location information needs to be embedded into the generated frame tokens. Therefore, frame token extraction can be represented as:

[0063] e(n,p)=Θ·x(n,p)+v(n,p)

[0064] Where, v(n,p)∈R d Representing a spatial location vector, it is generated through position encoding based on a sine function. Θ∈R d ×d Let e(n,p)∈R be a learnable matrix. d It is the frame token representation of the p-th patch in the n-th frame.

[0065] Step 102: Using the preset CLIPVQA model, perform frame awareness on the frame token and the preset pseudo-MOS token to generate a frame token representation, a pseudo-MOS token representation, and a fused token representation. Then, perform spatiotemporal quality aggregation on the pseudo-MOS token representation and the fused token representation to generate a video-level quality representation. Subsequently, generate a video language quality representation through the language aggregation module and the video content module within the model. Finally, fuse the video-level quality representation and the video language quality representation to obtain the quality assessment result of the video to be evaluated.

[0066] Furthermore, in the first embodiment of the present invention, the frame token and the preset pseudo-MOS token are subjected to frame awareness to generate a frame token representation, a pseudo-MOS token representation, and a fused token representation, specifically as follows:

[0067] The frame token and the preset pseudo MOS token are concatenated and processed using the frame-aware Transformer in the CLIPVQA model to generate frame markers and pseudo MOS tokens.

[0068] The pseudo-MOS tokens are processed using multi-head self-attention to obtain the fusion token;

[0069] The frame-aware Transformer is used to perform deep processing on the frame marker, pseudo-MOS, and fusion marker to generate a frame token representation, a pseudo-MOS token representation, and a first fusion token representation, respectively; wherein, the pseudo-MOS token representation includes the quality information of the video frame; and the fusion token representation includes the spatiotemporal information of the video frame.

[0070] In the first embodiment of the present invention, the frame marker and pseudo MOS of the video frame are extracted by using a frame-aware Transformer. The pseudo MOS token is processed by a self-attention mechanism to obtain the fusion marker and obtain the spatiotemporal information of the video frame. The frame marker, pseudo MOS and fusion marker are processed by the frame-aware Transformer to generate frame-level spatiotemporal quality features, which can reduce the amount of computation to obtain spatiotemporal features and improve computational efficiency.

[0071] In the first embodiment of this invention, a pseudo-MOS token is randomly generated for each video frame and concatenated with the corresponding frame token. The frame token and pseudo-MOS token are concatenated using a frame-aware Transformer in the CLIPVQA model to fully capture the latent spatiotemporal information of the video frame. Specifically, the frame-aware Transformer is a fixed set of CViT blocks. For the first CViT block, the generated pseudo-MOS token is fed into the first layer of a Visual Transformer (ViT) network. The ViT network extracts spatial information within the frame. The output is then rearranged into two matrices, corresponding to the frame token and the pseudo-MOS, respectively. The pseudo-MOS token is then further processed by a single-layer multi-head self-attention (MSA) layer to obtain a fused token. After deep processing by the frame-aware Transformer, a frame token representation, a pseudo-MOS token representation, and a first fused token representation can be generated.

[0072] In the first embodiment of this invention, the frame-aware Transformer in the CLIPVQA model enables information exchange between multiple video frames to capture inter-frame temporal information. To better capture the spatiotemporal features of the entire video, L CViT blocks are executed sequentially to process the concatenation of frame markers, pseudo-MOS markers, and fusion markers. As L increases, the number of fusion markers gradually increases. As this process proceeds, the pseudo-MOS marker gradually evolves into a state that not only reflects the quality of a single frame but also summarizes the overall video quality. This is achieved by enabling information exchange and integration between the pseudo-MOS markers of multiple video frames, as well as the correlation between the pseudo-MOS markers and the fusion marker.

[0073] Furthermore, in the first embodiment of the present invention, spatiotemporal quality aggregation is performed on the pseudo-MOS token representation and the fused token representation to generate a video-level quality representation, specifically as follows:

[0074] The location information of the video frame is embedded into the pseudo MOS token representation;

[0075] The fusion token representation is weighted.

[0076] The pseudo-MOS token representation with embedded location information and the weighted fused token representation are aggregated to generate a video-level quality representation.

[0077] In the first embodiment of the present invention, the pseudo MOS token representation containing quality information of video frames and the fused token representation containing spatiotemporal information of video frames are processed respectively, and the processed pseudo MOS token representation and fused token representation are aggregated to generate a video-level quality representation, which deeply integrates the quality information and spatiotemporal information of the video and improves the accuracy of video quality prediction.

[0078] In the first embodiment of this invention, after the frame-aware stage, pseudo-MOS token representations and fused token representations can be obtained, both carrying rich spatiotemporal information and video quality-related information. Therefore, a spatiotemporal quality aggregation Transformer (SAT) can fully utilize the pseudo-MOS token representations and fused token representations to generate a video-level quality representation. The SAT mainly consists of a single-layer multi-head self-attention (MSA) and a multilayer perceptron (MLP). First, the pseudo-MOS token representation needs further transformation before aggregation. To more efficiently obtain the underlying temporal information, the positional information of the video frames is embedded into the pseudo-MOS token representation. Then, the fused token representation is weighted. Finally, the processed pseudo-MOS token representation and fused token representation are aggregated together to generate a video-level quality representation.

[0079] Furthermore, in the first embodiment of the present invention, a video language quality representation is generated through a language aggregation module and a video content module within the model, specifically as follows:

[0080] Using the language aggregation module in the CLIPVQA model, the natural language in the video to be evaluated is coded to generate quality language codes;

[0081] The video content module in the CLIPVQA model is used to perform dimensionality reduction on the frame token representation to generate a video content representation;

[0082] The quality language encoding and the video content representation are aggregated to generate a video language quality representation.

[0083] Furthermore, in the first embodiment of the present invention, the frame token representation is subjected to dimensionality reduction processing to generate a video content representation, specifically as follows:

[0084] Using a two-dimensional convolutional network, the frame token representation is reduced in dimensionality to generate a video content representation, specifically expressed as follows:

[0085]

[0086] Among them, Y c For the representation of video content, M f L rame It is represented by a frame token.

[0087] In the first embodiment of the present invention, the language aggregation module in the CLIPVQA model can be used to generate quality language codes. The language aggregation module includes a CLIP-based text encoder, which is mainly composed of multiple MSAs. It can encode the natural language in the video to be evaluated to generate quality language codes, providing language supervision for subsequent representation learning.

[0088] In the first embodiment of this invention, a frame token representation carrying rich video content information can be obtained by executing a Frame Aware Transformer (FPT). First, to reduce redundant information in video frames and adapt to the input requirements of CLIP-based networks, the video content module in the CLIP VQA model can be used to perform dimensionality reduction processing on the frame token representation using a two-dimensional convolutional network to generate a video content representation. The quality language encoding and video content representation are then aggregated using a Video Content and Language Aggregation Transformer (VAT). Using quality language encoding as supervision, the aggregation of quality language encoding and video content representation can generate a video-language representation. Specifically, the Video Content and Language Aggregation Transformer (VAT) mainly consists of several content and language aggregation (CandLA) blocks. Each CandLA block contains a LayerNorm operation (LN) and a multi-head cross-attention (MCA) operation, and then includes a multilayer perceptron (MLP).

[0089] Furthermore, in the first embodiment of the present invention, the quality assessment result of the video to be evaluated is obtained by fusing the video-level quality representation and the video language quality representation, specifically as follows:

[0090] The video-level quality representation and the video language quality representation are fused together by element-wise multiplication to generate the prediction vector of the video to be evaluated.

[0091] The expression for the element-wise multiplication is:

[0092]

[0093] in, Let be the predicted vector of the video to be evaluated. For video language quality representation; y v represents the quality at the video level; Soft(·) is the Softmax activation function.

[0094] Furthermore, in the first embodiment of the present invention, the CLIPVQA model is specifically as follows:

[0095] Acquire several training videos; wherein the training videos include a first training video and a second training video;

[0096] The first training video is input into the CLIPVQA model to obtain the prediction vector of the first training video;

[0097] Obtain the quality score of the second training video;

[0098] Convert the mass score numerical value into a mass score vector;

[0099] A loss function is generated by combining the quality score vector and the prediction vector of the first training video;

[0100] The CLIPVQA model is optimized using the loss function.

[0101] In the first embodiment of this invention, in order for the objective VQA model to generate video quality predictions that are highly consistent with subjective ratings, a vector regression (VR) loss function can be used to optimize the CLIPVQA model to improve evaluation accuracy. The key to optimizing the model is to convert the true quality score into a vector, and then combine it with the predicted probability vector to generate the final loss function. After deriving the loss function, it can be used to optimize the CLIPVQA model.

[0102] As an example of the first embodiment of the present invention, see Figure 2 This is a schematic diagram of the CLIPVQA model provided by the present invention. The model includes a frame-aware Transformer (FPT), a Transformer (SAT) that fuses spatiotemporal information, a MOS2Language encoder, a video content and language fusion Transformer (VAT), and a feature fusion module.

[0103] Among them, the Frame Aware Transformer (FPT) is used to perform frame awareness on the frame token and the pseudo MOS token, and generate the frame token representation, the pseudo MOS token representation and the fused token representation;

[0104] The Transformer (SAT) that integrates spatiotemporal information is used to perform spatiotemporal quality aggregation on the pseudo MOS token representation and the fused token representation to generate a video-level quality representation;

[0105] The MOS2Language encoder is used to encode natural language in video, generating high-quality language encoding.

[0106] The Video Content and Language Fusion Transformer (VAT) is used to aggregate quality language encoding and video content representation to generate a video language quality representation.

[0107] The feature fusion module is used to fuse video-level quality representation and video language quality representation to obtain the quality assessment result of the video to be evaluated.

[0108] The CLIPVQA model can be optimized using a vectorized regression loss function.

[0109] In summary, the first embodiment of this invention provides a CLIP-based video quality assessment method. It extracts frame tokens from the video to be evaluated, and uses a pre-defined CLIP VQA model to concatenate the frame tokens and pre-defined pseudo-MOS tokens to generate a frame token representation, a pseudo-MOS token representation containing quality information of the video frames, and a fused token representation containing spatiotemporal information of the video frames. A spatiotemporal quality aggregation network is used to obtain the video-level quality representation. A CLIP-based encoder encodes the video's language description to extract and analyze spatiotemporal quality features. Then, video language quality representation is generated through video content and language aggregation. Finally, the video-level quality representation and the video language quality representation are fused to obtain the evaluation result of the video to be evaluated, thus achieving video quality assessment, improving the accuracy of video quality assessment, and expanding the application of CLIP in VQA scenarios.

[0110] Example 2

[0111] See Figure 3 This is a schematic diagram of an embodiment of the CLIP-based video quality assessment device provided by the present invention. The device includes an extraction module 201 and an assessment module 202.

[0112] Extraction module 201 is used to extract frame tokens from the video to be evaluated;

[0113] The evaluation module 202 is used to perform frame awareness on the frame token and the preset pseudo MOS token using a preset CLIPVQA model, generate a frame token representation, a pseudo MOS token representation and a fused token representation, and perform spatiotemporal quality aggregation on the pseudo MOS token representation and the fused token representation to generate a video-level quality representation. Then, a video language quality representation is generated through the language aggregation module and the video content module within the model, so as to fuse the video-level quality representation and the video language quality representation to obtain the quality evaluation result of the video to be evaluated.

[0114] Furthermore, in the second embodiment of the present invention, the extraction module 201 includes: a sampling unit and a frame token generation unit;

[0115] The sampling unit is used to sample video frames from the video to be evaluated;

[0116] The frame token generation unit is used to embed the position information of the video frame into the video frame using a two-dimensional convolutional network to generate a frame token.

[0117] Furthermore, in the second embodiment of the present invention, the evaluation module 202 includes: a concatenation unit, a fusion marker acquisition unit, and a depth processing unit;

[0118] The concatenation unit is used to concatenate the frame token and the preset pseudo MOS token, and process them using the frame-aware Transformer in the CLIPVQA model to generate frame markers and pseudo MOS.

[0119] The fusion token acquisition unit is used to process the pseudo MOS tokens using multi-head self-attention to obtain the fusion token;

[0120] The deep processing unit is used to perform deep processing on the frame marker, pseudo MOS and fusion marker using the frame-aware Transformer, and generate a frame token representation, a pseudo MOS token representation and a first fusion token representation, respectively; wherein, the pseudo MOS token representation includes the quality information of the video frame; and the fusion token representation includes the spatiotemporal information of the video frame.

[0121] Furthermore, in the second embodiment of the present invention, the evaluation module 202 includes: an embedding unit, a weighting unit, and an aggregation unit;

[0122] The embedding unit is used to embed the position information of the video frame into the pseudo MOS token representation;

[0123] The weighting unit is used to perform weighting processing on the fusion token representation;

[0124] The aggregation unit is used to aggregate the pseudo MOS token representation with embedded location information and the weighted fused token representation to generate a video-level quality representation.

[0125] Furthermore, in the second embodiment of the present invention, the evaluation module 202 includes: a quality language encoding generation unit, a video content representation generation unit, and a video language quality representation generation unit;

[0126] The quality language coding generation unit is used to encode the natural language in the video to be evaluated using the language aggregation module in the CLIPVQA model, and generate quality language codes.

[0127] The video content representation generation unit is used to perform dimensionality reduction processing on the frame token representation using the video content module in the CLIPVQA model to generate a video content representation.

[0128] The video language quality representation generation unit is used to aggregate the quality language encoding and the video content representation to generate a video language quality representation.

[0129] Furthermore, in the second embodiment of the present invention, the video content representation generation unit includes a dimensionality reduction processing subunit;

[0130] The dimensionality reduction subunit is used to perform dimensionality reduction processing on the frame token representation using a two-dimensional convolutional network to generate a video content representation, specifically expressed as follows:

[0131]

[0132] Among them, Y c For the representation of video content, It is represented by a frame token.

[0133] Furthermore, in the second embodiment of the present invention, the evaluation module 202 includes: a prediction vector generation unit;

[0134] The prediction vector generation unit is used to fuse the video-level quality representation and the video language quality representation through element-wise multiplication to generate the prediction vector of the video to be evaluated.

[0135] The expression for the element-wise multiplication is:

[0136]

[0137] in, Let be the predicted vector of the video to be evaluated. For video language quality representation; y v represents the quality at the video level; Soft(·) is the Softmax activation function.

[0138] Furthermore, in the second embodiment of the present invention, the CLIPVQA model is specifically as follows:

[0139] Acquire several training videos; wherein the training videos include a first training video and a second training video;

[0140] The first training video is input into the CLIPVQA model to obtain the prediction vector of the first training video;

[0141] Obtain the quality score of the second training video;

[0142] Convert the mass score numerical value into a mass score vector;

[0143] A loss function is generated by combining the quality score vector and the prediction vector of the first training video;

[0144] The CLIPVQA model is optimized using the loss function.

[0145] In summary, the second embodiment of the present invention provides a CLIP-based video quality assessment device. Based on the organic integration of modules, it extracts frame tokens from the video to be evaluated. Using a preset CLIP VQA model, the frame tokens and preset pseudo-MOS tokens are concatenated to generate a frame token representation, a pseudo-MOS token representation containing the quality information of the video frame, and a fused token representation containing the spatiotemporal information of the video frame. A spatiotemporal quality aggregation network is used to obtain the video-level quality representation. A CLIP-based encoder encodes the video's language description to extract and analyze spatiotemporal quality features. Then, a video language quality representation is generated through video content and language aggregation to achieve the video's language representation. Finally, the video-level quality representation and the video language quality representation are fused to obtain the evaluation result of the video to be evaluated, thus achieving video quality assessment, improving the accuracy of video quality assessment, and expanding the application of CLIP in VQA scenarios.

[0146] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A CLIP-based video quality assessment method, characterized in that, include: Extract frame tokens from the video to be evaluated; Using a pre-defined CLIPVQA model, the frame token and a pre-defined pseudo-MOS token are frame-aware, generating a frame token representation, a pseudo-MOS token representation, and a fused token representation. Spatiotemporal quality aggregation is then performed on the pseudo-MOS token representation and the fused token representation to generate a video-level quality representation. Subsequently, a video language quality representation is generated through the language aggregation module and video content module within the model. The video-level quality representation and the video language quality representation are then fused to obtain the quality assessment result of the video to be evaluated. The CLIPVQA model includes: a frame-aware Transformer, a Transformer fusing spatiotemporal information, a MOS2Language encoder, a video content and language fusion Transformer, and a feature fusion module. The frame-aware Transformer performs frame awareness on the frame token and the pseudo-MOS token, generating a frame token representation, a pseudo-MOS token representation, and a fused token representation. The spatiotemporal information fusion Transformer performs spatiotemporal quality aggregation on the pseudo-MOS token representation and the fused token representation to generate a video-level quality representation. MOS2Language... The encoder is used to encode the natural language in the video to generate a quality language code; the video content and language fusion transformer is used to aggregate the quality language code and the video content representation to generate a video language quality representation; the feature fusion module is used to fuse the video-level quality representation and the video language quality representation to obtain the quality assessment result of the video to be evaluated.

2. The CLIP-based video quality assessment method according to claim 1, characterized in that, The extraction of frame tokens from the video to be evaluated specifically involves: Sample video frames from the video to be evaluated; A two-dimensional convolutional network is used to embed the position information of the video frame into the video frame to generate a frame token.

3. The CLIP-based video quality assessment method according to claim 1, characterized in that, The step of performing frame awareness on the frame token and the preset pseudo-MOS token to generate a frame token representation, a pseudo-MOS token representation, and a fused token representation specifically involves: The frame token and the preset pseudo MOS token are concatenated and processed using the frame-aware Transformer in the CLIPVQA model to generate frame markers and pseudo MOS tokens. The pseudo-MOS tokens are processed using multi-head self-attention to obtain the fusion token; The frame-aware Transformer is used to perform deep processing on the frame marker, pseudo-MOS, and fusion marker to generate a frame token representation, a pseudo-MOS token representation, and a first fusion token representation, respectively; wherein, the pseudo-MOS token representation includes the quality information of the video frame; The fusion token represents spatiotemporal information including the video frame.

4. The CLIP-based video quality assessment method according to claim 2, characterized in that, The process of spatiotemporal quality aggregation of the pseudo-MOS token representation and the fused token representation to generate a video-level quality representation specifically involves: The location information of the video frame is embedded into the pseudo MOS token representation; The fusion token representation is weighted. The pseudo-MOS token representation with embedded location information and the weighted fused token representation are aggregated to generate a video-level quality representation.

5. The CLIP-based video quality assessment method according to claim 1, characterized in that, The generation of video language quality representation through the language aggregation module and video content module within the model specifically involves: Using the language aggregation module in the CLIPVQA model, the natural language in the video to be evaluated is coded to generate quality language codes; The video content module in the CLIPVQA model is used to perform dimensionality reduction on the frame token representation to generate a video content representation; The quality language encoding and the video content representation are aggregated to generate a video language quality representation.

6. The CLIP-based video quality assessment method according to claim 5, characterized in that, The step of performing dimensionality reduction processing on the frame token representation to generate a video content representation specifically involves: Using a two-dimensional convolutional network, the frame token representation is reduced in dimensionality to generate a video content representation, specifically expressed as follows: in, For the representation of video content, It is represented by a frame token.

7. The CLIP-based video quality assessment method according to claim 1, characterized in that, The process of fusing the video-level quality representation and the video language quality representation to obtain the quality assessment result of the video to be evaluated is as follows: The video-level quality representation and the video language quality representation are fused together by element-wise multiplication to generate the prediction vector of the video to be evaluated. The expression for the element-wise multiplication is: in, Let be the predicted vector of the video to be evaluated. For video language quality representation; This represents the quality at the video level. This is the Softmax activation function.

8. The CLIP-based video quality assessment method according to claim 1, characterized in that, The CLIPVQA model is as follows: Acquire several training videos; wherein the training videos include a first training video and a second training video; The first training video is input into the CLIPVQA model to obtain the prediction vector of the first training video; Obtain the quality score of the second training video; Convert the mass score numerical value into a mass score vector; A loss function is generated by combining the quality score vector and the prediction vector of the first training video; The CLIPVQA model is optimized using the loss function.

9. A CLIP-based video quality assessment device, characterized in that, include: Extraction module and evaluation module; The extraction module is used to extract frame tokens from the video to be evaluated; The evaluation module utilizes a preset CLIPVQA model to perform frame awareness on the frame token and a preset pseudo-MOS token, generating a frame token representation, a pseudo-MOS token representation, and a fused token representation. It then performs spatiotemporal quality aggregation on the pseudo-MOS token representation and the fused token representation to generate a video-level quality representation. Subsequently, a video language quality representation is generated through a language aggregation module and a video content module within the model. Finally, the video-level quality representation and the video language quality representation are fused to obtain the quality evaluation result of the video to be evaluated. The CLIPVQA model includes: a frame-aware Transformer, a Transformer fusing spatiotemporal information, a MOS2Language encoder, a video content and language fusion Transformer, and a feature fusion module. The frame-aware Transformer performs frame awareness on the frame token and the pseudo-MOS token, generating a frame token representation, a pseudo-MOS token representation, and a fused token representation. The spatiotemporal information fusion Transformer performs spatiotemporal quality aggregation on the pseudo-MOS token representation and the fused token representation to generate a video-level quality representation. The MOS2Language... The encoder is used to encode the natural language in the video to generate a quality language code; the video content and language fusion transformer is used to aggregate the quality language code and the video content representation to generate a video language quality representation; the feature fusion module is used to fuse the video-level quality representation and the video language quality representation to obtain the quality assessment result of the video to be evaluated.

10. The CLIP-based video quality assessment device according to claim 9, characterized in that, The evaluation module includes: an embedding unit, a weighting unit, and an aggregation unit; The embedding unit is used to embed the position information of the video frame into the pseudo MOS token representation; The weighting unit is used to weight the fusion token representation; The aggregation unit is used to aggregate the pseudo MOS token representation embedded with location information and the weighted fused token representation to generate a video-level quality representation.

Images