Zero sample video quality evaluation method based on auxiliary opinions

By using adversarial language-image pre-training model CLIP for combined joint training and migration knowledge in video quality evaluation, the problem of limited scale of data set based on opinion-driven methods is solved, and the generalization and perception capabilities of video quality evaluation are improved. , achieving a more comprehensive video quality evaluation.

CN118918505APending Publication Date: 2024-11-08FUZHOU UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410942902.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Opinion-driven video quality evaluation method relies on a large number of human annotations, resulting in limited data set size and weak generalization capabilities, making it difficult to show stable performance on new data sets.

Method used

Using the zero-sample video quality evaluation method based on auxiliary opinions, the adversarial language-image pre-training model CLIP is used to jointly train on perceptual quality evaluation, scene classification and distortion type classification tasks, combined with the migration knowledge of large-scale image quality evaluation data sets. Enhance the model's sensitivity to advanced semantic understanding and text cues, calculate the video's time and space natural scores, and scale through Gaussian normalization and Sigmoid function to ultimately obtain a more comprehensive video quality score.

Benefits of technology

The model's generalization ability of video quality is improved, the ability to perceive video distortion is enhanced, the problem of limited data set size is solved, and a more scalable and adaptable video quality evaluation is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918505A_ABST
    Figure CN118918505A_ABST
Patent Text Reader

Abstract

The invention relates to a zero sample video quality evaluation method based on auxiliary opinions, and belongs to the field of computer vision. Comprising the following steps: down-sampling video time, and calculating a spatial natural score; sampling is carried out on the same video space, a time natural score is calculated, and the space natural score and the time natural score are zoomed again through Gaussian normalization and a Sigmoid function; fine adjustment is carried out on an advisory language-image pre-training model CLIP on an image quality evaluation data set, labels of scenes and distortion types are added, text prompts are established, and a corresponding relation is established between the text prompts and features corresponding to the images; performing time-space down-sampling on the video, embedding a fine-tuned CLIP, extracting overall related advanced semantic features of the video, predicting a text suggestion quality score by using the CLIP, and re-scaling an alignment score through Gaussian normalization and a Sigmoid function; and adding the calculated time natural score, space natural score and text suggested quality score to obtain a more comprehensive video quality score.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a zero-sample video quality evaluation method based on auxiliary opinions. Background Art

[0002] In recent years, opinion-driven VQA (Video Quality Assessment) methods have made significant progress. However, these methods rely heavily on human opinions, resulting in models that can only adapt to the data distribution in the training dataset, which is a huge challenge. In order to obtain a reliable mean opinion score (MOS), it takes a lot of time and human resources to collect a large number of human opinions, usually requiring the efforts of at least 25 annotators, and sometimes even more. Therefore, the training dataset size of these methods is usually limited, resulting in their weak generalization ability on new datasets. For example, VQA methods trained using the KoNViD-1k dataset, which contains only 1,200 labeled videos, have poor correlation with human opinions in the YouTube-UGC dataset. Due to the limited dataset size, the generalization performance of these methods is unstable, which seriously limits their value in practical applications.

[0003] At present, opinion-driven quality assessment methods need to rely on a large amount of human quality annotated data. In this regard, researchers are actively exploring some solutions. One strategy is to improve the generalization ability of the model by increasing the size of the training data set. For example, a large-scale user-generated content (UGC) data set on a video sharing platform can be used, and more human opinions can be obtained by sharing evaluations on the Internet, and merged with the original training data set to increase the amount of learnable samples of the model, such as the collection of the LSVQ data set. Another strategy is the zero-shot learning method. Zero-shot learning can comprehensively check the quality issues of stereoscopic videos without human quality annotations. The introduction of this method provides a more scalable and universal solution for video quality assessment. Therefore, in the zero-shot method, how to improve the generalization ability and make the model pay attention to video distortion is the problem solved by the present invention. Summary of the invention

[0004] The purpose of the present invention is to provide a zero-sample video quality assessment method based on auxiliary opinions. By jointly training on perceptual quality assessment tasks, scene classification tasks and distortion type classification tasks, the model's ability to understand high-level semantics is enhanced, and knowledge is transferred from a large image quality assessment dataset to enhance the sensitivity of text prompts to distortion types.

[0005] To achieve the above object, the technical solution of the present invention is: a zero-sample video quality assessment method based on auxiliary opinions, comprising the following steps:

[0006] Step S1, downsample the video time and calculate the spatial natural score; downsample the same video space and calculate the temporal natural score, and rescale the spatial natural score and the temporal natural score by Gaussian normalization and Sigmoid function respectively;

[0007] Step S2, fine-tuning the adversarial language-image pre-training model CLIP on the image quality assessment dataset, adding labels of scenes and distortion types, establishing text prompts, and establishing a correspondence between the text prompts and the features corresponding to the images;

[0008] Step S3, downsample the video spatiotemporally, embed the CLIP fine-tuned in step S2, extract high-level semantic features related to the video as a whole, use CLIP to predict the text suggestion quality score, and rescale the alignment score through Gaussian normalization and Sigmoid function;

[0009] Step S4: Add the temporal naturalness score, spatial naturalness score and text suggestion quality score calculated in steps S1 and S3 to obtain a more comprehensive video quality score.

[0010] In one embodiment of the present invention, the step S1 is specifically as follows:

[0011] Step S11, downsample the video time and reduce the frame rate, that is, set the video to retain its original spatial resolution, but only keep a uniform number of frames in time, and then calculate the spatial natural score;

[0012] Step S12: Downsample the same video space to reduce the spatial complexity of the video, that is, all videos are spatially downsampled to a short size of 270 and keep the original aspect ratio, and then all frames are input into the temporal distortion calculation model to calculate the temporal natural score.

[0013] In one embodiment of the present invention, the spatial natural fraction calculation operation is as follows:

[0014] First, the NIQE index score Q is calculated using the NIQE algorithm NIQE,i . The NIQE algorithm is a no-reference image quality evaluation algorithm that aims to evaluate the naturalness of an image, that is, whether the image looks like a natural scene. The NIQE algorithm is based on a set of quality-perceptual features and fits them into the MVG model. The quality-perceptual features are derived from a simple but highly regularized NSS (Natural Scene Statistics) model. The NIQE indicator score for a given video frame is then expressed as: the distance between the MVG model of the quality-perceptual features extracted from the video frame and the MVG model of the quality-perceptual features extracted from the natural image corpus;

[0015] Then, the NIQE index score Q NIQE,iAfter average pooling, Gaussian normalization and Sigmoid function rescaling, the spatial natural score Q is obtained S :

[0016]

[0017] Among them, times is the total number of frames after the whole video is downsampled, σ is the standard deviation calculation operation, i represents the i-th frame, and σ(Q NIQE,i ) are the mean and standard deviation of the NIQE indicator scores in the entire set, respectively.

[0018] In one embodiment of the present invention, the time natural fraction calculation operation is as follows:

[0019] Based on the neural domain trajectories of three consecutive frames, the TPQI is used to obtain the non-smooth inter-frame changes between adjacent video frames; specifically, the simulated neural responses of the primary visual cortex V1 are calculated through 2D Gabor filters and the lateral geniculate nucleus LGN domain, and the TPQI quality score is then obtained from the curvature of the two domains:

[0020]

[0021] Where M is the total number of frames in the entire video, and are the curvatures of the three video frames (j-1, j, j+1) respectively; they are rescaled by Gaussian normalization and Sigmoid function, from Q TPQI The time naturalness fraction Q is mapped out T :

[0022]

[0023] in, and σ(Q TPQI,j ) are the mean and standard deviation of the TPQI quality scores in the entire set, respectively.

[0024] In one embodiment of the present invention, step S2 is specifically as follows:

[0025] Step S21, using the pre-trained CLIP to encode the image on the image quality assessment dataset;

[0026] Step S22, using pre-trained CLIP encoding for the set text prompt;

[0027] Step S23: perform cosine similarity on the features obtained by image encoding and the features obtained by text encoding, and the one with the largest cosine similarity is used as the output classification result; the quality classification quality is regarded as a weight and the score of the image is calculated.

[0028]

[0029] Where C = 5 is the number of quality levels, c∈C = {1, 2, 3, 4, 5}, is the estimated marginal probability of c, x = 5 means it is divided into 5 quality levels;

[0030] Step S2: The result obtained in step S23 is supervised by training using an image quality assessment dataset, and the CLIP model is fine-tuned.

[0031] In one embodiment of the present invention, the image quality evaluation data set is divided into 8 distortion types, 9 scenes, and 5 quality scores; wherein the distortion type is dis = {"blur", "color distortion", "JPEG compression", "noise", "spatial distortion", "underexposure", "overexposure" and "shake"}; the scene category is scene = {"animal", "city landscape", "human", "indoor scene", "landscape", "night scene", "plant", "still life" and "other"}; the quality score category is quality = {1, 2, 3, 4, 5} = {"bad", "poor", "average", "good" and "perfect"};

[0032] Then, we create an English template by putting together the labels from the three tasks of distortion type, scene, and quality score, resulting in a total of 8×9×5=360 candidate text descriptions.

[0033] In one embodiment of the present invention, step S3 is specifically as follows:

[0034] Step S31, down-sampling the video in time and space, performing inference every 3 frames, and for each frame, evenly cropping 8 image blocks with a size of 224×224×3 as a representative image of a video frame;

[0035] Step S32: Input the 8 image blocks into the CLIP fine-tuned in step S2 to obtain the text suggestion quality score of the video frame.

[0036] Step S32: Frame-level quality score Average pooling Then, after Gaussian normalization and rescaling with Sigmoid function, we get the frame-level natural score Q LSQI .

[0037] In one embodiment of the present invention, step S4 specifically includes: LSQI , Q S and Q T Based on the definition of comprehensive video quality score Q mix :

[0038] Q mix =Q LSQI +Q S +Q T

[0039] Among them, Q LSQI , Q S and Q T They represent frame-level natural fractions, spatial natural fractions, and temporal natural fractions respectively.

[0040] The present invention also provides a zero-sample video quality assessment system based on auxiliary opinions, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0041] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. The present invention ensures the correspondence between textual prompts and visual features based on the adversarial language-image pre-training network (CLIP), and obtains high-level semantic features of the video;

[0044] 2. Taking into account the impact of video scenes and distortion types on perceived quality, the present invention designs a multi-task learning method based on auxiliary knowledge, which improves the model's ability to understand images and perceive distortion;

[0045] 3. Based on the similarities between IQA datasets and VQA datasets, the present invention proposes to transfer knowledge from image quality assessment datasets with real distortion and optimize text to improve adaptability, thus solving the problem of limited datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0047] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0048] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0049] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0050] Please refer to Figure 1 The present invention provides a zero-sample video quality assessment method based on auxiliary opinions, comprising the following steps:

[0051] Step S1, downsample the video time and calculate the spatial natural score; downsample the same video space, calculate the temporal natural score, and rescale them by Gaussian normalization and Sigmoid function respectively;

[0052] Step S2: fine-tune the CLIP (Adversarial Language-Image Pre-training) model on the IQA dataset, add labels for scenes and distortion types, create appropriate text prompts, and establish a correspondence between the text prompts and the features corresponding to the image;

[0053] Step S3, downsample the video spatiotemporally, embed the CLIP model fine-tuned in step S2, extract high-level semantic features related to the video as a whole, use the model to predict the text suggestion quality score, and rescale the alignment score through Gaussian normalization and Sigmoid function;

[0054] Step S4: Add the temporal naturalness score, spatial naturalness score and text suggestion quality score calculated in steps S1 and S3 to obtain a more comprehensive video quality score.

[0055] In this embodiment, step S1 specifically includes the following steps:

[0056] Step S11, downsample the video time and reduce the frame rate, that is, set the video to retain its original spatial resolution, but only keep a uniform number of frames in time, and then calculate the spatial natural score; the spatial natural score calculation operation is as follows: First, calculate the NIQE index score Q by the NIQE algorithm NIQE,i. The NIQE algorithm is a no-reference image quality evaluation algorithm that aims to evaluate the naturalness of an image, that is, whether the image looks like a natural scene. The NIQE algorithm is based on a set of quality-perceptual features and fits them into the MVG model. The quality-perceptual features are derived from a simple but highly regularized NSS (Natural Scene Statistics) model. The NIQE indicator score for a given video frame is then expressed as: the distance between the MVG model of the quality-perceptual features extracted from the video frame and the MVG model of the quality-perceptual features extracted from the natural image corpus;

[0057] Then, the original NIQE score Q NIQE,i (i represents the i-th frame) After average pooling, Gaussian normalization and Sigmoid function rescaling, the spatial natural score Q is obtained S :

[0058]

[0059] Among them, times is the total number of frames after the entire video is downsampled, σ is the standard deviation calculation operation, and σ(Q NIQE,i ) are the mean and standard deviation of the raw NIQE scores in the entire set, respectively;

[0060] Step S12: downsample the same video space to reduce the spatial complexity of the video, that is, all videos are spatially downsampled to a short size of 270 and keep the original aspect ratio, and then all frames are input into the temporal distortion calculation model to calculate the temporal natural fraction; the temporal natural fraction calculation operation is as follows:

[0061] TPQI captures the non-smooth inter-frame changes between adjacent video frames, mainly based on the neural domain trajectories of three consecutive frames. Specifically, the simulated neural response of the primary visual cortex (V1) is calculated through 2D Gabor filters and the lateral geniculate nucleus (LGN) domain, and then the TPQI quality score is obtained from the curvature of the two domains:

[0062]

[0063] Where M is the total number of frames in the entire video, and are the curvatures of the three video frames (j-1, j, j+1). Similarly, through Gaussian normalization and Sigmoid function rescaling, the original score Q TPQI The time naturalness fraction Q is mapped out T :

[0064]

[0065] in, and σ(QTPQI,j ) are the mean and standard deviation of the raw TPQI scores in the entire set, respectively.

[0066] In this embodiment, step S2 specifically includes the following steps:

[0067] Step S21: Encode the image using a pre-trained CLIP (Contrastive Language-Image Pre-Training) model on the image quality assessment dataset;

[0068] Step S22, the set text prompt is encoded using the pre-trained CLIP model; the text prompt operation is as follows: the image data set is divided into 8 distortion types, 9 scenes, and 5 quality scores. Among them, the distortion type is dis = {"blur", "color distortion", "JPEG compression", "noise", "spatial distortion", "underexposure", "overexposure" and "shake"}; the scene category is scene = {"animal", "city landscape", "human", "indoor scene", "landscape", "night scene", "plant", "still life" and "other"}; the quality score category is quality = {1, 2, 3, 4, 5} = {"bad", "poor", "average", "good" and "perfect"}.

[0069] Then, we naturally create an English template by putting together the labels from the three tasks: “a photo of a {scene} with {dis} distortion, having a quality rating of {quality}.” There are a total of 8×9×5=360 candidate text descriptions.

[0070] Step S23: perform cosine similarity on the features obtained by image encoding and the features obtained by text encoding, and the one with the largest similarity is the output classification result; the quality classification quality is regarded as a weight to calculate the score of the image.

[0071]

[0072] Where C = 5 is the number of quality levels, c∈C = {1, 2, 3, 4, 5}, is the estimated marginal probability of c, x = 5 means it is divided into 5 quality levels;

[0073] Step S24, using the image quality assessment dataset to supervise the training of the result obtained in step S23, and fine-tuning the CLIP model;

[0074] In this embodiment, step S3 specifically includes the following steps:

[0075] Step S31, downsampling the video in time and space, performing inference every 3 frames, and for each frame, evenly cropping 8 image blocks with a size of 224×224×3 as a representative image of a video frame;

[0076] Step S32: Input the 8 image blocks into the CLIP model fine-tuned in step S2 to obtain the text suggestion quality score of the video frame.

[0077] Step S32: Frame-level quality score Average pooling Then, after Gaussian normalization and rescaling with Sigmoid function, we get the frame-level natural score Q LSQI .

[0078] In this embodiment, step S4 is specifically as follows: LSQI , Q S and Q T Based on the definition of comprehensive video quality score Q mix :

[0079] Q mix =Q LSQI +Q S +Q T .

[0080] The present invention also provides a zero-sample video quality assessment system based on auxiliary opinions, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0081] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0082] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0083] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0084] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0086] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A zero-sample video quality assessment method based on auxiliary opinions, characterized in that: The steps include: Step S1, downsample the video time and calculate the spatial natural score; downsample the same video space and calculate the temporal natural score, and rescale the spatial natural score and the temporal natural score by Gaussian normalization and Sigmoid function respectively; Step S2, fine-tuning the adversarial language-image pre-training model CLIP on the image quality assessment dataset, adding labels of scenes and distortion types, establishing text prompts, and establishing a correspondence between the text prompts and the features corresponding to the images; Step S3, downsample the video spatiotemporally, embed the CLIP fine-tuned in step S2, extract high-level semantic features related to the video as a whole, use CLIP to predict the text suggestion quality score, and rescale the alignment score through Gaussian normalization and Sigmoid function; Step S4: Add the temporal naturalness score, spatial naturalness score and text suggestion quality score calculated in steps S1 and S3 to obtain a more comprehensive video quality score.

2. The zero-sample video quality assessment method based on auxiliary opinions according to claim 1 is characterized in that: The step S1 is specifically as follows: Step S11, downsample the video time and reduce the frame rate, that is, set the video to retain its original spatial resolution, but only keep a uniform number of frames in time, and then calculate the spatial natural score; Step S12: Downsample the same video space to reduce the spatial complexity of the video, that is, all videos are spatially downsampled to a short size of 270 and keep the original aspect ratio, and then all frames are input into the temporal distortion calculation model to calculate the temporal natural score.

3. The zero-sample video quality assessment method based on auxiliary opinions according to claim 2 is characterized in that: The spatial natural fraction calculation operation is as follows: First, the NIQE index score Q of a given video frame is calculated using the NIQE algorithm. NIQE,i ,Then, the NIQE index score of a given video frame is expressed as: the distance between the MVG model of the quality-aware features extracted from the video frame and the MVG model of the quality-aware features extracted from the natural image corpus; Then, the NIQE index score Q NIQE,i After average pooling, Gaussian normalization, and rescaling with the Sigmoid function, the spatial natural score Qs is obtained: Among them, times is the total number of frames after the whole video is downsampled, σ is the standard deviation calculation operation, i represents the i-th frame, and σ(Q NIQE,i ) are the mean and standard deviation of the NIQE indicator scores in the entire set, respectively.

4. The zero-sample video quality assessment method based on auxiliary opinions according to claim 2 is characterized in that: The time natural fraction calculation operation is as follows: Based on the neural domain trajectories of three consecutive frames, the TPQI is used to obtain the non-smooth inter-frame changes between adjacent video frames; specifically, the simulated neural responses of the primary visual cortex V1 are calculated through 2D Gabor filters and the lateral geniculate nucleus LGN domain, and the TPQI quality score is then obtained from the curvature of the two domains: Where M is the total number of frames in the entire video, and are the curvatures of the three video frames (j-1, j, j+1) respectively; they are rescaled by Gaussian normalization and Sigmoid function, from Q TPQI The time naturalness fraction Q is mapped out T : in, and σ(Q TPQI,j ) are the mean and standard deviation of the TPQI quality scores in the entire set, respectively.

5. The zero-sample video quality assessment method based on auxiliary opinions according to claim 1, characterized in that: The step S2 is specifically as follows: Step S21, using the pre-trained CLIP to encode the image on the image quality assessment dataset; Step S22, using pre-trained CLIP encoding for the set text prompt; Step S23: perform cosine similarity on the features obtained by image encoding and the features obtained by text encoding, and the one with the largest cosine similarity is used as the output classification result; the quality classification quality is regarded as a weight and the score of the image is calculated. Where C = 5 is the number of quality levels, c∈U = {1, 2, , 4, 5}, is the estimated marginal probability of c, x = 5 means it is divided into 5 quality levels; Step S2: The result obtained in step S23 is supervised by training using an image quality assessment dataset, and the CLIP model is fine-tuned.

6. The zero-sample video quality assessment method based on auxiliary opinions according to claim 5, characterized in that: The image quality evaluation dataset is divided into 8 distortion types, 9 scenes, and 5 quality scores; the distortion type is dis = {"blur", "color distortion", "JPEG compression", "noise", "spatial distortion", "underexposure", "overexposure" and "shake"}; the scene category is scene = {"animal", "cityscape", "human", "indoor scene", "landscape", "night scene", "plant", "still life" and "other"}; the quality score category is quality = {1, 2, 3, 4, 5} = {"bad", "poor", "average", "good" and "perfect"}; Then, we create an English template by putting together the labels from the three tasks of distortion type, scene, and quality score, resulting in a total of 8×9×5=360 candidate text descriptions.

7. The zero-sample video quality assessment method based on auxiliary opinions according to claim 1, characterized in that: The step S3 is specifically as follows: Step S31, down-sampling the video in time and space, performing inference every 3 frames, and for each frame, evenly cropping 8 image blocks with a size of 224×224×3 as a representative image of a video frame; Step S32: Input the 8 image blocks into the CLIP fine-tuned in step S2 to obtain the text suggestion quality score of the video frame. Step S32: Frame-level quality score Average pooling Then, after Gaussian normalization and rescaling with Sigmoid function, we get the frame-level natural score Q LSQI .

8. The zero-sample video quality assessment method based on auxiliary opinions according to claim 1, characterized in that: The step S4 is specifically as follows: LSQI , Q S and Q T Based on the definition of comprehensive video quality score Q mix : Q mix =Q LSQI +Q S +Q T Among them, Q LSQI , Q S and Q T They represent frame-level natural fractions, spatial natural fractions, and temporal natural fractions respectively.

9. A zero-sample video quality assessment system based on auxiliary opinions, characterized in that: The method comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps as claimed in any one of claims 1 to 8 can be implemented.

10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.

Citation Information

Cited By

  • Video clipping method and clipped video quality evaluation method

    CN119342207A