AI video quality assessment method and device based on multi-scale and CLIP semantic supervision

Abstract

The present invention discloses an AI video quality assessment method and device based on multi-scale and CLIP semantic supervision, which combines CLIP semantic supervision and cross-attention mechanism. The specific implementation includes: in order to maximize the extraction and utilization of information embedded in the video content, video analysis is performed from three levels: single frame, frame fragment and entire video; in order to ensure that the generated video is consistent with the provided prompt, the present invention introduces a prompt semantic supervision module, which uses the encoding capability of the CLIP text encoder to semantically evaluate the consistency between the video and the prompt; considering the semantic changes of AI-generated videos, the present invention also introduces a semantic mutation perception module, which has the basic modeling capability to capture subtle semantic changes between frames. Ultimately, through our carefully designed model, the quality of AI-generated videos can be effectively evaluated.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to an AI video quality assessment method and device based on multi-scale and CLIP semantic supervision. Background Art

[0002] Because each frame, clip, and overall video contains features of varying granularity, multi-dimensional video feature analysis is crucial for evaluating the quality of AI-generated content. Traditional video quality assessment methods, all based on real-world video, fail to fully utilize technologies like data augmentation, semantic consistency supervision, and semantic mutation awareness, making them unable to accurately evaluate the complexity and semantic mutation of AI-generated videos between frames. Summary of the Invention

[0003] The purpose of this paper is to propose a novel video quality assessment framework that combines multi-scale analysis, data enhancement technology, semantic supervision module and semantic mutation perception to improve the quality assessment accuracy of AI-generated videos.

[0004] The object of the present invention is achieved through the following technical solutions: In the first aspect, the present invention provides an AI video quality assessment method based on multi-scale and CLIP semantic supervision, the method comprising the following steps:

[0005] (1) Obtain the raw video data generated by AI and process it into video frames and video segments;

[0006] (2) Data enhancement is performed on video frames and video segments. CLIP-based semantic supervision is used to evaluate the semantic consistency between video frames and video segments and between the original video content and its prompts. Semantic mutations between video frames are detected based on the semantic information of video frames, and scores of video frame data, video segment data, and original video data are predicted.

[0007] (3) The final prediction score is obtained by comprehensively scoring the video frames, video segments and original video data to achieve video quality assessment.

[0008] Furthermore, the semantic supervision based on CLIP is specifically as follows: CLIP's text encoder is used to encode the prompt words of the generated video, and by adding adapter training to the last two layers of transformer in the CLIP text encoder, the encoded information is made more semantically relevant.

[0009] Furthermore, in step (2), ConvNet is used as the backbone to extract video frame features and input them into the evaluation head to predict the score of the video frame data.

[0010] Furthermore, in step (2), the semantic mutation between video frames is detected as follows: the semantic information of the video frame is encoded using the CLIP image encoder based on the semantic mutation perception module SMA, and the cls feature of each video frame feature is used as the semantic information of the video frame. An adapter is added to the two transformer layers after the image encoder to make its encoded features more relevant to the evaluation features. A cross-attention layer is used to capture the semantic changes between frames, and the feature is fused into the evaluation head of the video branch model.

[0011] Furthermore, in step (2), temporal semantic information is extracted through slowfast motion features, and video segment data features are extracted based on the pre-trained Swin Transformer to predict the video segment data score.

[0012] Furthermore, in step (2), FastVQA is used as the backbone network to predict the original video data score.

[0013] Furthermore, in step (3), the prediction results of the video frame, video segment and original video data are transformed by applying the Sigmoid function, and the prediction results are combined by weighted summation to obtain the final prediction score.

[0014] In the second aspect, the present invention also provides an AI video quality assessment device based on multi-scale and CLIP semantic supervision, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the AI ​​video quality assessment method based on multi-scale and CLIP semantic supervision.

[0015] In a third aspect, the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it implements the AI ​​video quality assessment method based on multi-scale and CLIP semantic supervision.

[0016] In a fourth aspect, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the AI ​​video quality assessment method based on multi-scale and CLIP semantic supervision.

[0017] The beneficial effects of the present invention are as follows: the present invention effectively improves the comprehensiveness and accuracy of video quality assessment by adopting a multi-dimensional analysis method, combining frame-level, segment-level and video-level assessments. In particular, the introduction of a CLIP-based semantic consistency supervision module and a semantic mutation perception module enhances the model's ability to recognize semantic coherence and mutations in AI-generated videos, significantly improving the accuracy of video quality assessment. In addition, through a multi-layer model integration strategy, the present invention not only improves the robustness of the assessment model, but also optimizes the model's processing capabilities for different data dimensions, making the assessment results more stable and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is a schematic diagram of the overall architecture of the present invention. (a) The schematic diagram shows the three main components that capture video, paragraph, and frame-level features; (b) The schematic diagram shows the semantic consistency supervision (PSS) module based on the CLIP text encoder integrated into the model branch; (c) The schematic diagram shows the semantic variation awareness (SMA) module for modeling sudden semantic variation between video frames.

[0020] Figure 2 It shows that semantic mutations occur between AI-generated video frames.

[0021] Figure 3 This is a structural diagram of an AI video quality assessment device based on multi-scale and CLIP semantic supervision provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0023] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0024] like Figure 1 As shown, the present invention proposes a method for multi-dimensional evaluation of video quality of AI-generated content, which includes the following steps:

[0025] (1) Figure 1 As shown in (a) of the figure, a multidimensional analysis method is used to evaluate videos at three levels: frame level, video clip level, and video level: from top to bottom, they are video level, video clip level, and frame level. Data at each level is processed using specific enhancement and analysis methods to ensure comprehensiveness and accuracy of the evaluation. A loss function is carefully designed for each level to more comprehensively capture video information, thereby achieving accurate video evaluation. Specifically, different loss functions are designed based on the different characteristics of the video frame, segment, and overall video content to optimize the model's capture of information at each level, laying a solid foundation for video evaluation.

[0026] (2) In frame-level evaluation, in order to enhance the model's ability to cope with the inherent variability of AI-generated videos, we implemented a data augmentation strategy to broadcast the video quality score to the video frame to obtain the quality score of the video frame. At this level, we learned to score the local features of the video frame, which significantly enhanced the data. We used a deep convolutional architecture to detect key local features and textures to carefully evaluate the quality of each frame and improve the model's ability to identify complex video quality in frame-level enhancement. Specifically, in this branch, we used ConvNet as the backbone and input the features of the four stages of the network into the evaluation head. At the frame level, L1 smoothing loss was applied:

[0027]

[0028] (3) At the video segment level, we introduce the Swin Transformer pre-trained on the LSVQ dataset to enhance the features of the video segments, broadcast the video quality scores to the video segments, and obtain the quality scores of the video segments, thereby improving the performance of the SimpleVQA architecture and giving the Swin transformer the ability to initially perceive the video quality. The present invention divides the video into segments for centralized analysis, so that the specific features of the segments can be accurately enhanced. Specifically, the model's dependence on specific video segments is reduced by randomly initializing the starting points of the segments, improving the model's generalization ability in different time contexts, and further ensuring the consistency of features between segments through dynamic and spatial data alignment technology; thus optimizing the video quality assessment process. In addition, we also extract rich temporal semantic understanding through slowfast motion features, significantly improving the accuracy and detail of the model's analysis of video content. At the video segment level, we apply mean absolute value loss and ranking loss.

[0029]

[0030] (4) At the video level, we use FastVQA as the backbone network. To increase the resolution of videoSwinTransformers, we enlarge the patch size to 6 to increase the resolution of the backbone network and achieve better performance. The proposed model uses the Pearson Linear Correlation Coefficient (PLCC) loss and ranking loss to capture the linear relationship and hierarchical order of video-level features. The PLCC loss function quantifies the correlation and encourages the model to learn representations that are well consistent with the underlying structure of the data:

[0031]

[0032] (5) We introduce the Prompt Semantic Supervision Module (PSS) in all three branches, such as Figure 1 As shown in (b) of the figure, this module is used to assess the semantic consistency between a video and its prompt. This module uses CLIP's text encoder to encode the prompt word for the generated video. An adapter is added to the two transformer layers after the text encoder. The encoded text features are then integrated into the evaluation head of the video quality assessment model to monitor whether the video content is consistent with the text prompt. By adding adapter training (fully connected networks) to the last two layers of the CLIP text encoder, the encoded information becomes more semantically relevant, improving the ability to judge the semantic consistency of AI-generated videos.

[0033] (6) There will be semantic mutations between the video frames generated by the artificial intelligence algorithm, such as Figure 2 In order to accurately capture the semantic mutation between video frames, we introduce a semantic mutation awareness module (SMA) at the video level, as shown in Figure 1 As shown in (c) of the figure, SMA uses the CLIP image encoder to encode the semantic information of video frames. The CLS features of each video frame are used as the semantic information of the video frame. An adapter is added to the last two layers of the image encoder to make the encoded features more relevant to the evaluation features. A learnable query is used through a cross-attention layer to capture the semantic changes between frames, and this feature is fused into the evaluation head of the video branch model. The SMA module enhances the accuracy of our video quality assessment method.

[0034] (7) The prediction results of the three levels are transformed by applying the Sigmoid function to ensure that they fall within a common range. The prediction results of multiple models are combined to enhance the accuracy and robustness of the final model. Each level of branches operates independently during training, ensuring the specificity of the learning process and the optimization of the model. The prediction results from each branch are combined by weighted summation to obtain the final prediction score, thus achieving a comprehensive and effective evaluation of video quality.

[0035] Corresponding to the aforementioned embodiment of an AI video quality assessment method based on multi-scale and CLIP semantic supervision, the present invention also provides an embodiment of an AI video quality assessment device based on multi-scale and CLIP semantic supervision.

[0036] See also Figure 3 An embodiment of the present invention provides an AI video quality assessment device based on multi-scale and CLIP semantic supervision, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement an AI video quality assessment method based on multi-scale and CLIP semantic supervision in the above embodiment.

[0037] The embodiment of the AI ​​video quality assessment device based on multi-scale and CLIP semantic supervision provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 3 As shown in the figure, it is a hardware structure diagram of any device with data processing capability where the AI ​​video quality assessment device based on multi-scale and CLIP semantic supervision provided by the present invention is located. Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0038] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0039] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0040] An embodiment of the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, an AI video quality assessment method based on multi-scale and CLIP semantic supervision in the above embodiment is implemented.

[0041] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0042] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the AI ​​video quality assessment method based on multi-scale and CLIP semantic supervision.

[0043] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from the other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0044] The above description is only a preferred embodiment of the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can use the above disclosed methods and technical contents to make many possible changes and modifications to the technical solution of the present invention without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment with equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. An AI video quality assessment method based on multi-scale and CLIP semantic supervision, characterized in that: The method comprises the following steps: (1) Obtain the raw video data generated by AI and process it into video frames and video segments; (2) Perform data enhancement on video frames and video segments, and use CLIP-based semantic supervision to evaluate the semantic consistency between video frames and video segments and the original video content and its prompt words. Based on the semantic information of video frames, semantic mutations between video frames are detected, and the scores of video frame data, video segment data, and original video data are predicted. The specific semantic supervision based on CLIP is as follows: CLIP's text encoder is used to encode the prompt words of the generated video, and adapter training is added to the last two transformer layers of CLIP's text encoder to make the encoded information more semantically relevant; Detecting semantic mutations between video frames involves: Using the CLIP image encoder based on the semantic mutation awareness module (SMA), the semantic information of video frames is encoded. The CLS features of each video frame are used as the semantic information of the video frame. An adapter is added to the two transformer layers after the image encoder to make the encoded features more relevant to the evaluation features. A cross-attention layer is used to capture the semantic changes between frames, and this feature is integrated into the evaluation head of the video branch model. (3) The final prediction score is obtained by comprehensively scoring the video frames, video segments and original video data to achieve video quality assessment.

2. The AI ​​video quality assessment method based on multi-scale and CLIP semantic supervision according to claim 1 is characterized in that: In step (2), ConvNet is used as the backbone network to extract video frame features and input them into the evaluation head to predict the score of the video frame data.

3. The AI ​​video quality assessment method based on multi-scale and CLIP semantic supervision according to claim 1 is characterized in that: In step (2), temporal semantic information is extracted through slowfast motion features, and video segment data features are extracted based on the pre-trained SwinTransformer to predict the video segment data score.

4. The AI ​​video quality assessment method based on multi-scale and CLIP semantic supervision according to claim 1 is characterized in that: In step (2), FastVQA is used as the backbone network to predict the score of the original video data.

5. The AI ​​video quality assessment method based on multi-scale and CLIP semantic supervision according to claim 1 is characterized in that: In step (3), the prediction results of the video frame, video segment and original video data are transformed by the Sigmoid function, and the prediction results are combined by weighted summation to obtain the final prediction score.

6. An AI video quality assessment device based on multi-scale and CLIP semantic supervision, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it implements an AI video quality assessment method based on multi-scale and CLIP semantic supervision as described in any one of claims 1-5.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, an AI video quality assessment method based on multi-scale and CLIP semantic supervision is implemented as described in any one of claims 1 to 5.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements an AI video quality assessment method based on multi-scale and CLIP semantic supervision as described in any one of claims 1 to 5.

Images