A method for blind assessment of cartoon video quality
By constructing a blind evaluation model for cartoon video quality and combining self-supervised learning and multi-task learning frameworks, the problem of insufficient datasets in cartoon video quality evaluation is solved, resulting in more accurate evaluation results that conform to human visual perception.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ANHUI UNIV
- Filing Date
- 2023-12-26
- Publication Date
- 2026-05-19
AI Technical Summary
Existing no-reference video quality assessment algorithms cannot be effectively used for cartoon videos. The lack of large-scale labeled datasets leads to overfitting or poor generalization ability in model training, making it impossible to accurately assess the quality of cartoon videos.
A blind quality assessment model for cartoon videos is constructed. The model is pre-trained and fine-tuned using a self-supervised learning strategy through edge line, spatiotemporal features and texture features quality perception modules. Combined with a multi-task learning framework, edge, spatiotemporal and texture features of cartoon videos are extracted. The model is optimized using auxiliary tasks and loss functions trained by self-supervised pre-training.
It improves the reliability and generalization of cartoon video quality assessment, makes the assessment results more consistent with human visual perception, overcomes the problem of insufficient labeled datasets, and achieves more accurate cartoon video quality assessment.
Smart Images
Figure CN117676127B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of cartoon video quality assessment technology, specifically a blind assessment method for cartoon video quality. Background Technology
[0002] Video Quality Assessment (VQA) is a fundamental research topic in computer vision and image processing, aiming to evaluate the visual quality of videos by simulating human visual perception. In recent years, although many stable and efficient VQA algorithms have been proposed, they are primarily designed for natural videos. Due to the significant differences between natural and cartoon videos, these algorithms are not applicable to cartoon videos. Only a few studies have focused on non-natural videos, such as computer graphics (CG), virtual videos, and screen content videos.
[0003] Over the past few decades, with the rapid development of the cartoon video industry, many cartoon videos of varying quality have emerged. Furthermore, the quality of cartoon videos can change during production, compression, storage, and transportation. Therefore, quality assessment algorithms for cartoon videos have become increasingly important for processing, evaluating, and promoting the development of the cartoon video industry.
[0004] Objective VQA methods can be categorized into three types based on the degree of reliance on reference videos: full-reference, partial-reference, and no-reference. However, due to the limitation of lacking reference videos for comparison in practical applications, full-reference and partial-reference methods are not applicable. Therefore, the no-reference objective VQA (NR-VQA) method has greater research value and significance. Based on prior knowledge of distortion types, NR-VQA models can be further divided into two types: specific distortion types and general types. Specific distortion types are designed for known distortions, such as compression, block and blur artifacts, low bitrate, noise, and low contrast. Conversely, general NR-VQA methods do not consider specific distortion types and provide different strategies for general quality modeling, such as traditional manual statistical features and learning-based algorithms. Learning-based algorithms can extract high-level, highly discriminative features, enabling better alignment between objective and subjective evaluation results, and have become mainstream in VQA research. However, the performance of learning-based models, especially the recent deep neural network (DNN)-based methods, is largely limited by the size and quality of the dataset.
[0005] For specific video types like cartoons, there is still a lack of large-scale labeled datasets, which may lead to overfitting or poor generalization ability in model training. Currently, there is no no-reference video quality assessment device for cartoon videos, and existing no-reference VQA assessment algorithms are not effective for quality assessment of cartoon videos. Summary of the Invention
[0006] The purpose of this invention is to address the shortcomings of existing cartoon video evaluation methods in terms of poor reliability, and to provide a blind evaluation method for cartoon video quality to solve the above problems.
[0007] To achieve the above objectives, the technical solution of the present invention is as follows:
[0008] A blind evaluation method for cartoon video quality includes the following steps:
[0009] Acquisition and preprocessing of original cartoon video and corresponding distorted video: Acquire the original cartoon video and corresponding distorted video frames, and perform preprocessing;
[0010] Constructing a blind evaluation model for cartoon video quality: Based on the edge line quality perception module, the spatiotemporal feature quality perception module, and the texture feature quality perception module, a self-supervised learning framework and a blind evaluation module are constructed.
[0011] Pre-training of the blind quality assessment model for cartoon videos: Input the original cartoon video and the corresponding distorted video to perform self-supervised pre-training of the blind quality assessment model for cartoon videos, and save the pre-trained model to extract edge lines, spatiotemporal and texture quality features of the cartoon video;
[0012] Acquisition and preprocessing of cartoon videos with real quality labels and cartoon videos to be evaluated;
[0013] Obtaining the blind evaluation results of cartoon video quality: Input the pre-processed cartoon video with real quality labels into the pre-trained blind evaluation model of cartoon video quality, fine-tune the blind evaluation model of cartoon video quality, and save it for the evaluation of cartoon video quality; input the pre-processed cartoon video to be evaluated into the fine-tuned blind evaluation model of cartoon video quality to obtain the blind evaluation results of cartoon video quality.
[0014] The acquisition and preprocessing of the original cartoon video and the corresponding distorted video include the following steps:
[0015] The original cartoon video is used to generate corresponding distorted videos through a distortion enhancement strategy. That is, different types and degrees of corresponding distorted cartoon videos are generated from the original cartoon video by artificially synthesizing distortion.
[0016] The original cartoon video and the corresponding distorted video are frame-by-frame extracted, and the entire cartoon video is uniformly extracted.
[0017] Cartoon video frame I n n = (1, 2, ..., N), where N is the total number of input cartoon video frames. After cropping to a uniform resolution, let's say H × W. Then the total size of the input video frames is N × 3 × H × W, where 3 represents the three RGB channels.
[0018] The construction of the blind evaluation model for cartoon video quality includes the following steps:
[0019] Configure the edge line quality perception module;
[0020] Set up a spatiotemporal characteristic quality perception module;
[0021] Configure the texture feature quality perception module;
[0022] Design a multi-task self-supervised learning framework for cartoon videos:
[0023] Using the edge line quality perception module, the spatiotemporal feature quality perception module, and the texture feature quality perception module, the spatiotemporal feature differences between the original input cartoon video and its distorted video are calculated. Edge feature differences Texture feature differences The specific formula is as follows:
[0024]
[0025]
[0026]
[0027] in, and This represents the spatiotemporal, edge, and texture features extracted from the original input cartoon video, respectively. and These represent the spatiotemporal, edge, and texture features extracted from the input distorted cartoon video, respectively.
[0028] Edge similarity measurement is used as an auxiliary task for self-supervised pre-training of edge features, leveraging edge feature differences. The model focuses on the edge information of the input cartoon video;
[0029] Constructing the prediction head structure f pred Used for auxiliary tasks, predicting the head structure f pred The process involves passing the material through a Conv3d convolutional layer with a 1×1×1 kernel, then through the GELU activation function, then through another Conv3d convolutional layer with a 1×1×1 kernel, and finally through global average pooling (GAP).
[0030] For auxiliary tasks using self-supervised pre-training of texture features, texture complexity computation tasks are employed. By predicting the structure To calculate texture complexity The formula is as follows:
[0031]
[0032] For auxiliary tasks of self-supervised pre-training of spatiotemporal features, three tasks are used: distortion type and distortion degree judgment, frame rate type prediction, and so on.
[0033] For the task of judging the type and degree of distortion By predicting the head structure respectively and To predict the probability of distortion type and degree of distortion
[0034] For frame rate type prediction, use By predicting head structure To predict the probability of frame rate type The formula is as follows:
[0035]
[0036]
[0037]
[0038] Where T and R represent the total number of distortion types and the total number of frame rate types, respectively;
[0039] Set up a blind evaluation module for cartoon video quality:
[0040] The F-values of the input cartoon video are extracted by setting up an edge line quality perception module, a spatiotemporal feature quality perception module, and a texture feature quality perception module. edge F s-t and F texture By performing concatenation and fusion, the final feature F representing the quality of the cartoon video is obtained. fusion Its size is The formula is as follows:
[0041] F fusion =Concat(F edge ,F s-t ,F texture (7)
[0042] Using equation (8) to F fusion The quality score of the input cartoon video is mapped to the following formula:
[0043] Score = f out (F fusion (8)
[0044] Among them, f outThe quality prediction module first extracts quality-sensitive features of the cartoon video using a Conv3d convolutional network with a kernel of 1×1×1. Then, it predicts the quality score using the GELU activation function and another Conv3d convolutional network with a kernel of 1×1×1. Finally, it uses global average pooling (GAP) to obtain the final cartoon video quality score.
[0045] The pre-training of the aforementioned blind evaluation model for cartoon video quality includes the following steps:
[0046] The original cartoon video and the corresponding distorted video are input into the blind evaluation model for cartoon video quality for training.
[0047] For the distortion prediction and texture complexity calculation auxiliary tasks, ranking loss is used as the loss functions l1 and l2 for distortion prediction and texture complexity calculation, respectively, as shown in the following formulas:
[0048]
[0049]
[0050]
[0051] Where c and d are the true labels for texture complexity and distortion level, respectively. and These are the predicted values for texture complexity and distortion level, respectively. yes The covariance between and d and σ d They are and the standard deviation of d, and The calculation methods are the same;
[0052] For the edge similarity measurement auxiliary task, the established edge consistency loss is used as the loss function l3 for edge similarity measurement, as shown in the following formula:
[0053]
[0054] Where Sigmoid(·) is the Sigmoid activation function, and GAP(·) is the global equal pooling operation;
[0055] For the auxiliary tasks of distortion type prediction and frame rate type prediction, cross-entropy loss is used as the loss function l4 and l5 for distortion type prediction and frame rate type prediction, respectively, and the formulas are as follows:
[0056]
[0057]
[0058] Among them, t i and r i These are the actual labels for distortion type and frame rate type, respectively. and These are the predicted probability values for distortion type and frame rate type, where T and R are the total number of distortion types and the total number of frame rate types, respectively.
[0059] The weighted sum of the losses from the five auxiliary tasks is used as the joint loss function L for self-supervised pre-training. pre The specific formula is as follows:
[0060] L pre =l1+α×l2+β×l3+λ×l4+ω×l5, (22)
[0061] Here, α, β, λ, and ω are all positive values, used as weighting parameters to adjust the relative importance of the five components in the joint loss function;
[0062] A self-supervised pre-training method is used to pre-train a blind evaluation model for cartoon video quality. The gradient of each parameter in the blind evaluation model is calculated using the backpropagation method, and the parameters in the blind evaluation model are updated using the stochastic gradient descent method. The parameters of the blind evaluation model for cartoon video quality when the PLCC index is at its best are saved, which are then used to extract edge, spatiotemporal, and texture features.
[0063] The edge line quality sensing module includes the following steps:
[0064] The EDTER network was set as the cartoon edge extractor. The cartoon edge extractor was first pre-trained on the BSDS500 dataset and then fine-tuned on the AnimeRun dataset with cartoon outline maps to extract the edge maps of cartoon video frames.
[0065] The input cartoon video frames are processed by a cartoon edge extractor to obtain their edge maps. The total size of the edge graph is N×3×H×W, as shown in the following formula:
[0066]
[0067] Where f EDTER It is a finely tuned cartoon edge extractor;
[0068] An edge line quality-aware network was designed, with InceptionNeXt-Tiny as the backbone network. The first two stages consist of four InceptionNeXt blocks.
[0069] For each input cartoon edge image, edge features are extracted through the first two stages of InceptionNeXt-Tiny to obtain a deep, high-level edge line feature map (EM). n Its size is The specific formula is as follows:
[0070]
[0071] Among them, f Icn Indicates the InceptionNeXt block used;
[0072] Equation (3) is used to further process and extract effective edge line quality-perceived features, and the depth, height, and width channel dimensions are aligned to promote the merging of edge features with spatiotemporal and texture feature channels, thereby obtaining the final edge line quality-perceived feature F. edge Its size is The number of output feature channels is represented by the following formula:
[0073]
[0074] Among them, f p This represents a post-processing module. First, it performs dimensionality reduction using a Conv3d convolutional network with a kernel size of 2×4×4 and a stride of 2×4×4. Then, it uses the GELU activation function and a Conv3d convolutional network with a kernel size of 1×1×1 to extract deep edge line quality features.
[0075] The aforementioned spatiotemporal characteristic quality perception module includes the following steps:
[0076] A spatiotemporal feature quality perception network was designed, using the Video Swin Transformer-Tiny network as the backbone. This network was used in four stages to extract the deep spatiotemporal features F of the input cartoon video clip V. s-t Its size is The specific formula is as follows:
[0077] F s-t =f vst (V) (4)
[0078] Where f vst This refers to the Video Swin Transformer-Tiny network.
[0079] The texture feature quality perception module includes the following steps:
[0080] Using a grayscale converter, each frame of the input cartoon video is converted into a grayscale image. n Convert to grayscale image The total size of the input grayscale image is N×1×H×W, where 1 represents a single channel;
[0081] Configure a texture feature quality perception module.
[0082] Using ConvNeXt-Tiny as the backbone network, four ConvNeXt blocks are used to extract grayscale images from each frame. Deep and highly complex local texture features TM n Its size is Use post-processing module f p Further processing and extraction of effective texture features yields the final texture feature F. texture Its size is The specific formula is as follows:
[0083]
[0084]
[0085] Where f cb This indicates the ConvNeXt block used.
[0086] The blind evaluation results of the cartoon video quality are obtained through the following steps:
[0087] The pre-processed cartoon video with a real quality label is input into a pre-trained blind quality assessment model for cartoon videos. Edge, spatiotemporal, and texture features are extracted through an edge line quality perception module, a spatiotemporal feature quality perception module, and a texture feature quality perception module, and then the predicted quality score Q of the input cartoon video is obtained. pred ;
[0088] Using the distortion prediction loss function as the loss function L during fine-tuning tune The specific formula is as follows:
[0089] L tune =log(1-PLCC(Q) pred Q label )), (twenty three)
[0090] Q label It is the actual quality score of the cartoon video;
[0091] The cartoon video quality assessment model is fine-tuned. The gradient of each parameter in the model is calculated using the backpropagation method, and the parameters in the model are updated using the stochastic gradient descent method. The model parameters when the PLCC index is best are saved to evaluate the quality of cartoon videos.
[0092] The preprocessed cartoon video to be evaluated is input into the fine-tuned blind evaluation model for cartoon video quality to obtain the blind evaluation results of cartoon video quality.
[0093] Beneficial effects
[0094] The present invention provides a blind evaluation method for cartoon video quality, which overcomes the problem of insufficient cartoon labeled datasets and improves the reliability and generalization of cartoon video quality evaluation compared with the prior art, making the evaluation of cartoon video quality more in line with human visual perception.
[0095] 1. This invention takes into account the importance of edges and lines for cartoon visual perception, and the fact that cartoon videos have less complex local textures compared to natural videos. It designs a cartoon video quality assessment model with three branches. This model takes into account the influence of texture and edges on cartoon video quality, thereby better characterizing the quality of cartoon videos.
[0096] 2. In order to overcome the problem of insufficient labeled samples in cartoon videos, this invention uses a self-supervised learning strategy to pre-train the designed model. It formulates edge similarity measurement and texture complexity calculation tasks to enable the model to focus on the edge and texture information of cartoon videos. Through distortion type and distortion degree prediction tasks, the model learns deep spatial features. The frame rate type prediction task enables the model to learn distortion-sensitive features in the time dimension.
[0097] 3. After self-supervised pre-training, this invention fine-tunes the model to enable it to evaluate the quality of cartoon videos, thereby improving the reliability and robustness of the model in evaluating cartoon video quality and making it more consistent with human visual perception. Attached Figure Description
[0098] Figure 1 This is a sequence diagram of the method of the present invention;
[0099] Figure 2 This is a diagram of the cartoon quality assessment model involved in this invention;
[0100] Figure 3 This is a structural diagram of the edge line quality sensing module involved in this invention;
[0101] Figure 4 This is a structural diagram of the texture feature quality perception module involved in this invention;
[0102] Figure 5 This is a diagram of the self-supervised pre-training framework involved in this invention;
[0103] Figure 6 This is a graph showing the results of the cartoon video quality assessment. Detailed Implementation
[0104] To provide a better understanding of the structural features and effects achieved by the present invention, a detailed description is provided below, accompanied by preferred embodiments and accompanying drawings:
[0105] like Figure 1 As shown, the blind evaluation method for cartoon video quality according to the present invention includes the following steps:
[0106] The first step is to acquire and preprocess the original cartoon video and the corresponding distorted video: acquire the original cartoon video and the corresponding distorted video frames, and perform preprocessing.
[0107] (1) The original cartoon video is used to generate corresponding distorted videos through distortion enhancement strategies, that is, the original cartoon video is artificially synthesized to generate corresponding distorted cartoon videos of different types and degrees.
[0108] (2) Extract frames from the input original cartoon video and the corresponding distorted video, and extract frames uniformly from the entire cartoon video.
[0109] (3) I will convert the cartoon video frame I n n = (1, 2, ..., N), where N is the total number of input cartoon video frames. After cropping to a uniform resolution, let's say H × W. Then the total size of the input video frames is N × 3 × H × W, where 3 represents the three RGB channels.
[0110] The second step is to construct a blind evaluation model for cartoon video quality: such as Figure 2 As shown, a self-supervised learning framework and a blind evaluation module for cartoon video quality are constructed based on the edge line quality perception module, the spatiotemporal feature quality perception module, and the texture feature quality perception module. Since cartoon videos consist of cartoon images made up of lines and flat surfaces, they have sharper edge lines and fewer local complex textures compared to natural videos. The human visual system pays more attention to the edge quality of cartoons and artifacts on planar color blocks. Therefore, combining the characteristics of cartoons to construct a three-branch blind evaluation model for cartoon video quality provides a more comprehensive representation of cartoon video quality. Furthermore, in the research on cartoon video quality evaluation, cartoon datasets lack sufficient labeled samples to train deep models, which increases the probability of overfitting and reduces the model's generalization ability. Therefore, a self-supervised learning strategy is adopted to overcome this problem, and different auxiliary tasks are formulated according to the roles of the three branches for pre-training, constructing a multi-task self-supervised learning framework. The specific steps are as follows:
[0111] (1) Set up an edge line quality perception module, such as Figure 3 As shown.
[0112] The human visual system is highly sensitive to edge quality, especially in cartoon videos, where the main elements are outlined by lines, and the quality of these lines has a significant impact on subjective visual perception. Therefore, an edge line quality perception module was designed to specifically characterize the edge sharpness and quality of cartoon videos.
[0113] A1) The EDTER network is set as the cartoon edge extractor. The cartoon edge extractor is first pre-trained on the BSDS500 dataset and then fine-tuned on the AnimeRun dataset with cartoon outline maps to extract the edge maps of cartoon video frames.
[0114] A2) Obtain the edge map of the input cartoon video frame using a cartoon edge extractor. The total size of the edge graph is N×3×H×W, as shown in the following formula:
[0115]
[0116] Where f EDTER It is a finely tuned cartoon edge extractor;
[0117] A3) Define an edge line quality-aware network with InceptionNeXt-Tiny as the backbone network. The first two stages consist of four InceptionNeXt blocks.
[0118] For each input cartoon edge image, edge features are extracted through the first two stages of InceptionNeXt-Tiny to obtain a deep, high-level edge line feature map (EM). n Its size is The specific formula is as follows:
[0119]
[0120] Among them, f Icn Indicates the InceptionNeXt block used;
[0121] A4) Equation (3) is used to further process and extract effective edge line quality-perceived features, and the depth, height, and width channel dimensions are aligned to promote the merging of edge features with spatiotemporal and texture feature channels, thereby obtaining the final edge line quality-perceived feature F. edge Its size is The number of output feature channels is represented by the following formula:
[0122]
[0123] Among them, f pThis represents a post-processing module. First, it performs dimensionality reduction using a Conv3d convolutional network with a kernel size of 2×4×4 and a stride of 2×4×4. Then, it uses the GELU activation function and a Conv3d convolutional network with a kernel size of 1×1×1 to extract deep edge line quality features.
[0124] (2) Setting up a spatiotemporal feature quality perception module. Videos contain rich static and dynamic information. By modeling the spatiotemporal aspects of the video, task-related information can be automatically captured, thus becoming a key component of various video-related tasks. Therefore, a spatiotemporal feature quality perception module is designed to obtain in-depth spatiotemporal information of cartoon videos that is sensitive to video quality.
[0125] A spatiotemporal feature quality perception network was designed, using the Video Swin Transformer-Tiny network as the backbone. This network was used in four stages to extract the deep spatiotemporal features F of the input cartoon video clip V. s-t Its size is The specific formula is as follows:
[0126] F s-t =f vst (V) (4)
[0127] Where f vst This refers to the Video Swin Transformer-Tiny network.
[0128] (3) Configure the texture feature quality perception module, such as Figure 4 As shown, compared to natural videos, cartoon videos have simpler local texture details. Some distortions inevitably affect the video's clarity and local details, and the quality of cartoon videos is more easily perceived through local texture detail analysis. Therefore, a texture feature quality perception module was designed to analyze the local texture information of cartoon videos.
[0129] C1) Using a grayscale converter, each frame of the input cartoon video is converted into an image. n Convert to grayscale image The total size of the input grayscale image is N×1×H×W, where 1 represents a single channel;
[0130] C2) Configure the texture feature quality perception module.
[0131] Using ConvNeXt-Tiny as the backbone network, four ConvNeXt blocks are used to extract grayscale images from each frame. Deep and highly complex local texture features TM n Its size is Use post-processing module f pFurther processing and extraction of effective texture features yields the final texture feature F. texture Its size is The specific formula is as follows:
[0132]
[0133]
[0134] Where f cb This indicates the ConvNeXt block used.
[0135] (4) Design a multi-task self-supervised learning framework for cartoon videos, such as... Figure 5 As shown, self-supervised learning strategies can overcome the problem of insufficient labeled samples on a large scale. In the field of self-supervised learning, multi-task learning methods are widely used. This is because, compared to single-task learning, multi-task learning can enhance the model's feature representation and generalization ability by sharing features and knowledge among multiple tasks. Therefore, a multi-task self-supervised learning framework for cartoon videos was designed.
[0136] D1) Using the edge line quality perception module, the spatiotemporal feature quality perception module, and the texture feature quality perception module, calculate the spatiotemporal feature differences between the original input cartoon video and its distorted video. Edge feature differences Texture feature differences The specific formula is as follows:
[0137]
[0138]
[0139]
[0140] in, and This represents the spatiotemporal, edge, and texture features extracted from the original input cartoon video, respectively. and These represent the spatiotemporal, edge, and texture features extracted from the input distorted cartoon video, respectively.
[0141] D2) uses edge similarity measurement as an auxiliary task for self-supervised pre-training of edge features, through edge feature differences. The model focuses on the edge information of the input cartoon video;
[0142] D3) Construct the prediction head structure f pred Used for auxiliary tasks, predicting the head structure f predThe process involves passing the material through a Conv3d convolutional layer with a 1×1×1 kernel, then through the GELU activation function, then through another Conv3d convolutional layer with a 1×1×1 kernel, and finally through global average pooling (GAP).
[0143] D4) For auxiliary tasks of self-supervised pre-training of texture features, texture complexity calculation tasks are used. By predicting the structure To calculate texture complexity The formula is as follows:
[0144]
[0145] D5) For auxiliary tasks of self-supervised pre-training of spatiotemporal features, three tasks are used: distortion type and distortion degree judgment, frame rate type prediction, etc.
[0146] For the task of judging the type and degree of distortion By predicting the head structure respectively and To predict the probability of distortion type and degree of distortion
[0147] For frame rate type prediction, use By predicting head structure To predict the probability of frame rate type The formula is as follows:
[0148]
[0149]
[0150]
[0151] T and R represent the total number of distortion types and the total number of frame rate types, respectively.
[0152] (5) Set up a blind evaluation module for cartoon video quality: Input a cartoon video, and the model extracts cartoon video features from multiple angles through the edge line quality perception module, the spatiotemporal feature quality perception module and the texture feature quality perception module. The extracted edge, spatiotemporal and texture features are fused to represent the features of the input cartoon video quality, and the quality prediction module is used to obtain the quality score of the cartoon video.
[0153] E1) Extract the F-values of the input cartoon video by setting the edge line quality perception module, setting the spatiotemporal feature quality perception module, and setting the texture feature quality perception module. edge F s-t and F textureBy performing concatenation and fusion, the final feature F representing the quality of the cartoon video is obtained. fusion Its size is The formula is as follows:
[0154] F fusion =Concat(F edge ,F s-t ,F texture (7)
[0155] E2) Using formula (8) to F fusion The quality score of the input cartoon video is mapped to the following formula:
[0156] Score = f out (F fusion (8)
[0157] Among them, f out The quality prediction module first extracts quality-sensitive features of the cartoon video using a Conv3d convolutional network with a kernel of 1×1×1. Then, it predicts the quality score using the GELU activation function and another Conv3d convolutional network with a kernel of 1×1×1. Finally, it uses global average pooling (GAP) to obtain the final cartoon video quality score.
[0158] The third step is the pre-training of the blind quality assessment model for cartoon videos: The original cartoon video and its corresponding distorted version are input to perform self-supervised pre-training on the blind quality assessment model. The pre-trained model is then saved to extract edge lines, spatiotemporal, and texture quality features from the cartoon video. The specific steps are as follows:
[0159] (1) Input the original cartoon video and the corresponding distorted video into the cartoon video quality blind evaluation model for training.
[0160] (2) For the distortion prediction auxiliary task and the texture complexity calculation auxiliary task, the ranking loss is used as the loss function l1 and l2 for distortion prediction and texture complexity calculation, respectively, as shown in the following formula:
[0161]
[0162]
[0163]
[0164] Where c and d are the true labels for texture complexity and distortion level, respectively. and These are the predicted values for texture complexity and distortion level, respectively. yes The covariance between and d and σ d They are and the standard deviation of d, and The calculation method is the same.
[0165] (3) For the edge similarity measurement auxiliary task, the established edge consistency loss is used as the loss function l3 for edge similarity measurement, as shown in the following formula:
[0166]
[0167] Here, Sigmoid(·) is the Sigmoid activation function, and GAP(·) is the global equal pooling operation.
[0168] (4) For the auxiliary tasks of distortion type prediction and frame rate type prediction, cross-entropy loss is used as the loss function l4 and l5 for distortion type prediction and frame rate type prediction, and the formulas are as follows:
[0169]
[0170]
[0171] Among them, t i and r i These are the actual labels for distortion type and frame rate type, respectively. and These are the predicted probability values for distortion type and frame rate type, where T and R are the total number of distortion types and the total number of frame rate types, respectively.
[0172] (5) The weighted sum of the losses from the five auxiliary tasks is used as the joint loss function L for self-supervised pre-training. pre The specific formula is as follows:
[0173] L pre =l1+α×l2+β×l3+λ×l4+ω×l5, (22)
[0174] Here, α, β, λ, and ω are all positive values, used as weighting parameters to adjust the relative importance of the five components in the joint loss function.
[0175] (6) Self-supervised pre-training of the cartoon video quality blind evaluation model is performed. The gradient of each parameter in the cartoon video quality blind evaluation model is calculated using the backpropagation method, and the parameters in the cartoon video quality blind evaluation model are updated using the stochastic gradient descent method. The parameters of the cartoon video quality blind evaluation model when the PLCC index is best are saved to extract edge, spatiotemporal and texture features.
[0176] The fourth step involves acquiring cartoon videos with real quality labels and cartoon videos to be evaluated, and performing the same frame-by-frame preprocessing.
[0177] Step 5: Obtaining the blind evaluation results of cartoon video quality: Input the preprocessed cartoon video with real quality labels into the pre-trained blind evaluation model of cartoon video quality, fine-tune the blind evaluation model of cartoon video quality, save the evaluation for cartoon video quality, and input the preprocessed cartoon video to be evaluated into the fine-tuned blind evaluation model of cartoon video quality to obtain the blind evaluation results of cartoon video quality.
[0178] (1) Input the pre-processed cartoon video with real quality labels into the pre-trained cartoon video quality blind evaluation model. Extract edge, spatiotemporal and texture features through the edge line quality perception module, the set spatiotemporal feature quality perception module and the set texture feature quality perception module, and then obtain the predicted quality score Q of the input cartoon video. pred .
[0179] (2) After self-supervised pre-training, the model has the ability to capture the distortion change characteristics of cartoon videos. By fine-tuning, the model can be made suitable for evaluating the quality of cartoon videos.
[0180] Using the distortion prediction loss function as the loss function L during fine-tuning tune The specific formula is as follows:
[0181] L tune =log(1-PLCC(Q) pred Q label )), (twenty three)
[0182] Q label It is the actual quality score of the input cartoon video.
[0183] (3) Fine-tune the cartoon video quality assessment model, use the backpropagation method to calculate the gradient of each parameter in the model, and use the stochastic gradient descent method to update the parameters in the model. Save the model parameters when the PLCC index is at its best to evaluate the quality of cartoon videos.
[0184] (4) Input the preprocessed cartoon video to be evaluated into the fine-tuned cartoon video quality blind evaluation model to obtain the blind evaluation results of the cartoon video quality.
[0185] like Figure 6 As shown, this is a sequence of cartoon video frames. The right side is clearly of higher quality than the left. A higher score indicates better quality. (retrain) means retraining with a labeled cartoon dataset; no (retrain) means using the model parameters provided by the algorithm to predict the score. Figure 6It can be seen that these natural video algorithms may mistakenly tend to generate higher scores for low-quality cartoon videos, while the proposed model for cartoon video quality assessment is able to generate reasonable quality scores consistent with human visual perception.
[0186] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.
Claims
1. A blind evaluation method for the quality of cartoon videos, characterized in that, Includes the following steps: 11) Acquisition and preprocessing of original cartoon video and corresponding distorted video: Acquire the original cartoon video and corresponding distorted video frames, and perform preprocessing; 12) Construct a blind evaluation model for cartoon video quality: Based on the edge line quality perception module, the spatiotemporal feature quality perception module, and the texture feature quality perception module, a self-supervised learning framework and a blind evaluation module are constructed. The steps involved in constructing a blind quality assessment model for cartoon videos are as follows: 121) Set up the edge line quality perception module; 122) Set up a spatiotemporal characteristic quality perception module; 123) Set up the texture feature quality perception module; 124) Design a multi-task self-supervised learning framework for cartoon videos: 1241) Using the edge line quality perception module, the spatiotemporal feature quality perception module, and the texture feature quality perception module, calculate the spatiotemporal feature differences between the original input cartoon video and its distorted video. Differences in edge features Texture feature differences The specific formula is as follows: ， ， ， in, , and This represents the spatiotemporal, edge, and texture features extracted from the original input cartoon video, respectively. , and These represent the spatiotemporal, edge, and texture features extracted from the input distorted cartoon video, respectively. 1242) Edge similarity measurement is used as an auxiliary task for self-supervised pre-training of edges, through edge feature differences. The model focuses on the edge information of the input cartoon video; 1243) Constructing the prediction head structure Used for auxiliary tasks, prediction head structure The process involves passing the material through a Conv3d convolutional layer with a 1x1x1 kernel, then through the GELU activation function, then through another Conv3d convolutional layer with a 1x1x1 kernel, and finally through global average pooling (GAP). 1244) For auxiliary tasks involving self-supervised pre-training of texture features, texture complexity computation tasks are used. By predicting the structure To calculate texture complexity The formula is as follows: ； 1245) For auxiliary tasks of self-supervised pre-training of spatiotemporal features, three tasks are used: distortion type and distortion degree judgment, frame rate type prediction, etc. For the task of judging the type and degree of distortion By predicting the head structure respectively and To predict the probability of distortion type and degree of distortion ; For frame rate type prediction, use By predicting head structure To predict the probability of frame rate type The formula is as follows: ， ， ， Where T and R represent the total number of distortion types and the total number of frame rate types, respectively; 125) Set up a blind evaluation module for cartoon video quality: 1251) Extract the quality of the input cartoon video by setting an edge line quality perception module, a spatiotemporal feature quality perception module, and a texture feature quality perception module. , as well as By performing concatenation and fusion, the final features representing the quality of the cartoon video are obtained. Its size is *(768+2C)* * The formula is as follows: ； 1252) will The quality score of the input cartoon video is mapped to the following formula: ， in, The quality prediction module first extracts quality-sensitive features of the cartoon video through a Conv3d convolutional network with a kernel of 1*1*1, then predicts the quality score through a GELU activation function and a Conv3d convolutional network with a kernel of 1*1*1, and finally obtains the final cartoon video quality score by using global average pooling (GAP). 13) Pre-training of the blind quality assessment model for cartoon videos: Input the original cartoon video and the corresponding distorted video to perform self-supervised pre-training of the blind quality assessment model for cartoon videos, and save the pre-trained model to extract edge lines, spatiotemporal and texture quality features of the cartoon video. 14) Acquisition and preprocessing of cartoon videos with real quality labels and cartoon videos to be evaluated; 15) Obtaining the blind evaluation results of cartoon video quality: Input the preprocessed cartoon video with real quality labels into the pre-trained blind evaluation model of cartoon video quality, fine-tune the blind evaluation model of cartoon video quality, and save it for the evaluation of cartoon video quality; input the preprocessed cartoon video to be evaluated into the fine-tuned blind evaluation model of cartoon video quality to obtain the blind evaluation results of cartoon video quality.
2. The blind evaluation method for cartoon video quality according to claim 1, characterized in that, The acquisition and preprocessing of the original cartoon video and the corresponding distorted video include the following steps: 21) The original cartoon video is used to generate corresponding distorted videos through distortion enhancement strategies, that is, the original cartoon video is artificially synthesized to generate corresponding distorted cartoon videos of different types and degrees. 22) Extract frames from the input original cartoon video and the corresponding distorted video, and then extract frames uniformly from the entire cartoon video. 23) Convert cartoon video frames , N is the total number of input cartoon video frames. After cropping to a uniform resolution, let's say H*W. Then the total size of the input video frames is N*3*H*W, where 3 represents the three RGB channels.
3. The blind evaluation method for cartoon video quality according to claim 2, characterized in that, The pre-training of the aforementioned blind evaluation model for cartoon video quality includes the following steps: 31) Input the original cartoon video and the corresponding distorted video into the cartoon video quality blind evaluation model for training; 32) For the distortion prediction auxiliary task and the texture complexity calculation auxiliary task, ranking loss is used as the loss function for distortion prediction and texture complexity calculation. , The formula is as follows: ， ， ， Where c and d are the true labels for texture complexity and distortion level, respectively. and These are the predicted values for texture complexity and distortion level, respectively. yes The covariance between and d and They are and the standard deviation of d, and The calculation methods are the same; 33) For edge similarity measurement auxiliary tasks, the established edge consistency loss is used as the loss function for edge similarity measurement. The formula is as follows: ， in, It is the Sigmoid activation function. It is a global equal pooling operation; 34) For the auxiliary tasks of distortion type prediction and frame rate type prediction, cross-entropy loss is used as the loss function for distortion type prediction and frame rate type prediction. , The formula is as follows: ， ， in, and These are the actual labels for distortion type and frame rate type, respectively. and These are the predicted probability values for distortion type and frame rate type, where T and R are the total number of distortion types and the total number of frame rate types, respectively. 35) Use the weighted sum of the losses from the five auxiliary tasks as the joint loss function for self-supervised pre-training. The specific formula is as follows: ， in, , , as well as All values are positive and are used as weighting parameters to adjust the relative importance of the five components in the joint loss function. 36) Self-supervised pre-training is performed on the blind evaluation model of cartoon video quality. The gradient of each parameter in the blind evaluation model of cartoon video quality is calculated using the backpropagation method, and the parameters in the blind evaluation model of cartoon video quality are updated using the stochastic gradient descent method. The parameters of the blind evaluation model of cartoon video quality when the PLCC index is best are saved to extract edge, spatiotemporal and texture features.
4. The blind evaluation method for cartoon video quality according to claim 3, characterized in that, The edge line quality sensing module includes the following steps: 41) The EDTER network is set as the cartoon edge extractor. The cartoon edge extractor is first pre-trained on the BSDS500 dataset and then fine-tuned on the AnimeRun dataset with cartoon outline maps to extract the edge maps of cartoon video frames. 42) Obtain the edge map of the input cartoon video frame using a cartoon edge extractor. The total size of the edge graph is N*3*H*W, as shown in the following formula: ， in It is a finely tuned cartoon edge extractor; 43) Set up an edge line quality-aware network, with InceptionNeXt-Tiny as the backbone network. The first two stages include four InceptionNeXt blocks. For each input cartoon edge image, edge features are extracted through the first two stages of InceptionNeXt-Tiny to obtain a deep, high-level edge line feature map. Its size is N*192* * The specific formula is as follows: ， in, Indicates the InceptionNeXt block used; 44) Further process and extract effective edge line quality-perceived features, and align the depth, height, and width channel dimensions to promote the merging of edge features with spatiotemporal and texture feature channels, thereby obtaining the final edge line quality-perceived features. Its size is *C* * , representing the number of output feature channels, is given by the following formula: ， in, This represents a post-processing module. First, it performs dimensionality reduction using a Conv3d convolutional network with a kernel size of 2*4*4 and a stride of 2*4*4. Then, it uses the GELU activation function and a Conv3d convolutional network with a kernel size of 1*1*1 to extract deep edge line quality features.
5. The blind evaluation method for cartoon video quality according to claim 4, characterized in that, The aforementioned spatiotemporal characteristic quality perception module includes the following steps: 51) Design a spatiotemporal feature quality perception network, using the Video Swin Transformer-Tiny network as the backbone network, and use the network in four stages to extract the deep spatiotemporal features of the input cartoon video clip V. Its size is *768* * The specific formula is as follows: ， in This refers to the Video Swin Transformer-Tiny network.
6. The blind evaluation method for cartoon video quality according to claim 5, characterized in that, The texture feature quality perception module includes the following steps: 61) Use a grayscale converter to convert each frame of the input cartoon video. Convert to grayscale image The total grayscale image size is N*1*H*W, where 1 represents a single channel. 62) Configure the texture feature quality perception module. Using ConvNeXt-Tiny as the backbone network, four ConvNeXt blocks are used to extract grayscale images from each frame. Deep and highly complex local texture features Its size is N*192* * and use post-processing module Further processing and extraction of effective texture features yields the final texture features. Its size is *C* * The specific formula is as follows: ， ， in This indicates the ConvNeXt block used.
7. The blind evaluation method for cartoon video quality according to claim 6, characterized in that, The blind evaluation results of the cartoon video quality are obtained through the following steps: 71) Input the pre-processed cartoon video with real quality labels into the pre-trained cartoon video quality blind evaluation model. Extract edge, spatiotemporal, and texture features through the edge line quality perception module, the set spatiotemporal feature quality perception module, and the set texture feature quality perception module, and then obtain the predicted quality score of the input cartoon video. ; 72) Use the distortion prediction loss function as the loss function for fine-tuning. The specific formula is as follows: ， in It is the actual quality score of the cartoon video; 73) Fine-tune the cartoon video quality assessment model, use the backpropagation method to calculate the gradient of each parameter in the model, and use the stochastic gradient descent method to update the parameters in the model. Save the model parameters when the PLCC index is at its best to evaluate the quality of cartoon videos. 74) Input the preprocessed cartoon video to be evaluated into the fine-tuned blind evaluation model of cartoon video quality to obtain the blind evaluation results of cartoon video quality.