A dual stability video quality evaluation method, system, device and medium

Abstract

This application provides a dual-stability video quality assessment method, system, apparatus, and medium, belonging to the field of video quality assessment technology. It involves decomposing a video data sequence into frames and further dividing each frame into blocks; using a pre-trained visual deep learning model ViT to extract image feature vectors from each block; calculating the inter-block distance and inter-frame distance of the video data sequence based on the image feature vectors; fusing the inter-block and inter-frame distances to obtain the average quality score of each image block; aggregating frame-level and video-level scores; and marking quality frames for feedback optimization. Frame-level scores are uniformly aggregated, while video-level scores are time-weighted aggregated. Therefore, this method achieves the technical effect of improving overall video quality without complete regeneration, requiring only a single forward propagation to detect and provide feedback on low-stability frames, thus improving generation efficiency and quality.

Description

Technical Field

[0001] This application belongs to the field of video quality assessment technology, and specifically relates to a dual-stability video quality assessment method, system, device and medium. Background Technology

[0002] In the field of video generation, such as when using diffusion-transformer (DiT) models or similar architectures to generate video sequences, problems such as noise interference, blurring, or temporal inconsistencies are often encountered. These problems lead to a decrease in the quality of the generated video, affecting the usability of the final output in surveillance, entertainment, or multimodal generation systems. Traditional video quality assessment methods typically rely on trained models or reference images (such as PSNR and SSIM), which are computationally complex and unsuitable for scenarios without references. Existing methods lack efficient quantification of the spatiotemporal stability in the generation process (such as denoising stages), and cannot provide real-time feedback on low-quality frames to optimize the generation process, resulting in low-quality video generation. Summary of the Invention

[0003] Given the lack of efficient quantification of spatiotemporal stability during the generation process, and the inability to provide real-time feedback on low-quality frames to optimize the generation process, resulting in low video generation quality, this application provides a dual-stability video quality assessment method, system, device, and medium.

[0004] One embodiment of this application provides a dual-stability video quality assessment method, including:

[0005] Based on the intermediate state video data sequence obtained from the model that is generating video, the video data sequence is decomposed into frames, and each frame of the video data sequence is divided to obtain a block patch result.

[0006] The pre-trained visual deep learning model ViT is used to extract the image feature vector patchbedding from the block results of each frame;

[0007] The inter-block distance and inter-frame distance of the video data sequence are obtained by performing dual distance calculation based on the image feature vector patchbedding;

[0008] The average quality score of the image block is calculated by fusing the inter-block distance and inter-frame distance.

[0009] Frame-level scores and video-level scores are aggregated, and quality frames are labeled for feedback optimization. Frame-level scores are aggregated uniformly, and video-level scores are aggregated temporally weighted.

[0010] Furthermore, the step of decomposing the video data sequence into frames includes:

[0011] The intermediate state video data sequence obtained in the video generation model is decomposed to obtain multiple video images;

[0012] The multi-frame video images correspond to the intermediate state sequence arrangement of the video being generated.

[0013] The step of dividing the video data sequence into frames to obtain a patch result includes:

[0014] The multi-frame video images obtained from the video data sequence are segmented into non-overlapping video blocks;

[0015] Flatten each non-overlapping video block to obtain the corresponding image block.

[0016] Furthermore, the step of extracting the image feature vector patchbedding from each frame block result using the pre-trained visual deep learning model ViT includes:

[0017] The image blocks obtained after partitioning are input into a pre-trained ViT network. Linear projection and position embedding are performed on the image blocks. Semantic feature representations of specific deep networks are extracted from the image blocks through forward propagation.

[0018] Furthermore, the extraction of semantic feature representations of specific deep networks from image patches through forward propagation includes:

[0019] Each non-overlapping image patch is mapped to a fixed-dimensional embedding vector, and the fixed-dimensional embedding vector is used as the input to the subsequent quality assessment module.

[0020] The fixed-dimensional embedding vectors of all image patches are arranged according to their spatial positions to form a frame-level structured representation, resulting in the image feature vector patch embedding.

[0021] Furthermore, the step of calculating the inter-block distance and inter-frame distance of the video data sequence based on the image feature vector patchbedding includes:

[0022] The image feature vectors obtained from frame-level structured representation are normalized, and the normalized image feature vectors are used for dual distance calculation.

[0023] The distance between blocks is determined based on the feature distance between the image feature vectors of the image blocks in adjacent Transformer layers;

[0024] The inter-frame distance is determined by extracting the features of the last layer of the network based on a visual deep learning model.

[0025] Furthermore, the normalization process for the image feature vectors obtained from the frame-level structured representation includes:

[0026] The image feature vector is processed using a normalization formula, which is:

[0027]

[0028] in, It is the norm for calculating the double distance between image feature vectors, that is, the absolute length of the vector in space; The image feature vector before normalization. This is the normalized image feature vector.

[0029] Furthermore, determining the inter-block distance based on the feature distance of the image feature vector of the image block in adjacent Transformer layers includes:

[0030] Calculate the feature position of the feature vector of the same image patch at the first time step of the Transformer layer;

[0031] Calculate the feature vector of the same image patch at the feature position of the second time step adjacent to the first time step in the Transformer layer;

[0032] The adjacent feature distances of the feature vectors of the same image block in the Transformer layer are determined based on the feature positions at the first and second time points.

[0033] Furthermore, the method of determining the inter-block distance based on the feature distance of the image feature vector of the image block in adjacent Transformer layers further includes:

[0034] Calculate the distances between all adjacent features of the same image patch in a selected layer of the Transformer layer;

[0035] The spatial distance between image blocks, i.e., the block spacing, is obtained by calculating the average of all adjacent feature distances.

[0036] Furthermore, the step of determining inter-frame distance by extracting the last layer features of the network based on the visual deep learning model includes:

[0037] Extract the features of the last layer of the network, and calculate the feature distance between two adjacent frames at the same spatial location based on the features of the last layer.

[0038] Furthermore, the step of determining inter-frame distance by extracting the last layer features of the network based on the visual deep learning model also includes:

[0039] Extract the feature distance between all adjacent frames at the same spatial location.

[0040] The inter-frame distance of the video data sequence is obtained by calculating the average inter-frame distance based on the feature distance between any two adjacent frames at the same spatial location.

[0041] Furthermore, the step of calculating the average quality score of image blocks by fusing the inter-block distance and inter-frame distance includes:

[0042] Map the inter-block distance to an inter-block distance score, and map the inter-frame distance to an inter-frame distance score;

[0043] The average quality score of the image block is obtained by fusing the inter-block distance score and the inter-frame distance score.

[0044] Furthermore, mapping the inter-block distance to an inter-block distance score includes:

[0045] The inter-block distance score is obtained by mapping using the inter-block distance mapping formula;

[0046] The inter-block distance mapping formula is as follows:

[0047]

[0048] Where α is the scaling parameter, α>0; The inter-block distance score. p is the distance between blocks; p is the side length of a single image block; f is the number of frames.

[0049] Furthermore, mapping the inter-frame distance to an inter-frame distance score includes:

[0050] The inter-frame distance score is obtained by mapping using the inter-frame distance mapping formula;

[0051] The inter-frame distance mapping formula is as follows:

[0052]

[0053] Where α is the scaling parameter, α>0; Inter-frame distance score, p is the inter-frame distance; p is the side length of a single image patch; f is the number of frames.

[0054] Furthermore, the average quality score of the image block is obtained by fusing the inter-block distance score and the inter-frame distance score, including:

[0055] The average quality score is obtained by performing dual fusion using the dual fusion score formula.

[0056] The formula for the dual fusion score is:

[0057]

[0058] in, β is the average quality score; β is the adjustable weight. The inter-block distance score; is the inter-frame distance score; p is the side length of a single image patch; f is the number of frames.

[0059] Furthermore, aggregate frame-level scores and video-level scores, including:

[0060] The frame-level score is obtained by aggregating all image frames based on the average quality score using a frame-level aggregation formula.

[0061] Calculate the frame weights of all frame difference sequences based on the frame-level scores;

[0062] The frame weights based on the difference sequences of all frames are aggregated using a video-level aggregation formula to obtain a video-level score.

[0063] Furthermore, the frame-level aggregation formula is as follows:

[0064]

[0065] in, Frame-level score; is the average quality score; p is the side length of a single image patch; N is the total number of image patches into which the image is segmented; f is the number of frames.

[0066] Furthermore, the frame weights of all frame difference sequences are calculated based on the frame-level scores, including:

[0067] Through the formula:

[0068]

[0069] Obtain the frame weights;

[0070] in, Frame weights; For frame-level scores, f is the frame score of the previous frame; f is the number of frames.

[0071] Furthermore, the video-level aggregation formula is as follows:

[0072]

[0073] Where Q is the video-level score; Q ranges from 0 to 1; Frame weights; f is the frame score; fF is the total number of frames.

[0074] Furthermore, the labeled quality frame is used for feedback optimization, including:

[0075] Set a frame-level score threshold and set the frame-level score. Image frames smaller than the frame-level score threshold are marked and fed back into video frame generation;

[0076] For frame-level scores For image frames with a frame-level score threshold, the number of iterations is increased and noise removal is enhanced.

[0077] Based on the same inventive concept, another aspect of the embodiments of this application provides a dual-stability video quality assessment system, including:

[0078] The video segmentation unit is used to obtain an intermediate state video data sequence based on the model that is generating video, decompose the video data sequence into frames, and divide each frame of the video data sequence to obtain a segmentation patch result.

[0079] The feature extraction unit is used to extract the image feature vector patchbedding from the block results of each frame using the pre-trained visual deep learning model ViT;

[0080] The dual computation unit is used to perform dual distance calculations based on the image feature vector patchbedding to obtain the inter-block distance and inter-frame distance of the video data sequence;

[0081] The quality score unit is used to calculate the average quality score of an image block by fusing the inter-block distance and the inter-frame distance;

[0082] The aggregation optimization unit aggregates frame-level scores and video-level scores, and marks quality frames for feedback optimization. Frame-level scores are aggregated uniformly, while video-level scores are aggregated temporally weighted.

[0083] Preferably, the video segmentation unit is used to decompose the video data sequence into frames, including:

[0084] The intermediate state video data sequence obtained in the video generation model is decomposed to obtain multiple video images;

[0085] The multi-frame video images correspond to the intermediate state sequence arrangement of the video being generated.

[0086] Preferably, the video segmentation unit, used to divide each frame of the video data sequence to obtain a segmentation patch result, further includes:

[0087] The multi-frame video images obtained from the video data sequence are segmented into non-overlapping video blocks;

[0088] Flatten each non-overlapping video block to obtain the corresponding image block.

[0089] Based on the same inventive concept, another aspect of the embodiments of this application provides an electronic device, characterized in that it includes: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0090] Memory, which stores computer programs;

[0091] The processor, when executing a program stored in memory, implements a dual-stability video quality assessment method.

[0092] Based on the same inventive concept, another aspect of the embodiments of this application provides a computer-readable storage medium, characterized in that it stores a computer program, which, when executed by a processor, implements a dual-stability video quality assessment method.

[0093] The beneficial effects of this application are:

[0094] Based on the above technical solution, the video data sequence is decomposed into frames, and each frame of the video data sequence is further divided to obtain block results. Image feature vectors are extracted from each frame block result using a pre-trained visual deep learning model ViT. The inter-block distance and inter-frame distance of the video data sequence are calculated using dual distance calculations based on the image feature vectors. The average quality score of the image blocks is obtained by fusing the inter-block distance and inter-frame distance. Frame-level scores and video-level scores are aggregated, and quality frames are marked for feedback optimization. Frame-level scores are uniformly aggregated, and video-level scores are temporally weighted aggregated. Therefore, this method achieves the technical effect of improving overall video quality without complete regeneration, requiring only a single forward propagation. It can be integrated into video generation systems (such as the WanVideo model) to detect low-stability frames and feed them back to the denoising process, improving generation efficiency and quality.

[0095] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description and the accompanying drawings. Attached Figure Description

[0096] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0097] Figure 1A schematic diagram of a dual-stability video quality assessment method is shown.

[0098] Figure 2 A schematic diagram of a dual-stability video quality assessment method system is shown.

[0099] Figure 3 A schematic diagram of the dual-stability video data sequence processing procedure is shown.

[0100] Figure 4 A schematic diagram of an electronic device is shown. Detailed Implementation

[0101] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0102] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. In this application, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," "longitudinal," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings.

[0103] This application provides a dual-stability video quality assessment method, see [link to relevant documentation]. Figure 1 ,include:

[0104] S101: Based on the intermediate state video data sequence obtained from the model that is generating video, the video data sequence is decomposed into frames, and each frame of the video data sequence is divided to obtain a block patch result.

[0105] S102: Use the pre-trained visual deep learning model ViT to extract the image feature vector patchbedding from the block results of each frame;

[0106] S103: Calculate the inter-block distance and inter-frame distance of the video data sequence by performing dual distance calculation based on the image feature vector patchbedding;

[0107] S104: The average quality score of the image block is obtained by fusing the inter-block distance and inter-frame distance.

[0108] S105: Aggregate frame-level scores and video-level scores, and mark quality frames for feedback optimization. Frame-level scores are aggregated uniformly, and video-level scores are aggregated temporally weighted.

[0109] The dual-stability video quality assessment method described in the embodiments of this specification is typically applied in servers, which can be cloud servers and local servers. Servers can be electronic devices such as laptops, desktop computers, tablets, and all-in-one computers.

[0110] This specification's embodiments are based on a training-free method using a pre-trained VisionTransformer (ViT) (computer vision) model. It achieves efficient video quality assessment by calculating the dual stability of the patchbedding across both block-wise and frame-wise transformer segments as a quality surrogate. Requiring only a single forward pass, it can be integrated into video generation systems (such as the WanVideo model) to detect low-stability frames and feed them back into the denoising process, improving generation efficiency and quality.

[0111] Specifically, the embodiments of this application include the following steps:

[0112] Input video sequence preprocessing: Decompose the video data sequence into frames and divide each frame into patches.

[0113] ViT forward propagation: Extracts image feature vector patch embeddings for each frame using a pre-trained ViT model.

[0114] Dual distance calculation: Calculate (see details below) inter-block distance and inter-frame distance, and quantize spatiotemporal stability.

[0115] The core assumption of this embodiment is that "stability is equivalent to high quality". The smaller the evolution of spatial features between adjacent network layers (smaller inter-block distance), the more stable the intra-frame spatial structure; the smaller the feature differences at the same position between adjacent frames (smaller inter-frame distance), the more consistent the temporal sequence. Therefore, smaller distance values ​​are quantified as high stability, i.e., high quality.

[0116] Specifically, the feature vectors (embedded representations) of a specific Transformer layer in the pre-trained ViT model are extracted, firstly normalized by L2, and then the Euclidean distance (L2 distance) formula is used to calculate the "feature distance between adjacent network layers of the same frame (inter-block distance)" and the "feature distance between adjacent frames in the last layer (inter-frame distance)".

[0117] Fusion and score calculation (detailed calculation method follows): Fusion of dual distances yields patch-level quality scores.

[0118] First, using a mapping function of the Sigmoid form. The calculated inter-block distance and inter-frame distance are mapped to scores q_block and q_frame, respectively, ranging from 0 to 1, where a smaller distance d corresponds to a higher score q. Subsequently, the two are linearly weighted and fused using an adjustable weight β.

[0119]

[0120] This fusion formula is based on the premise that video quality is jointly determined by "single-frame spatial quality" and "multi-frame temporal smoothness". The weight β is set to 0.5 by default to balance the contributions of the two, but in time-sensitive tasks (such as videos with rapid action), β can be reduced to increase the weight of inter-frame stability in the overall score.

[0121] Aggregation and Output: Aggregate to frame-level and video-level scores, and mark low-quality frames for feedback optimization.

[0122] The core innovation lies in introducing dual stability: inter-block stability captures the evolution of intra-frame spatial features, while inter-frame stability captures temporal continuity. Small distances correspond to high stability (high quality), and large distances correspond to low stability (low quality). This method is suitable for video quality assessment scenarios, such as in the denoising stages of the WanVideo model, where the quality of the generated sequence is evaluated in real time, and denoising iterations are enhanced for low-quality frames.

[0123] It should be noted that the technical English terms or mathematical formulas used in this specification are terms that can be understood by those skilled in the art, and the embodiments in this specification further explain them.

[0124] • VIT / VisionTransformer stands for "Vision Transformer (ViT)," which is a video model.

[0125] • Patch: An image patch or block.

[0126] •Patchembedding: Image patch embedding representation or image patch feature vector.

[0127] •PosEmbedding: Position embedding.

[0128] •LinearProj / LinearProjection: Linear projection.

[0129] • TransformerBlock: A Transformer encoding layer or Transformer block.

[0130] • Denoising stages: These are the denoising stages.

[0131] •latent: the latent representation or latent feature space.

[0132] • pipeline: refers to the processing flow or processing pipeline.

[0133] •VAE: Variational Autoencoder.

[0134] In some specific embodiments, the step of decomposing the video data sequence into frames includes:

[0135] The intermediate state video data sequence obtained in the video generation model is decomposed to obtain multiple video images;

[0136] The multi-frame video images correspond to the intermediate state sequence arrangement of the video being generated.

[0137] The step of dividing the video data sequence into frames to obtain a patch result includes:

[0138] The multi-frame video images obtained from the video data sequence are segmented into non-overlapping video blocks;

[0139] Flatten each non-overlapping video block to obtain the corresponding image block.

[0140] It should be noted that the input video sequence V = {I1, I2, ..., I_F}, where I_f ∈ ℝ^{H×W×3}, I_f is the f-th frame, and F is the frame number. (Here, I_f (uppercase I, subscript f) represents the f-th frame in the video sequence V).

[0141] Each frame I_f is divided into N = HW / P² non-overlapping patches of size P × P, and each patch is flattened into p_i^{(f)}∈ℝ^{P²·3}.

[0142] In some specific embodiments, the step of extracting the image feature vector patchbedding from the block results of each frame using the pre-trained visual deep learning model ViT includes:

[0143] The image blocks obtained after partitioning are input into a pre-trained ViT network. Linear projection and position embedding are performed on the image blocks. Semantic feature representations of specific deep networks are extracted from the image blocks through forward propagation.

[0144] Furthermore, the extraction of semantic feature representations of specific deep networks from image patches through forward propagation includes:

[0145] Each non-overlapping image patch is mapped to a fixed-dimensional embedding vector, and the fixed-dimensional embedding vector is used as the input to the subsequent quality assessment module.

[0146] The fixed-dimensional embedding vectors of all image patches are arranged according to their spatial positions to form a frame-level structured representation, resulting in the image feature vector patch embedding.

[0147] The embedding process employs a lightweight convolutional encoder, balancing computational efficiency with feature representation capabilities.

[0148] Specifically, each frame is processed independently using a pre-trained ViT model (with L transformer blocks):

[0149] Linear projection and position embedding:

[0150]

[0151] Where Y∈ℝ^{D×(P²·3)} is the projection matrix, b∈ℝ^D is the bias, E_pos∈ℝ^{N×D} is the position embedding, z0^{(f)}∈ℝ^{N×D}, and D is the embedding dimension.

[0152] This formula describes the flattening and projection process of the input image. Its physical meaning is as follows: After dividing the f-th frame image into N image blocks, each image block is multiplied by the projection matrix and a bias is added to perform a linear mapping operation; then all the mapped results are concatenated together and the position embedding information representing the original spatial position of the image blocks is superimposed to obtain the first layer feature representation z0 input to the Transformer.

[0153] Symbolic meaning:

[0154] Y: Linear projection matrix, used to map image patches to a high-dimensional feature space.

[0155] b: Bias vector.

[0156] Epos: Position embedding tensor, used to preserve the position information of image patches in the original image.

[0157] N: The total number of image blocks into which an image is segmented.

[0158] D: The dimension of feature embedding in the Transformer model.

[0159] P: Side length of a single image patch.

[0160] Transformer layer: For the l-th layer (l=1 to L).

[0161]

[0162] Among them, MHA stands for Multi-head Self-Attention, MLP stands for Multilayer Perceptron, and LN stands for Layer Normalization.

[0163] Extract embeddings z_t^{(f,p)}∈ℝ^D (where p is the patch index) from a selected subset of layers T={t0,t1,…,t_{T-1}} (adjacent layers, such as intermediate layers). T is a typical transformer subset selected from ViT.

[0164] In some specific embodiments, the step of calculating the inter-block distance and inter-frame distance of the video data sequence based on the image feature vector patchbedding includes:

[0165] The image feature vectors obtained from frame-level structured representation are normalized, and the normalized image feature vectors are used for dual distance calculation.

[0166] The distance between blocks is determined based on the feature distance between the image feature vectors of the image blocks in adjacent Transformer layers;

[0167] The inter-frame distance is determined by extracting the features of the last layer of the network based on a visual deep learning model.

[0168] Furthermore, the normalization process for the image feature vectors obtained from the frame-level structured representation includes:

[0169] The image feature vector is processed using a normalization formula, which is:

[0170]

[0171] in, It is the norm for calculating the double distance between image feature vectors, that is, the absolute length of the vector in space; The image feature vector before normalization. This is the normalized image feature vector. L2 normalization is performed on the extracted feature vectors to eliminate the difference in absolute numerical magnitude (length) of feature vectors between different layers or frames. This makes the subsequently calculated L2 distance actually equivalent to measuring the cosine similarity in the direction of the feature vectors, which helps to more stably and purely reflect the evolution of image content and semantic features.

[0172] In some specific embodiments, determining the inter-block distance based on the feature distance of the image feature vector of the image block in adjacent Transformer layers includes:

[0173] Calculate the feature position of the feature vector of the same image patch at the first time step of the Transformer layer;

[0174] Calculate the feature vector of the same image patch at the feature position of the second time step adjacent to the first time step in the Transformer layer;

[0175] The adjacent feature distances of the feature vectors of the same image block in the Transformer layer are determined based on the feature positions at the first and second time points.

[0176] Furthermore, the method of determining the inter-block distance based on the feature distance of the image feature vector of the image block in adjacent Transformer layers further includes:

[0177] Calculate the distances between all adjacent features of the same image patch in a selected layer of the Transformer layer;

[0178] The spatial distance between image blocks, i.e., the block spacing, is obtained by calculating the average of all adjacent feature distances.

[0179] Specifically, the distance between adjacent features:

[0180]

[0181] in, is the distance between adjacent features; p is the side length of a single image patch; f is the number of frames.

[0182] Inter-block distance:

[0183]

[0184] is the inter-block distance; p is the side length of a single image block; f is the number of frames; and T is a typical transformer sublayer selected from ViT.

[0185] In some specific embodiments, the step of determining inter-frame distance by extracting the last layer features of the network based on the visual deep learning model includes:

[0186] Extract the features of the last layer of the network, and calculate the feature distance between two adjacent frames at the same spatial location based on the features of the last layer.

[0187] Furthermore, the step of determining inter-frame distance by extracting the last layer features of the network based on the visual deep learning model also includes:

[0188] Extract the feature distance between all adjacent frames at the same spatial location.

[0189] The inter-frame distance of the video data sequence is obtained by calculating the average inter-frame distance based on the feature distance between any two adjacent frames at the same spatial location.

[0190] Specifically, the feature distance between two adjacent frames:

[0191]

[0192] Inter-frame distance:

[0193]

[0194] in, The feature distance between two adjacent frames; p is the inter-frame distance; p is the side length of a single image patch; f is the number of frames. The image feature vector before normalization. This is the normalized image feature vector; and The frames are adjacent.

[0195] In some specific embodiments, the step of calculating the average quality score of image blocks by fusing the inter-block distance and inter-frame distance includes:

[0196] Map the inter-block distance to an inter-block distance score, and map the inter-frame distance to an inter-frame distance score;

[0197] The average quality score of the image block is obtained by fusing the inter-block distance score and the inter-frame distance score.

[0198] Furthermore, mapping the inter-block distance to an inter-block distance score includes:

[0199] The inter-block distance score is obtained by mapping using the inter-block distance mapping formula;

[0200] The inter-block distance mapping formula is as follows:

[0201]

[0202] Where α is the scaling parameter, α>0; The inter-block distance score. p is the distance between blocks; p is the side length of a single image block; f is the number of frames.

[0203] Furthermore, mapping the inter-frame distance to an inter-frame distance score includes:

[0204] The inter-frame distance score is obtained by mapping using the inter-frame distance mapping formula;

[0205] The inter-frame distance mapping formula is as follows:

[0206]

[0207] Where α is the scaling parameter, α>0; Inter-frame distance score, p is the inter-frame distance; p is the side length of a single image patch; f is the number of frames.

[0208] Furthermore, the average quality score of the image block is obtained by fusing the inter-block distance score and the inter-frame distance score, including:

[0209] The average quality score is obtained by performing dual fusion using the dual fusion score formula.

[0210] The formula for the dual fusion score is:

[0211]

[0212] in, β is the average quality score; β is an adjustable weight, defaulting to 0.5, which balances the spatiotemporal contribution and is adjusted according to the generation scenario, such as reducing β in time-sensitive tasks. The inter-block distance score; is the inter-frame distance score; p is the side length of a single image patch; f is the number of frames.

[0213] In some specific embodiments, the aggregated frame-level score and video-level score include:

[0214] The frame-level score is obtained by aggregating all image frames based on the average quality score using a frame-level aggregation formula.

[0215] Calculate the frame weights of all frame difference sequences based on the frame-level scores;

[0216] The frame weights based on the difference sequences of all frames are aggregated using a video-level aggregation formula to obtain a video-level score.

[0217] Furthermore, the frame-level aggregation formula is as follows:

[0218]

[0219] in, Frame-level score; is the average quality score; p is the side length of a single image patch; N is the total number of image patches into which the image is segmented; f is the number of frames.

[0220] In some specific embodiments, the frame weights of all frame difference sequences are calculated based on frame-level scores, including:

[0221] Through the formula:

[0222]

[0223] Obtain the frame weights;

[0224] in, Frame weights; For frame-level scores, f is the frame score of the previous frame; f is the number of frames.

[0225] Furthermore, the video-level aggregation formula is as follows:

[0226]

[0227] Where Q is the video-level score; Q ranges from 0 to 1, with 1 representing the highest quality. Frame weights; f is the frame score; fF is the total number of frames.

[0228] Specifically, the frame weight w_f is calculated (e.g., softmax based on inter-frame score differences: w_f = softmax(-|Q_{f+1}-Q_f|), with the boundaries padded with the mean). The softmax is applied to all frame difference sequences, with lower differences (stable temporal sequences) corresponding to higher weights. Boundary handling: the differences between the first and last frames are padded with the overall mean of the sequence to avoid edge bias.

[0229] In some specific embodiments, the marked quality frame is used for feedback optimization, including:

[0230] Set a frame-level score threshold and set the frame-level score. Image frames smaller than the frame-level score threshold are marked and fed back into video frame generation;

[0231] For frame-level scores For image frames with a frame-level score threshold, the number of iterations is increased and noise removal is enhanced.

[0232] The technical solution described in this specification will be explained in detail below with reference to the accompanying drawings.

[0233] See Figure 3This embodiment designs a quality evaluation system for a typical DiT (Diffusion Transformer) architecture video generation process (such as the Wan2.2 series models). This system intervenes in real-time during the denoising stages of video generation, evaluating the generation quality by analyzing the stability of intermediate latent representations and providing feedback to optimize the denoising process. The system architecture is tightly integrated with the generation workflow. Figure 3 In this context, Noise Frame represents a noisy frame; DiT represents the Diffusion Transformer large model; Video1 latent Space represents the initial video latent representation space; VAE represents the Variational Autoencoder; VIT represents the Visual Transformer Feature Extractor; and Frame Interpolate represents the frame interpolation module.

[0234] like Figure 3 As shown, in the video generation process of the Diffusion Model (DiT), the input noisy frames are processed by DiT to obtain a preliminary video latent space (Video1 latent space). This representation can be decoded by VAE to obtain preliminary Video1. At this time, the latent representation or Video1 is fed into the ViT model, and the dual stability algorithm of this invention is used for quality evaluation. The system will identify and remove low-quality frames, and then use the frame interpolation module to regenerate supplementary frames based on the retained adjacent high-quality frames. Finally, the final high-quality Video2 is uniformly decoded by VAE and output.

[0235] Example workflow (integrated into WanVideo denoising pipeline):

[0236] 1. Input: A 30-frame video sequence (224x224 RGB or corresponding latent representation, e.g., intermediate output from high-noise expert to low-noise expert) generated by WanVideo (DiT-based).

[0237] 2. Preprocessing: Each frame (or latent frame) is divided into 196 patches (P=16) to form a patch sequence.

[0238] 3. Extraction: Select the intermediate block subset T={6,7,8,9,10,11}, and run ViT forward propagation to obtain the patch embeddings of each layer.

[0239] 4. Distance Calculation: For each patch, calculate the average inter-block distance and the local inter-frame distance.

[0240] 5. Fusion: Obtain the patch score (e.g., 0.8~0.95 range, small distance → high score).

[0241] 6. Aggregation: Frame-level score Q_f (obtained by uniform aggregation weighting) and video-level score; frames 10-15 are marked as low quality (Q_f < 0.7, these frames may correspond to instability in the denoising transition phase, such as layout collapse or noise residue).

[0242] Feedback: In the WanVideo system, the latent representations of marked frames 10-15 are directly deleted (the sequence temporarily becomes 25 frames). Then, a frame interpolation model (such as FILM) is used to interpolate five new latent frames from the adjacent retained frames, which are then inserted back into their corresponding positions to restore the complete 30-frame sequence. Finally, it is sent to VAE for decoding, resulting in high-quality Video2 with no obvious artifacts.

[0243] Steps 1 through 7 constitute a complete closed-loop logic of "monitoring-evaluation-feedback-repair", and the specific relationships are as follows:

[0244] • Step 1-2 (Data Preparation): First, extract the intermediate state data sequence from the model that is generating video (such as WanVideo), and divide it into image patches of fixed size to prepare for subsequent data format conversion.

[0245] • Step 3 (Feature Awareness): Input the preprocessed data into the pre-trained ViT network and extract the semantic feature representation of specific deep networks through forward propagation.

[0246] • Steps 4-6 (Analysis and Qualitative Analysis): These three steps are the core evaluation process. First, the spatial and temporal difference distances are calculated based on the extracted features (Step 4). Then, the distances are fused and converted into an intuitive quality score (Step 5). Next, a threshold judgment is performed using the aggregated scores (Qf < 0.7) to accurately locate the specific frame numbers with poor generation quality, layout collapse, or residual noise (Step 6).

[0247] Step 7 (Feedback Control): Perform error correction and repair. The "poor frames" identified in Step 6 are deleted from the original sequence. The frame interpolation model is then invoked to regenerate replacement frames based on the surrounding good frames, thus achieving the technical effect of improving overall video quality without requiring a complete regeneration.

[0248] Based on the same inventive concept, another aspect of the embodiments of this application provides a dual-stability video quality assessment system, see [link to relevant documentation]. Figure 2 ,include:

[0249] The video segmentation unit 201 is used to obtain an intermediate state video data sequence based on the model that is generating video, decompose the video data sequence into frames, and divide each frame of the video data sequence to obtain a segmentation patch result.

[0250] Feature extraction unit 202 is used to extract image feature vector patchbedding from the block results of each frame using the pre-trained visual deep learning model ViT;

[0251] The dual calculation unit 203 is used to perform dual distance calculations based on the image feature vector patchbedding to obtain the inter-block distance and inter-frame distance of the video data sequence;

[0252] Quality score unit 204 is used to calculate the average quality score of image blocks by fusing the inter-block distance and inter-frame distance;

[0253] The aggregation optimization unit 205 is used to aggregate frame-level scores and video-level scores, and to mark quality frames for feedback optimization. Frame-level scores are aggregated uniformly, and video-level scores are aggregated temporally weighted.

[0254] Preferably, the video segmentation unit 201 is used to decompose the video data sequence into frames, including:

[0255] The intermediate state video data sequence obtained in the video generation model is decomposed to obtain multiple video images;

[0256] The multi-frame video images correspond to the intermediate state sequence arrangement of the video being generated.

[0257] Preferably, the video segmentation unit 201, used to divide each frame of the video data sequence to obtain a segmentation patch result, further includes:

[0258] The multi-frame video images obtained from the video data sequence are segmented into non-overlapping video blocks;

[0259] Flatten each non-overlapping video block to obtain the corresponding image block.

[0260] Based on the same inventive concept, this disclosure also provides an electronic device 161, see [link to previous document]. Figure 4 It includes a processor 164, a communication interface 165, a memory 162, and a communication bus, wherein the processor 164, the communication interface 165, and the memory 162 communicate with each other through the communication bus;

[0261] Memory 162 stores computer program 163;

[0262] When processor 164 executes the program stored in memory 162, it implements the aforementioned dual-stability video quality assessment method.

[0263] The aforementioned communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0264] The communication interface 165 is used for communication between the aforementioned electronic device 161 and other devices.

[0265] The memory 162 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 162 may also be at least one storage device located remotely from the aforementioned processor 164.

[0266] The processor 164 mentioned above can be a general-purpose processor 164, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0267] Based on the same inventive concept, another aspect of the present disclosure provides a computer-readable storage medium storing a computer program 163, which, when executed by a processor 164, implements a dual-stability video quality assessment method.

[0268] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or it may exist independently and not assembled into the device / apparatus. The computer-readable storage medium carries one or more programs that, when executed, implement a dual-stability video quality assessment method according to embodiments of this disclosure.

[0269] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A dual-stability video quality assessment method, characterized in that, include: Based on the intermediate state video data sequence obtained from the model that is generating video, the video data sequence is decomposed into frames, and each frame of the video data sequence is divided to obtain a block patch result. The pre-trained visual deep learning model ViT is used to extract the image feature vector patchbedding from the block results of each frame; The inter-block distance and inter-frame distance of the video data sequence are obtained by performing dual distance calculation based on the image feature vector patchbedding; The average quality score of the image block is calculated by fusing the inter-block distance and inter-frame distance. Frame-level scores and video-level scores are aggregated, and quality frames are labeled for feedback optimization. Frame-level scores are aggregated uniformly, while video-level scores are aggregated temporally weighted.

2. The method according to claim 1, characterized in that, The step of decomposing the video data sequence into frames includes: The intermediate state video data sequence obtained in the video generation model is decomposed to obtain multiple video images; The multi-frame video images correspond to the intermediate state sequence arrangement of the video being generated.

3. The method according to claim 2, characterized in that, The step of dividing the video data sequence into frames to obtain a patch result includes: The multi-frame video images obtained from the video data sequence are segmented into non-overlapping video blocks; Flatten each non-overlapping video block to obtain the corresponding image block.

4. The method according to claim 1, characterized in that, The step of extracting image feature vector patchbedding from each frame block result using the pre-trained visual deep learning model ViT includes: The image blocks obtained after partitioning are input into a pre-trained ViT network. Linear projection and position embedding are performed on the image blocks. Semantic feature representations of specific deep networks are extracted from the image blocks through forward propagation.

5. The method according to claim 4, characterized in that, The extraction of semantic feature representations of specific deep networks from image patches through forward propagation includes: Each non-overlapping image patch is mapped to a fixed-dimensional embedding vector, and the fixed-dimensional embedding vector is used as the input to the subsequent quality assessment module. The fixed-dimensional embedding vectors of all image patches are arranged according to their spatial positions to form a frame-level structured representation, resulting in the image feature vector patch embedding.

6. The method according to claim 1, characterized in that, The step of calculating the inter-block distance and inter-frame distance of the video data sequence based on the image feature vector patchbedding includes: The image feature vectors obtained from frame-level structured representation are normalized, and the normalized image feature vectors are used for dual distance calculation. The distance between blocks is determined based on the feature distance between the image feature vectors of the image blocks in adjacent Transformer layers; The inter-frame distance is determined by extracting the features of the last layer of the network based on a visual deep learning model.

7. The method according to claim 6, characterized in that, The normalization process for the image feature vectors obtained from frame-level structured representation includes: The image feature vector is processed using a normalization formula, which is: in, It is the norm for calculating the double distance between image feature vectors, that is, the absolute length of the vector in space; The image feature vector before normalization. This is the normalized image feature vector.

8. The method according to claim 7, characterized in that, The method of determining the inter-block distance based on the feature distance of the image feature vector of the image block in adjacent Transformer layers includes: Calculate the feature position of the feature vector of the same image patch at the first time step of the Transformer layer; Calculate the feature vector of the same image patch at the feature position of the second time step adjacent to the first time step in the Transformer layer; The adjacent feature distances of the feature vectors of the same image block in the Transformer layer are determined based on the feature positions at the first and second time points.

9. The method according to claim 8, characterized in that, The method of determining the inter-block distance based on the feature distance of the image feature vector of the image block in adjacent Transformer layers further includes: Calculate the distances between all adjacent features of the same image patch in a selected layer of the Transformer layer; The spatial distance between image blocks, i.e., the block spacing, is obtained by calculating the average of all adjacent feature distances.

10. The method according to claim 7, characterized in that, The method of determining inter-frame distance by extracting the last layer features of the network based on a visual deep learning model includes: Extract the features of the last layer of the network, and calculate the feature distance between two adjacent frames at the same spatial location based on the features of the last layer.

11. The method according to claim 10, characterized in that, The method for determining inter-frame distance by extracting the last layer features of the network based on a visual deep learning model also includes: Extract the feature distance between all adjacent frames at the same spatial location. The inter-frame distance of the video data sequence is obtained by calculating the average inter-frame distance based on the feature distance between adjacent two frames at all the same spatial locations.

12. The method according to claim 1, characterized in that, The step of calculating the average quality score of image blocks by fusing the inter-block distance and inter-frame distance includes: Map the inter-block distance to an inter-block distance score, and map the inter-frame distance to an inter-frame distance score; The average quality score of the image block is obtained by fusing the inter-block distance score and the inter-frame distance score.

13. The method according to claim 12, characterized in that, Mapping the inter-block distance to an inter-block distance score includes: The inter-block distance score is obtained by mapping using the inter-block distance mapping formula; The inter-block distance mapping formula is as follows: Where α is the scaling parameter, α>0; The inter-block distance score. p is the distance between blocks; p is the side length of a single image block; f is the number of frames.

14. The method according to claim 12, characterized in that, Mapping the inter-frame distance to an inter-frame distance fraction includes: The inter-frame distance score is obtained by mapping using the inter-frame distance mapping formula; The inter-frame distance mapping formula is as follows: Where α is the scaling parameter, α>0; Inter-frame distance score, p is the inter-frame distance; p is the side length of a single image patch; f is the number of frames.

15. The method according to claim 12, characterized in that, The average quality score of the image block is obtained by fusing the inter-block distance score and the inter-frame distance score, including: The average quality score is obtained by performing dual fusion using the dual fusion score formula. The formula for the dual fusion score is: in, β is the average quality score; β is the adjustable weight. The inter-block distance score; is the inter-frame distance score; p is the side length of a single image patch; f is the number of frames.

16. The method according to claim 1, characterized in that, Aggregate frame-level scores and video-level scores, including: The frame-level score is obtained by aggregating all image frames based on the average quality score using a frame-level aggregation formula. Calculate the frame weights of all frame difference sequences based on the frame-level scores; The frame weights based on the difference sequences of all frames are aggregated using a video-level aggregation formula to obtain a video-level score.

17. The method according to claim 16, characterized in that, The frame-level aggregation formula is as follows: in, Frame-level score; is the average quality score; p is the side length of a single image patch; N is the total number of image patches into which the image is segmented; f is the number of frames.

18. The method according to claim 16, characterized in that, Calculate the frame weights for all frame difference sequences based on the frame-level scores, including: Through the formula: Obtain the frame weights; in, Frame weights; For frame-level scores, f is the frame score of the previous frame; f is the number of frames.

19. The method according to claim 16, characterized in that, The video-level aggregation formula is as follows: Where Q is the video-level score; Q ranges from 0 to 1; Frame weights; f is the frame score; fF is the total number of frames.

20. The method according to claim 1, characterized in that, The labeled quality frame is used for feedback optimization, including: Set a frame-level score threshold and set the frame-level score. Image frames smaller than the frame-level score threshold are marked and fed back into video frame generation; For frame-level scores For image frames with a frame-level score threshold, the number of iterations is increased and noise removal is enhanced.

21. A dual-stability video quality assessment system, characterized in that, include: The video segmentation unit is used to obtain an intermediate state video data sequence based on the model that is generating video, decompose the video data sequence into frames, and divide each frame of the video data sequence to obtain a segmentation patch result. The feature extraction unit is used to extract the image feature vector patchbedding from the block results of each frame using the pre-trained visual deep learning model ViT; The dual computation unit is used to perform dual distance calculations based on the image feature vector patchbedding to obtain the inter-block distance and inter-frame distance of the video data sequence; The quality score unit is used to calculate the average quality score of an image block by fusing the inter-block distance and the inter-frame distance; The aggregation optimization unit aggregates frame-level scores and video-level scores, and marks quality frames for feedback optimization. Frame-level scores are aggregated uniformly, while video-level scores are aggregated temporally weighted.

22. The system according to claim 21, characterized in that, The video segmentation unit is used to decompose the video data sequence into frames, including: The intermediate state video data sequence obtained in the video generation model is decomposed to obtain multiple video images; The multi-frame video images correspond to the intermediate state sequence arrangement of the video being generated.

23. The system according to claim 22, characterized in that, The video segmentation unit is used to divide each frame of the video data sequence to obtain a segmentation patch result, and further includes: The multi-frame video images obtained from the video data sequence are segmented into non-overlapping video blocks; Flatten each non-overlapping video block to obtain the corresponding image block.

24. An electronic device, characterized in that, include: The processor, communication interface, memory, and communication bus are connected, with the processor, communication interface, and memory communicating with each other via the communication bus. Memory, which stores computer programs; When the processor executes a program stored in memory, it implements the dual-stability video quality assessment method according to any one of claims 1 to 20.

25. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the dual-stability video quality assessment method according to any one of claims 1 to 20.

Images