A method, program, device, and storage medium for evaluating the quality of user-generated videos.

Abstract

This invention divides the complete video to be evaluated into multiple video segments of equal length. By calculating the proportion of static frames, global dynamic frames, and local dynamic frames in each video segment, the video segments are defined as static-dominated segments, object-motion-dominated segments, and camera-motion-dominated segments. Each type of segment is extracted and concatenated to form a corresponding segment set. After preprocessing, the segment set is input into a feature extraction network to obtain corresponding feature vectors. The feature vectors are then weighted and fused before being input into a regressor to obtain the quality score of the complete video to be evaluated. This invention, by classifying the motion features of video content, can more accurately match the motion patterns of UGC videos. It uses dedicated models for different motion types, effectively improving the accuracy of video feature extraction and quality regression. The weighted fusion strategy based on type proportions makes the overall quality assessment more consistent with visual perception patterns.

Description

Technical Field

[0001] This invention belongs to the field of video quality evaluation technology, specifically relating to a user-generated video quality evaluation method, program, device, and storage medium. Background Technology

[0002] With the booming development of mobile internet and short video platforms, user-generated content (UGC) videos have become the mainstream of online videos. Compared with traditional professionally produced videos, UGC videos originate from non-professional creators shooting in complex and ever-changing real-world environments, and their content movement patterns have a more significant diversity, such as static backgrounds, camera shake, and rapid movement of objects within the scene. This diversity poses a significant challenge to traditional Video Quality Assessment (VQA) methods when evaluating UGC videos.

[0003] Existing deep learning-based VQA methods, while adapted for UGC videos, lack consideration for the impact of diverse motion patterns on video quality assessment, and suffer from issues such as untargeted feature extraction and overly simplistic scoring strategies. Therefore, there is an urgent need for a VQA method capable of perceiving and adapting to the diverse motion patterns in UGC videos, thereby achieving more accurate quality assessments that better align with human subjective experience. Summary of the Invention

[0004] The purpose of this invention is to solve the problem that existing video quality evaluation methods are poorly adaptable to complex motion types in UGC videos, and to provide a user-generated video quality evaluation method, program, device and storage medium.

[0005] A method for evaluating the quality of user-generated videos includes the following steps:

[0006] The complete video to be evaluated is divided into video segments, and each video segment is a sequence of frames of the same length.

[0007] For each video segment, after grayscale processing, the average absolute difference of pixels between adjacent frames is calculated as the total inter-frame motion intensity. If the total inter-frame motion intensity between a frame and its previous frame is greater than a threshold, the frame is classified as a dynamic frame; otherwise, it is classified as a static frame. For dynamic frames, the feature point set between the dynamic frame and its previous frame is obtained through feature point matching. The feature point set is then processed using an affine transformation to obtain an affine transformation matrix. Based on the affine transformation matrix, the global motion intensity and local motion intensity of the dynamic frame are calculated. If the global motion intensity is greater than the local motion intensity, the dynamic frame is classified as a global dynamic frame; otherwise, it is classified as a local dynamic frame.

[0008] Calculate the proportion of static frames in a video clip. If the proportion of static frames is greater than a threshold, the video clip is classified as a static-dominated clip. Otherwise, count the number of global and local dynamic frames in the video clip. If the number of local dynamic frames is greater than the number of global dynamic frames, the video clip is classified as an object-movement-dominated clip. Otherwise, the video clip is classified as a camera-movement-dominated clip.

[0009] Video clips of the same category are concatenated end to end in chronological order to obtain sets of static dominant clips, camera motion dominant clips, and object motion dominant clips. After preprocessing the three sets of clips using different temporal sampling and spatial cropping strategies, they are input into a feature extraction network to obtain the corresponding feature vectors.

[0010] Based on the duration proportions of the static dominant segment set, the camera motion dominant segment set, and the object motion dominant segment set in the complete video to be evaluated, the feature vectors are weighted and fused to obtain the complete video fused feature vector; the complete video fused feature vector is then input into a regressor to obtain the score of the complete video to be evaluated.

[0011] Furthermore, for each video segment, after grayscale processing, the mean absolute difference between pixels in adjacent frames is calculated as the total inter-frame motion intensity.

[0012]

[0013] in, The first video clip Frame and the Total inter-frame motion intensity; and These represent the width and height of the image, respectively; P is the total number of pixels in the image. ; The first video clip In frame grayscale images Pixel value at the location;

[0014] like If it is greater than the threshold, then the first The frame is classified as a dynamic frame; otherwise, the first frame will be classified as a dynamic frame. The frame is classified as a static frame;

[0015] If the first The frame is a dynamic frame, obtained through feature point matching. Frame and the Feature point set of a frame and Affine transformation is used on the feature point set and After processing, the affine transformation matrix is ​​obtained. Based on affine transformation matrix Calculate global motion intensity With local motion intensity ;

[0016] ,

[0017] in, and Affine transformation matrix The last two elements in the last column;

[0018] If global motion intensity Greater than local motion intensity Then the first The frame is classified as a global dynamic frame; otherwise, the first frame is classified as a global dynamic frame. The frame is classified as a local dynamic frame.

[0019] Furthermore, by connecting all the static dominant segments in chronological order, we obtain a set of static dominant segments;

[0020] Randomly extract continuous segments from the static dominant fragment set Frames, to obtain a continuous sequence of frames. ;

[0021] Equal-interval sampling of the static dominant fragment set Frames, resulting in discrete frame sequences. ;

[0022] For consecutive frame sequences The center position of each frame is cropped to a size of [size missing]. Patch fragments;

[0023] For discrete frame sequences Uniform grid sampling is used to evenly divide each frame into... A grid, with random cropping within each grid. The patch, with consistent clipping positions within each grid across different frames, is then stitched together according to the grid positions to obtain... Patch fragments;

[0024] A continuous sequence of frames Patch fragments and discrete frame sequences The patch fragments are input into the feature extraction network to obtain the feature vectors of the static dominant fragment set. ;

[0025] in, Cut to the preset target size; As a preset odd number, ; .

[0026] Furthermore, by connecting all the camera motion-dominant segments in chronological order, a set of camera motion-dominant segments is obtained.

[0027] For the set of segments dominated by camera motion, perform two random, non-overlapping samplings, each sampling consecutively. Frames, to obtain a continuous sequence of frames. and ;

[0028] For consecutive frame sequences Non-uniform grid sampling is used, through a scaling factor. Each frame is divided into sections with denser centers and sparser edges. A grid, with random cropping within each grid. The patch, with consistent clipping positions within each grid across different frames, is then stitched together according to the grid positions to obtain... Patch fragments; for consecutive frame sequences Perform the same operation;

[0029] A continuous sequence of frames and The patch fragments are input into the feature extraction network to obtain the feature vectors of the set of camera motion-dominant fragments. .

[0030] Furthermore, by connecting all the dominant motion segments of the objects in chronological order, we obtain a set of dominant motion segments of the objects.

[0031] Random sampling is performed on the set of dominant motion segments of the object, and continuous sampling is performed. Frames, to obtain a continuous sequence of frames. ;

[0032] For consecutive frame sequences Non-uniform grid sampling is used, through a scaling factor. Each frame is divided into sections with denser centers and sparser edges. A grid, with random cropping within each grid. The patch, with consistent clipping positions within each grid across different frames, is then stitched together according to the grid positions to obtain... Patch fragments;

[0033] Analyze continuous frame sequences The inter-frame differences are analyzed to generate a comprehensive motion heatmap, the bounding boxes of the motion regions are located, and random cropping is performed within these bounding boxes to ensure that the moving subject is completely preserved in the output patch fragment, resulting in another set of... Patch fragments;

[0034] The two sets of patch fragments are input into the feature extraction network to obtain the feature vector of the set of object motion-dominant fragments. .

[0035] Furthermore, the non-uniform grid sampling method is used, through a scaling factor. Each frame is divided into sections with denser centers and sparser edges. The grid is as follows:

[0036]

[0037] in, , ;

[0038] Based on the proportional coefficient Construct a symmetrical proportional sequence with the smallest center. ;

[0039]

[0040]

[0041] in, ; Calculate the proportional sequence The sum of the elements , ;

[0042] Construct a sequence of segmentation points in the width direction Sequence of split points in the height direction ;

[0043]

[0044]

[0045] in, Represents a comparative sequence Center front Summing the elements of the terms; , indicating the starting point in the width and height directions; , , indicating the endpoints in the width and height directions;

[0046] Sequence of split points and A non-uniform mesh was uniquely defined.

[0047] Furthermore, the analysis involves continuous frame sequences. The inter-frame differences are analyzed to generate a comprehensive motion heatmap, the bounding boxes of the motion regions are located, and random cropping is performed within these bounding boxes to ensure that the moving subject is completely preserved in the output patch fragment, resulting in another set of... The patch fragment is as follows:

[0048] Calculate frame sequence Two consecutive frames and absolute difference plot between ;

[0049]

[0050] Use sensitivity-based adaptive threshold For absolute difference graph Perform binarization to generate a binary motion mask. ;

[0051]

[0052] in, The preset color value;

[0053] All binary motion masks are aggregated using an OR operation. , to obtain the frame sequence Integrated motion mask ;

[0054] From the integrated motion mask Extract the coordinate set of all non-zero pixels. Calculate the coordinate set minimum bounding rectangle ;

[0055] , , ,

[0056] For rectangle Expand the bounding box; the expanded bounding box is: ;

[0057] ,

[0058] ,

[0059] in, , , , ;

[0060] In the expanded bounding box Randomly select the starting point for cropping. :

[0061]

[0062]

[0063] in, , ; Indicates the interval Generate a random integer;

[0064] From frame sequence In all frames, with Starting from the point of origin, cut out a piece of size [size to be specified]. The region yields a set The patch fragment.

[0065] A computer device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the user-generated video quality evaluation method described above.

[0066] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the user-generated video quality evaluation method described above.

[0067] A computer program product includes computer instructions that, when executed by a processor, implement the steps of the user-generated video quality evaluation method described above.

[0068] The beneficial effects of this invention are as follows:

[0069] This invention classifies video content by motion features, which can more accurately match the actual motion patterns of UGC videos. It uses dedicated models for different motion types, which effectively improves the accuracy of video feature extraction and quality regression. Based on the type proportion, a weighted fusion strategy is implemented, which makes the overall quality assessment more in line with the laws of visual perception. This invention has strong applicability and can be widely used in various practical scenarios such as short video platforms, video surveillance, and mobile video communication. Attached Figure Description

[0070] Figure 1 This is the overall flowchart of the present invention.

[0071] Figure 2 This is a schematic diagram of video motion feature classification in an embodiment of the present invention.

[0072] Figure 3 This is a schematic diagram of the video quality evaluation method in an embodiment of the present invention. Detailed Implementation

[0073] The present invention will now be further described with reference to the accompanying drawings.

[0074] This invention designs a method for evaluating the quality of user-generated videos. Based on the motion characteristics of the videos, UGC videos are divided into three categories of video segments: static-dominated segments, camera-motion-dominated segments, and object-motion-dominated segments. Then, pre-trained feature extraction networks are used to extract multi-scale features from each of the three types of video segments—a spatiotemporal multi-scale feature extraction network is used for static-dominated segments, while dedicated networks are used for camera-motion and object-motion-dominated segments. Finally, the extracted features are weighted and fused according to the proportion of their corresponding video segments in the video, thereby achieving efficient feature representation and accurate quality evaluation of video segments with different motion scenes.

[0075] A method for evaluating the quality of user-generated videos includes the following steps:

[0076] Step 1: Divide the complete video to be evaluated into video segments, each video segment containing continuous segments. A frame sequence;

[0077] The segmentation process starts from the first frame of the video. If the last video segment has fewer than [a certain number of frames], [the process continues]. The frames are then filled using bilinear interpolation. frame;

[0078] Step 2: Categorize all video clips into static-dominated clips, camera-movement-dominated clips, and object-movement-dominated clips;

[0079] For each video segment, perform the following steps:

[0080] Step 2.1: After converting the video clip to grayscale, calculate the mean absolute difference between pixels in adjacent frames as the total motion intensity between frames;

[0081]

[0082] in, and These are the width and height of the image, respectively; P is the total number of pixels in the image. ; The first video clip In frame grayscale images Pixel value at the location;

[0083] If the first Frame and the Total inter-frame motion intensity Greater than the exercise intensity threshold Then the first The frame is classified as a dynamic frame; otherwise, the first frame will be classified as a dynamic frame. The frame is classified as a static frame;

[0084] Step 2.2: If the first If the frame is a dynamic frame, then the first frame is obtained through feature point matching. Frame and the Feature point set of a frame and Affine transformation is used on the feature point set and After processing, the affine transformation matrix is ​​obtained. ;

[0085] Based on affine transformation matrix Calculate global motion intensity With local motion intensity If the overall motion intensity Greater than local motion intensity Then the first The frame is classified as a global dynamic frame; otherwise, the first frame is classified as a global dynamic frame. The frame is classified as a local dynamic frame;

[0086] ,

[0087] in, and Affine transformation matrix The last two elements in the last column;

[0088] Step 2.3: Calculate the proportion of static frames in the video segment. If it is greater than the static proportion threshold, then classify the video segment as a static-dominant segment.

[0089] Otherwise, count the number of global and local dynamic frames in the video clip. If the number of local dynamic frames is greater than the number of global dynamic frames, classify the video clip as a segment dominated by object motion; otherwise, classify the video clip as a segment dominated by camera motion.

[0090] Step 3: Connect all static dominant segments in chronological order to obtain a set of static dominant segments;

[0091] Randomly extract continuous segments from the static dominant fragment set Frames, to obtain a continuous sequence of frames. ;

[0092] Equal-interval sampling of the static dominant fragment set Frames, resulting in discrete frame sequences. ;

[0093] For consecutive frame sequences The center position of each frame is cropped to a size of [size missing]. Patch fragments;

[0094] For discrete frame sequences Uniform grid sampling is used to evenly divide each frame into... A grid, with random cropping within each grid. The patch, with consistent clipping positions within each grid across different frames, is then stitched together according to the grid positions to obtain... Patch fragments;

[0095] A continuous sequence of frames Patch fragments and discrete frame sequences The patch fragments are input into the feature extraction network to obtain the feature vectors of the static dominant fragment set. ;

[0096] in, Cut to the preset target size; As a preset odd number, ; ;

[0097] Step 4: Connect all the camera motion-dominant clips in chronological order to obtain a set of camera motion-dominant clips;

[0098] For the set of segments dominated by camera motion, perform two random, non-overlapping samplings, each sampling consecutively. Frames, to obtain a continuous sequence of frames. and ;

[0099] For consecutive frame sequences Non-uniform grid sampling is used, through a scaling factor. Each frame is divided into sections with denser centers and sparser edges. A grid, with random cropping within each grid. The patch, with consistent clipping positions within each grid across different frames, is then stitched together according to the grid positions to obtain... Patch fragments; for consecutive frame sequences Perform the same operation;

[0100] A continuous sequence of frames and The patch fragments are input into the feature extraction network to obtain the feature vectors of the set of camera motion-dominant fragments. ;

[0101] Among them, non-uniform grid sampling is used, and the scaling factor is used. Each frame is divided into sections with denser centers and sparser edges. The grid is as follows:

[0102]

[0103] in, , ;

[0104] Based on the proportional coefficient Construct a symmetrical proportional sequence with the smallest center. ;

[0105]

[0106]

[0107] in, ; Calculate the proportional sequence The sum of the elements , ;

[0108] Construct a sequence of segmentation points in the width direction Sequence of split points in the height direction ;

[0109]

[0110]

[0111] in, Represents a comparative sequence Center front Summing the elements of the terms; , indicating the starting point in the width and height directions; , , indicating the endpoints in the width and height directions;

[0112] Sequence of split points and A non-uniform mesh was uniquely defined;

[0113] Step 5: Connect all the dominant motion segments of the objects in chronological order to obtain a set of dominant motion segments of the objects;

[0114] Random sampling is performed on the set of dominant motion segments of the object, and continuous sampling is performed. Frames, to obtain a continuous sequence of frames. ;

[0115] For consecutive frame sequences Non-uniform grid sampling is used, through a scaling factor. Each frame is divided into sections with denser centers and sparser edges. A grid, with random cropping within each grid. The patch, with consistent clipping positions within each grid across different frames, is then stitched together according to the grid positions to obtain... Patch fragments;

[0116] Analyze continuous frame sequences The inter-frame differences are analyzed to generate a comprehensive motion heatmap, the bounding boxes of the motion regions are located, and random cropping is performed within these bounding boxes to ensure that the moving subject is completely preserved in the output patch fragment, resulting in another set of... Patch fragments;

[0117] The two sets of patch fragments are input into the feature extraction network to obtain the feature vector of the set of object motion-dominant fragments. ;

[0118] Calculate frame sequence Two consecutive frames and absolute difference plot between ;

[0119]

[0120] Use sensitivity-based adaptive threshold For absolute difference graph Perform binarization to generate a binary motion mask. ;

[0121]

[0122] in, The preset color value;

[0123] All binary motion masks are aggregated using an OR operation. , to obtain the frame sequence Integrated motion mask ;

[0124] From the integrated motion mask Extract the coordinate set of all non-zero pixels. Calculate the coordinate set minimum bounding rectangle ;

[0125] , , ,

[0126] For rectangle Expand the bounding box; the expanded bounding box is: ;

[0127] ,

[0128] ,

[0129] in, , , , ;

[0130] In the expanded bounding box Randomly select the starting point for cropping. :

[0131]

[0132]

[0133] in, , ; Indicates the interval Generate a random integer;

[0134] From frame sequence In all frames, with Starting from the point of origin, cut out a piece of size [size to be specified]. The region yields a set Patch fragments;

[0135] Step 6: Based on the duration proportions of the static dominant segment set, the camera motion dominant segment set, and the object motion dominant segment set in the complete video to be evaluated, calculate the feature vector of the static dominant segment set. Feature vectors of the set of camera motion-dominant segments Feature vectors of the set of dominant motion segments of an object Weighted fusion is performed to obtain the complete video fusion feature vector. ;

[0136]

[0137]

[0138]

[0139]

[0140] in, The total duration of the complete video to be evaluated. The total duration of the static dominant segment set. The total duration of the set of segments dominated by camera motion. The total duration of the set of segments that dominate the motion of an object;

[0141] Step 7: Fuse the feature vectors from the complete video Input the regressor to obtain a score for the complete video to be evaluated. .

[0142] Example 1:

[0143] S1 divides the complete video to be evaluated into multiple parts consisting of continuous... A video clip composed of frames.

[0144] The video segments are generated by dividing the video into segments, and each video segment contains continuous segments. (In this embodiment) The frame sequence of the video. The segmentation process starts from the first frame of the video. If the last video segment has fewer than 10 frames... Frames, filled with bilinear interpolation Frames are processed to obtain multiple video clips.

[0145] S2, calculate the total inter-frame motion intensity of the video segment, estimate the global motion intensity and local motion intensity, and classify each video segment into a static-dominant segment, a camera motion-dominant segment or an object motion-dominant segment according to the relative relationship of motion intensity.

[0146] Calculate the mean absolute difference between pixels in adjacent frames of each video segment to obtain the total inter-frame motion intensity. Then, a preliminary judgment of motion frames is performed, using a binary decision process to filter out static frames and dynamic frames that require further refined motion analysis. The judgment criteria are as follows:

[0147]

[0148] in, It is a static frame. For dynamic frames, It is a preset motion intensity threshold (in this embodiment) (), used to distinguish between static frames and dynamic frames;

[0149] For dynamic frames, global motion caused by camera movement is estimated through feature point matching and an affine transformation model. Global motion intensity. From the estimated affine transformation matrix The derivation leads to:

[0150]

[0151]

[0152] in, It is The affine transformation matrix, and They represent in and Translation in the direction, and They are the first Frame and the The set of feature points matched in the frame It is a two-dimensional affine transformation function.

[0153] Local movement intensity The independent motion of objects within the scene is approximated by subtracting the global motion from the total motion:

[0154]

[0155] Total inter-frame motion intensity Mainly due to camera movement and the movement of objects in the scene It consists of two superimposed parts;

[0156] Within the dynamic frame range, fine-grained frame-level classification is performed based on the relative strength of global and local motion:

[0157]

[0158] in, For global dynamic frames, This is a local dynamic frame.

[0159] For video length is A video segment is a frame, and the type of each video segment is determined by the type statistics of all the frames within it.

[0160] First, calculate the video segment. static frame percentage :

[0161]

[0162] Then, based on the static percentage threshold (in this embodiment) Perform a preliminary classification, if the proportion of static frames... Greater than the static percentage threshold If the condition is met, the video segment is classified as a static dominant segment; otherwise, it is classified as a non-static dominant segment.

[0163] Finally, for non-static dominant segments, the final classification is based on the number of camera motion frames and object motion frames: count the number of camera motion frames and object motion frames in the video segment. If the number of object motion frames is greater than the number of camera motion frames, the video segment is classified as an object motion dominant segment; otherwise, it is classified as a camera motion dominant segment.

[0164] S3 preprocesses each type of video segment using different temporal sampling and spatial cropping strategies, and extracts features for each type of video segment using three feature extractors.

[0165] S3.1 For static dominant segments, two different frame sampling strategies are adopted to obtain local and global information of the video;

[0166] The set of static dominant segments is obtained by concatenating all static dominant segments of a single video in chronological order. Then, consecutive segments are randomly selected from this set. Frame (in this embodiment) ) to obtain the frame sequence To obtain local information; to perform equal-interval sampling on the static dominant fragment set, and to perform equal-interval sampling across the entire frame sequence. Frames are obtained as discrete frame sequences. To obtain global information;

[0167] For consecutive frame sequences The center position of each frame is cropped to a size of 1. Patch fragments; for discrete frame sequences Using uniform grid sampling, each frame is first evenly divided into... A grid, with random cropping within each grid. The patches are arranged so that the clipping position within each grid is consistent across different frames. Finally, the patches are stitched together according to their grid positions to obtain the final result. The patch fragment.

[0168] The two sets of patch fragments were then fed into a 3D Swin Transformer network for feature extraction to obtain the features of the static dominant fragment. .

[0169] S3.2, concatenate all camera motion-dominant segments of a single video in chronological order to obtain a set of camera motion-dominant segments. Perform two disjoint random cuts on this set to obtain two segments of length [missing information]. Frame sequence .

[0170] For segments dominated by camera motion, two non-overlapping samples are randomly performed with a sampling interval of 1, and each sample is taken consecutively. Frames, obtaining a continuous sequence of frames on the timeline. .

[0171] For frame sequences Non-uniform grid sampling is used. For a size of The frames are divided into grids using a scaling factor. Divide the image into sections that are densely packed in the center and sparsely packed at the edges. A grid, consistent with human image viewing habits, can better capture key information in an image:

[0172] In this embodiment, , This ensures that the non-uniformity of the grid can be adaptively adjusted according to the image size. When, the mesh degenerates into a uniform mesh; when At that time, the central grid shrinks more significantly, increasing non-uniformity.

[0173] Based on the proportional coefficient Construct a symmetrical proportional sequence with the smallest center. :

[0174]

[0175] The sequence contains One element, An odd number represents the number of rows or columns in the grid. In this embodiment... :

[0176]

[0177] Through the proportional sequence To obtain the segmentation point location, calculate the proportion sequence. The sum :

[0178]

[0179] Calculate the sequence of segmentation points in the width direction Sequence of split points in the height direction Sequence of split points and A non-uniform grid is uniquely defined, with random cropping within each grid. The patches are arranged so that the clipping position within each grid is consistent across different frames. Finally, the patches are stitched together according to their grid positions to obtain the final result. The patch fragment.

[0180] The two sets of patch fragments were then fed into a 3D Swin Transformer network for feature extraction, ultimately yielding the features of the camera motion-dominant fragment. .

[0181] S3.3, concatenate all motion-dominant segments of a single video in chronological order to obtain a set of motion-dominant segments. Random sampling with an interval of 2 is performed on this set to obtain a length of... Frame sequence .

[0182] For frame sequences Using the non-uniform grid sampling method in step S3.2, a set of samples was obtained. The patch segments are then analyzed. Next, by analyzing the inter-frame differences in the video sequence, a comprehensive motion heatmap is generated. Boundary boxes of motion regions are located, and random cropping is performed within these bounding boxes to ensure that the moving subject is completely preserved in the output patch segments, resulting in another set of... The patch fragment.

[0183] Calculate two consecutive frames in the input video and absolute difference plot Using sensitivity-based adaptive thresholds Binarize the difference image to generate a binary motion mask. :

[0184]

[0185]

[0186] in, This is the basic threshold parameter, in this embodiment ; It is a motion sensitivity parameter, in this embodiment , Represents pixel coordinates.

[0187] By iterating through all consecutive frame pairs in the video and aggregating their motion masks using an OR operation, the composite motion mask for the entire video segment is obtained. :

[0188]

[0189] From the integrated motion mask Extract the coordinate set of all non-zero pixels. :

[0190]

[0191] Calculate the minimum bounding rectangle of this coordinate set. :

[0192]

[0193] in, , , , ;

[0194] To ensure the output dimensions, the bounding box needs to be... Expand to define the width of the bounding box. and height :

[0195]

[0196]

[0197] if or Smaller than the target cutting size Then it expands symmetrically to both sides according to the following formula:

[0198]

[0199]

[0200] Expanded bounding box for:

[0201]

[0202]

[0203]

[0204]

[0205] In the expanded bounding box Randomly select the starting point for cropping. :

[0206]

[0207]

[0208] in, and These are the width and height of the expanded bounding box. Indicates the interval Generate a random integer.

[0209] Then, from all frames of the original video, Starting from the point of origin, cut out a piece of size [size to be specified]. The region is used to form the output patch fragment.

[0210] Finally, the two sets of patch fragments are input into a 3D Swin Transformer network for feature extraction to obtain the features of the segment that dominates the object's motion. .

[0211] S4, the features of the static dominant segment, camera motion dominant segment, and object motion dominant segment are weighted and fused according to the duration ratio of their respective video segments in the complete video to obtain the complete video fused feature vector. ;

[0212] The fused feature vector Input the regressor to obtain the complete video quality score. :

[0213]

[0214] in, This represents a trained regression model, which is a multilayer perceptron that maps high-dimensional features to a one-dimensional score.

[0215] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for evaluating the quality of user-generated videos, characterized in that: The complete video to be evaluated is divided into video segments, and each video segment is a sequence of frames of the same length. For each video segment, after grayscale processing, the average absolute difference of pixels between adjacent frames is calculated as the total inter-frame motion intensity. If the total inter-frame motion intensity between a frame and its previous frame is greater than the threshold, the frame is classified as a dynamic frame; otherwise, the frame is classified as a static frame. For a dynamic frame, the feature point set of the dynamic frame and its previous frame is obtained by feature point matching. The feature point set is processed by affine transformation to obtain the affine transformation matrix. Based on the affine transformation matrix, the global motion intensity and local motion intensity of the dynamic frame are calculated. If the global motion intensity is greater than the local motion intensity, the dynamic frame is classified as a global dynamic frame; otherwise, the dynamic frame is classified as a local dynamic frame. Calculate the proportion of static frames in a video clip. If the proportion of static frames is greater than a threshold, the video clip is classified as a static-dominated clip. Otherwise, count the number of global and local dynamic frames in the video clip. If the number of local dynamic frames is greater than the number of global dynamic frames, the video clip is classified as an object-movement-dominated clip. Otherwise, the video clip is classified as a camera-movement-dominated clip. Video clips of the same category are concatenated end to end in chronological order to obtain sets of static dominant clips, camera motion dominant clips, and object motion dominant clips. After preprocessing the three sets of clips using different temporal sampling and spatial cropping strategies, they are input into a feature extraction network to obtain the corresponding feature vectors. Based on the duration proportions of the static dominant segment set, the camera motion dominant segment set, and the object motion dominant segment set in the complete video to be evaluated, the feature vectors are weighted and fused to obtain the complete video fused feature vector. The feature vector of the complete video is fused and input into the regressor to obtain the quality score of the complete video to be evaluated.

2. The method for evaluating the quality of user-generated videos according to claim 1, characterized in that: For each video segment, after grayscale processing, the mean absolute difference between pixels in adjacent frames is calculated as the total inter-frame motion intensity. in, The first video clip Frame and the Total inter-frame motion intensity; and These represent the width and height of the image, respectively; P is the total number of pixels in the image. ; The first video clip In frame grayscale images Pixel value at the location; like If it is greater than the threshold, then the first The frame is classified as a dynamic frame; otherwise, the first frame will be classified as a dynamic frame. The frame is classified as a static frame; If the first The frame is a dynamic frame, obtained through feature point matching. Frame and the Feature point set of a frame and Affine transformation is used on the feature point set and After processing, the affine transformation matrix is ​​obtained. Based on affine transformation matrix Calculate global motion intensity With local motion intensity ; ， in, and Affine transformation matrix The last two elements in the last column; If global motion intensity Greater than local motion intensity Then the first The frame is classified as a global dynamic frame; otherwise, the first frame is classified as a global dynamic frame. The frame is classified as a local dynamic frame.

3. The method for evaluating the quality of user-generated videos according to claim 1, characterized in that: By concatenating all the static dominant segments in chronological order, we obtain a set of static dominant segments. Randomly extract continuous segments from the static dominant fragment set Frames, to obtain a continuous sequence of frames. ; Equal-interval sampling of the static dominant fragment set Frames, resulting in discrete frame sequences. ; For consecutive frame sequences The center position of each frame is cropped to a size of [size missing]. Patch fragments; For discrete frame sequences Uniform grid sampling is used to evenly divide each frame into... A grid, with random cropping within each grid. The patch, with consistent clipping positions within each grid across different frames, is then stitched together according to the grid positions to obtain... Patch fragments; A continuous sequence of frames Patch fragments and discrete frame sequences The patch fragments are input into the feature extraction network to obtain the feature vectors of the static dominant fragment set. ; in, Cut to the preset target size; As a preset odd number, ; .

4. The method for evaluating the quality of user-generated videos according to claim 3, characterized in that: By connecting all the camera motion-dominant segments in chronological order, we obtain a set of camera motion-dominant segments. For the set of segments dominated by camera motion, perform two random, non-overlapping samplings, each sampling consecutively. Frames, to obtain a continuous sequence of frames. and ; For consecutive frame sequences Non-uniform grid sampling is used, through a scaling factor. Each frame is divided into sections with denser centers and sparser edges. A grid, with random cropping within each grid. The patch, with consistent clipping positions within each grid across different frames, is then stitched together according to the grid positions to obtain... Patch fragments; for consecutive frame sequences Perform the same operation; A continuous sequence of frames and The patch fragments are input into the feature extraction network to obtain the feature vectors of the set of camera motion-dominant fragments. .

5. The method for evaluating the quality of user-generated videos according to claim 4, characterized in that: By connecting all the dominant motion segments of the objects in chronological order, we obtain a set of dominant motion segments of the objects. Random sampling is performed on the set of dominant motion segments of the object, and continuous sampling is performed. Frames, to obtain a continuous sequence of frames. ; For consecutive frame sequences Non-uniform grid sampling is used, through a scaling factor. Each frame is divided into sections with denser centers and sparser edges. A grid, with random cropping within each grid. The patch, with consistent clipping positions within each grid across different frames, is then stitched together according to the grid positions to obtain... Patch fragments; Analyze continuous frame sequences The inter-frame differences are analyzed to generate a comprehensive motion heatmap, the bounding boxes of the motion regions are located, and random cropping is performed within these bounding boxes to ensure that the moving subject is completely preserved in the output patch fragment, resulting in another set of... Patch fragments; The two sets of patch fragments are input into the feature extraction network to obtain the feature vector of the set of object motion-dominant fragments. .

6. The method for evaluating the quality of user-generated videos according to claim 5, characterized in that: The non-uniform grid sampling method is used, and a scaling factor is applied. Each frame is divided into sections with denser centers and sparser edges. The grid is as follows: in, , ; Based on the proportional coefficient Construct a symmetrical proportional sequence with the smallest center. ; in, ; Calculate the proportional sequence The sum of the elements , ; Construct a sequence of segmentation points in the width direction Sequence of split points in the height direction ; in, Represents a comparative sequence Center front Summing the elements of the terms; , indicating the starting point in the width and height directions; , , indicating the endpoints in the width and height directions; Sequence of split points and A non-uniform mesh was uniquely defined.

7. The method for evaluating the quality of user-generated videos according to claim 5, characterized in that: The analysis is a continuous frame sequence The inter-frame differences are analyzed to generate a comprehensive motion heatmap, the bounding boxes of the motion regions are located, and random cropping is performed within these bounding boxes to ensure that the moving subject is completely preserved in the output patch fragment, resulting in another set of... The patch fragment is as follows: Calculate frame sequence Two consecutive frames and absolute difference plot between ; Use sensitivity-based adaptive threshold For absolute difference graph Perform binarization to generate a binary motion mask. ; in, The preset color value; All binary motion masks are aggregated using the "OR" operation. , to obtain the frame sequence Integrated motion mask ; From the integrated motion mask Extract the coordinate set of all non-zero pixels. Calculate the coordinate set minimum bounding rectangle ; ， ， ， For rectangle Expand the bounding box; the expanded bounding box is: ; ， ， in, , , , ; In the expanded bounding box Randomly select the starting point for cropping. : in, , ; Indicates the interval Generate a random integer; From frame sequence In all frames, with Starting from the point, cut out a piece of size [size missing]. The region yields a set The patch fragment.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that: When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 1 to 7.

Images