UGC video quality evaluation method and system based on spatial-temporal characteristics

Abstract

The invention discloses a UGC video quality evaluation method and system based on spatial-temporal characteristics, and the method comprises the following steps: obtaining a to-be-evaluated UGC video, and carrying out the video partitioning; performing spatial-temporal feature extraction on each video block to obtain corresponding time features and spatial features; performing weighted fusion and linear splicing on the time features and the space features by adopting an attention mechanism to obtain space-time cross fusion features of each video block; adopting a technical regression branch, an aesthetic regression branch and a semantic regression branch to carry out regression processing on the space-time cross fusion features to obtain quality scores corresponding to the three branches; and performing weighted summation processing on the three quality scores of all the video blocks to obtain an evaluation score of the UGC video. According to the invention, video feature extraction and feature cross fusion are carried out based on space and time dimensions; and three different quality regression branches are introduced, UGC video quality scores are regressed from different aspects of technology, aesthetics and semantics, and the video quality is evaluated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video quality evaluation, and more particularly to a UGC video quality evaluation method and system based on space-time features. BACKGROUND

[0002] In recent years, the improvement of Internet speed, the optimization of coding technology and the rapid growth of the number of smartphone users have jointly contributed to the exponential growth of user-generated video content and consumption.

[0003] User-generated content (UGC) video is a video content created, shot and shared by ordinary users on network platforms. With its unique authenticity and diversity, UGC video has been warmly welcomed by the general public. UGC video has become an important part of the digital media field, and watching, shooting and sharing UGC video has become one of people's daily activities.

[0004] Video quality assessment (VQA) predicts the subjective feedback of human watching video by training a model to automatically evaluate the quality of input video. It has been widely studied in evaluating compression distortion, transmission error and overall quality of video picture. Relatively poor video quality often leads to user dissatisfaction, reduced participation and negative user experience.

[0005] The emergence and development of UGC video have brought new viewing experience to users, but also pose new challenges to video service providers. Although the general video quality evaluation model has shown its effectiveness in various scenarios, it faces more complex problems when applied to UGC video. First, the content of UGC video is very rich, from natural scenes to animations, games, screen contents, etc. The features of different video contents differ greatly. Second, since UGC videos are mostly shot by non-professional users, various distortions are often introduced during the video shooting process. Different distortions are intertwined and cannot be exhaustively enumerated all distortion-related features to evaluate video quality. At the same time, there is a complex interaction between video content and video distortion, which brings great challenges to the UGC video quality evaluation task.

[0006] With the continuous development of the video content industry, high-quality UGC video content has become the key to attracting users and traffic. However, the complex interactions between people's perception of videos, different video content and video distortion have brought great challenges to the UGC video quality evaluation task. Poor quality UGC videos not only affect the visual effect of the video, but also may have a negative impact on the emotional experience of the audience. With the rapid growth of UGC videos, the problem of uneven video quality is increasingly prominent, which not only affects the user's viewing experience, but also poses a challenge to the entire video quality ecosystem and healthy development.

[0007] Therefore, it is an urgent problem for those skilled in the art to provide a scientific, objective and accurate UGC video quality evaluation method. SUMMARY

[0008] Therefore, the present application provides a UGC video quality evaluation method and system based on spatio-temporal features. Through the technical solutions of feature extraction, spatio-temporal feature cross fusion and quality score regression, the problem of inaccurate UGC video quality evaluation in the prior art is solved.

[0009] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0010] In a first aspect, the present application provides a UGC video quality evaluation method based on spatio-temporal features, comprising the following steps:

[0011] S1, obtaining a UGC video to be evaluated, and performing video blocking; performing spatio-temporal feature extraction on each video block to obtain corresponding temporal features and spatial features;

[0012] S2, using an attention mechanism to perform weighted fusion and linear splicing on the temporal features and spatial features to obtain spatio-temporal cross fusion features of each video block;

[0013] S3, using a technical regression branch, an aesthetic regression branch and a semantic regression branch to perform regression processing on the spatio-temporal cross fusion features to obtain quality scores corresponding to the three branches;

[0014] S4, performing weighted summation processing on the three quality scores of all video blocks to obtain an evaluation score of the UGC video.

[0015] Further, step S1 specifically comprises:

[0016] S11, obtaining a UGC video to be evaluated, and performing video blocking based on the number of video frames and the video frame rate of the UGC video;

[0017] S12, performing spatial feature extraction on each video block by a Conformer model to obtain multi-layer spatial feature representation;

[0018] S13, select multi-scale features from the multi-layer spatial feature representation to splice and integrate to obtain corresponding spatial features, with the same number of layers being spaced apart;

[0019] S14, after the video frames of each video block are processed by downsampling, time features are extracted by SlowFast to obtain corresponding time features.

[0020] Further, step S12 specifically comprises:

[0021] spatial feature extraction is performed on each video block by the Conformer model to obtain multi-layer feature maps and feature vectors;

[0022] The global average and standard deviation of each layer of the feature map are calculated, and each layer of the feature vector is cropped to obtain the corresponding class label;

[0023] The global average, standard deviation and class label of each layer are spliced to obtain a multi-layer spatial feature representation.

[0024] Further, step S2 specifically comprises:

[0025] S21, align the dimensions of the time features and the spatial features;

[0026] S22, generate a query using the aligned spatial features and generate a key and a value using the aligned time features; calculate and weight fuse the spatial attention to obtain the spatial features fused with the time features;

[0027] S23, generate a query using the aligned time features and generate a key and a value using the aligned spatial features; calculate and weight fuse the time attention to obtain the time features fused with the spatial features;

[0028] S24, splice the spatial features and the time features after the fusion processing, and regress to the initial dimension through a linear layer to obtain the spatio-temporal cross-fusion features.

[0029] Further, the calculation and weight fusion of the spatial or time attention specifically comprises:

[0030] The similarity between each query and the key is calculated by dot product, and a weight matrix is generated by applying Softmax; the weight matrix is multiplied by the value to obtain the corresponding fusion result.

[0031] Further, step S3 specifically comprises:

[0032] S31, adopting a technology regression branch, performing technology regression processing on the spatio-temporal cross-fusion feature to obtain a technology branch score; the technology regression branch includes two linear layers, an activation function, and a score interval mapping;

[0033] The formula is:

[0034]

[0035] wherein QT represents the technology branch score, represents the spatio-temporal cross-fusion feature of the C i th video block, and M represents the score mapping interval [0, M];

[0036] S32, adopting an aesthetic regression branch, performing aesthetic regression processing on the spatio-temporal cross-fusion feature to obtain an aesthetic branch score; the aesthetic regression branch includes a multi-layer perception machine (MLP), an activation function, and a score interval mapping;

[0037] The formula is:

[0038]

[0039] wherein QA represents the aesthetic branch score;

[0040] S33, adopting a semantic regression branch, performing semantic regression processing on the spatio-temporal cross-fusion feature to obtain a semantic branch score; the semantic regression branch includes a CLIP text encoder, an activation function, and a score interval mapping.

[0041] Further, step S33 specifically includes:

[0042] S331, using a pair of antonyms as prompts to obtain features t1 and t2 of the two text prompts through the CLIP text encoder;

[0043] S332, performing cosine similarity calculation on the features of the two text prompts and the spatio-temporal cross-fusion feature respectively to obtain two similarity scores st1 and st2;

[0044] The formula is:

[0045]

[0046] S333, based on the similarity scores, using a Softmax function to calculate the probability that the video belongs to high quality, and performing activation function and score interval mapping to obtain the semantic branch score;

[0047] The formula is:

[0048]

[0049] wherein QS represents a semantic branch score.

[0050] Further, the step S4 specifically comprises:

[0051] S41, performing weighted summation on the three quality scores corresponding to each video block to obtain a quality score of each video block;

[0052] which is expressed by a formula as:

[0053]

[0054] wherein a, β and γ respectively represent weights of the technical regression branch, the aesthetic regression branch and the semantic regression branch, represents a quality score of the i-th video block; i

[0055] S42, performing averaging on the quality scores of all video blocks to obtain an evaluation score of the complete video;

[0056] which is expressed by a formula as:

[0057]

[0058] wherein C represents the number of video blocks, i represents the i-th video block, and Q represents the evaluation score of the complete video.

[0059] In a second aspect, the present application provides a UGC video quality evaluation system based on spatio-temporal features, which adopts the UGC video quality evaluation method based on spatio-temporal features according to any one of the first aspect, and comprises the following modules:

[0060] a spatio-temporal feature extraction module: obtaining a UGC video to be evaluated, performing video blocking; performing spatio-temporal feature extraction on each video block to obtain corresponding time features and space features;

[0061] a spatio-temporal feature fusion module: performing weighted fusion and linear splicing on the time features and space features by using an attention mechanism to obtain spatio-temporal cross fusion features of each video block;

[0062] a quality score regression module: performing regression processing on the spatio-temporal cross fusion features by using a technical regression branch, an aesthetic regression branch and a semantic regression branch to obtain quality scores corresponding to the three branches;

[0063] an evaluation score output module: performing weighted summation processing on the three quality scores of all video blocks to obtain an evaluation score of the UGC video.

[0064] ​Compared with the prior art, the UGC video quality evaluation method and system based on space-time features provided by the technical solution has the following beneficial effects:

[0065] 1) The present application is aimed at the problems of UGC video content diversity and distortion type complexity. In order to reduce the dependence of the quality prediction model on the distortion type or video content, the present application directly extracts the quality perception features of the video from two different dimensions of space and time. The spatial quality perception features reflect the detail level and richness of visual information of the video, and the temporal quality perception features reflect the fluency of motion, frame rate stability and continuity of objects or scenes over time. The video features are extracted in multiple dimensions to enrich the feature representation of the model.

[0066] 2) The present application fuses features of different dimensions through a space-time cross-fusion module, helping the model to consider information of spatial and temporal dimensions simultaneously in the feature extraction process, capturing the relevance between different dimensions in the video and improving the accuracy and robustness of the quality evaluation task.

[0067] 3) The present application introduces three different quality regression branches to regress the UGC video quality score from different aspects of technology, aesthetics and semantics, avoiding the one-sidedness of single-dimensional evaluation. At the same time, different weight selection mechanisms are used to let the model adaptively adjust the weights of different branches in the learning process, more comprehensively reflecting the quality of UGC content. BRIEF DESCRIPTION OF DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0069] Figure 1 A flowchart of the UGC video quality evaluation method based on space-time features provided by the embodiment of the present application.

[0070] Figure 2 A framework diagram of the space-time feature extraction stage provided by the embodiment of the present application.

[0071] Figure 3 A framework diagram of the space-time feature cross-fusion stage provided by the embodiment of the present application.

[0072] Figure 4 A framework diagram of the quality score regression stage provided by the embodiment of the present application.

[0073] Figure 5A UGC video quality evaluation system structure diagram based on space-time features is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0074] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application.

[0075] The technical regression branch focuses on the jitter, noise, compression and other distortions occurring in the UGC video shooting or transmission process. The aesthetic regression branch focuses on the abstract aesthetic concepts of video content, composition, color and the like. The semantic regression branch simulates the habit of human subjective evaluation of video quality, and evaluates the UGC video quality from the perspective of video understanding by constructing an antonym pair. The quality scores output by the three branches are fused by weighting to obtain a video quality score, avoiding the one-sidedness of single-dimensional evaluation, and more comprehensively reflecting the quality of the UGC video.

[0076] Embodiment 1

[0077] The embodiment of the present application discloses a UGC video quality evaluation method based on space-time features, as shown in Figure 1 The method comprises the following steps:

[0078] S1, acquiring a UGC video to be evaluated, and performing video blocking; performing space-time feature extraction on each video block to obtain corresponding time features and space features;

[0079] S2, using an attention mechanism to perform weighted fusion and linear splicing on the time features and the space features to obtain space-time cross fusion features of each video block;

[0080] S3, using a technical regression branch, an aesthetic regression branch and a semantic regression branch to perform regression processing on the space-time cross fusion features to obtain quality scores corresponding to the three branches;

[0081] S4, performing weighted summation processing on the three quality scores of all video blocks to obtain an evaluation score of the UGC video.

[0082] The embodiment of the present invention is applied in the quality monitoring and optimization management of the video content platform, automatically evaluating the quality of UGC videos on the platform, identifying low-quality content; providing quality feedback and improvement suggestions for content creators; improving the quality of platform content and enhancing user experience. This embodiment first obtains the UGC video to be evaluated on the platform, enters the spatiotemporal feature extraction stage, performs video block processing, and extracts quality perception features from two different dimensions of space and time, respectively, and enhances the feature expression ability of the model by using different types of features; obtains the time features and spatial features of each video block. Afterwards, it enters the spatiotemporal feature cross-fusion stage to obtain the spatiotemporal cross-fusion features of each video block; finally, it enters the quality score regression stage, regresses the fused spatiotemporal features into scores, and obtains the evaluation score of the UGC video to be evaluated on the platform. This method can accurately evaluate the quality scores of different types of UGC videos, so that they are in line with human subjective opinions to the greatest extent, thereby providing valuable feedback and guidance to video creators and platforms.

[0083] The implementation steps of this embodiment are described in detail below.

[0084] In this embodiment, UGC videos shot by users are obtained, and their contents include but are not limited to natural scenes, animations, games, screen contents, etc., and may also introduce various distortions.

[0085] The first step is to refer to Figure 2 As shown, we enter the spatiotemporal feature extraction stage.

[0086] This embodiment first divides the input UGC video into blocks and extracts the temporal features and spatial features of the video using different feature extractors.

[0087] 1.1、Video segmentation.

[0088] This embodiment reads a UGC video to be evaluated, with a resolution of 1280×720 pixels, a frame rate of 30 fps, and 240 frames. The video is divided into C video blocks, which are expressed as follows:

[0089]

[0090] Wherein, N is the number of video frames, and t is the video frame rate; in this embodiment, N=240, t=30, and C=8.

[0091] 1.2. Spatial feature extraction.

[0092] This embodiment reads C i video blocks, where i is an integer from 1 to 8 in this embodiment.

[0093] For spatial features, there is a lot of similar spatial information between adjacent frames. In order to reduce the amount of calculation, the spatial features of the same frame are extracted for each video block to represent the spatial information of the entire video block.

[0094] The first frame is taken in this embodiment, and the first frame is taken for each video block. The first frame is taken in this embodiment, and the first frame is taken for each video block. The 448x448 pixel size picture is extracted by the Conformer model to extract the spatial features, and the feature map f k and the feature vector T k of the kth layer of the Conformer model are obtained.

[0095] The Conformer model used in this example is a hybrid network architecture that combines the advantages of CNN and Transformer, which can capture both local detailed features and global context dependencies of input data.

[0096] For the feature map f k of the kth layer of this embodiment, the global average value and the standard deviation of the kth layer feature vector T k are calculated. The three are spliced together as the spatial features of the kth layer

[0097]

[0098]

[0099] wherein f k represents the kth layer feature map, and represent the global average value and the standard deviation of the feature map f k , respectively, T k represents the kth layer feature vector, and represents the category label of the feature vector.

[0100] ​This embodiment then selects the 4th, 8th, and 12th spatial features extracted by the Conformer and combines them as multi-scale features for multi-scale feature fusion. By integrating features at different scales, this embodiment captures high-level global semantic features while retaining low-level local texture details. Furthermore, by selecting features at intervals of four layers, a multi-scale pyramid can be formed covering 4×, 16×, and 64× downsampling ratios, balancing information richness and model efficiency.

[0101] The spatial features of these three layers are spliced ​​together to obtain the C i Spatial features of video blocks The formula is:

[0102]

[0103] in, Represent the spatial features of the 4th, 8th, and 12th layers of Conformer respectively; Indicates C i Spatial features of video blocks.

[0104] 1.3. Temporal feature extraction.

[0105] Temporal features reflect the coherence of motion between frames. To reduce the amount of computation, all video frames in the video block are downsampled to 224×224 pixels, and temporal features are extracted using SlowFast.

[0106] The SlowFast model used in this example implements multi-scale feature modeling in the time dimension through a dual-path network structure.

[0107] This embodiment reads C i Video blocks, all the video frames in the video block are extracted with SlowFast to obtain the temporal features of the Cth video block. i Temporal features of video chunks

[0108] Step 2: Refer to Figure 3 As shown, we enter the stage of cross-fusion of spatiotemporal features.

[0109] 2.1. Align the spatiotemporal feature dimensions; Align the spatial features obtained in the first step and time characteristics Align to the same dimension dim, in this example dim = 768, and use it for subsequent attention calculations.

[0110] 2.2. Perform spatial attention calculation.

[0111] First, perform spatial feature query and generate temporal feature keys and values. Generate query Q through linear layer s , time feature Generate key K and value V through different linear layers t t Then, the calculation and weighting of the spatial attention score are performed.

[0112] In this embodiment, the similarity between each Q s and K t is calculated through dot product, and a weight matrix Attention s is generated by applying Softmax. Attention s is multiplied by V t to obtain spatial feature fused with time feature The formula is expressed as:

[0113]

[0114] Among them, Attention s represents the weight matrix, Q s represents the spatial feature generated through the linear layer, K t and V t represent the key and value of the time feature generated through different linear layers, respectively, represents the scaling factor, represents the processed spatial feature.

[0115] 2.3, time attention calculation is performed.

[0116] First, the generation of time feature query, spatial feature key and value is performed. The time feature is generated through a linear layer to generate query Q t ; the spatial feature is generated through a linear layer to generate K s and value V s . Then, the calculation and weighting of the time attention score are performed.

[0117] In this embodiment, the similarity between each Q t and K s is calculated through dot product, and a weight matrix Attention t is generated by applying Softmax; Attention t is multiplied by V s to obtain time feature fused with spatial feature The formula is expressed as:

[0118]

[0119] Among them, Attention t ​represents the weight matrix, Q t represents the query generated by the time feature through the linear layer, K s and V s Represents the keys and values ​​generated by spatial features through different linear layers, represents the scaling factor, Represents the time characteristics after processing.

[0120] 2.4. Concatenate the spatial and temporal features calculated by attention.

[0121] The spatial features and temporal features after attention processing are spliced ​​together and regressed to the initial dimension through the linear layer to obtain the cross-fused spatiotemporal features. The formula is:

[0122]

[0123] in, and Represent the processed spatial features and temporal features respectively, Represents the spatiotemporal features after cross-fusion.

[0124] Step 3: Refer to Figure 4 As shown, the quality score regression stage begins.

[0125] The quality regression stage is used to regress the fused spatiotemporal features into fractions, which includes the following steps:

[0126] 3.1. Perform score regression on each video block.

[0127] The technical regression branch, the aesthetic regression branch, and the semantic regression branch are used to regress the spatiotemporal fusion features of each video block, obtaining three different quality scores. These three scores are then weighted to obtain the quality score of each video block.

[0128] In this embodiment, the implementation steps of the technical regression branch are as follows:

[0129] The spatiotemporal cross-fusion feature of this embodiment After two different linear layers and sigmoid, the score is mapped to the interval [0, M]. In this embodiment, M = 100, so the mapping interval is [0, 100]. The technical branch score QT is obtained, which is expressed as follows:

[0130]

[0131] Linear mid Regress the spatiotemporal features from 768 dimensions to 128 dimensions, Linear out Regress the features from 128 dimensions to 1.

[0132] wherein, represents the spatio-temporal cross-fusion feature, and QT represents the technical branch score.

[0133] In this embodiment, the implementation steps of the aesthetic regression branch are as follows:

[0134] The spatio-temporal fusion feature of this embodiment After passing through a multi-layer perception machine (MLP), the score is mapped to the interval [0, 100] after sigmoid, and the aesthetic branch score QA is obtained, which is expressed by the formula:

[0135]

[0136] wherein, represents the spatio-temporal cross-fusion feature, and QA represents the aesthetic branch score.

[0137] This example uses a multi-layer perception machine (MLP) to implement the quality evaluation function of the aesthetic branch. The MLP is initialized by the weight parameters obtained by training the aesthetic evaluation data set, so that the branch can provide effective aesthetic quality scoring ability.

[0138] In this embodiment, the implementation steps of the semantic regression branch are as follows:

[0139] First, feature extraction is performed through text prompt words; this embodiment introduces an opposite prompt pairing strategy, using paired opposite words prompts, such as "a high-quality video", "a low-quality video", or "a clear video", "a fuzzy video", etc.

[0140] Through paired opposite word prompts, this embodiment can not only evaluate the quality of the entire video, but also score specific dimensions of the video.

[0141] Then, let the opposite word pair get the features t1 and t2 of the two text prompts through the CLIP text encoder.

[0142] After that, the two text features are respectively calculated with the spatio-temporal fusion feature to obtain the similarity scores st1 and st2, which are expressed by the formula:

[0143]

[0144] wherein, represents the spatio-temporal cross-fusion feature, and t1 and t2 represent two text feature values, and st1 and st2 represent the values after cosine similarity calculation of the two text features and the spatio-temporal fusion feature.

[0145] Finally, the probability of the video belonging to the positive class, i.e. belonging to "a high-quality video", is calculated using Softmax; the semantic score QS is obtained by passing the score through a sigmoid function and mapping it to the [0, 100] interval, which is expressed by the formula:

[0146]

[0147] wherein QS represents the semantic branch score.

[0148] 3.2, Calculate the score of each video block.

[0149] In this embodiment, for the obtained scores of the three branches QT, QA and QS, three different weights a, b and g are set, and the sum of the three weights is 1, so that the model automatically adjusts the weight values of different branches in the learning process. The optimal values in this example are: a = 0.3987, b = 0.3066, g = 0.2947, and the final score of the video block is which is expressed by the formula:

[0150]

[0151] wherein QT, QA and QS represent the technical branch score, the aesthetic branch score and the semantic branch score, respectively, and a, b and g represent the weights of the technical branch, the aesthetic branch and the semantic branch, respectively, QCi represents the quality score of the C i th video block.

[0152] 3.3, Calculate the complete video score. The quality score Q of the entire video is obtained by averaging the scores of all video blocks. The score interval of Q is [0, 100], and the higher the score, the better the quality of the video.

[0153] The calculation formula is:

[0154]

[0155] wherein C represents the number of video blocks, i represents the i th video block, i QCi represents the quality score of the C th video block, and Q represents the final score of the entire video.

[0156] This method addresses the diverse content and complex distortion types of UGC videos. To reduce the quality prediction model's dependence on distortion type or video content, it directly extracts quality-perceived features from two distinct dimensions: spatial and temporal. A spatiotemporal cross-fusion module then fuses features from different dimensions, enabling the model to simultaneously consider spatial and temporal information during feature extraction and capture the correlations between different dimensions in the video. Finally, three distinct quality regression branches are introduced to regress UGC video quality scores from technical, aesthetic, and semantic perspectives, providing a more comprehensive reflection of the quality of UGC content.

[0157] Example 2

[0158] The embodiment of the present invention discloses a UGC video quality evaluation system based on spatiotemporal features, referring to Figure 5 As shown, a UGC video quality evaluation method based on spatiotemporal features as described in any one of Embodiment 1 is adopted, including the following modules:

[0159] Spatiotemporal feature extraction module: obtains the UGC video to be evaluated and divides the video into blocks; extracts spatiotemporal features from each video block to obtain the corresponding temporal and spatial features;

[0160] Spatiotemporal feature fusion module: uses the attention mechanism to perform weighted fusion and linear splicing of temporal and spatial features to obtain spatiotemporal cross-fusion features of each video block;

[0161] Quality score regression module: This module uses the technical regression branch, aesthetic regression branch, and semantic regression branch to regress the spatiotemporal cross-fusion features and obtain the corresponding quality scores of the three branches.

[0162] Evaluation score output module: Perform weighted summation on the three quality scores of all video blocks to obtain the evaluation score of the UGC video.

[0163] When evaluating UGC video quality, the system of this embodiment first uses the spatiotemporal feature extraction module to segment the input video and extract its temporal and spatial features. The spatiotemporal feature fusion module then fuses these extracted features. Finally, the quality score regression module regresses the quality scores of the video blocks based on three branches: technical, aesthetic, and semantic. The quality scores of the video blocks are then calculated through a weighted summation. Finally, the evaluation score output module averages the quality scores of the video blocks to obtain the overall video quality score.

[0164] The system of the present invention serves as an evaluation tool to help content creators perform quality checks before video release, and can also provide quality monitoring and optimization suggestions for video platforms.

[0165] Through the application, the overall quality of the UGC video of the video content platform can be effectively improved, and a more rich watching experience can be provided for users.

[0166] The various embodiments are described in the specification by way of progression, each building on the last to facilitate ease of understanding. The same or similar reference numerals are used in the drawings and description to refer to the same or like parts, components and operations throughout the various embodiments. Numbered embodiments are described with emphasis on the differences from the previously described embodiments.

[0167] The above description of disclosed embodiments provides enough information to enable one of ordinary skill in the art to practice or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Accordingly, the application is not to be restricted based on the specific embodiments shown and described, as defined in the appended claims and their equivalents.

Claims

1. A UGC video quality evaluation method based on spatiotemporal features, characterized in that: The following steps are involved: S1. Obtain the UGC video to be evaluated and divide it into blocks; extract spatiotemporal features of each video block to obtain corresponding temporal features and spatial features; S2. Use the attention mechanism to perform weighted fusion and linear splicing on the temporal features and spatial features to obtain the spatiotemporal cross-fusion features of each video block; S3, using the technical regression branch, the aesthetic regression branch, and the semantic regression branch to regress the spatiotemporal cross-fusion features to obtain quality scores corresponding to the three branches; S4. Perform weighted summation on the three quality scores of all video blocks to obtain an evaluation score of the UGC video.

2. A UGC video quality evaluation method based on spatiotemporal features as claimed in claim 1, characterized in that: Step S1 specifically includes: S11, obtaining a UGC video to be evaluated, and dividing the video into blocks based on the number of video frames and the video frame rate of the UGC video; S12, extract spatial features from each video block through the Conformer model to obtain a multi-layer spatial feature representation; S13, selecting multi-scale features from the multi-layer spatial feature representation at intervals of the same number of layers, and performing concatenation and integration to obtain corresponding spatial features; S14. After downsampling the video frames of each video block, extract the temporal features through SlowFast to obtain corresponding temporal features.

3. A UGC video quality evaluation method based on spatiotemporal features as claimed in claim 2, characterized in that: Step S12 specifically includes: The spatial features of each video block are extracted through the Conformer model to obtain multi-layer feature maps and feature vectors; Calculate the global mean and standard deviation of the feature map of each layer, and clip the feature vector of each layer to obtain the corresponding category label; The global mean, standard deviation and category label of each layer are concatenated to obtain a multi-layer spatial feature representation.

4. The UGC video quality evaluation method based on spatiotemporal features according to claim 1, wherein: Step S2 specifically includes: S21, dimensional alignment of the temporal features and the spatial features; S22. Generate a query using the aligned spatial features and generate keys and values ​​using the aligned temporal features; perform spatial attention calculation and weighted fusion to obtain spatial features fused with temporal features; S23. Generate a query using the aligned temporal features and generate keys and values ​​using the aligned spatial features; perform temporal attention calculation and weighted fusion to obtain temporal features fused with spatial features; S24. Concatenate the fused spatial features and temporal features, and regress them to the initial dimension through a linear layer to obtain spatiotemporal cross-fusion features.

5. A UGC video quality evaluation method based on spatiotemporal features as claimed in claim 4, characterized in that: The calculation and weighted fusion of the spatial or temporal attention specifically include: Calculate the similarity between each query and the key through dot product, and apply Softmax to generate a weight matrix; multiply the weight matrix with the value to obtain the corresponding fusion result.

6. A UGC video quality evaluation method based on spatiotemporal features as claimed in claim 1, characterized in that: Step S3 specifically includes: S31. Using a technical regression branch, perform technical regression processing on the spatiotemporal cross-fusion features to obtain a technical branch score; the technical regression branch includes two linear layers, an activation function, and a score interval mapping; The formula is: Among them, QT represents the technical branch score, Indicates C i The spatiotemporal cross-fusion features of video blocks, M represents the score mapping interval [0, M]; S32, using an aesthetic regression branch to perform aesthetic regression processing on the spatiotemporal cross-fusion features to obtain an aesthetic branch score; the aesthetic regression branch includes a multi-layer perceptron (MLP), an activation function, and a score interval mapping; The formula is: Among them, QA represents the aesthetic branch score; S33. Use a semantic regression branch to perform semantic regression processing on the spatiotemporal cross-fusion features to obtain a semantic branch score; the semantic regression branch includes a CLIP text encoder, an activation function, and a score interval mapping.

7. A UGC video quality evaluation method based on spatiotemporal features as claimed in claim 6, characterized in that: Step S33 specifically includes: S331, using paired antonym prompts, obtain features t1 and t2 of two text prompts through the CLIP text encoder; S332, performing cosine similarity calculation on the features of the two text prompts and the spatiotemporal cross-fusion features, respectively, to obtain two similarity scores st1 and st2; The formula is: S333: Based on the similarity score, a Softmax function is used to calculate the probability that the video is of high quality, and a semantic branch score is obtained through activation function and score interval mapping. The formula is: Among them, QS represents the semantic branch score.

8. A UGC video quality evaluation method based on spatiotemporal features as claimed in claim 7, characterized in that: Step S4 specifically includes: S41. Perform a weighted summation on the three quality scores corresponding to each video block to obtain a quality score for each video block; The formula is: Among them, α, β and γ represent the weights of the technical regression branch, aesthetic regression branch and semantic regression branch respectively. Indicates C i The quality score of each video block; S42, averaging the quality scores of all video blocks to obtain an evaluation score for the complete video; The formula is: Where C represents the number of video blocks, i represents the i-th video block, and Q represents the evaluation score of the complete video.

9. A UGC video quality evaluation system based on spatiotemporal features, characterized in that: A UGC video quality assessment method based on spatiotemporal features according to any one of claims 1 to 8 is adopted, comprising the following modules: Spatiotemporal feature extraction module: obtains the UGC video to be evaluated and divides the video into blocks; extracts spatiotemporal features from each video block to obtain the corresponding temporal and spatial features; Spatiotemporal feature fusion module: uses the attention mechanism to perform weighted fusion and linear splicing of the temporal and spatial features to obtain the spatiotemporal cross-fusion features of each video block; Quality score regression module: uses the technical regression branch, aesthetic regression branch, and semantic regression branch to regress the spatiotemporal cross-fusion features and obtain the quality scores corresponding to the three branches; Evaluation score output module: performs weighted summation on the three quality scores of all video blocks to obtain the evaluation score of the UGC video.

Images