A method and device for evaluating the quality of user-generated videos

Abstract

The present invention discloses a method and device for evaluating the quality of user-generated videos. By constructing a video quality evaluation model, the decrease in evaluation accuracy caused by neglecting the filtering mechanism is avoided. A dual-threshold adaptive adjustment weight threshold strategy is adopted to avoid the insufficient modeling ability of the evaluation model for distortion information. A new spatial domain window is designed through a visual focusing strategy to supplement the original spatial domain window, so that the spatial and temporal domain distortion information of the evaluation model can fully interact, the spatial domain information can be fully perceived in the initial stage, and the temporal domain distortion information can be fully learned in the final stage. On the basis of the window attention module, a learnable window relative position offset table based on the foveal effect is added to represent the global relative positions of different windows, so as to avoid the loss of global position information. In addition, a regional area factor proportional to the area is set in the local self-attention to balance the attention weights of the channel groups of windows of different areas, so as to avoid the deficiency of the evaluation model in global modeling.

Description

Technical Field

[0001] The present invention belongs to the technical field of user-generated content video processing, and in particular relates to a method and device for evaluating the quality of user-generated videos. Background Art

[0002] Since 2012, video data has consistently accounted for over half of global mobile data traffic. In recent years, the amount of user-generated content (UGC) videos on social media has exploded. The relative proportions of each commercial ad type have also shifted significantly. According to relevant reports, while photos still account for a large proportion of commercial ads, this proportion has decreased significantly compared to 2020. This decline has almost entirely shifted to videos. The rapid growth of video data has led to an exponential increase in demand for high-quality video. However, UGC video content is created and post-processed by users. During the creation process, it is subject to a variety of complex and mixed real-world distortions, such as hue shift, excessive contrast, overexposure, and blur. During post-processing, users may employ video enhancement algorithms (such as color enhancement and beauty enhancement) to beautify and modify the video. However, this often results in unknown quality impairments, which significantly reduce the consumer's visual experience. Therefore, to enhance the subjective experience of video, the production of high-quality UGC video requires an objective and effective quality feedback mechanism. One of the key issues is how to build an efficient and accurate video quality assessment (VQA) model. Furthermore, unlike professional videos, where the main types of distortion are compression and transmission distortion, real and more complex distortion makes quantifying the quality loss of UGC videos very challenging.

[0003] In the past few years, many no-reference image quality assessment methods (NR-IQA) and no-reference video quality assessment methods (NR-VQA) have emerged. Compared with reference video quality assessment methods (FR-VQA) that compare with the original video to evaluate the video quality, NR-VQA does not require any comparison. Therefore, NR video quality monitoring can change the video processing and interpretation of smartphones, social media, telemedicine, surveillance, and vision-guided robots in a way that FR-VQA models cannot do. However, due to the diversity of UGC video library content and the complexity of distortion, NR-VQA designed for evaluating compression distortion and transmission artifacts in the past often does not achieve satisfactory results on UGC videos.

[0004] Looking back at the development of NR-VQA, with the increasing advancements in deep learning theory and hardware technology, deep learning-based models have achieved significant progress in numerous computer vision applications. However, due to the scale of video subjective quality databases, previous VQA models using deep learning have exhibited poor robustness. The recent release of several large-scale video subjective quality databases has accelerated the application of deep learning in perceptual quality modeling. For example, Varga et al. employed LSTM (Long Short-term Memory) temporal networks to enhance the temporal information perception of spatial features extracted by CNNs. Li et al. proposed VSFA, an algorithm that includes a content-evoked feature extractor and modeling of long-term memory effects. The first part, consisting of a ResNet-50 and a global pooling layer, extracts high-dimensional spatial perceptual features from a single frame. The second part consists of a feature aggregation module and a temporal pooling layer. It primarily uses a GRU (Gate Recurrent Unit) to aggregate temporal distortion information. Subsequently, a temporal pooling layer inspired by temporal lag effects is used to determine video quality.

[0005] Wang et al. proposed a no-reference video quality assessment model based on 3D-CNN, which effectively learns and captures spatial and temporal distortion information through 3D blocks. Liu et al. proposed the V-MEON model. This method adopts a multi-task learning framework and uses branching tasks to jointly optimize 3D-CNN to introduce spatiotemporal features with better representation capabilities. Yi et al. introduced an attention mechanism to model spatiotemporal features and enhance the model's ability to perceive local distortion information. Tu et al. proposed RAPIQUE, which combines manual NSS features with CNN features to jointly capture spatial distortion information. Wang et al. proposed a ColNVQ, which attempts to analyze the quality of UGC videos from different aspects such as semantic content, technical quality, and compression level. The model consists of EfficientNet and a pre-trained D3D model. Zhu et al. proposed an end-to-end video quality assessment model, which consists of a spatial feature extractor and a temporal modeling module, the latter of which includes a local motion feature extractor and a temporal quality aggregation module.

[0006] While these models have made significant progress in simulating perceptual theory to process visual information and extract high-level features of content and semantics, the neglect of the filtering mechanism of useless visual information associated with HVS feature extraction has resulted in these modeling approaches increasing model complexity and potentially even compromising performance. Therefore, exploring the information processing mechanisms of the human visual system and simultaneously simulating perceptual processing and distillation processes are essential for efficient and accurate video quality assessment models. Furthermore, global information is crucial for many computer vision tasks, such as image classification and action recognition. In particular, in video quality assessment, simulating the human visual system to capture spatial and temporal distortions places high demands on the model's ability to resolve long-term dependencies. The Transformer architecture, which utilizes a single self-attention layer for long-term spatiotemporal modeling, has attracted widespread attention. While the Transformer architecture demonstrates strong performance in modeling global context, it also faces computational complexity challenges. To further reduce computational cost, some researchers have proposed restricting self-attention computation to a local window, known as local attention. This significantly reduces computational complexity, but also reduces the model's ability to capture global information. To compensate for the loss of global information, the Swin Transformer proposes a window transformation strategy, alternating self-attention calculations on shifted local windows between successive layers. However, studies have shown that the Swin Transformer lacks sufficient global information in the initial stages, resulting in a lack of visual focus. It is not until the final stages that it can focus on the primary object. This can be attributed to three main reasons. First, the spatial distortion information of visual features in the spatial domain is not fully exchanged based on the two existing spatial window partitioning methods. Second, in the initial stages of video processing and analysis, due to the large amount and complex structure of video data, fully utilizing spatial information can help the model more quickly establish a holistic understanding of the scene. However, in all four stages, the Swin Transformer's window transformation strategy uses simultaneous transformations in the spatial and temporal domains. This indicates that a single visual feature cannot perform two consecutive self-attention calculations with most spatially adjacent visual features, resulting in insufficient spatial information being perceived by the model in the initial stages. Finally, in the human eye system, the foveal effect reveals that visual acuity varies across different regions, and global position information influences the allocation of human attention. However, in the window self-attention calculation module, the neglect of window area and relative position information leads to inaccurate attention allocation. Therefore, in video quality assessment applications, the low-complexity Transformer structure has the problem of insufficient ability to depict global information.

[0007] To address these issues, the present invention introduces psychological and biological theories, and studies the no-reference video quality evaluation method in UGC scenarios by simultaneously simulating the multi-stage information processing and filtering process in human visual perception and optimizing the global modeling capability of the low-complexity Transformer architecture based on the foveal effect. Summary of the Invention

[0008] To address the shortcomings of existing technologies, simulate the multi-stage information processing and filtering process in human visual perception, accurately and dynamically distinguish noise features from key features in each processing stage, and enhance the global modeling capability in video quality assessment, the present invention adopts the following technical solutions:

[0009] A method for evaluating the quality of user-generated videos comprises the following steps:

[0010] Step S1: In psychological filtering theory, only a limited amount of visual information is allowed to be fully processed. This limitation requires that attention forms a filter through the information flow of the cognitive system. During the information flow through the cognitive system, some visual information is filtered out, thereby allowing other information to be processed more deeply. However, in the prior art, such as the NR-VQA model, the neglect of the filtering mechanism leads to a decrease in the accuracy of the video quality assessment model. Therefore, based on the Swin transformer, the present invention combines the self-attention calculation module to construct a video quality assessment model combined with the Transformer network and feature distillation. The video sequence is frame-recombined based on block sampling to obtain a recombined frame set with distorted information aggregation. Multi-stage visual information processing and filtering are performed on the recombined frame sequence. The self-attention module is used to group the channels of the visual features. Different channel groups represent different visual information of the visual features. In each layer of each stage, the multi-head self-attention module obtains the attention weight matrix of each channel group. Based on the attention weight matrix, the attention weight of the channel group of the visual features is obtained. The visual feature filter matrix of the current layer is obtained in combination with the weight threshold. The visual features of this layer are distilled using the filter matrix. The video quality score is generated by the distilled visual feature set obtained at each stage.

[0011] Step S2: For different visual information, the human visual system can accurately distinguish key visual information during each processing stage, thereby performing differentiated processing. However, in deep learning, as visual processing deepens, the receptive field of visual features expands and the representational power increases. The attention distribution of each stage and the attention distribution pattern of each layer of the Swin transformer change. In addition, simple threshold decision methods cannot meet the requirements for information type discrimination in complex visual scenes, resulting in insufficient modeling of distortion information. Therefore, in the multi-stage information processing and filtering process of the model simulating human visual perception, the attention weight distribution pattern of visual features at each stage is modeled. Obtaining an accurate weight threshold can effectively enhance the model's distortion perception ability. The present invention uses a dual-threshold adaptive weight threshold strategy to introduce a minimum error Bayesian decision method to analyze the attention distribution pattern at each stage and determine the state threshold of different channel groups in each stage. At the same time, combined with biological research on the nervous system, the present invention regards the self-attention calculation process as the activity process of neurons, determines the total energy of each stage from the perspective of energy distribution, and thus obtains a proportional threshold. Based on the state threshold and proportional threshold, the weight threshold of different channel groups in each stage is accurately and dynamically determined;

[0012] In video quality assessment applications, the low-complexity Transformer structure using local self-attention calculation has the problem of insufficient global information characterization ability. Therefore, the present invention needs to optimize the window conversion strategy of each stage, design a new window division method and an efficient window attention module based on the foveal theory through steps 3 and 4 to improve the efficiency of local self-attention calculation;

[0013] Step S3: The Swin Transformer lacks focus in the initial stage due to a lack of sufficient global information. It is not until the final stage that it can focus on the main object. This can be attributed to three main reasons. First, the two spatial window division methods are insufficient to fully interact with the spatial and temporal distortion information. Second, in the initial stage, spatial information is more abundant and more important than temporal information. The Swin Transformer's window transformation strategy performs simultaneous spatial and temporal transformations in all four stages, which means that the spatial information is not fully perceived by the model in the initial stage. Therefore, through the visual focus strategy, two new spatial windows are designed to supplement the original spatial windows. The effective combination of the two new spatial windows and the temporal window allows the edge information and the center information to interact efficiently. Second, different window transformation strategies are adopted at different stages, allowing the model to learn spatial and temporal distortion information in a targeted manner. In the initial stage, the model focuses on perceiving spatial information. In the subsequent third stage, the model, having learned spatial information, preliminarily perceives spatial and temporal distortion information. Based on the first three stages, the model focuses on learning temporal distortion information in the final fourth stage.

[0014] Step S4: Although Swin Transformer introduces relative position encoding to represent the image block in the window (QK T ), but from the perspective of the foveal effect, global position information is crucial to visual perception. Therefore, the lack of global position information weakens the global modeling ability of the model. On the basis of the window attention module of Swin Transformer, a learnable window relative position offset table based on the foveal effect is added to represent the relative position of different windows in the world, thereby simulating the visual acuity of different windows. In addition, according to the attention calculation formula, the channel group attention weight is positively correlated with the number of visual features in the window, while in many spatial window division methods, the number of visual features in different windows is different, which shows that in the global scope, the attention weight is more inclined to be allocated to the visual features in the large area window, while the visual features in the small window are easily ignored, resulting in the deficiency of the model in global modeling. In order to eliminate the influence of window area on the attention weight allocation, the present invention sets a regional area factor proportional to the area in the local self-attention to balance the channel group attention weights of windows of different areas.

[0015] Furthermore, in step S1, based on the consideration of computational complexity, video information is often represented by a set of sampled frames. However, the spatial and temporal features representing video distortion information are mostly scattered in each video frame V in the form of block level. m In order to maximize the preservation of distortion information, the block-level sampling strategy adopted in the present invention allocates limited computational resources to the picture blocks that reflect the degree of video distortion. By quantifying the degree of distortion of each picture block and performing time-series comparison on the picture blocks at the same spatial position, the time-series block with the minimum distortion is obtained. The sequence segments corresponding to the reconstructed frames are constructed based on the minimum time-series blocks at each spatial position.

[0016] Furthermore, the image block division formula is as follows:

[0017]

[0018] Among them, H and W represent the height and width of the video respectively, h and w represent the number of rows and columns of the corresponding picture block of the video, k represents that the small block belongs to the kth frame of the sequence segment, V m represents the mth sequence segment, Represents the image block in row i and column j of frame k;

[0019] Then, the SSIM value between the i-th row and j-th column image blocks of the k-th frame and the r-th frame is used. The total number of frames in the video sequence is t, and the TRS value of the image block in the i-th row and j-th column of the k-th frame is Then use the TRS indicator to quantify the degree of distortion of each image block (the smaller the value, the smaller the distortion), perform time sequence comparison TC on the image blocks at the same spatial position, and obtain the time sequence block with the smallest TRS value The formula is as follows:

[0020]

[0021]

[0022] The time series blocks of each spatial position (i, j) Forming set F m , pieced together into sequence fragment V m Corresponding reassembled frame RF m , the recombined frame sequence of the video is represented as {RF1,…,RF m ,…,RF n};

[0023]

[0024] Among them, AS{·} represents the splicing operation, RF m Represents the recombined frame corresponding to the mth sequence segment.

[0025] Furthermore, the multi-head self-attention module MSA obtains the attention weight matrix of each channel group:

[0026]

[0027] in, represents the set of visual feature vectors after noise feature filtering in the l-th layer of the k-th stage, T k,l is the set of visual feature vectors in the l-th layer of the k-th stage that have not been filtered by noise features, W k,l The attention weight of the nth channel group is

[0028] based on middle Characterize the physical meaning of the importance between visual features, and sum the columns of the matrix to obtain the row vector of the attention weight distribution of the nth channel group

[0029]

[0030]

[0031] in, represents the channel group attention weight obtained by the j-th visual feature of the n-th channel group, and t represents the number of visual features;

[0032] According to the weight threshold Mark the redundant features and noise features of each channel group to obtain the feature filter matrix of the current layer

[0033]

[0034]

[0035] in, represents the weight threshold of the nth channel group, represents the label value of the jth visual feature of the nth channel group, Represents a vector of labeled values;

[0036]

[0037] Where cat{·} represents the concatenation operation, and FC represents the total number of channel groups;

[0038] Obtained by splicing For the k,l-th layer visual feature set T k,l Perform distillation to obtain the visual feature set after distillation of the current layer

[0039]

[0040] Among them, ⊙ represents matrix dot product;

[0041] After the processing and filtering stages are completed, the video quality score is obtained through the regressor:

[0042] Q = FC(3DConv(T))

[0043] Among them, FC represents the fully connected layer, 3DConv represents the three-dimensional convolution kernel with convolution kernel sizes of 8, 7, 7, T represents the visual feature set, and Q represents the video quality score.

[0044] Furthermore, in step S2, in each stage of the Swin transformer, when a channel group g l When the following formula is satisfied, the channel group g l It is classified as having the ability to distinguish noise characteristics c1, otherwise it is classified as having no distinguishing ability c2:

[0045]

[0046] in, Represents the channel group g at stage k l The attention weight variance, ε represents the preset parameters;

[0047] Based on the number of two categories c1 and c2 Sum(c1) and Sum(c2), the probability of each category in each stage is obtained:

[0048]

[0049]

[0050] Where P(c1) represents the probability of channel group c1 with resolution capability, and P(c2) represents the probability of channel group c2 without resolution capability.

[0051] Based on the minimum error rate Bayesian criterion, the attention weight variance corresponding to the intersection of the probability density function curves of the two categories is the state threshold α k , if channel group g belongs to c i class, then the following equations are satisfied:

[0052]

[0053]

[0054] Among them, V g represents the attention weight variance of g, P(V g ) indicates that the attention weight variance of channel group g is V g The probability of P(V g |c i ) means that when channel group g belongs to c i When the class is , the attention weight variance of g is V g probability;

[0055] The state threshold α in defining each stage k Finally, the model divides the channel group into two subsets, c1 and c2. Among them, c2 is not effective in distinguishing noise features based on the attention weight distribution, so c2 will not participate in the distillation process.

[0056] Furthermore, in step S2, the noise feature ratios in different stages are different, so it is necessary to further establish the ratio threshold β of each stage. k Neurons are divided into suprathreshold and subthreshold states. From a statistical point of view, subthreshold neurons and suprathreshold neurons consume 20% and 75% of the total energy, respectively, which are called total subthreshold energy and total suprathreshold energy. In this paper, the self-attention calculation process is regarded as the activity process of neurons, while the neurons responsible for noise feature processing are in the subthreshold state. The total energy consumed by self-attention calculation is calculated as follows:

[0057] E=E1+......+E i +......+E n

[0058] E k =N k ·H k ·L k ·E s

[0059]

[0060] Among them, E i are the energy consumed in a single stage, n represents the number of stages, H k Indicates the number of channel groups, N k Indicates the number of visual features, L k represents the number of noise feature distillation layers, E s represents the energy consumed by each visual feature in a single self-attention calculation, E b represents the total subthreshold energy, β k Indicates the ratio threshold.

[0061] Furthermore, in step S2, in each layer in each stage, based on the ratio threshold β k and state threshold α k Determine the weight threshold for each channel group

[0062] In the multi-head self-attention module MSA at each stage, the attention weights of each channel group are distributed as row vectors Splicing them together, we get the attention weight row vector of the nth channel group:

[0063]

[0064] Among them, m represents the number of channel groups in MSA, and then the sorted channel group attention weight vector matrix is ​​obtained by arranging them in ascending order in each row:

[0065]

[0066] Among them, sort(·) represents the ascending order operation of rows. Represents a single vector in the sequence after ascending order;

[0067]

[0068]

[0069] Based on B k,l The variance of each row and state threshold α k Get the noise characteristic ratio of each channel group according to Determine the weight thresholds of channel groups in each layer at different stages

[0070] Furthermore, in step S3, the spatial domain window and temporal domain window of the Swin transformer are constructed, and different self-attention calculations are used in different areas of the window, including spatial domain window a, spatial domain window b, spatial domain window c, spatial domain window d, temporal domain window e and temporal domain window f; when there are t′ pieces of feature map sequence and the spatial super size of each feature map is length H′×width W′, the spatial domain window a divides each feature map into four equal areas, and in each area The window self-attention calculation is performed in ; the spatial domain window b divides each feature map into nine regions, and the size of the area in the four corners is The sizes of the upper and lower horizontal strips are The size of the two longitudinal strip areas on the left and right is The middle area is The spatial window c divides each feature map into six regions, with two vertically connected upper and lower regions of size Long strip area, with two sizes of Square area; The spatial window d divides each feature map into six areas, with two horizontally connected left and right areas of size Long strip area, the two sizes of the left and right sides of the middle area are both The time domain window e cuts the t′ feature map sequence into segments at equal intervals k, and each segment has a total of k feature maps. In each segment, the front Zhang is an area, The time domain window f cuts the t′ feature map sequence into segments at equal intervals k, and each segment has a total of k feature maps. In each segment, the previous Zhang is an area, the middle Zhang is an area, Zhang is an area;

[0071] Spatial windows c and d are constructed to complement spatial windows a and b, allowing efficient interaction between edge and center information. Secondly, different window transformation strategies are adopted at different stages to enable the model to specifically learn spatial and temporal distortion information. In the first two stages, to enhance the ability of features to characterize spatial distortion, the windows of two adjacent layers use spatial windows a and b in the spatial domain, and only temporal window e in the temporal domain. After the first two stages, since the temporal windows are not transformed, the receptive field of the features in the temporal domain is limited by the size of the windows. However, in the spatial domain, the distortion information is fully interacted with each other. In the third stage, the windows of two adjacent layers use spatial windows c and d in the spatial domain, and temporal windows e and f in the temporal domain. As the spatial features in the first three stages continue to merge, modeling temporal distortion information becomes the main concern of the model. Therefore, in the fourth stage, the windows of two adjacent layers use spatial windows b in the spatial domain, and temporal windows e and f in the temporal domain.

[0072] Furthermore, in step S4, the local self-attention calculation formula is as follows:

[0073]

[0074] Q=W q T

[0075] K=W k T

[0076] V=W v T

[0077] Among them, T represents the input token set, P W Represents the relative position offset table of the learnable window, σ i =1 / S i represents the area factor, S i represents the window area, QK T represents the matrix product of the transpose of Q and K, B represents the relative position encoding table within the window, Q, K, V represent, V represents the feature matrix after T passes through three linear transformations, d represents the scaling factor, W q 、W k and W v are three learnable weight matrices.

[0078] A user-generated video quality assessment device includes a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement a user-generated video quality assessment method.

[0079] The advantages and beneficial effects of the present invention are:

[0080] The present invention introduces psychological and biological theories, and based on the Swin transformer, simultaneously simulates the multi-stage information processing and filtering process in human visual perception to build a video quality evaluation model; the present invention introduces statistical and biological theories into the modeling process of noise feature filtering, adopts the minimum error Bayesian decision method to measure the noise significance of the features, and introduces the neuron energy loss mechanism to quantify the proportion of noise features in each processing stage, so as to accurately and dynamically distinguish the noise features from the key features in each processing stage; the present invention improves the efficiency of local self-attention calculation by optimizing the window conversion strategy of each stage, designing a new window division method and an efficient window attention module based on the foveal theory, thereby enhancing the global modeling capability of the low-complexity Transformer structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 2 is a schematic diagram of the structure of a video quality assessment model using a Transformer network combined with feature distillation in an embodiment of the present invention.

[0082] Figure 2a This is a diagram showing the effect of the minimum error Bayesian threshold decision in stage one of an embodiment of the present invention.

[0083] Figure 2b This is a diagram showing the effect of the minimum error Bayesian threshold decision in stage two of an embodiment of the present invention.

[0084] Figure 2c This is a diagram showing the effect of the minimum error Bayesian threshold decision in stage three of an embodiment of the present invention.

[0085] Figure 2d 3 is a diagram showing the effect of the minimum error Bayesian threshold decision in stage four of an embodiment of the present invention.

[0086] Figure 3a Schematic diagram of spatial window a in an embodiment of the present invention.

[0087] Figure 3b Schematic diagram of the spatial window b in an embodiment of the present invention.

[0088] Figure 3c Schematic diagram of the spatial window c in an embodiment of the present invention.

[0089] Figure 3d Schematic diagram of the spatial window d in an embodiment of the present invention.

[0090] Figure 3e 2 is a schematic diagram of a time domain window e in an embodiment of the present invention.

[0091] Figure 3f FIG. 2 is a schematic diagram of a time domain window f in an embodiment of the present invention.

[0092] Figure 4 Schematic diagram of the structure of the device in an embodiment of the present invention. DETAILED DESCRIPTION

[0093] The following describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0094] like Figure 1 As shown, the present invention proposes a method for assessing the quality of user-generated videos using a Transformer network combined with feature distillation. This method is a reference-free quality assessment method suitable for quality assessment of user-generated content videos on various current video platforms. In order to simultaneously simulate the multi-stage information processing and filtering process in human visual perception, the present invention first uses block-level sampling to aggregate distortion information to obtain a reconstructed frame set, and then reconstructs the frame sequence to perform multi-stage visual information processing and filtering. The entire process is divided into four stages, and the efficient window attention module and visual focus strategy proposed in the present invention are used for global information modeling. Finally, a regressor consisting of a 3D adaptive average pooling layer and a fully connected layer is introduced to obtain the video score.

[0095] 1. Video quality evaluation model based on Transformer network combined with feature distillation

[0096] Considering the computational complexity, video information is often represented by a set of sampled frames. However, the spatial and temporal features that represent video distortion information are mostly scattered in each video frame V in the form of block level. m In order to maximize the preservation of distortion information, the present invention adopts a block-level sampling strategy to allocate the limited amount of calculation to the image blocks that reflect the degree of video distortion. The formula is as follows:

[0097]

[0098] Where H and W represent the height and width of the video, k represents the kth frame of the sequence segment to which the small block belongs, and V m represents the mth sequence segment, Represents the small block in the i-th row and j-th column of the k-th frame.

[0099] Then, the TRS index is used to quantify the degree of distortion of each small block (the smaller the value, the smaller the distortion), and the small blocks at the same spatial position are compared in time sequence to obtain the time sequence block with the smallest TRS value.

[0100]

[0101]

[0102] in, Indicates the TRS value of the small block in the i-th row and j-th column of the k-th frame, represents the SSIM value between the i-th row and j-th column blocks of the k-th frame and the r-th frame, and t represents the total number of frames in the video sequence.

[0103] The time series blocks of each spatial position (i, j) Forming set F m , and finally pieced together into sequence fragment V m Corresponding reassembled frame RF m The recombined frame sequence of the video can be represented as {RF1,…,RF m ,…,RF n}.

[0104]

[0105] Where AS{·} represents the concatenation operation, RF m Represents the recombined frame corresponding to the mth sequence segment.

[0106] Afterwards, the reassembled frame sequence undergoes multi-stage visual information processing and filtering. The entire process is divided into four stages, with the number of layers in each stage being (2, 2, 6, 4), and the efficient window attention module and visual focus strategy proposed in this paper are used for global information modeling.

[0107] In each layer of each stage, the multi-head self-attention module (MSA) obtains the attention weight matrix W of each channel group through the SoftMax function.

[0108]

[0109] in represents the set of visual feature vectors after noise feature filtering in the l-th layer of the k-th stage, T k,l is the set of visual feature vectors in the l-th layer of the k-th stage without filtering of noise features, W k,l The attention weight of the nth channel group is

[0110] based on middle Characterize the physical meaning of the importance between visual features, and sum the columns of the matrix to obtain the row vector of the attention weight distribution of the nth channel group

[0111]

[0112]

[0113] in, represents the channel group attention weight obtained by the j-th visual feature of the n-th channel group, and t is the number of visual features. For convenience, this method simply refers to the channel group attention weight threshold as the weight threshold.

[0114] Then according to the weight threshold Mark the redundant features and noise features of each channel group to obtain the feature filter matrix of the current layer

[0115]

[0116]

[0117] in represents the weight threshold of the nth channel group, represents the label value of the jth visual feature of the nth channel group, A vector of labeled values.

[0118]

[0119] Among them, cat{·} represents the concatenation operation, and FC represents the total number of channel groups. For the k,l-th layer visual feature set T k,l Perform distillation to obtain the visual feature set after distillation of the current layer

[0120]

[0121] where ⊙ represents matrix dot product.

[0122] After the four processing and filtering stages, the video quality score is obtained through the regressor.

[0123] Q = FC(3DConv(T))

[0124] Among them, FC is the fully connected layer, 3DConv is the three-dimensional convolution kernel with kernel sizes of 8, 7, and 7, T is the visual feature set, and Q is the video quality score.

[0125] 2. Dual-threshold adaptive adjustment weight threshold strategy

[0126] It is worth noting that as visual processing deepens, the receptive field of visual features expands and the representation power increases, and the attention distribution of each stage and each layer of the Swin transformer changes. Therefore, in the process of simulating the multi-stage information processing and filtering in human visual perception, the attention weight distribution law of each stage of visual features is modeled to obtain an accurate weight threshold. It can effectively enhance the distortion perception ability of the model. The present invention proposes a dual-threshold adaptive adjustment weight threshold strategy, introduces the minimum error Bayesian decision method to analyze the attention distribution law of each stage, and thus determine the state thresholds of different channel groups in each stage. At the same time, combined with the biological research on the nervous system, the present invention regards the self-attention calculation process as the activity process of neurons, determines the total energy of each stage from the perspective of energy distribution, and thus obtains the proportional threshold. Finally, based on the two thresholds, the weight thresholds of different channel groups in each stage are accurately and dynamically determined.

[0127] In each stage of the Swin transformer, the channel groups are divided into two categories: those with the ability to distinguish noise features c1 and those without the ability to distinguish c2. l When the following equation is satisfied, the channel group g l is classified as c1, otherwise it is c2.

[0128]

[0129] in, represents the g of the kth stage l The attention weight variance is ε, and ε is a parameter.

[0130] Then, the probability of each class in each stage is obtained based on the number of two classes c1 and c2 (Sum(c1), Sum(c2)).

[0131]

[0132]

[0133] Among them, P(c1) is the probability of channel group c1 with resolution ability, and P(c2) is the probability of channel group c2 without resolution ability.

[0134] like Figures 2a to 2d As shown, based on the minimum error rate Bayesian criterion, the attention weight variance corresponding to the intersection of the probability density function curves of the two types of samples is the state threshold α k Among them, if the channel group g belongs to c i class, then the following equation is satisfied.

[0135]

[0136]

[0137] Where V g represents the attention weight variance of g.

[0138] In accurately defining the state threshold α of each stage kAfterwards, the model divides the channel group into two subsets, c1 and c2. Among them, c2 is not enough to effectively distinguish noise features based on the attention weight distribution, so c2 will not participate in the distillation process. The proportion of noise features contained in different stages is different, so it is necessary to further establish the ratio threshold β of each stage. k .

[0139] Biological research shows that neurons in the nervous system are divided into suprathreshold and subthreshold states. Statistically, subthreshold and suprathreshold neurons consume 20% and 75% of the total energy, respectively, referred to as total subthreshold and total suprathreshold energy. This paper considers the self-attention computation process as a neuronal activity, while neurons responsible for processing noise features are in a subthreshold state. The total energy consumed by self-attention computation can be calculated as follows.

[0140] E=E1+E2+E3+E4

[0141] E k =N k ·H k ·L k ·E s

[0142] Among them, E1, E2, E3, and E4 are the energy consumed in stages 1, 2, 3, and 4 respectively, and H k is the number of channel groups, N k is the number of visual features, L k is the number of noise feature distillation layers, E s The energy consumed by each visual feature in a single self-attention computation.

[0143]

[0144] In each layer of each stage, based on the ratio threshold β k and state threshold α k Determine the weight threshold for each channel group Specifically, in the multi-headed self-attention (MSA) module at each stage, the attention weight distribution row vector of each channel group is Splice them together and then sort them in ascending order in each row to get B k,l .

[0145]

[0146] Among them, B k,l is the sorted channel group attention weight vector matrix, sort(·) is the row ascending operation, A k,l The expression is as follows.

[0147]

[0148] Where m is the number of channel groups in MSA, is the row vector of attention weights for the nth channel group.

[0149] The present invention is based on B k,l The variance of each row and state threshold α k Get the noise characteristic ratio of each channel group Finally, according to Determine the weight thresholds of channel groups in each layer at different stages

[0150]

[0151]

[0152] 3. Visual focus strategy

[0153] The Swin transformer has two spatial domain window division methods and two temporal domain window division methods, as shown in Figures a and b, and Figures e and f, where different colors represent different window self-attention calculation areas. The present invention newly adds two spatial domain windows, as shown in Figures c and d. Specifically, assuming that there are t′ feature map sequences composed of tokens, and the spatial size of each feature map is H′×W′, in Figure a, each feature map is divided into four equal regions, and in each region Window self-attention calculation is performed in ; In Figure b, each feature map is divided into nine areas, and the size of the small square area on the four edges is The sizes of the upper and lower strip areas are The sizes of the two long strip areas on the left and right are The middle area is In Figure c, each feature map is divided into six regions, with two vertically connected long strip regions on the left and right sides. The size of the long strip region is There are two square areas in the middle area with the size of In Figure d, each feature map is divided into six areas, with two horizontally connected long strip areas on the upper and lower sides. The size of the long strip area is The two squares arranged on the left and right of the middle area are both The difference from Figure c is that the long strip area and the square area are located in different spatial positions; in Figure e, the t′ feature map sequence is cut into segments at equal intervals of k, and each segment has a total of k feature maps. In each segment, the front Zhang is an area, In Figure f, the t′ feature map sequence is cut into segments at equal intervals of k, and each segment has a total of k feature maps. In each segment, the front Zhang is an area, the middle Zhang is an area, Zhang is an area.

[0154] The present invention first designs two spatial windows ( Figure 3c )、( Figure 3d ). These two spatial windows are the Swintransformer model ( Figure 3a )、( Figure 3b ) is supplemented by ( Figure 3c )、( Figure 3d ) can make the edge information and the center information interact efficiently. Secondly, at different stages, the present invention adopts different window conversion strategies to enable the model to learn the spatial-temporal distortion information in a targeted manner. In the initial stage, the model focuses on perceiving the spatial information. In the third stage, the model that has learned the spatial information preliminarily perceives the distortion information in the spatial-temporal domain. On the basis of the first three stages, the fourth stage model focuses on learning the time-domain distortion information. Specifically, in the first two stages, in order to enhance the ability of features to depict spatial distortion, the windows of the two adjacent layers are converted into ( Figure 3a )and( Figure 3b ). In the time domain, only ( Figure 3e After the first two stages, since the time domain window is not transformed, the receptive field of the feature in the time domain is limited by the size of the window, but in the spatial domain, the distortion information is fully interacted. In the third stage, the windows of the two adjacent layers are spatially transformed using ( Figure 3c )and( Figure 3d ), the corresponding time domain is ( Figure 3e )and( Figure 3f As the features in the spatial domain continue to merge in the first three stages, the modeling of the time domain distortion information becomes the main issue of the model. Therefore, in the fourth stage, the windows of the two adjacent layers are spaced using ( Figure 3b ), in the time domain, ( Figure 3e )and( Figure 3f ).

[0155] 4. Efficient Window Attention Module

[0156] Although Swin Transformer introduces relative position encoding to represent the image block in the window (QK T), but from the perspective of the foveal effect, global position information is crucial to visual perception, so the lack of global position information weakens the global modeling ability of the model. The present invention adds a learnable window relative position offset table P based on the foveal effect on the basis of the window attention module of Swin Transformer. W To represent the relative position of different windows in the world, thus simulating the visual acuity of different windows. In addition, in order to eliminate the influence of window area on the distribution of attention weight, the present invention sets a regional area factor σ proportional to the area in the local self-attention formula. i =1 / S i , used to balance the attention weights of channel groups of windows with different areas, where S i is the window area. The local self-attention calculation formula is as follows:

[0157]

[0158] like Figure 4 As shown, a user-generated video quality assessment device includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the user-generated video quality assessment method.

[0159] The implementation of this part of the content is similar to that of the above method embodiment and will not be repeated here.

[0160] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for assessing the quality of user-generated videos, characterized in that The steps include: Step S1: Construct a video quality evaluation model combining a Transformer network and feature distillation, perform frame reorganization on the video sequence based on block sampling, obtain a set of reorganized frames with aggregated distortion information, perform multi-stage visual information processing and filtering on the reorganized frame sequence, and use the self-attention module to group the channels of the visual features. Different channel groups represent different visual information of the visual features. In each layer of each stage, the multi-head self-attention module obtains the attention weight matrix of each channel group, obtains the attention weight of the channel group of the visual features based on the attention weight matrix, and combines the weight threshold to obtain the visual feature filter matrix of the current layer. The filter matrix is ​​used to distill the visual features of this layer, and the video quality score is generated through the distilled visual feature set obtained at each stage. Step S2: Adaptively adjust the weight threshold strategy through dual thresholds, introduce the minimum error Bayesian decision method to analyze the attention distribution law of each stage, determine the state threshold of different channel groups in each stage, and at the same time, regard the self-attention calculation process as the activity process of neurons, determine the total energy of each stage from the perspective of energy distribution, and thus obtain the proportional threshold. Based on the state threshold and the proportional threshold, the weight threshold of different channel groups in each stage is dynamically determined; Step S3: Through the visual focus strategy, a new spatial window is designed to supplement the original spatial window. The combination of the new spatial window and the temporal window allows the edge information to interact with the center information. Different window conversion strategies are adopted at different stages. In the initial stage, the model focuses on perceiving spatial information. In the subsequent stages, the model that has learned spatial information initially perceives distortion information in the spatial and temporal domains. In the final stage, the model focuses on learning distortion information in the temporal domain. Step S4: Based on the window attention module, a learnable window relative position offset table based on the foveal effect is added to represent the relative positions of different windows in the world. A regional area factor proportional to the area is set in the local self-attention to balance the attention weights of the channel groups of windows with different areas.

2. The method for assessing user-generated video quality according to claim 1, wherein: In step S1, a block-level sampling strategy is adopted to distribute the computational effort to the image blocks reflecting the degree of video distortion. By quantifying the degree of distortion of each image block and performing a time sequence comparison on the image blocks at the same spatial position, the time sequence block with the minimum degree of distortion is obtained. Sequence segments corresponding to the reconstructed frames are formed based on the minimum time sequence blocks at each spatial position.

3. The method for assessing user-generated video quality according to claim 2, wherein: The division formula of the picture block is as follows: Among them, H and W represent the height and width of the video respectively, h and w represent the number of rows and columns of the corresponding picture block of the video, k represents that the small block belongs to the kth frame of the sequence segment, V m represents the mth sequence segment, Represents the image block in row i and column j of frame k; Then, the SSIM value between the i-th row and j-th column image blocks of the k-th frame and the r-th frame is used. The total number of frames in the video sequence is t, and the TRS value of the image block in the i-th row and j-th column of the k-th frame is Then use the TRS indicator to quantify the degree of distortion of each image block, perform time sequence comparison TC on the image blocks at the same spatial position, and obtain the time sequence block with the smallest TRS value The formula is as follows: The time series blocks of each spatial position (i, j) Forming set F m , pieced together into sequence fragment V m Corresponding reassembled frame RF m ; Among them, AS{·} represents the splicing operation, RF m Represents the recombined frame corresponding to the mth sequence segment.

4. The method for assessing user-generated video quality according to claim 1, wherein: The multi-head self-attention module MSA obtains the attention weight matrix of each channel group: in, represents the set of visual feature vectors after noise feature filtering in the l-th layer of the k-th stage, T k,l is the set of visual feature vectors in the l-th layer of the k-th stage that have not been filtered by noise features, W k,l The attention weight of the nth channel group is based on middle Characterize the physical meaning of the importance between visual features, and sum the columns of the matrix to obtain the row vector of the attention weight distribution of the nth channel group in, represents the channel group attention weight obtained by the j-th visual feature of the n-th channel group, and t represents the number of visual features; According to the weight threshold Mark the redundant features and noise features of each channel group to obtain the feature filter matrix of the current layer in, represents the weight threshold of the nth channel group, represents the label value of the jth visual feature of the nth channel group, Represents a vector of labeled values; Among them, cat{·} represents the splicing operation; Obtained by splicing For the k,l-th layer visual feature set T k,l Perform distillation to obtain the visual feature set after distillation of the current layer Among them, ⊙ represents matrix dot product; After the processing and filtering stages are completed, the video quality score is obtained through the regressor: Q = FC(3DConv(T)) Among them, FC represents the fully connected layer, 3DConv represents the three-dimensional convolution kernel, T represents the visual feature set, and Q represents the video quality score.

5. The method for assessing user-generated video quality according to claim 1, wherein: In step S2, in each stage of the Swin transformer, when a channel group g l When the following equation is satisfied, the channel group g l It is classified as having the ability to distinguish noise characteristics c1, otherwise it is classified as having no distinguishing ability c2: in, Represents the channel group g at stage k l The attention weight variance, ε represents the preset parameters; Based on the number of two categories c1 and c2 Sum(c1) and Sum(c2), the probability of each category in each stage is obtained: Where P(c1) represents the probability of channel group c1 with resolution capability, and P(c2) represents the probability of channel group c2 without resolution capability. Based on the minimum error rate Bayesian criterion, the attention weight variance corresponding to the intersection of the probability density function curves of the two categories is the state threshold α k , if channel group g belongs to c i class, then the following equations are satisfied: Among them, V g represents the attention weight variance of g, P(V g ) indicates that the attention weight variance of channel group g is V g The probability of P(V g |c i ) means that when channel group g belongs to c i When the class is , the attention weight variance of g is V g probability; The state threshold α in defining each stage k Finally, the model divides the channel group into two subsets, c1 and c2. Among them, c2 is not effective in distinguishing noise features based on the attention weight distribution, so c2 will not participate in the distillation process.

6. The method for assessing the quality of user-generated videos according to claim 1, wherein: In step S2, neurons are divided into suprathreshold and subthreshold states. The self-attention calculation process is regarded as the activity process of neurons, while the neurons responsible for noise feature processing are in the subthreshold state. The total energy consumed by the self-attention calculation is calculated as follows: E=E1+......+E i +......+E n E k =N k ·H k ·L k ·E s Among them, E i are the energy consumed in a single stage, n represents the number of stages, H k Indicates the number of channel groups, N k Indicates the number of visual features, L k represents the number of noise feature distillation layers, E s represents the energy consumed by each visual feature in a single self-attention calculation, E b represents the total subthreshold energy, β k Indicates the ratio threshold.

7. The method for assessing user-generated video quality according to claim 4, wherein: In step S2, in each layer in each stage, based on the ratio threshold β k and state threshold α k Determine the weight threshold for each channel group In the multi-head self-attention module MSA at each stage, the attention weights of each channel group are distributed as row vectors Splicing them together, we get the attention weight row vector of the nth channel group: Among them, m represents the number of channel groups in MSA, and then the sorted channel group attention weight vector matrix is ​​obtained by arranging them in ascending order in each row: Among them, sort(·) represents the ascending order operation of rows. Represents a single vector in the sequence after ascending order; Based on B k,l The variance of each row and state threshold α k Get the noise characteristic ratio of each channel group according to Determine the weight thresholds of channel groups in each layer at different stages 8. The method for assessing user-generated video quality according to claim 1, wherein: In step S3, a spatial domain window and a temporal domain window are constructed, and different self-attention calculations are used for different areas of the window, including spatial domain window a, spatial domain window b, spatial domain window c, spatial domain window d, temporal domain window e, and temporal domain window f. When there are t′ pieces of feature map sequence and the spatial super size of each feature map is length H′×width W′, the spatial domain window a divides each feature map into four equal areas, and in each area The window self-attention calculation is performed in ; the spatial domain window b divides each feature map into nine regions, and the size of the area in the four corners is The sizes of the upper and lower horizontal strips are The size of the two longitudinal strip areas on the left and right is The middle area is The spatial window c divides each feature map into six regions, with two vertically connected upper and lower regions of size Long strip area, with two sizes of Square area; The spatial window d divides each feature map into six areas, with two horizontally connected left and right areas of size Long strip area, the two sizes of the left and right sides of the middle area are both The time domain window e cuts the t′ feature map sequence into segments at equal intervals k, and each segment has a total of k feature maps. In each segment, the front Zhang is an area, The time domain window f cuts the t′ feature map sequence into segments at equal intervals k, and each segment has a total of k feature maps. In each segment, the previous Zhang is an area, the middle Zhang is an area, Zhang is an area; Spatial window a and spatial window b are supplemented by constructing spatial window c and spatial window d. In the first two stages, the windows of the two adjacent layers use spatial window a and spatial window b in the spatial domain, and only time domain window e in the time domain. In the third stage, the windows of the two adjacent layers use spatial window c and spatial window d in the spatial domain, and time domain window e and time domain window f in the time domain. In the fourth stage, the windows of the two adjacent layers use spatial window b in the spatial domain, and time domain window e and time domain window f in the time domain.

9. The method for assessing the quality of user-generated videos according to claim 1, wherein: In step S4, the local self-attention calculation formula is as follows: Q=W q T K=W k T V=W v T Among them, T represents the input token set, P W Represents the relative position offset table of the learnable window, σ i =1 / S i represents the area factor, S i represents the window area, QK T represents the matrix product of the transpose of Q and K, B represents the relative position encoding table within the window, Q, K, V represent, V represents the feature matrix after T passes through three linear transformations, d represents the scaling factor, W q 、W k and W v are three learnable weight matrices.

10. A user-generated video quality assessment device, characterized in that The system comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the system is used to implement a user-generated video quality assessment method according to any one of claims 1 to 9.

Images