Content-based guidance for no-reference video quality assessment

Abstract

The application provides a content guidance-based no-reference video quality evaluation method, which comprises the following steps: obtaining a video segment to be evaluated; constructing a feature extraction network and inputting the video segment to be evaluated into the feature extraction network so as to obtain corresponding frame-level features and semantic content features; constructing a time sequence information capturing network so as to obtain time sequence dependency between the frame-level features according to the time sequence information capturing network; constructing a super network and using the super network to reshape the semantic content features into a query and a quality perception head respectively; inputting the time sequence dependency between the frame-level features and the query into an encoder so as to obtain frame-level quality features with time sequence dependency guided by video content; and using the quality perception head to predict the frame-level quality features so as to obtain a video quality score corresponding to the video segment to be evaluated. Thus, the allocation of the time sequence attention can be guided by the video content, and the quality score can be generated, so that the accuracy of the video quality evaluation is improved.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a content-guided no-reference video quality assessment method and a computer device. Background Technology

[0002] In related technologies, no-reference video quality assessment methods mainly target specific types of distortion, such as compression and transmission distortion, to quantify video quality. However, due to the diversity and complexity of video content, their effectiveness is limited when dealing with various distortion types. In recent years, with the rapid development of deep learning technology, deep learning-based no-reference video quality assessment methods have gradually emerged. Deep learning-based methods use deep neural networks to extract high-level, distinguishable features from videos, thereby enabling better evaluation of video quality. These methods typically employ 2D-CNN and 3D-CNN structures, combined with temporal modules to capture the temporal relationships between video frames. By fusing the spatial and temporal features of the video, deep learning-based methods have achieved some success in no-reference video quality assessment. However, recurrent neural networks (RNNs) are not ideal in terms of temporal information fusion and cannot fully consider the impact of the dependency between the video's main content and frame content on human subjective quality perception. Furthermore, no-reference video quality assessment still lacks an efficient and mature method. Although deep learning-based methods have made some progress, further research and improvement are still needed. Summary of the Invention

[0003] This invention aims to at least partially address one of the technical problems in the aforementioned technologies. To this end, one objective of this invention is to propose a content-guided, no-reference video quality assessment method that utilizes video content to guide the allocation of temporal attention and generate quality scores, thereby improving the accuracy of video quality assessment.

[0004] The second objective of this invention is to provide a computer device.

[0005] To achieve the above objectives, a first aspect of the present invention proposes a content-guided no-reference video quality assessment method, comprising the following steps: acquiring a video segment to be evaluated; constructing a feature extraction network based on SwinTransformerV2 and inputting the video segment to be evaluated into the feature extraction network to obtain frame-level features and semantic content features corresponding to the video segment to be evaluated; constructing a temporal information capture network based on a Transformer encoder to obtain the temporal dependencies between the frame-level features according to the temporal information capture network; constructing a hypernetwork for transmitting the semantic content features and using the hypernetwork to reshape the semantic content features into a query and a quality-aware head in the Transformer encoder respectively; inputting the temporal dependencies between the frame-level features and the query into the Transformer encoder to obtain frame-level quality features with temporal dependencies guided by video content; and using the quality-aware head to predict the frame-level quality features to obtain a video quality score corresponding to the video segment to be evaluated.

[0006] According to an embodiment of the present invention, a content-guided no-reference video quality assessment method first acquires the video segment to be evaluated; then, a feature extraction network based on SwingTransformerV2 is constructed, and the video segment to be evaluated is input into the feature extraction network to obtain corresponding frame-level features and semantic content features; next, a temporal information capture network based on a Transformer encoder is constructed to obtain the temporal dependencies between frame-level features; then, a hypernetwork for transmitting semantic content features is constructed, and the hypernetwork is used to reshape the semantic content features into queries and quality-aware heads in the Transformer encoder; then, the temporal dependencies between frame-level features and the queries are input into the Transformer encoder to obtain frame-level quality features with temporal dependencies guided by video content; finally, the quality-aware head is used to predict the frame-level quality features to obtain the video quality score corresponding to the video segment to be evaluated; thus, the allocation of temporal attention guided by video content and the generation of quality scores can be achieved, thereby improving the accuracy of video quality assessment.

[0007] In addition, the content-guided no-reference video quality assessment method proposed in the above embodiments of the present invention may also have the following additional technical features:

[0008] Optionally, a feature extraction network based on SwinTransformerV2 is constructed, and the video segment to be evaluated is input into the feature extraction network to obtain frame-level features and semantic content features corresponding to the video segment to be evaluated. This includes: pre-training SwinTransformerV2 on ImageNet to obtain the feature extraction network; using the feature extraction network to extract features frame by frame from the video segment to be evaluated to obtain depth features corresponding to four different stages; using a pyramid pooling strategy to compress the depth features of the first three stages into the same features as the fourth stage using global average pooling, and concatenating the depth features corresponding to the four different stages and then compressing the feature channels through convolution, and using global average pooling to compress them into a one-dimensional vector as the frame-level features; using temporal pooling to compress the time scale of the depth features corresponding to the fourth stage to one as the semantic content features.

[0009] Optionally, a temporal information capture network based on a Transformer encoder is constructed to obtain the temporal dependencies between the frame-level features according to the temporal information capture network, including: using a sliding window based on a graph convolutional network to compress several frames in the frame-level features into a token to obtain multiple tokens; using Sinusoidal positional encoding to obtain the positional information corresponding to the multiple tokens, and adding it to the corresponding token to obtain multiple tokens carrying positional information.

[0010] Optionally, each token can be obtained according to the following formula:

[0011] token = TP(Aδ(AXW1)W2)

[0012] Where TP(·) represents temporal pooling; A represents the adjacency matrix; δ(·) represents the activation function; X represents the feature combination of several frames within the input sliding window; and W1 and W2 represent trainable weight matrices.

[0013] Optionally, each token carrying location information can be obtained according to the following formula:

[0014] p k,2i =sin(k / 10000) 2i / d )

[0015] p k,2i+1 =cos(k / 10000) 2i / d )

[0016] token k =token k +p k

[0017] Where i represents the dimension index in the position encoding matrix p, 2i represents even positions, and 2i+1 represents odd positions; d represents the embedding dimension in the Transformer model, k represents the k-th token, and sin(·) and cos(·) represent sine and cosine calculations, respectively.

[0018] Optionally, the Transformer encoder includes at least one Transformer encoder module, which includes at least one attention head and one feedforward neural network.

[0019] Optionally, the hypernetwork used to transmit the semantic content features includes several convolutional layers, fully connected layers, and pooling layers.

[0020] Optionally, the quality-aware head is used to predict the frame-level quality features to obtain the video quality score corresponding to the video segment to be evaluated, including: concatenating the frame-level quality features together along the frame dimension and performing temporal pooling processing before inputting them into the quality-aware head for prediction; the quality-aware head performs weighted multiplication and bias addition on the frame-level quality features and outputs the video quality score corresponding to the video segment to be evaluated.

[0021] Optionally, the feature extraction network based on SwinTransformerV2, the temporal information capture network based on Transformer encoder, and the supernetwork for transmitting video content features are trained based on the following parameter settings: the feature extraction network based on SwinTransformerV2 is initialized by obtaining the pre-trained weights publicly available in SwinTransformerV2, and the remaining trainable parameters in the model are randomly initialized; the mean absolute error is used as the loss function to measure the difference between the predicted score and the true quality score; the model parameters are updated using the AdamW optimizer with an initial learning rate of 1e-3, and a total of 300 iterations are trained.

[0022] To achieve the above objectives, a second aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the content-guided no-reference video quality assessment method described above.

[0023] According to an embodiment of the present invention, a computer device stores a content-guided video quality evaluation program in a memory. When the content-guided video quality evaluation program is executed by a processor, it implements the above-described content-guided no-reference video quality evaluation method, which can utilize video content to guide the allocation of temporal attention and generate quality scores, thereby improving the accuracy of video quality evaluation. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a content-guided, no-reference video quality assessment method according to an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of a content-guided, no-reference video quality assessment method according to an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of the internal structure of a Transformer encoder according to an embodiment of the present invention. Detailed Implementation

[0027] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0028] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0029] like Figure 2 As shown, the video segment to be evaluated input to the model consists of T frames; stages 1-4 of SwinTransformerV2 are derived from the pre-trained model framework of SwinTransformerV2; GAP represents global average pooling; TP represents temporal pooling; 1×1Conv represents a convolutional layer with a kernel size of 1; 3×3Conv represents a convolutional layer with a kernel size of 3; K, Q, and V represent key, query, and value; GCN represents graph convolutional network, used to shield the influence of future frames on the current frame when aggregating multiple frames into a token in a sliding window; FC represents a fully connected layer; t represents the token; and p represents positional encoding. The final output of the model is the quality score of the input video segment to be evaluated.

[0030] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0031] Figure 1 This is a flowchart illustrating a content-guided, no-reference video quality assessment method according to an embodiment of the present invention. Figure 1 As shown, the content-guided no-reference video quality assessment method of this invention includes the following steps:

[0032] S101, Obtain the video segment to be evaluated.

[0033] As a specific example, the video segment to be evaluated consists of T frames.

[0034] S102, construct a feature extraction network based on SwinTransformerV2, and input the video segment to be evaluated into the feature extraction network in order to obtain the frame-level features and semantic content features corresponding to the video segment to be evaluated.

[0035] As an example, a feature extraction network based on SwinTransformerV2 is constructed, and the video segment to be evaluated is input into the feature extraction network to obtain the frame-level features and semantic content features corresponding to the video segment to be evaluated. The process includes: S21, pre-training SwinTransformerV2 on ImageNet to obtain the feature extraction network; using the feature extraction network to extract features frame by frame from the video segment to be evaluated to obtain the depth features corresponding to four different stages; S22, using a pyramid pooling strategy to compress the depth features of the first three stages into the same features as the fourth stage using global average pooling, and concatenating the depth features corresponding to the four different stages, then compressing the feature channels through convolution, and using global average pooling to compress them into a one-dimensional vector for use as frame-level features; S23, using temporal pooling to compress the temporal scale of the depth features corresponding to the fourth stage to one for use as semantic content features.

[0036] As a specific implementation, in S21 above, the torchvision.models library is used to call the pre-trained SwinTransformerV2Tiny model; the T frames of the video segment to be evaluated are packaged into a tensor and input into SwinTransformerV2; the torchvision's built-in create_feature_extractor tool is used to extract the block outputs of features.1, features.3, features.5, and features.7 from the SwinTransformerV2 model, which are the four stages of depth features of the video frame extracted by SwinTransformerV2. In S22, a pyramid pooling strategy is used to compress the features of the first three stages into the same form as the features of the fourth stage using global average pooling, and then they are concatenated. In a specific application scenario, the concatenated frame-level depth feature vector has a feature dimension of 1440 dimensions, with a width and height of 30 and 17, respectively. Then, a convolutional layer with a kernel size of 1 is used to compress the feature channels to 512 dimensions. Global average pooling is then used again to compress its width and height to 1, resulting in a 1-dimensional vector (per frame) of length 512, which serves as the frame-level feature of the video. In S23, temporal pooling averages and compresses the feature maps of each frame output from the fourth stage along the time dimension (i.e., the frame dimension), synthesizing them into a single-frame feature map, which is considered the representation of the video's main content.

[0037] S103, Construct a temporal information capture network based on a Transformer encoder, so as to obtain the temporal dependencies between frame-level features according to the temporal information capture network.

[0038] As an example, a temporal information capture network based on a Transformer encoder is constructed to obtain the temporal dependencies between frame-level features according to the temporal information capture network, including: S31, using a sliding window based on a graph convolutional network to compress several frames in the frame-level features into a token to obtain multiple tokens; S32, using Sinusoidal position encoding to obtain the position information corresponding to multiple tokens, and adding it to the corresponding token to obtain multiple tokens carrying position information.

[0039] It should be noted that each token is obtained according to the following formula:

[0040] token=TP(Aδ(AXW1)W2) (1)

[0041] Where TP(·) represents temporal pooling; A represents the adjacency matrix; δ(·) represents the activation function; X represents the feature combination of several frames within the input sliding window; and W1 and W2 represent trainable weight matrices.

[0042] Each token carrying location information is obtained using the following formula:

[0043] p k,2i =sin(k / 10000) 2i / d (2)

[0044] p k,2i+1 =cos(k / 10000) 2i / d (3)

[0045] token k =token k +p k (4)

[0046] Where i represents the dimension index in the position encoding matrix p, 2i represents even positions, and 2i+1 represents odd positions; d represents the embedding dimension in the Transformer model, k represents the k-th token, and sin(·) and cos(·) represent sine and cosine calculations, respectively.

[0047] As a specific implementation, two layers of graph convolutional networks are encapsulated within the sliding window in S31. Assuming the total number of frames in the video segment input to the model is T (T = video segment duration * frame rate), then after step S22, T one-dimensional vectors of length 512 are obtained, which are the frame-level features of the video. Assuming that m one-dimensional vectors are fed into the sliding window each time, the sliding window slides on the sequence T with a stride of m (if the end of the sequence T is less than m, then zero vectors are used to pad it. Generally speaking, m is set to half the frame rate. That is, assuming the video format is 10 seconds 30fps (T = 300), then m can be set to 15, which is equivalent to feeding half a second of sequence into the sliding window each time). Then in formula (1), the size of X is (m, 512), the size of A is (m, m), and the sizes of W1 and W2 are (512, 128) and (128, 64), respectively. A is designed as a matrix with all 1s in the lower triangle (including the diagonal) and all 0s in the upper triangle. This allows its product with X to mask all subsequent frames, meaning the current perception of the video is only influenced by previously seen frames (segments), while unseen frames (segments) need to be masked. The activation function δ(.) is designed as ReLU to provide a non-linear mapping between layers. After two layers of graph convolution, temporal pooling is used to compress m to 1. The final sliding window outputs a token of length 64. Furthermore, after all frame-level features have been compressed into tokens by the sliding window, all tokens are averaged to form the sequence start embedding t0, indicating the start position of the sequence.

[0048] As a specific implementation, in S32, it is assumed that after the sliding window finishes sliding on the video sequence, a total of n tokens are generated (where n = T / m + 1). Sinusoidal position encoding is performed on the indices of these n tokens, resulting in n one-to-one corresponding position information points, indicating the position of a certain token in the sequence. Since the position encoding and the token have the same size, the position information is carried by the token through direct addition.

[0049] As a specific embodiment, in S33, the token is input into the Transformer encoder module as V and K, while Q is generated by the hypernet in S104.

[0050] As an example, a Transformer encoder includes at least one Transformer encoder module, which includes at least one attention head and one feedforward neural network.

[0051] It should be noted that, as Figure 3 As shown, the masking multi-head attention layer is represented by the following formula:

[0052]

[0053] head i =Attention(KW i K QW i Q VW i V (6)

[0054] Output = Concat(head1, ..., head) h W i O (7)

[0055] Where i represents the i-th attention head (1≤i≤h, where h is the total number of attention heads). KQV represent key, query, and value, respectively. head represents the attention head. T It is the matrix transpose, d k It is the embedding dimension, which is 64 in some embodiments. It is the activation function SoftMax. K W Q W V W OThese represent the key, query, value, and trainable weight matrices for the output layer, respectively. `mask` is a mask computation, designed as a triangular matrix operation, also used to provide positional masking, ensuring that the current frame (token) is only temporally dependent on the previous frame (token). `Attention(.)` represents the attention layer, and `Concat(.)` is the concatenation operation. In some embodiments, four attention heads are designed, i.e., h = 4. The outputs of these four attention heads are concatenated and then compressed back to 64 using the weight matrix. The subsequent structure is the standard Transformer structure; residual connections and layer normalization operations are used to preserve feature information across trainable layers and to normalize the data distribution. The forward propagation layer consists of two fully connected layers; in some embodiments, the hidden layer has 128 neurons, and the output layer has 64 neurons.

[0056] In some embodiments, there are two Transformer encoder modules connected in series, with identical internal structure and data format.

[0057] S104, construct a hypernetwork for transmitting semantic content features, and use the hypernetwork to reshape the semantic content features into queries and quality-aware heads in the Transformer encoder, respectively.

[0058] As an example, the hypernetwork used to deliver semantic content features includes several convolutional layers, fully connected layers, and pooling layers.

[0059] In other words, several convolutional layers, fully connected layers, and pooling layers are used to reshape the semantic content features into query Q in the Transformer encoder and the final quality-aware head, respectively.

[0060] As a specific implementation, the output feature size of the fourth stage of SwinTransformerV2 is (B, T, C, H, W), where B is the batch size, i.e., the number of video segments input to the model; T is the number of frames; C is the number of feature channels; and H and W are the height and width of the feature map. First, temporal pooling is used to compress the frames, i.e., T, to 1, resulting in a video content feature size of (B, C, H, W). Then, three consecutive convolutional layers are used to compress the feature channels to 64, resulting in a feature size of (B, 64, H, W). The kernel size of each of the three convolutional layers is 1, with 512, 128, and 64 kernels respectively. Next, two consecutive convolutional layers with a kernel size of 3 and a stride of 2 are used to compress the height and width. Global average pooling is then used to further compress the width and height to 1, resulting in a feature vector size of (B, 64), consistent with the token. This vector is broadcast to the same size as the key-value pair and used as the query Q input to the Transformer encoder. Furthermore, global average pooling is directly used to compress the height and width of the video content features to 1, and then two fully connected networks are used to generate the weights and biases of the quality-aware head. The first fully connected network has 64 neurons in both the hidden and output layers, and the weights have a size of (B, 64, 1). The second fully connected network has 64 neurons in the hidden layer and 1 in the output layer, and the biases have a size of (B, 1, 1).

[0061] S105, the temporal dependencies between frame-level features and the query are input into the Transformer encoder to obtain frame-level quality features with temporal dependencies guided by video content.

[0062] S106, a quality-aware head is used to predict frame-level quality features to obtain the video quality score corresponding to the video segment to be evaluated.

[0063] As one embodiment, a quality-aware head is used to predict frame-level quality features to obtain the video quality score corresponding to the video segment to be evaluated. This includes: concatenating the frame-level quality features along the frame dimension and performing temporal pooling processing before inputting them into the quality-aware head for prediction; and the quality-aware head performing weighted multiplication and bias addition on the frame-level quality features before outputting the video quality score corresponding to the video segment to be evaluated.

[0064] In other words, the n-1 tokens output by the last Transformer encoder module are first temporally pooled (the first token, t0, is discarded because it comes from the artificial sequence start embedding in S31), resulting in a tensor of size (B, 1, 64), which is the frame-level quality feature with temporal dependencies guided by the video content; then the frame-level quality feature is fed into the aforementioned quality-aware head, multiplied by the weights, and then added to the bias; the final output tensor has a size of (B), which is the predicted video quality score (a batch of B video segments is input, and B corresponding quality scores are output).

[0065] It's important to note that the human visual system exhibits significant stimuli in response to preferred video content; specifically, for human observers, distortion in a video segment (frame) is only meaningful if the content of that segment (frame) is meaningful. Most existing technical solutions neglect to explore "meaningful" video segments and allow them to impact the overall video quality.

[0066] As an example, the feature extraction network based on SwinTransformerV2, the temporal information capture network based on Transformer encoder, and the supernetwork for conveying video content features are trained with the following parameter settings: the SwinTransformerV2-based feature extraction network is initialized using publicly available pre-trained weights from SwinTransformerV2, and the remaining trainable parameters in the model are randomly initialized; the mean absolute error is used as the loss function to measure the difference between the predicted score and the true quality score; the model parameters are updated using the AdamW optimizer with an initial learning rate of 1e-3, and a total of 300 iterations are trained.

[0067] As a specific implementation, firstly, the pre-trained weights publicly available from SwinTransformerV2 are downloaded to initialize the feature extraction network based on SwinTransformerV2. The remaining trainable parameters in the model are then randomly initialized using random samples from a standard normal distribution. Next, the mean absolute error is used as the loss function to measure the difference between the predicted score and the true quality score, expressed by the following formula:

[0068]

[0069] Where loss is the prediction loss, and y is the quality score of the video segment predicted by the model. It is the true quality score of the video clip, ‖.‖ l1This is the L1 norm. As a preferred implementation, mean absolute error (MAE) is widely used in quality assessment. This method eliminates the influence of extreme outliers on the gradient and also provides some gradient at the end of model training. Finally, the AdamW optimizer is used to update the model parameters, with an initial learning rate of 1e-3, and a total of 300 iterations are performed. As a preferred implementation, cosine annealing is used to update the learning rate, where the cosine half-cycle is set to one-tenth of the total number of iterations.

[0070] It should be noted that all modules of this invention are not dependent on the format of the input video. That is, provided that the hardware device allows it, this invention is applicable to video input of any size, any duration, and any resolution, and can generalize well among various types of videos.

[0071] In summary, the content-guided no-reference video quality assessment method proposed in this invention utilizes deep learning and attention mechanisms, combined with multi-scale features, semantic content features, and temporal dependencies of the video, to more comprehensively and accurately evaluate video quality. By constructing a feature extraction network, a temporal information capture network, and a supernetwork for transmitting semantic content features, it is possible to use video content to guide the allocation of temporal attention and generate quality scores, thereby improving the accuracy of video quality assessment.

[0072] In addition, this invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the content-guided no-reference video quality assessment method described above.

[0073] According to an embodiment of the present invention, a computer device stores a content-guided video quality evaluation program in a memory. When the content-guided video quality evaluation program is executed by a processor, it implements the above-described content-guided no-reference video quality evaluation method, which can utilize video content to guide the allocation of temporal attention and generate quality scores, thereby improving the accuracy of video quality evaluation.

[0074] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0078] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0079] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0080] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0081] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0082] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0083] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0084] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0085] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A content-guided, no-reference video quality assessment method, characterized in that, Includes the following steps: Obtain the video clips to be evaluated; A feature extraction network based on SwinTransformerV2 is constructed, and the video segment to be evaluated is input into the feature extraction network to obtain the frame-level features and semantic content features corresponding to the video segment to be evaluated; A temporal information capture network based on a Transformer encoder is constructed so as to obtain the temporal dependencies between the frame-level features according to the temporal information capture network; A hypernetwork is constructed to transmit the semantic content features, and the hypernetwork is used to reshape the semantic content features into the query and quality-aware head in the Transformer encoder, respectively. The temporal dependencies between the frame-level features and the query are input into the Transformer encoder to obtain frame-level quality features with temporal dependencies guided by video content. The quality-aware head is used to predict the frame-level quality features to obtain the video quality score corresponding to the video segment to be evaluated. Specifically, a feature extraction network based on SwinTransformerV2 is constructed, and the video segment to be evaluated is input into the feature extraction network to obtain the frame-level features and semantic content features corresponding to the video segment to be evaluated, including: The feature extraction network is obtained by pre-training SwinTransformerV2 on ImageNet; The feature extraction network is used to extract features frame by frame from the video segment to be evaluated, so as to obtain the depth features corresponding to four different stages; The pyramid pooling strategy is used to compress the depth features of the first three stages into the same features as the fourth stage using global average pooling. The depth features corresponding to the four different stages are concatenated and then compressed into feature channels through convolution. Global average pooling is then used to compress them into a one-dimensional vector so that they can be used as the frame-level features. Temporal pooling is used to compress the time scale of the deep features corresponding to the fourth stage into one, so as to serve as the semantic content features; Specifically, a temporal information capture network based on a Transformer encoder is constructed to obtain the temporal dependencies between the frame-level features, including: A sliding window based on a graph convolutional network is used to compress several frames in the frame-level features into a single token in order to obtain multiple tokens. The location information corresponding to the multiple tokens is obtained by using Sinusoidal location encoding, and then added to the corresponding token to obtain multiple tokens carrying location information.

2. The content-guided no-reference video quality assessment method as described in claim 1, characterized in that, Each token is obtained using the following formula: in, TP ( () indicates sequential pooling; A Represents the adjacency matrix; ( () represents the activation function; X This represents a combination of features from several frames within the input sliding window. W 1 and W 2 represents a trainable weight matrix.

3. The content-guided no-reference video quality assessment method as described in claim 2, characterized in that, Each token carrying location information is obtained using the following formula: in, i Represents the position encoding matrix p Dimension index in, 2 i Indicates even digits, 2 i +1 indicates an odd-numbered position; d This represents the embedding dimension in the Transformer model. k Indicates the first k token, sin( ) and cos( ) represent the calculation of sine and cosine respectively.

4. The content-guided no-reference video quality assessment method as described in claim 3, characterized in that, The Transformer encoder includes at least one Transformer encoder module, and the Transformer encoder module includes at least one attention head and one feedforward neural network.

5. The content-guided no-reference video quality assessment method as described in claim 1, characterized in that, The hypernetwork used to transmit the semantic content features includes several convolutional layers, fully connected layers, and pooling layers.

6. The content-guided no-reference video quality assessment method as described in claim 1, characterized in that, The quality-aware head is used to predict the frame-level quality features to obtain the video quality score corresponding to the video segment to be evaluated, including: The frame-level quality features are concatenated along the frame dimension and then subjected to temporal pooling before being input into the quality-aware head for prediction. The quality-aware head performs weighted multiplication and bias addition on the frame-level quality features and then outputs the video quality score corresponding to the video segment to be evaluated.

7. The content-guided no-reference video quality assessment method as described in claim 1, characterized in that, The feature extraction network based on SwinTransformerV2, the temporal information capture network based on Transformer encoder, and the supernetwork for transmitting video content features are trained with the following parameter settings: the feature extraction network based on SwinTransformerV2 is initialized using the pre-trained weights publicly available in SwinTransformerV2, and the remaining trainable parameters in the model are randomly initialized; the mean absolute error is used as the loss function to measure the difference between the predicted score and the true quality score; the model parameters are updated using the AdamW optimizer with an initial learning rate of 1e-3, and a total of 300 iterations are trained.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the content-guided no-reference video quality assessment method as described in any one of claims 1-7.

Images