A video quality evaluation method and device based on multi-dimensional graph association and a medium

Abstract

This invention discloses a video quality assessment method, device, and medium based on multi-dimensional graph association, relating to the field of video quality assessment technology. The method includes: acquiring the video to be evaluated and preprocessing it to obtain a video tensor; inputting the video tensor into a video quality feature extraction network to extract spatiotemporal quality features; constructing sub-attribute branches corresponding to multiple quality dimensions, inputting the spatiotemporal quality features into these branches to obtain sub-attribute features for multiple quality dimensions; converting these sub-attribute features into quality attribute nodes, and constructing a multi-dimensional quality association graph based on these nodes; inputting the multi-dimensional quality association graph into a graph association module to obtain an overall quality representation; and inputting the overall quality representation into a regression prediction layer to obtain an overall quality score. This invention improves the prediction accuracy of overall video quality.

Description

Technical Field

[0001] This invention belongs to the field of video quality evaluation technology, specifically relating to a video quality evaluation method, device, and medium based on multi-dimensional graph correlation. Background Technology

[0002] With the rapid development of applications such as short videos, live streaming, cloud gaming, online conferencing, and video surveillance, video content may be affected by various factors during acquisition, compression, transmission, storage, and playback, including blurring, noise, compression artifacts, color distortion, timing jitter, and unstable frame rates. How to automatically evaluate the subjective perceived quality of videos has become a crucial issue in video processing, content distribution, and multimedia system optimization.

[0003] Existing video quality assessment methods typically input the video to be evaluated into a deep neural network, extract spatiotemporal features through a video quality assessment backbone, and then directly regress an overall quality score, such as MOS (Mean Opinion Score) or overall quality score. These methods can learn the mapping relationship between video content, distortion type, and subjective quality to some extent, but they still have the following shortcomings: First, most existing methods use the overall quality score as the primary supervised objective, lacking explicit modeling of different quality attributes or sub-dimensions. For example, factors such as color, noise, blur, artifacts, and temporal stability in a video all affect subjective quality, but these factors influence overall quality in different ways. If the overall score is regressed directly from a single backbone feature, the model may struggle to fully utilize the structural relationships between different quality dimensions.

[0004] Second, existing methods are insufficient in modeling the interrelationships between multidimensional quality attributes. Real-world video quality degradation is rarely caused by a single factor. For example, compression can simultaneously introduce block artifacts, texture loss, and edge blurring; low-light scenes can simultaneously cause increased noise and color shift. Different quality attributes influence each other, and simply using ordinary feature stitching or simple MLP (multilayer perceptron) fusion is insufficient to fully represent the dynamic dependencies between attributes.

[0005] Third, existing video quality assessment methods have weak interpretability. Single-branch models typically only output an overall quality score, making it difficult to explain which quality dimensions primarily influence this score, and also hindering further analysis of the correlations between different quality dimensions. This limits their application in video enhancement, coding parameter optimization, quality diagnostics, and content distribution strategies. Summary of the Invention

[0006] The present invention aims to at least solve the above-mentioned technical problems existing in the prior art. To this end, the present invention provides a video quality evaluation method, device and medium based on multi-dimensional graph correlation.

[0007] The first aspect of this invention proposes a video quality assessment method based on multi-dimensional graph correlation, the method comprising: The video to be evaluated is acquired and preprocessed to obtain a video tensor. The video tensor is input into a video quality feature extraction network to extract the spatiotemporal quality features of the video to be evaluated; Multiple sub-attribute branches corresponding to multiple quality dimensions are constructed, and the spatiotemporal quality features are input into the multiple sub-attribute branches to obtain sub-attribute features of multiple quality dimensions; the quality dimensions include color quality dimension, noise quality dimension, artifact quality dimension, blur quality dimension and temporal quality dimension; The multiple sub-attribute features are converted into quality attribute nodes, and a multi-dimensional quality association graph is constructed based on the quality attribute nodes; The multidimensional quality association graph is input into the graph association module to obtain the overall quality characterization; the graph association module is used to propagate node information between nodes of different quality attributes. The overall quality characterization is input into the regression prediction layer to obtain the overall quality score of the video to be evaluated.

[0008] Optionally, the step of converting multiple sub-attribute features into quality attribute nodes and constructing a multi-dimensional quality association graph based on the quality attribute nodes includes: A multidimensional quality association graph is constructed using multiple sub-attribute features as quality attribute nodes and the association relationships between the quality attribute nodes as edges. The association relationships are fully connected relationships, relationships based on attribute similarity, relationships based on attribute semantic association, or relationships dynamically generated based on learnable attention weights. The edge weights between the quality attribute nodes are obtained automatically based on the features of the quality attribute nodes.

[0009] Optionally, inputting the multidimensional quality correlation graph into the graph correlation module to obtain the overall quality characterization includes: In the graph association module, a multi-head attention mechanism is adopted. For each target node in the multidimensional quality association graph, the target node is used as the query and all nodes in the multidimensional quality association graph are used as keys and values. The association strength between each target node and all nodes is obtained through attention weights. After processing the association strength through multi-head attention propagation, residual connection, normalization, and feedforward transformation, an updated node representation is obtained; the updated node representation is the output of the graph association module. If the multidimensional quality correlation graph does not have an overall node, the updated node representations are fused to obtain the overall quality representation. If the multidimensional quality association graph is configured with an overall node, the overall node representation after propagation through the graph association module is used as the overall quality representation; the overall node is a node constructed based on the spatiotemporal quality features.

[0010] Optionally, fusing the updated node representations to obtain the overall quality representation includes: The updated node representations are then subjected to attention pooling, weighted pooling, or concatenation to obtain the overall quality representation.

[0011] Optionally, after obtaining the overall quality score of the video to be evaluated, the method further includes: Output the sub-attribute features obtained based on multiple sub-attribute branches.

[0012] Optionally, the video quality feature extraction network is a FastVQA-style backbone network; the step of inputting the video tensor into the video quality feature extraction network to extract the spatiotemporal quality features of the video to be evaluated includes: The video tensor is input into a FastVQA-style backbone network for spatiotemporal encoding, and spatiotemporal encoded features are output. The spatiotemporal coding features are pooled or aggregated to obtain the spatiotemporal quality features.

[0013] Optionally, the preprocessing of the video to be evaluated includes: The video to be evaluated is sampled using a FastVQA-style quality-preserving spatiotemporal segment construction method based on uniform grid local micro-block sampling to obtain multiple video segments. Spatial cropping, resizing, normalization, and tensorization are performed on video frames from multiple video segments to obtain video tensors.

[0014] Optionally, it also includes: During the training phase, multiple sub-attribute branches are trained using a dataset with sub-attribute labels to obtain well-trained sub-attribute branches. Load the weights of the trained sub-attribute branches, and use the sub-attribute features output by each sub-attribute branch as input to the graph association module for the quality attribute nodes; In the overall quality score prediction stage, the parameters of the trained sub-attribute branches are frozen, and the graph association module and the regression prediction layer are trained under supervision using the overall quality score labels; the loss function of the supervised training is regression loss, correlation loss or ranking loss.

[0015] A second aspect of the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the video quality evaluation method as described in the first aspect.

[0016] A third aspect of the present invention provides a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the video quality evaluation method as described in the first aspect.

[0017] The present invention has at least the following beneficial effects: 1. This invention constructs different sub-attribute features as graph nodes and learns the dynamic relationship between attribute nodes through a graph association module. Compared with simple splicing or ordinary MLP fusion, it is more suitable for describing the mutual influence between multiple quality degradation factors. This invention extends the overall video quality evaluation task from a single overall regression to a multi-dimensional attribute association modeling task, which can make full use of the influence of different quality dimensions on the overall subjective quality, thereby improving the prediction accuracy of overall video quality. 2. For video quality evaluation tasks with multiple quality attribute labels, this invention can construct corresponding quality attribute nodes based on the attribute label definition, and learn the relationship between attribute nodes through the graph association module, which has good structural scalability. Attached Figure Description

[0018] Figure 1 This is a flowchart of a video quality evaluation method based on multi-dimensional graph correlation provided in an embodiment of the present invention. Figure 1 ; Figure 2 This is a flowchart of a video quality evaluation method based on multi-dimensional graph correlation provided in an embodiment of the present invention. Figure 2 ; Figure 3 This is a schematic diagram of the video quality feature extraction network in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the process of obtaining quality attribute nodes in an embodiment of the present invention; Figure 5 This is a schematic diagram of the graph association module in an embodiment of the present invention; Figure 6 This is a flowchart of the model training process in an embodiment of the present invention. Detailed Implementation

[0019] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0020] This invention provides a video quality evaluation method based on multi-dimensional graph correlation, such as... Figure 1 and Figure 2 As shown, it includes the following steps: Step 101: Obtain the video to be evaluated and preprocess it to obtain the video tensor.

[0021] In one possible implementation, the preprocessing of the video to be evaluated includes: Step 1011: The video to be evaluated is sampled using a FastVQA-style quality-preserving spatiotemporal segment construction method based on uniform grid local micro-block sampling to obtain multiple video segments.

[0022] Step 1012: Perform spatial cropping, resizing, normalization, and tensorization on the video frames in the multiple video segments to obtain video tensors.

[0023] Specifically, the video to be evaluated is first acquired. This video can be user-generated content video, short internet video, compressed video, enhanced video, or other videos requiring quality evaluation. Segment sampling is performed on the video to be evaluated to obtain several video segments. In this embodiment of the invention, a FastVQA-style quality-preserving spatiotemporal segment construction method based on uniform grid local micro-block sampling can be used to sample video segments from the video to be evaluated that reflect both local and global video quality. Then, the video frames are spatially cropped, resized, normalized, and tensorized to obtain a video tensor that can be input into a deep neural network.

[0024] Step 102: Input the video tensor into the video quality feature extraction network to extract the spatiotemporal quality features of the video to be evaluated.

[0025] In one possible implementation, the video quality feature extraction network is a FastVQA-style backbone network; the step of inputting the video tensor into the video quality feature extraction network to extract the spatiotemporal quality features of the video to be evaluated includes: Step 1021: Input the video tensor into a FastVQA-style backbone network for spatiotemporal coding and output spatiotemporal coding features.

[0026] Step 1022: Pool or aggregate the spatiotemporal coding features to obtain the spatiotemporal quality features.

[0027] Specifically, the video tensor obtained through preprocessing is input as follows: Figure 3The video quality feature extraction network shown is used to extract spatiotemporal quality features from video tensors. These spatiotemporal quality features include information related to video content, texture, motion, distortion, and perceived quality.

[0028] In this embodiment of the invention, a FastVQA-style backbone network can be used as a video quality feature extraction network. This backbone network performs spatiotemporal encoding on the input video tensor, including spatial quality pattern encoding and temporal quality variation encoding. The final output is spatiotemporal quality features, which serve as the basis for subsequent overall quality prediction or attribute branch modeling. Alternatively, other deep neural networks capable of extracting video spatiotemporal quality features can also be used.

[0029] Step 103: Construct sub-attribute branches corresponding to multiple quality dimensions, input the spatiotemporal quality features into the multiple sub-attribute branches, and obtain sub-attribute features of multiple quality dimensions; the quality dimensions include color quality dimension, noise quality dimension, artifact quality dimension, blur quality dimension and temporal quality dimension.

[0030] Specifically, refer to Figure 4 Based on the spatiotemporal quality features of the video, multiple sub-attribute branches are constructed, each corresponding to a quality dimension. In this embodiment, the sub-attribute branches include a color branch, a noise branch, an artifact branch, a blur branch, and a temporal branch; the multiple quality dimensions include color quality dimension, noise quality dimension, artifact quality dimension, blur quality dimension, and temporal quality dimension. Each sub-attribute branch can be pre-trained using its corresponding sub-attribute label to extract features related to the corresponding quality dimension. After training, each branch outputs a sub-attribute feature.

[0031] Step 104: Convert the multiple sub-attribute features into quality attribute nodes, and construct a multi-dimensional quality association graph based on the quality attribute nodes.

[0032] In one possible implementation, the step of converting the multiple sub-attribute features into quality attribute nodes and constructing a multi-dimensional quality association graph based on the quality attribute nodes includes: A multidimensional quality association graph is constructed using multiple sub-attribute features as quality attribute nodes and the association relationships between the quality attribute nodes as edges. The association relationships are fully connected relationships, relationships based on attribute similarity, relationships based on attribute semantic association, or relationships dynamically generated based on learnable attention weights. The edge weights between the quality attribute nodes are obtained automatically based on the features of the quality attribute nodes.

[0033] Specifically, the sub-attribute features output from multiple sub-attribute branches are converted into graph nodes, namely quality attribute nodes, to construct a multi-dimensional quality correlation graph, such as... Figure 2 As shown, Figure 2 In the multidimensional quality association graph, the dashed lines represent learnable edge weight relationships between attributes. Let the first... The node features corresponding to each quality attribute are: Then the multidimensional quality correlation diagram It can be represented as:

[0034] in, This represents a set of nodes, specifically a set of quality attribute nodes. This represents the set of edges, which is the set of relationships between nodes with quality attributes.

[0035] In an embodiment of the present invention, It includes at least color nodes, noise nodes, artifact nodes, blur nodes, and temporal nodes. In this embodiment of the invention, an "overall" node can also be set to fuse information from multiple attribute nodes propagated in subsequent steps and output the overall quality prediction result.

[0036] The edge set can be a fully connected relationship, or it can be a dynamically generated relationship based on attribute similarity, attribute semantic relationship, or learnable attention weights. In this embodiment of the invention, the edge weights are not manually fixed, but are automatically learned based on node features during model training.

[0037] Step 105: Input the multidimensional quality association graph into the graph association module to obtain the overall quality characterization; the graph association module is used to propagate node information between nodes of different quality attributes.

[0038] In one possible implementation, the step of inputting the multidimensional quality correlation graph into the graph correlation module to obtain the overall quality characterization includes: Step 1051: In the graph association module, a multi-head attention mechanism is adopted. For each target node in the multidimensional quality association graph, the target node is used as the query and all nodes in the multidimensional quality association graph are used as keys and values. The association strength between each target node and all nodes is obtained through attention weights.

[0039] Step 1052: After processing the association strength through multi-head attention propagation, residual connection, normalization and feedforward transformation, the updated node representation is obtained; the updated node representation is the output of the graph association module.

[0040] Specifically, the expression for the node update process is:

[0041] In the formula, Indicates the first Node representation of each quality attribute node Indicates the first Node representation of each quality attribute node Indicates the first The quality attribute node is the first The association weights of each quality attribute node, where W represents the learnable transformation matrix. Indicates the updated number Each quality attribute node represents a quality attribute.

[0042] Step 1053: If the multidimensional quality correlation graph does not have an overall node, the updated node representations are fused to obtain the overall quality representation.

[0043] Step 1054: If the multidimensional quality correlation graph is configured with an overall node, the overall node representation after propagation through the graph correlation module is used as the overall quality representation; the overall node is a node constructed based on the spatiotemporal quality features.

[0044] In one possible implementation, fusing the updated node representations to obtain the overall quality representation includes: The updated node representations are then subjected to attention pooling, weighted pooling, or concatenation to obtain the overall quality representation.

[0045] Specifically, refer to Figure 5 The multidimensional quality correlation graph is input into the graph correlation module, which propagates information between different quality attribute nodes, enabling each attribute node to perceive the quality information of other attribute nodes. For example, a noise node can receive information from color nodes, artifact nodes, and blur nodes to characterize the combined effect of noise and other distortion factors. The graph correlation module employs a multi-head self-attention mechanism. For any attribute node, the model uses that node as the query (Q) and all attribute nodes as the key and value, calculating the correlation strength between that node and other attribute nodes through attention weights. After multi-head attention propagation, residual connections, normalization, and feedforward transformation, the updated node representation, i.e., the output of the graph correlation module, is obtained. The graph correlation module models the relationships between different quality dimensions. The graph correlation module can also employ one or more layers of graph neural networks, graph attention networks, or other learnable relationship propagation modules. This module learns the correlation weights between attribute nodes to achieve information interaction between different quality dimensions.

[0046] If the multidimensional quality correlation graph does not have an overall node, the quality attribute nodes in the multidimensional quality correlation graph are color nodes, noise nodes, artifact nodes, fuzzy nodes, and time-series nodes. After the multidimensional quality correlation graph is processed by the graph correlation module, the updated color node representation, noise node representation, artifact node representation, fuzzy node representation, and time-series node representation are obtained.

[0047] If the multidimensional quality correlation graph is set with an overall node, the nodes in the multidimensional quality correlation graph are color nodes, noise nodes, artifact nodes, fuzzy nodes, time-series nodes, and overall nodes. The overall node represents the overall quality. After the multidimensional quality correlation graph is input into the graph correlation module for processing, the updated color node representation, noise node representation, artifact node representation, fuzzy node representation, time-series node representation, and overall node representation are obtained.

[0048] The updated node representations output by the graph association module are fused. Fusion methods may include reading the overall node representation, weighted pooling of multiple attribute nodes, concatenation, or attention pooling. In other words, if the multidimensional quality association graph has an overall node, the overall node representation propagated by the graph association module is used as the overall quality representation; if the multidimensional quality association graph does not have an overall node, the updated node representations are fused using weighted pooling, concatenation, or attention pooling, and the fused representation is used as the overall quality representation.

[0049] It should be noted that, based on the requirements of video quality evaluation, implementers may also add other attribute nodes on the basis of the quality attribute nodes in the embodiments of the present invention, and the embodiments of the present invention do not impose specific restrictions on this.

[0050] Step 106: Input the overall quality characterization into the regression prediction layer to obtain the overall quality score of the video to be evaluated.

[0051] Specifically, the overall quality score of the video to be evaluated is obtained through a regression prediction layer.

[0052] In one possible implementation, after obtaining the overall quality score of the video to be evaluated, the method further includes: Output the sub-attribute features obtained based on multiple sub-attribute branches.

[0053] In this embodiment of the invention, the overall video quality score is used as the core output, and sub-attribute features can also be output as auxiliary explanatory information.

[0054] In one possible implementation, it also includes: During the training phase, multiple sub-attribute branches are trained using a dataset with sub-attribute labels to obtain well-trained sub-attribute branches. Load the weights of the trained sub-attribute branches, and use the sub-attribute features output by each sub-attribute branch as input to the graph association module for the quality attribute nodes; In the overall quality score prediction stage, the parameters of the trained sub-attribute branches are frozen, and the graph association module and the regression prediction layer are trained under supervision using the overall quality score labels; the loss function of the supervised training is regression loss, correlation loss or ranking loss.

[0055] Specifically, refer to Figure 6 During the training phase, the following training strategy can be adopted: First, train five sub-attribute branches separately using the FineVD training set with sub-attribute labels, so that each sub-attribute branch learns the discriminative features of the corresponding quality dimension; where sub-attribute labels include color, noise, artifacts, blur, and temporal sequence; second, load the trained sub-attribute weights, and use the features output by each sub-attribute branch as input to the graph association module of the multi-dimensional attribute node; third, freeze the sub-attribute branch parameters, and train only the graph association module and the regression prediction layer, using the overall quality label to supervise the training of the graph association module and the regression prediction layer, and the loss function can be regression loss, correlation loss, ranking loss, or a combination thereof. In other words, if the multi-dimensional quality association graph has an overall node, first train multiple sub-attribute branches, then freeze the output features of the sub-attribute branches, and use the output features of the sub-attribute branches as input features of the overall branch, i.e., the graph association module. After graph association, train the overall branch, i.e., train the graph association module, so that the model can make full use of the sub-attribute quality representation to improve the overall quality prediction performance. Use the FineVD dataset for validation, select the optimal model according to the validation metrics, and save the optimal model parameters.

[0056] During the inference phase, after inputting the video to be evaluated, the model in this embodiment of the invention sequentially performs video sampling, spatiotemporal quality feature extraction, sub-attribute node construction, graph association propagation, and overall quality regression, ultimately outputting an overall video quality evaluation score. This score can be used in scenarios such as video quality monitoring, live streaming quality monitoring, cloud gaming quality assessment, video encoding optimization, video enhancement effect evaluation, content distribution strategy adjustment, video transmission link quality diagnosis, and user experience quality analysis.

[0057] Using the methods described above, an overall quality score can be automatically generated for the input video. At the same time, quality explanations at the sub-attribute level can be provided, thereby helping the platform to determine the source of video quality degradation, optimize video processing strategies, and improve video service quality and user viewing experience.

[0058] To verify the effectiveness of the video quality assessment method in this embodiment of the invention, three experimental routes were compared on the FineVD dataset. All three routes used the same FineVD training / validation set partitioning, the same video sampling method, the same pre-trained weights from the Kinetics-400 dataset, the same number of training epochs, and the same overall evaluation caliber. They also employed the same small-shift window Transformer backbone network and a spatiotemporal attention mechanism for video quality feature extraction. Specifically, the number of training epochs was 25, the batch size was 8, the optimizer learning rate was 1e-4, and the weight decay was 0.05.

[0059] The first type of experimental route is the overall direct prediction route, which involves inputting the video into the video quality feature extraction network and then directly outputting the overall quality score through the overall head.

[0060] The second category is the sub-attribute MLP fusion route, which involves obtaining five sub-attribute features (color, noise, artifacts, blur, and time series) and overall branch features, fusing multiple sub-attribute features through MLP, and finally outputting the overall quality score.

[0061] The third category is the sub-attribute GNN (Graph Neural Network) graph association route, which is the method in this embodiment of the invention. Multiple sub-attribute features are used as graph nodes, and attribute relationships are propagated and fused through the graph association module, finally outputting the overall quality score. In this experiment, the graph association module adopts a two-layer graph association structure.

[0062] The comparison results of the three types of experimental routes are shown in Table 1 below.

[0063] Table 1

[0064] As shown in Table 1 above, compared to the overall direct prediction route, the sub-attribute MLP fusion route improves on SROCC (Spearman Rank-Order Correlation Coefficient), PLCC (Pearson Linear Correlation Coefficient), and KROCC (Kendall Rank-Order Correlation Coefficient), while reducing RMSE (Root Mean Square Error). The sub-attribute GNN graph association route continues to improve compared to the sub-attribute MLP fusion route, indicating that explicit modeling of graph relationships between multi-dimensional quality attributes can further improve the overall video quality assessment performance. SROCC, PLCC, and KROCC are used to measure the correlation between the predicted overall quality score and the true subjective quality score, while RMSE is used to measure the error between the predicted overall quality score and the true subjective quality score.

[0065] Specifically, based on optimal metrics, the sub-attribute MLP fusion route, compared to the overall direct prediction route, shows a 0.0053 improvement in SROCC, a 0.0043 improvement in PLCC, a 0.0069 improvement in KROCC, and a 0.0011 decrease in RMSE. The sub-attribute GNN graph association route, compared to the overall direct prediction route, shows a 0.0092 improvement in SROCC, a 0.0089 improvement in PLCC, a 0.0102 improvement in KROCC, and a 0.0023 decrease in RMSE. The sub-attribute GNN graph association route, compared to the sub-attribute MLP fusion route, shows a 0.0039 improvement in SROCC, a 0.0046 improvement in PLCC, a 0.0033 improvement in KROCC, and a 0.0012 decrease in RMSE. These results further demonstrate that the video quality assessment method based on multi-dimensional graph association in this embodiment of the invention, compared to direct overall prediction and ordinary MLP fusion, can effectively improve the overall quality prediction performance and has better accuracy, structural scalability, and interpretability.

[0066] In summary, in this embodiment of the invention, different sub-attribute features are constructed as graph nodes, and the relationship between attribute nodes is propagated and fused using a graph association module. This overcomes the problem that existing methods mainly rely on single overall regression and lack multi-dimensional quality attribute relationship modeling. Compared with simple splicing or ordinary MLP fusion, it is more suitable for describing the mutual influence between multiple quality degradation factors. By extending the overall video quality evaluation task from single overall regression to a multi-dimensional attribute association modeling task, the influence of different quality dimensions on the overall subjective quality can be fully utilized, thereby improving the prediction accuracy of overall video quality.

[0067] For video quality evaluation tasks with multiple quality attribute labels, the embodiments of the present invention construct corresponding quality attribute nodes according to the attribute label definitions, and learn the relationship between attribute nodes through a graph association module, which has good structural scalability.

[0068] In this embodiment of the invention, multiple quality attribute nodes are explicitly constructed before overall quality prediction, and the relationships between attribute nodes are learned. Therefore, it is possible to further analyze which quality attributes affect the overall quality score and the strength of the relationships between different quality attributes, which has a certain degree of interpretability.

[0069] In another embodiment of the present invention, an electronic device is also provided, the electronic device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the video quality evaluation method in the embodiment of the present invention.

[0070] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or at least one program being loaded and executed by a processor to implement the video quality evaluation method of the present invention.

[0071] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0072] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A video quality assessment method based on multi-dimensional graph association, characterized in that, include: The video to be evaluated is acquired and preprocessed to obtain a video tensor. The video tensor is input into a video quality feature extraction network to extract the spatiotemporal quality features of the video to be evaluated; Multiple sub-attribute branches corresponding to multiple quality dimensions are constructed, and the spatiotemporal quality features are input into the multiple sub-attribute branches to obtain sub-attribute features of multiple quality dimensions; the quality dimensions include color quality dimension, noise quality dimension, artifact quality dimension, blur quality dimension and temporal quality dimension; The multiple sub-attribute features are converted into quality attribute nodes, and a multi-dimensional quality association graph is constructed based on the quality attribute nodes; The multidimensional quality correlation graph is input into the graph correlation module to obtain the overall quality characterization; The graph association module is used to propagate node information between nodes with different quality attributes; The overall quality characterization is input into the regression prediction layer to obtain the overall quality score of the video to be evaluated.

2. The video quality evaluation method based on multi-dimensional graph association according to claim 1, characterized in that, The step of converting multiple sub-attribute features into quality attribute nodes and constructing a multi-dimensional quality association graph based on the quality attribute nodes includes: A multidimensional quality association graph is constructed using multiple sub-attribute features as quality attribute nodes and the association relationships between the quality attribute nodes as edges. The association relationships are fully connected relationships, relationships based on attribute similarity, relationships based on attribute semantic association, or relationships dynamically generated based on learnable attention weights. The edge weights between the quality attribute nodes are obtained automatically based on the features of the quality attribute nodes.

3. The video quality evaluation method based on multi-dimensional graph association according to claim 2, characterized in that, The step of inputting the multidimensional quality correlation graph into the graph correlation module to obtain the overall quality characterization includes: In the graph association module, a multi-head attention mechanism is adopted. For each target node in the multidimensional quality association graph, the target node is used as the query and all nodes in the multidimensional quality association graph are used as keys and values. The association strength between each target node and all nodes is obtained through attention weights. After processing the association strength through multi-head attention propagation, residual connection, normalization, and feedforward transformation, an updated node representation is obtained; the updated node representation is the output of the graph association module. If the multidimensional quality correlation graph does not have an overall node, the updated node representations are fused to obtain the overall quality representation. If the multidimensional quality association graph is configured with an overall node, the overall node representation after propagation through the graph association module is used as the overall quality representation; the overall node is a node constructed based on the spatiotemporal quality features.

4. The video quality evaluation method based on multi-dimensional graph association according to claim 3, characterized in that, The process of fusing the updated node representations to obtain the overall quality representation includes: The updated node representations are then subjected to attention pooling, weighted pooling, or concatenation to obtain the overall quality representation.

5. The video quality evaluation method based on multi-dimensional graph association according to claim 1, characterized in that, After obtaining the overall quality score of the video to be evaluated, the process further includes: Output the sub-attribute features obtained based on multiple sub-attribute branches.

6. The video quality evaluation method based on multi-dimensional graph association according to claim 1, characterized in that, The video quality feature extraction network is a FastVQA-style backbone network; the step of inputting the video tensor into the video quality feature extraction network to extract the spatiotemporal quality features of the video to be evaluated includes: The video tensor is input into a FastVQA-style backbone network for spatiotemporal encoding, and spatiotemporal encoded features are output. The spatiotemporal coding features are pooled or aggregated to obtain the spatiotemporal quality features.

7. The video quality evaluation method based on multi-dimensional graph association according to claim 1, characterized in that, The preprocessing of the video to be evaluated includes: The video to be evaluated is sampled using a FastVQA-style quality-preserving spatiotemporal segment construction method based on uniform grid local micro-block sampling to obtain multiple video segments. Spatial cropping, resizing, normalization, and tensorization are performed on video frames from multiple video segments to obtain video tensors.

8. The video quality evaluation method based on multi-dimensional graph association according to claim 4, characterized in that, Also includes: During the training phase, multiple sub-attribute branches are trained using a dataset with sub-attribute labels to obtain well-trained sub-attribute branches. Load the weights of the trained sub-attribute branches, and use the sub-attribute features output by each sub-attribute branch as input to the graph association module for the quality attribute nodes; In the overall quality score prediction stage, the parameters of the trained sub-attribute branches are frozen, and the graph association module and the regression prediction layer are trained under supervision using the overall quality score labels; the loss function of the supervised training is regression loss, correlation loss or ranking loss.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the video quality evaluation method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the video quality evaluation method as described in any one of claims 1-8.

Images