Method for non-reference quality assessment of uhd image and video based on graph convolution and spatio-temporal correlation

Abstract

The application discloses a method for evaluating the quality of UHD images and videos based on graph convolution and space-time correlation, which comprises the following steps: dividing an input ultra-high-definition image according to a preset grid, and obtaining node features with uniform dimensions through linear mapping. Based on the spatial distance between the normalized central coordinates, the node features and a normalized adjacency matrix are input into a multi-layer residual graph convolution network for feature propagation to obtain node representation. The node representation is subjected to attention weighted pooling to obtain a standardized quality prediction value. The quality prediction value is subjected to learnable affine calibration, and is subjected to inverse standardization according to statistical parameters of quality scores of a training set to obtain a final image quality score. The method can adapt to the high-dimensional input characteristics of ultra-high-definition images, and can combine the time sequence correlation between video frames to realize a high-precision and high-time-efficiency method for evaluating the quality of ultra-high-definition images and videos without reference.

Description

Technical Field

[0001] This invention relates to no-reference image quality assessment and graph convolutional neural network technology, and particularly focuses on no-reference ultra-high-definition image and video quality assessment methods and apparatus using graph convolutional neural networks and spatiotemporal correlation. Background Technology

[0002] With the popularization of Ultra-High Definition (UHD) imaging technology, 4K and 8K resolution image display and video production are gradually becoming the mainstream of next-generation broadcast television and streaming media content. Several television stations have officially launched 4K UHD channels, marking the entry of my country's UHD image and video technology into a stage of large-scale application. As display terminal resolutions improve and content production diversifies, objective and automated image and video quality assessment methods are becoming increasingly important in media production, transmission monitoring, and intelligent post-processing scenarios.

[0003] Blind Image Quality Assessment (BIQA) is a key research area in image quality assessment. However, this technology still faces many challenges when dealing with ultra-high-definition images and videos, and it is not yet possible to achieve reliable subjective quality prediction under high resolution and complex distortion conditions. (1) The prediction accuracy of BIQA models based on feature extraction and numerical regression is limited. These methods rely on manually constructed statistical features and transform domain indicators, such as contrast, gradient, and texture entropy, which are difficult to characterize the complex semantic content and cross-regional relationships in ultra-high-definition images, and are particularly unstable on datasets with large resolution variations. (2) BIQA models based on deep learning have structural limitations in ultra-high-definition input scenarios. Most mainstream convolutional neural network (CNN) methods process the input image size through patch-based sampling or image scaling to reduce computational complexity. However, the local block input mode cannot cover the complete spatial context, the model's receptive field is limited, and it is difficult to capture global semantic consistency, resulting in the predicted values ​​being concentrated in the middle range. The estimation bias for extremely high or low quality samples is obvious, and the image compression method will lose a lot of structural details of the original image, making it difficult to accurately express the quality of the original image. (3) The method based on graph convolutional network (GCN) can model the relationship between local regions of the image to a certain extent, but it is still difficult to make full use of its large field of view and high information density when processing ultra-high-definition images. Ultra-high-definition images usually contain far more local regions than ordinary images. There are both fine-grained local differences and long-distance cross-regional correlations between these regions. However, the existing GCN graphing method is too simplified, often relying on fixed-scale block division and a single adjacency strategy, which cannot cover the global relationship across regions and across semantic levels in ultra-high-definition images, resulting in a large amount of effective information not participating in the modeling. (4) The existing BIQA method has insufficient scalability in multi-resolution scenes. Most algorithms are designed for specific resolutions or specific distortion types, making it difficult to maintain consistent performance across standard definition, high definition (HD), and ultra-high definition (UHD) resolutions. This makes it difficult for BIQA models to adapt to the image quality assessment needs of different devices, bitrates, or compression conditions during deployment. (5) At the video quality assessment level, Blind Video Quality Assessment (BVQA) faces higher requirements in terms of accuracy and timeliness. The subjective quality of ultra-high definition video sequences depends not only on the clarity of a single frame but also on the combined effects of temporal factors such as inter-frame consistency, motion blur, compression artifacts, and content changes.Traditional BVQA methods based on optical flow or fixed-frame sampling have high computational complexity, making them difficult to meet real-time processing requirements and unable to accurately reflect the overall spatiotemporal quality perception of the video. Single-frame static feature models have inherent defects in temporal consistency modeling, leading to discrepancies between prediction results and subjective perception.

[0004] To address the aforementioned issues, there is an urgent need for a unified framework that integrates the image feature representation capabilities of deep learning with the spatial structure modeling advantages of graph convolutional neural networks. This framework should be able to adapt to the high-dimensional input characteristics of ultra-high-definition images and combine the temporal correlations between video frames to achieve a high-precision and high-timeliness no-reference ultra-high-definition image and video quality assessment method. Summary of the Invention

[0005] This invention proposes a no-reference quality prediction method and apparatus for ultra-high-definition images (UHD-BIQA), and then, based on this no-reference quality prediction method and apparatus for ultra-high-definition images, this invention extends to a no-reference quality assessment method and apparatus for ultra-high-definition video data (UHD-BVQA).

[0006] A method for quality assessment of unreferenced UHD images and videos based on graph convolution and spatiotemporal correlation includes the following steps:

[0007] The input ultra-high-definition image is divided into multiple local image patches according to a preset grid, and the center coordinates of each patch are normalized according to the image size. These image patches are then input into a pre-trained residual neural network to extract block-level depth features, and node features of a uniform dimension are obtained through linear mapping. Based on the spatial distance between the normalized center coordinates, a K-nearest neighbor graph structure is constructed for each node, generating an adjacency matrix and performing normalization. The node features and the normalized adjacency matrix are input into a multi-layer residual graph convolutional network for feature propagation to obtain node representations. Attention-weighted pooling is applied to the node representations to obtain image-level feature vectors, which are then passed through a regression network to output standardized quality prediction values. The quality prediction values ​​are subjected to learnable affine calibration and de-normalized based on the statistical parameters of the training set quality scores to obtain the final image quality score.

[0008] Furthermore, the image patch is obtained from a square cropping window, and its center coordinates are normalized to the image width and height. Interval.

[0009] Furthermore, the aforementioned The nearest neighbor graph structure is built based on the Euclidean distance between the center coordinates of image patches, and when the number of image patches is insufficient, the number of nearest neighbors is adaptively limited to no more than the total number of nodes minus one.

[0010] Furthermore, self-loop connections are added to the adjacency matrix, and a normalized adjacency matrix for graph convolution propagation is generated according to row normalization.

[0011] Furthermore, each layer of the residual graph convolutional network includes adjacency propagation, linear transformation, batch normalization, nonlinear activation, and residual fusion path. When the input and output dimensions are inconsistent, residual alignment is achieved through linear mapping.

[0012] Furthermore, the attention-weighted pooling obtains attention weights by performing learnable scoring on the features of each node and normalizing them, and then sums the node features by weight to obtain an image-level feature vector.

[0013] Furthermore, the quality prediction values ​​output by the regression network are scaled and offset by a learnable affine transformation to improve prediction stability under different data distributions.

[0014] Furthermore, the subjective quality scores are standardized during the training phase, and the prediction results are destandardized during the inference phase based on the mean and standard deviation of the quality scores in the training set.

[0015] Furthermore, the training process employs a joint loss function, which includes at least a regression error term, a correlation constraint term between the prediction and the true quality score, a ranking loss term based on paired samples, and a variance constraint term to suppress the collapse of the prediction distribution.

[0016] Furthermore, the ranking loss term adopts the ranking constraint form of pairwise samples within the batch, and the ranking stability of the prediction results is enhanced by setting a margin parameter.

[0017] Furthermore, the AdamW optimizer with grouped learning rates is used to set learning rates for the backbone network and other network modules respectively, and training is performed by combining the learning rate scheduling strategy of warm-up phase and cosine annealing.

[0018] Furthermore, at least one of mixed precision computation, gradient accumulation, and gradient clipping is employed during training to improve training stability and computational efficiency.

[0019] Furthermore, during the model evaluation phase, an exponential moving average method is used to integrate the model parameters to improve the stability of the prediction results.

[0020] Furthermore, during the testing phase, multiple inferences are performed on the input image, and the arithmetic mean of the quality prediction results obtained from the multiple inferences is taken, which includes at least one enhancement method of horizontally flipping the image patch.

[0021] Furthermore, the method is extended to video quality evaluation scenarios, by performing quality prediction on each frame or frame segment in a video sequence and temporally fusing the frame-level prediction results to obtain video-level quality prediction results.

[0022] By employing a complete technical process of "grid sampling - pre-trained residual feature extraction - K-nearest neighbor graph construction based on spatial coordinates - residual graph convolution - attention-weighted pooling - regression - learnable affine calibration", and combining a joint training objective function of "mean squared error, negative Pearson correlation, pairwise ranking hinge loss and prediction variance alignment regularization term", UHD-BIQA can effectively alleviate the problem of prediction distribution compression (variance collapse) in ultra-high-definition scenes by traditional methods, improve the ranking stability between samples, and improve the Spearman's Rank-Ordered Correlation Coefficient (SROCC) and Pearson's Linear Correlation Coefficient (PLCC) through engineering optimization processes such as warm-up-cosine learning rate scheduling, exponential moving average (EMA), mixed precision training, gradient accumulation and pruning, and test-time augmentation (TTA).

[0023] First, the input image is subjected to grid sampling processing, and the image is divided into... Mesh generation, determining the sampling starting point for each mesh cell, and clipping the side length within the mesh. square image blocks, obtained Each image patch is a local image patch; during the training phase, a random perturbation proportional to the side length of the image patch can be applied to the cropping starting point to achieve data augmentation; simultaneously, the center coordinates of each image patch are recorded and normalized to [0,1] spatial coordinates according to the image width. Used for subsequent graph structure construction.

[0024] A pre-trained residual network (preferably ResNet50 or ResNet34) is used to extract deep features for each image patch. These features are then processed by global average pooling to obtain a block-level feature vector (2048 dimensions for ResNet50), and finally mapped to a low-dimensional feature space through a linear projection layer. Dimension, forming the initial feature representation of the nodes. The backbone network can use weights pre-trained on general large-scale image data and be optimized end-to-end on this task to adapt to quality assessment.

[0025] In terms of graph structure construction, this invention employs a K-nearest neighbor (KNN) graph construction strategy based on spatial coordinates. For any two image block nodes... The spatial distance is defined as the Euclidean distance between normalized coordinates:

[0026]

[0027] For each node Select the closest Each node is used as its nearest neighbor set. (Adaptive adjustment when the number of nodes is insufficient) No more than Construct the adjacency matrix And add a self-loop to get To ensure numerical stability, this invention performs row normalization on the adjacency matrix to obtain a normalized adjacency matrix used for graph convolution propagation:

[0028]

[0029] in To prevent division by zero of constants.

[0030] This invention employs a multi-layer residual graph convolutional network for feature propagation. Each residual graph convolutional unit first performs adjacency propagation and linear transformation, and then combines batch normalization, ReLU nonlinear activation, and Dropout regularization to obtain propagated features. Subsequently, the input and output are fused in the form of weighted residuals to improve the stability of deep training.

[0031]

[0032] in For linear mapping parameters, Activated for ReLU It is a residual branch (an identity mapping when the dimensions are consistent, and a linear mapping when they are inconsistent). These are the residual fusion coefficients. Through multi-layer residual map convolution, the model is able to capture the spatial dependencies between image patches and enhance globally consistent representations.

[0033] To aggregate graph-level node features into image-level quality representations, this invention employs an attention-weighted pooling mechanism: firstly, a learnable gating scoring network is used to generate scalar weights for each node. The attention coefficients are then obtained by normalizing the values ​​at the node dimension using softmax. Finally, the node features are weighted and summed to obtain the image-level feature vector:

[0034]

[0035] in This is an image-level representation vector. Image-level features. The input consists of a two-layer sensing structure forming a regression head, and the output is a standardized spatial quality prediction value. Furthermore, this invention introduces a learnable affine calibration module to perform linear calibration on the regression output:

[0036]

[0037] in These are learnable parameters used to align the scale and offset of the prediction distribution, improving prediction stability under different data distributions. During the inference phase, the final subjective quality score is obtained through inverse standardization of the training set statistical parameters.

[0038]

[0039] in and These are the mean and standard deviation of the subjective quality scores for the training set, respectively.

[0040] In terms of training objective design, a joint optimization strategy is adopted, comprehensively considering regression accuracy, correlation consistency, ranking stability, and prediction distribution variance constraints. The objective quality score is... Standardized as:

[0041]

[0042] The joint loss function is defined as:

[0043] in For the batch Nippon correlation coefficient (denominator added) (To prevent numerical instability). The sorting loss adopts an in-batch pairwise hinge sorting method: randomly sampling several pairs of samples. ,make ,but

[0044]

[0045] in Here, represents the margin parameter. To suppress prediction variance collapse, this invention introduces a prediction variance alignment regularization term, making the within-batch prediction standard deviation approximate the unit standard deviation of the standardized target distribution: This alleviates the problem of reduced sample differentiation at both ends caused by the compression of the prediction distribution in ultra-high-definition scenarios.

[0046] Regarding optimization strategies, this invention employs the AdamW optimizer with grouped learning rates, where the backbone network learning rate is scaled proportionally to the head network to preserve pre-trained features; learning rate scheduling combines a warm-up phase and a cosine annealing strategy. During training, mixed precision and gradient accumulation are used to reduce memory pressure, and gradient pruning is performed before parameter updates to improve training stability. To enhance evaluation stability, exponential moving average (EMA) is used for ensemble evaluation of model parameters; simultaneously, during the validation and testing phases, test-time augmentation (TTA) is employed, averaging prediction results under augmentations such as horizontal flipping to improve evaluation robustness and repeatability.

[0047] Building upon the aforementioned Ultra-High Definition Image Quality Assessment (UHD-BIQA) model based on graph convolutional networks, this invention addresses the challenge of accurately reflecting the quality of ultra-high definition video sequences through single-frame static features. It proposes a referenceless ultra-high definition video quality prediction framework based on single-frame scoring and spatiotemporal correlation modeling. This framework is grounded in an image-level quality-aware model, incorporating temporal structure to capture inter-frame dependencies, ultimately generating an overall video quality prediction.

[0048] The overall model process includes four stages:

[0049] 1. Video content segmentation: Automatically divides the video into multiple semantically consistent segments based on inter-frame feature variation measurement;

[0050] 2. Single-frame quality prediction: Using an ultra-high-definition image quality assessment network, features are extracted for each frame and the frame quality is predicted;

[0051] 3. Segment-level temporal modeling: Based on keyframe detection and frame-level scoring, the overall score of the segment is obtained through Gaussian kernel weighting or LSTM network;

[0052] 4. Video-level aggregated prediction: Weighted or attention-based fusion of all segment scores to obtain the final video quality score.

[0053] Unlike traditional fixed-frame-interval sampling, this method first adaptively segments the video based on changes in video content. Let the total number of video frames be T. This represents the t-th frame of the video, and the frame sequence is denoted as:

[0054]

[0055] Calculate the content difference between adjacent frames :

[0056]

[0057] in, This represents the global features extracted by the pre-trained backbone network ResNet. Let L be the L2 norm, representing the Euclidean distance in the feature space.

[0058] like If a significant scene change is observed at that location, a new segment boundary is defined. This results in the segment set:

[0059]

[0060] in It is the starting frame number of the i-th video segment, which corresponds to Local peaks on the curve that exceed the threshold θ are "scene transition markers" in the video segmentation algorithm, while N is the automatically determined number of segments. This represents the i-th video segment.

[0061] Regarding the first The j-th frame of the segment Quality features are extracted using a graph convolution-based ultra-high-definition image quality assessment module:

[0062]

[0063] in, This represents a single-frame image quality assessment model. Represents the frame-level quality feature vector;

[0064] The single-frame prediction score is output through a fully connected regression layer. This represents the predicted frame quality score of the i-th segment in frame j. These are the weight parameters of the fully connected layer. For bias terms:

[0065]

[0066] At this point, a fragment is obtained. The set of scores for all frames within the set. Represents the set of frames for the i-th segment:

[0067]

[0068] These scores reflect the trend of inter-frame quality fluctuations over time.

[0069] To highlight the core visual changes within a segment, a keyframe detection mechanism based on content differences is introduced.

[0070] when Define a keyframe when the maximum value is reached in the current segment:

[0071]

[0072] Indicates the first Keyframe index for each segment The characteristic considered to best represent the quality change state of this segment is denoted as: The score is .

[0073] To obtain an overall quality perception of the fragments, two fragment-level modeling strategies are designed:

[0074] (a) Gaussian-weighted fusion;

[0075] It is assumed that the subjective perception influence of frames within a segment follows a Gaussian distribution with respect to the time center; therefore, a weighted sum of frame scores can be obtained using a Gaussian kernel.

[0076]

[0077] in Control the width of the weight distribution. For the first Each segment scores a point.

[0078] Frames near keyframes have higher weights, thus highlighting areas of visual change.

[0079] (b) LSTM fusion based on temporal relationships

[0080] Another more dynamic strategy is LSTM modeling based on frame sequences:

[0081]

[0082] in, This represents the hidden state and the memory state of the LSTM.

[0083] The segment score is an attention-weighted output of the hidden state:

[0084]

[0085]

[0086] in, , , q, For the LSTM attention layer parameters, These are frame-level attention weights. Both methods can integrate information from multiple frames within a segment to obtain a unified quality assessment, with the Gaussian kernel method having higher computational efficiency and the LSTM method having stronger expressive power.

[0087] Score all segments Perform global aggregation to obtain video quality predictions.

[0088] The integration method can be designed as content-related weighting or attention aggregation:

[0089] (a) Content-aware weighted average

[0090] Based on the duration of each segment Weights are determined by the characteristic variance (which reflects the complexity of image changes):

[0091]

[0092] The overall score for the video is:

[0093]

[0094] This method provides an overall predicted score for the video, taking into account both segment duration and content stability.

[0095] (b) Fragment-level attention aggregation

[0096] To adaptively learn the contribution of each segment to the overall quality, an attention-based fusion mechanism is introduced:

[0097]

[0098]

[0099] in This represents the "segment importance weights" learned by the model.

[0100] This method allows the model to automatically focus on segments with significant changes in image quality or severe distortion.

[0101] During training, the optimization objective includes three levels of constraints: frame-level, segment-level, and video-level.

[0102]

[0103] in:

[0104] .

[0105]

[0106]

[0107] , , This represents the three-layer MSE loss. The weights for each loss item. Attached Figure Description

[0108] Figure 1The overall flowchart of the no-reference ultra-high-definition image quality prediction method is as follows (mesh cropping → block feature extraction → KNN graph construction → residual graph convolution → gated attention pooling → regression and affine calibration → inverse normalization output).

[0109] Figure 2 This is a schematic diagram of grid sampling and image patch cropping, showing the image according to... The grid generates sampling locations and clips the side lengths. The image patch, and the normalization method for the center coordinates of the image patch.

[0110] Figure 3 This diagram illustrates the construction and normalization of a coordinate-based K-nearest neighbor graph (calculating Euclidean distance between coordinates → selecting the K nearest neighbors for each node → constructing a 0 / 1 adjacency matrix → adding self-loops → normalizing by row). ).

[0111] Figure 4 This is a schematic diagram of the residual graph convolutional unit structure (adjacency propagation). →Linear transformation→BN→ReLU→Dropout, and correlated with the residual branch by coefficients. (Integration).

[0112] Figure 5 This is a schematic diagram of a gated attention pooling structure (scalar weights are generated for each node's features and then normalized using softmax to obtain a graph-level vector). ).

[0113] Figure 6 This is a schematic diagram of the regression head and affine calibration, showing the regression output. The process of calibration and denormalization to obtain the MOS quality score.

[0114] Figure 7 This is a schematic diagram of the overall process of no-reference ultra-high-definition video quality assessment based on single-frame scoring and spatiotemporal correlation, showing the overall structure from video input, content segmentation, frame-level scoring to segment aggregation and video-level regression.

[0115] Figure 8 This is a schematic diagram of the video segmentation and keyframe detection process, demonstrating the steps to automatically determine segment boundaries and keyframe positions based on changes in inter-frame features.

[0116] Figure 9 This is a schematic diagram of a single-frame quality prediction structure, demonstrating the process of extracting frame-level features and outputting prediction scores using a graph convolution-based ultra-high-definition image quality assessment network.

[0117] Figure 10 This is a schematic diagram of Gaussian kernel weighted fusion within a segment, illustrating the mechanism by which the scores of multiple frames within a segment are fused using Gaussian weights to form the overall score of the segment.

[0118] Figure 11 This is a schematic diagram of the video-level score aggregation process, showing the relationship between the scores of multiple segments after attention weighting or content relevance weighting, and the fusion into the final video quality prediction result. Detailed Implementation

[0119] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0120] A referenceless ultra-high-definition image quality prediction method includes:

[0121] The input ultra-high-definition image is divided into multiple local image patches according to a preset grid, and the center coordinates of each image patch are normalized according to the image size. The image patches are then input into a pre-trained residual neural network to extract block-level depth features, and node features of a unified dimension are obtained through linear mapping. Based on the spatial distance between the normalized center coordinates, a K-nearest neighbor graph structure is constructed for each node, generating an adjacency matrix and performing normalization. The node features and the normalized adjacency matrix are input into a multi-layer residual graph convolutional network for feature propagation to obtain node representations. Attention-weighted pooling is applied to the node representations to obtain image-level feature vectors, which are then passed through a regression network to output standardized quality prediction values. The quality prediction values ​​are subjected to learnable affine calibration and de-standardized based on the statistical parameters of the training set quality scores to obtain the final image quality score.

[0122] Furthermore, the image patch is obtained from a square cropping window, and its center coordinates are normalized to the image width and height. Interval.

[0123] Furthermore, the aforementioned The nearest neighbor graph structure is built based on the Euclidean distance between the center coordinates of image patches, and when the number of image patches is insufficient, the number of nearest neighbors is adaptively limited to no more than the total number of nodes minus one.

[0124] Furthermore, self-loop connections are added to the adjacency matrix, and a normalized adjacency matrix for graph convolution propagation is generated according to row normalization.

[0125] Furthermore, each layer of the residual graph convolutional network includes adjacency propagation, linear transformation, batch normalization, nonlinear activation, and residual fusion path. When the input and output dimensions are inconsistent, residual alignment is achieved through linear mapping.

[0126] Furthermore, the attention-weighted pooling obtains attention weights by performing learnable scoring on the features of each node and normalizing them, and then sums the node features by weight to obtain an image-level feature vector.

[0127] Furthermore, the quality prediction values ​​output by the regression network are scaled and offset by a learnable affine transformation to improve prediction stability under different data distributions.

[0128] Furthermore, the subjective quality scores are standardized during the training phase, and the prediction results are destandardized during the inference phase based on the mean and standard deviation of the quality scores in the training set.

[0129] Furthermore, the training process employs a joint loss function, which includes at least a regression error term, a correlation constraint term between the prediction and the true quality score, a ranking loss term based on paired samples, and a variance constraint term to suppress the collapse of the prediction distribution.

[0130] Furthermore, the ranking loss term adopts the ranking constraint form of pairwise samples within the batch, and the ranking stability of the prediction results is enhanced by setting a margin parameter.

[0131] Furthermore, the AdamW optimizer with grouped learning rates is used to set learning rates for the backbone network and other network modules respectively, and training is performed by combining the learning rate scheduling strategy of warm-up phase and cosine annealing.

[0132] Furthermore, at least one of mixed precision computation, gradient accumulation, and gradient clipping is employed during training to improve training stability and computational efficiency.

[0133] Furthermore, during the model evaluation phase, an exponential moving average method is used to integrate the model parameters to improve the stability of the prediction results.

[0134] Furthermore, during the testing phase, multiple inferences are performed on the input image, and the arithmetic mean of the quality prediction results obtained from the multiple inferences is taken, which includes at least one enhancement method of horizontally flipping the image patch.

[0135] Furthermore, the method is extended to video quality evaluation scenarios, by performing quality prediction on each frame or frame segment in a video sequence and temporally fusing the frame-level prediction results to obtain video-level quality prediction results.

[0136] Implementation details

[0137] 1. Data Standardization and Sampling: During the training phase, the mean is calculated using the subjective quality score (MOS) of the training set. with standard deviation and standardize the labels to ,in To prevent division by zero constants, the model output is denormalized during the inference phase to obtain a subjective quality score. For input ultra-high-definition images, press The grid generates sampling points and the side length is clipped. A local square image patch is obtained. Each image block is recorded; the center coordinates of each image block are recorded and normalized to the image width and height. Two-dimensional coordinates During the training phase, the cutting starting point can be adjusted accordingly. Proportional random perturbation is used to augment the data; this perturbation is disabled during the validation and testing phases to maintain sampling determinism. Pixel normalization standardizes the RGB channels using preset mean and variance. Test-time enhancement (TTA) improves robustness by performing multiple inferences on the same input and taking the arithmetic mean of the predictions; in this embodiment, TTA includes at least horizontal flip enhancement of image patches and averaging of the outputs.

[0138] 2. Feature Extraction and Linear Projection: A pre-trained residual network is used as the backbone feature extractor to extract block-level deep feature vectors for each image patch (e.g., 2048 dimensions for ResNet50, or 512 dimensions for ResNet34, optionally). These block-level features are then mapped to a uniform dimension via a linear projection layer. The initial feature representation of the node is obtained. And form a node feature matrix. .

[0139] 3. Graph Construction and Normalization: Construct a K-nearest neighbor graph based on normalized coordinates. For any two nodes... Spatial distance is defined as: For each node Choose the closest one Build a nearest neighbor set for each node. Based on this, construct an adjacency matrix. When the number of nodes is insufficient, adaptive limiting is implemented. To ensure that nodes retain their own information, add self-loops to the adjacency matrix: Then, row normalization is performed on the adjacency matrix to obtain the normalized adjacency matrix:

[0140]

[0141] 4. Residual Image Convolution and Pooling: Stacking A single-layer residual graph convolutional network propagates node features. The single-layer residual graph convolution includes adjacency propagation, linear transformation, batch normalization, nonlinear activation, and Dropout regularization, and obtains the output through residual fusion.

[0142]

[0143] in It is the ReLU activation function. It is a residual branch mapping (when the input and output dimensions are the same, it is an identity mapping; when they are different, it is a linear mapping). These are the residual fusion coefficients. The propagated node representations are aggregated into image-level vectors through attention-weighted pooling. Specifically, a learnable scoring network outputs scalar weights for each node. Attention coefficients are obtained by softmax normalization. The image-level representation vector is obtained by weighted summation of the node features. :

[0144]

[0145] 5. Regression Output and Learnable Affine Calibration: Transforming Image-Level Vectors Input regression head; output standardized quality prediction To align the scale and offset of the predicted distribution, a learnable affine calibration module is introduced:

[0146] in These are learnable parameters. During the inference phase, the parameters are determined based on the statistics of the training set. Destandardization for final quality score .

[0147] 6. Loss Function and Optimized Training: The training objective employs a joint optimization strategy, comprehensively considering regression accuracy, correlation consistency, ranking stability, and prediction variance constraints.

[0148] in The intra-batch Pearson correlation coefficient; the ranking loss is the intra-batch pairwise hinge ranking loss: randomly sampled sample pairs. ,make ,but: ,in Here is the margin parameter. To suppress prediction output variance collapse, a prediction variance alignment regularization term is introduced:

[0149] This regularization term utilizes the unit scale characteristics of the label standardization to ensure that the predicted distribution maintains a reasonable variance, thereby improving the discrimination and ranking stability of extreme samples.

[0150] The optimizer uses AdamW, and learning rates are set in groups for the backbone network parameters and the parameters of other modules. The backbone network learning rate is scaled proportionally to the regression and graph modules. Learning rate scheduling employs a combination of a warm-up phase and cosine annealing. The training process may use mixed precision computation, gradient accumulation, and gradient clipping to improve training stability and resource utilization efficiency.

[0151] 7. Evaluation and Testing: In the evaluation phase, exponential moving average (EMA) can be used to perform integrated evaluation of model parameters. In the testing phase, augmentation during test (TTA) is used to perform multiple inferences on the same input and take the arithmetic mean of the prediction results. The output is a destandardized quality score, and evaluation indicators such as SROCC, PLCC, and RMSE can be calculated.

[0152] Example

[0153] Example 1 (Default Configuration): Grid size is preferably selected The preferred side length of the image block Hidden dimensions are preferred The optimal number of convolutional layers is [missing information]. The nearest neighbor number is preferred. The residual fusion coefficient is preferably... The dropout value is preferably 0.15. The number of training epochs can be set to 120, the batch size to 6, and gradient accumulation can be used to effectively increase the batch size. The joint loss weights can preferably be set as follows: Sorting margins The number of enhancements during testing is preferably 12, including at least horizontal flip enhancements.

[0154] Example 2 (Resource-Constrained Example): While keeping the overall process unchanged, the backbone network can be replaced with ResNet34 to reduce the computational load, or the image patch side length can be adjusted to 224 and the grid size can be adjusted to a smaller value to reduce memory usage; the rest of the process and joint loss remain unchanged.

[0155] Example 3 (Parameter Adjustment Example): With other hyperparameters fixed, adjust the nearest neighbor number separately. Hidden Dimensions Variance Alignment Weights Perform single-factor adjustment; when the observed decrease in ranking ability due to excessively small predicted output variance, the adjustment can be appropriately increased. To enhance the suppression of prediction variance collapse.

Claims

1. A method for quality assessment of no-reference UHD images and videos based on graph convolution and spatiotemporal correlation, characterized in that: Includes the following steps: The input ultra-high-definition image is divided into multiple local image patches according to a preset grid, and the center coordinates of each image patch are normalized according to the image size. The image patches are then input into a pre-trained residual neural network to extract block-level depth features, and node features of a unified dimension are obtained through linear mapping. Based on the spatial distance between the normalized center coordinates, a K-nearest neighbor graph structure is constructed for each node, generating an adjacency matrix and performing normalization. The node features and the normalized adjacency matrix are input into a multi-layer residual graph convolutional network for feature propagation to obtain node representations. Attention-weighted pooling is applied to the node representations to obtain image-level feature vectors, which are then passed through a regression network to output standardized quality prediction values. The quality prediction values ​​are subjected to learnable affine calibration and de-standardized based on the statistical parameters of the training set quality scores to obtain the final image quality score.

2. The method for quality assessment of no-reference UHD images and videos based on graph convolution and spatiotemporal correlation according to claim 1, characterized in that, The image patch is obtained by a square cropping window, and its center coordinates are normalized to the image width and height. The interval; the The nearest neighbor graph structure is constructed based on the Euclidean distance between the center coordinates of image blocks, and when the number of image blocks is insufficient, the number of nearest neighbors is adaptively limited to no more than the total number of nodes minus one; self-loop connections are added to the adjacency matrix, and a normalized adjacency matrix for graph convolution propagation is generated according to the row normalization method.

3. The method for quality assessment of no-reference UHD images and videos based on graph convolution and spatiotemporal correlation according to claim 1, characterized in that, Each layer of the residual graph convolutional network includes adjacency propagation, linear transformation, batch normalization, nonlinear activation, and residual fusion path. When the input and output dimensions are inconsistent, residual alignment is achieved through linear mapping.

4. The method for quality assessment of no-reference UHD images and videos based on graph convolution and spatiotemporal correlation according to claim 1, characterized in that, The attention-weighted pooling obtains attention weights by performing learnable scoring on the features of each node and normalizing them, and then sums the node features by weight to obtain an image-level feature vector. The quality predictions output by the regression network are scaled and offset by a learnable affine transformation to improve prediction stability under different data distributions. During the training phase, subjective quality scores are standardized, and during the inference phase, prediction results are destandardized based on the mean and standard deviation of the quality scores in the training set. The training process employs a joint loss function, which includes at least a regression error term, a correlation constraint term between the prediction and the true quality score, a ranking loss term based on paired samples, and a variance constraint term to suppress the collapse of the prediction distribution. The ranking loss term adopts a ranking constraint form based on paired samples within a batch, and the ranking stability of the prediction results is enhanced by setting a margin parameter. The AdamW optimizer with grouped learning rates is used to set learning rates for the backbone network and other network modules respectively, and training is performed by combining a warm-up phase and a cosine annealing learning rate scheduling strategy.

5. The method for quality assessment of no-reference UHD images and videos based on graph convolution and spatiotemporal correlation according to claim 1, characterized in that, During training, at least one of mixed precision computation, gradient accumulation, and gradient clipping is used to improve training stability and computational efficiency; during the model evaluation phase, exponential moving average is used to integrate model parameters to improve the stability of prediction results. During the testing phase, multiple inferences are performed on the input image, and the arithmetic mean of the quality prediction results obtained from the multiple inferences is taken, including at least the enhancement method of horizontally flipping the image patch; by performing quality prediction on each frame or frame segment in the video sequence, and performing temporal fusion on the frame-level prediction results, the video-level quality prediction result is obtained.

6. The method for quality assessment of no-reference UHD images and videos based on graph convolution and spatiotemporal correlation according to claim 1, characterized in that, Perform grid sampling on the input image, dividing the image according to... Mesh generation, determining the sampling starting point for each mesh cell, and clipping the side length within the mesh. square image blocks, obtained Each image patch is a local image patch; during the training phase, a random perturbation with an amplitude proportional to the side length of the image patch is applied to the cropping starting point to achieve data augmentation; the center coordinates of each image patch are recorded and normalized to [0,1] spatial coordinates according to the image width. Used for subsequent graph structure construction; A pre-trained residual network is used to extract deep features for each image patch. These features are then obtained through global average pooling to obtain block-level feature vectors, which are then mapped to a low-dimensional feature space via a linear projection layer. Dimension, forming the initial feature representation of the nodes. The backbone network uses weights pre-trained on general large-scale image data and is optimized end-to-end on this task to adapt to quality assessment. In terms of graph structure construction, a K-nearest neighbor (KNN) graph construction strategy based on spatial coordinates is adopted; for any two image block nodes... The spatial distance is defined as the Euclidean distance between normalized coordinates: ； For each node Select the closest Each node is used as its nearest neighbor set. Construct the adjacency matrix And add a self-loop to get Row normalization of the adjacency matrix yields the normalized adjacency matrix used for graph convolution propagation: ； in To prevent division by zero of constants.

7. The method for quality assessment of no-reference UHD images and videos based on graph convolution and spatiotemporal correlation according to claim 1, characterized in that, A multi-layer residual graph convolutional network is used for feature propagation. Each residual graph convolutional unit first performs adjacency propagation and linear transformation, and then combines batch normalization, ReLU nonlinear activation, and Dropout regularization to obtain the propagated features. Subsequently, the input and output are fused in the form of weighted residuals to improve the stability of deep training. ； in For linear mapping parameters, Activated for ReLU For residual branches, The residual fusion coefficient; An attention-weighted pooling mechanism is employed: a learnable gating scoring network is used to generate scalar weights for each node. The attention coefficients are then obtained by normalizing the values ​​at the node dimension using softmax. Finally, the node features are weighted and summed to obtain the image-level feature vector: ； in Image-level representation vector; image-level features The input consists of a two-layer sensing structure forming a regression head, and the output is a standardized spatial quality prediction value. A learnable affine calibration module is introduced to perform linear calibration on the regression output. ； in, These are learnable parameters; during the inference phase, the final subjective quality score is obtained by denormalizing the statistical parameters of the training set. ； in and These are the mean and standard deviation of the subjective quality scores for the training set, respectively. In terms of training objective design, a joint optimization strategy is adopted, comprehensively considering regression accuracy, correlation consistency, ranking stability, and prediction distribution variance constraints; the objective quality score is included. Standardized as: ； The joint loss function is defined as: ; in The Pearson correlation coefficient is used within the batch; the ranking loss adopts the pairwise hinge ranking method within the batch: several pairs of samples are randomly sampled. ,make ,but ； in The margin parameter is used to suppress prediction variance collapse. A prediction variance alignment regularization term is introduced to make the intra-batch prediction standard deviation approximate the unit standard deviation of the standardized target distribution, thus alleviating the problem of reduced sample differentiation ability at both ends caused by the compression of the prediction distribution in ultra-high-definition scenarios.

8. The method for quality assessment of no-reference UHD images and videos based on graph convolution and spatiotemporal correlation according to claim 1, characterized in that, The AdamW optimizer with grouped learning rates is used, where the learning rate of the backbone network is scaled relative to that of the head network to preserve pre-trained features. The learning rate scheduling combines a warm-up phase and a cosine annealing strategy; the exponential moving average (EMA) is used to integrate and evaluate the model parameters; and time-of-test (TTA) enhancement is employed during the validation and testing phases. Based on an image-level quality-aware model, and combined with temporal structure to capture inter-frame dependencies, the overall video quality prediction is finally generated. The overall model process includes four stages: The first stage is video content segmentation: the video is automatically divided into multiple semantically consistent segments based on the inter-frame feature change measurement. The second stage, single-frame quality prediction: using an ultra-high-definition image quality assessment network, features are extracted for each frame and frame quality is predicted. The third stage is segment-level temporal modeling: based on keyframe detection and frame-level scoring, the overall score of the segment is obtained through Gaussian kernel weighting or LSTM network. The fourth stage is video-level aggregation prediction: weighting or attention fusion of all segment scores to obtain the final video quality score.

9. The method for quality assessment of no-reference UHD images and videos based on graph convolution and spatiotemporal correlation according to claim 8, characterized in that, First, the video is adaptively segmented based on changes in video content; let the total number of video frames be T. This represents the t-th frame of the video, and the frame sequence is denoted as: ； Calculate the content difference between adjacent frames : ； in, This represents the global features extracted by the pre-trained backbone network ResNet. Let L2 be the norm, representing the Euclidean distance in the feature space; like If a significant scene change is observed at that location, a new segment boundary is defined; thus, the segment set is obtained: ； in It is the starting frame number of the i-th video segment, which corresponds to Local peaks on the curve that exceed the threshold θ are scene transition markers in the video segmentation algorithm, while N is the automatically determined number of segments. This represents the i-th video segment; Regarding the first The j-th frame of the segment Quality features are extracted using a graph convolution-based ultra-high-definition image quality assessment module: ； in, This represents a single-frame image quality assessment model. Represents the frame-level quality feature vector; The single-frame prediction score is output through a fully connected regression layer. This represents the predicted frame quality score of the i-th segment in frame j. These are the weight parameters of the fully connected layer. For bias terms: ； Get fragment The set of scores for all frames within the set. Represents the set of frames for the i-th segment: ； These scores reflect the trend of inter-frame quality fluctuations over time.

10. The method for quality assessment of no-reference UHD images and videos based on graph convolution and spatiotemporal correlation according to claim 9, characterized in that, To highlight the core visual changes within a segment, a keyframe detection mechanism based on content differences is introduced; when Define a keyframe when the maximum value is reached in the current segment: ； Indicates the first Keyframe index for each segment It is considered to best represent the quality change state of this segment. Its characteristics are denoted as The score is ; To obtain an overall quality perception of the fragments, two fragment-level modeling strategies are designed: (a) Gaussian kernel weighted fusion; It is assumed that the subjective perception influence of frames within a segment follows a Gaussian distribution with respect to the time center, and the frame scores are weighted and summed using a Gaussian kernel: ； in Control the width of the weight distribution. For the first Each segment score; (b) LSTM fusion based on temporal relationships; Another more dynamic strategy is LSTM modeling based on frame sequences: ； in, Representing the hidden and memory states of the LSTM; The segment score is an attention-weighted output of the hidden state: ； ； in, , , q, For the LSTM attention layer parameters, These are frame-level attention weights; Score all segments Perform global aggregation to obtain video quality prediction; The integration method is designed as either content-related weighting or attention aggregation: (a) Content-aware weighted average; Based on the duration of each segment Weights are determined by the characteristic variance: ； The overall score for the video is: ； The overall prediction score for the video; (b) Fragment-level attention aggregation; To adaptively learn the contribution of each segment to the overall quality, an attention-based fusion mechanism is introduced: ； ； in This represents the importance weights of segments learned by the model; During training, the optimization objective includes three levels of constraints: frame-level, segment-level, and video-level.

Images