Non-reference image quality evaluation method and system based on multi-scale image structure

Abstract

The invention discloses a non-reference image quality evaluation method and system based on a multi-scale image structure, and the method comprises the steps: firstly, for an input image, employing a multi-branch convolutional neural network, and respectively extracting multi-dimensional quality feature representations; secondly, on the basis of multi-dimensional quality feature representation, a quality relation graph is constructed on different spatial granularities, and global pooling is carried out respectively to obtain global quality representation on different spatial granularities; and then constructing a graph attention fusion module, carrying out weighted fusion on the global quality representations on different spatial granularities, and generating a comprehensive quality representation. Finally, the comprehensive quality features are sent into a multi-layer perceptron regression head, and a final image quality score is output. And training is carried out. According to the invention, the one-sidedness of single-dimensional evaluation is avoided, the quality evaluation is more comprehensive and accurate, and the accuracy and generalization ability of no-reference image quality evaluation are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image quality assessment, and particularly relates to a multi-scale graph structure based no-reference image quality assessment method and system. BACKGROUND

[0002] Image quality assessment, as a basic research direction in the field of computer vision and image processing, has important application value in the fields of multimedia content distribution, video communication, medical imaging, remote sensing image processing, etc. Unlike the full-reference (FR) and reduced-reference (RR) methods, the no-reference image quality assessment method does not require an original reference image, but only relies on the image itself to be evaluated for quality judgment, which is more in line with the needs of actual application scenarios, and thus has attracted widespread attention from the academic and industrial communities.

[0003] However, no-reference image quality assessment faces many challenges. First, image quality is a multi-dimensional concept, covering clarity, texture, noise, color and other aspects, and a single-dimensional quality feature is difficult to fully reflect the overall quality level of the image. Second, the perception of image quality has obvious spatial hierarchy, and the influence weight of local quality defects (such as local blur, noise particles) and global quality distribution (such as overall contrast, color balance) on the final quality judgment is different, and existing methods often have difficulty in effectively modeling this multi-scale spatial relationship.

[0004] Existing no-reference image quality assessment methods mainly include natural scene statistics (NSS) based methods and deep learning based methods. Mittal et al. proposed the BRISQUE method, which extracts normalized luminance coefficient statistical features for quality assessment. Zhang et al. proposed a feature similarity index based on phase consistency and gradient amplitude. Although these traditional methods are computationally efficient, they have limited feature expression capability and are difficult to handle complex mixed distortion scenarios.

[0005] In recent years, the development of deep learning technology has brought new breakthroughs to no-reference image quality assessment. Kang et al. first applied a convolutional neural network to no-reference quality assessment. Bosse et al. designed an end-to-end deep neural network framework. However, most of these methods use single-scale feature extraction and global pooling strategies, ignoring the difference in image quality at different spatial granularities, and cannot effectively capture the correlation between local quality details and global quality distribution.

[0006] As a new deep learning architecture, graph neural network (GNN) can effectively model the relationship structure between data. However, the introduction of graph structure into the research of image quality assessment is still in its infancy. How to construct an effective quality relationship graph, how to perform graph feature fusion at multiple spatial scales, and how to adaptively learn the importance weights of different scales are key issues that have not been systematically solved.

[0007] If the above problems can be effectively solved, the quality relationship graph can be constructed at multiple spatial scales, the local quality details and global quality distribution of the image can be captured, and the multi-scale features can be adaptively fused through the attention mechanism. This can significantly improve the accuracy and generalization ability of no-reference image quality assessment, thereby providing more accurate guidance for image processing and optimization. This will effectively bridge the gap between image quality assessment technology and actual application needs, and promote practical applications. SUMMARY

[0008] To overcome the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a no-reference image quality assessment method and system based on multi-scale graph structure. By constructing a multi-granularity quality relationship graph, the local quality details and global quality distribution of the image can be captured simultaneously. By designing a graph attention fusion module, the importance weights of different scale quality graphs can be adaptively learned, and effective fusion of multi-scale features can be achieved. The no-reference image quality assessment method proposed in the present application solves the technical problems that single-scale methods are difficult to comprehensively represent image quality and existing deep learning methods are difficult to effectively model spatial quality relationships, without the need for a reference image, through multi-dimensional quality feature extraction and multi-scale graph structure modeling.

[0009] To achieve the above-mentioned purpose, the technical solution adopted by the present application is as follows:

[0010] In one aspect of the present application, a no-reference image quality assessment method based on multi-scale graph structure is provided, comprising the following steps:

[0011] Step 1: For the input image, use a multi-branch convolutional neural network to extract multi-dimensional quality feature representations respectively;

[0012] Step 2: Based on the multi-dimensional quality feature representations, construct quality relationship graphs at different spatial granularities, including fine-grained local quality graphs, medium-grained regional quality graphs, and coarse-grained global quality graphs. Calculate the quality similarity between different spatial positions as the edge attribute in the three quality graphs, and perform global pooling on the quality relationship graphs constructed at different spatial granularities to obtain the corresponding global quality representations;

[0013] Step 3: Construct a graph attention fusion module, perform statistical pooling on the global quality representations of the three scales respectively, adaptively fuse the multi-scale graph features by learning the importance weights of different scale quality graphs, and generate a comprehensive quality representation;

[0014] Step 4: The fused comprehensive quality features are input into a multi-layer perception regression head to output the final image quality score; a multi-objective joint loss function is used for model training, including subjective quality regression loss and ranking loss.

[0015] In step 1, the input image I is input into a multi-branch convolutional neural network, and through different branches in the multi-branch convolutional neural network, the input image is respectively extracted for clarity feature F sharp , texture feature F texture , noise feature F noise and color feature F color ; each branch in the multi-branch convolutional neural network includes a feature preprocessing module and a MobileNetV2 model in turn; the feature preprocessing module includes the following four implementation methods:

[0016] Method one, clarity feature extraction: through a frequency domain analysis module, combined with discrete cosine transform and gradient statistics extraction, represented as:

[0017]

[0018] Wherein, represents a frequency domain feature encoding neural network, DCT represents a discrete cosine transform function, represents a gradient feature encoding neural network, represents a gradient calculation function, and Φ1 represents a feature fusion function.

[0019] Method two, texture feature extraction: a multi-direction Gabor filter bank is used to extract at different scales and directions, represented as:

[0020]

[0021] Wherein, G s,θ represents a Gabor filter of scale s and direction θ, and * represents a convolution operation.

[0022] Method three, noise feature extraction: through high-pass filtering and statistical feature analysis extraction, including Gaussian noise, salt and pepper noise and JPEG compression noise features;

[0023] represented as:

[0024] F noise =Φ2(Var(H(I)),Kurt(H(I)),Skew(H(I)),Ent(H(I)))

[0025] where H(I) denotes the high-pass filtered residual image, Var, Kurt, Skew, Ent denote the functions of calculating variance, kurtosis, skewness and entropy, respectively, and Φ2 denotes the feature fusion function.

[0026] Way four, color feature extraction: color feature in multiple color space (RGB, HSV, Lab) extraction color distribution F dist , color contrast F contrast And color saturation F saturation Feature, the output is expressed as:

[0027] F color = Φ (F dist ,F contrast ,F saturation )

[0028] Where:

[0029]

[0030] c∈R,G,B,H,S,V,L,a,b, The expected value (mean), and σ is the standard deviation.

[0031]

[0032] Indicates the gradient calculation in Lab color space.

[0033]

[0034] Where I S Indicates the saturation channel in HSV space.

[0035] Then the clarity feature, texture feature, noise feature and color feature are respectively extracted by MobileNetV2 model, and then the feature splicing and fusion are carried out to obtain multi-dimensional quality feature representation.

[0036] In step 2, the fine-grained local quality map is constructed in a space resolution of a x a, and for each spatial position i and j, the structural similarity of its quality feature is calculated as the edge attribute, that is, the fine-grained graph edge attribute Is expressed as:

[0037]

[0038] Where, f i And f j Respectively represent the quality feature vectors of positions i and j, μ, σ and σ ij Respectively represent the mean, standard deviation and covariance, and C1 and C2 are stable constants.

[0039] The medium-grained quality map is constructed at a spatial resolution of 2a x 2a, and a perceptual distance metric edge attribute, i.e., a medium-grained map edge attribute, is used

[0040] The coarse-grained global quality map is constructed at a spatial resolution of 4a x 4a, and a cosine similarity metric is used as an edge attribute, i.e., a coarse-grained map edge attribute

[0041] The global quality representation g of each scale map scale is obtained by performing global average pooling on the quality features of each scale.

[0042] In step 3, a graph attention fusion module is constructed to perform statistical pooling on the edge attributes e and global quality representations g at different spatial granularities, respectively, to adaptively fuse multi-scale map features by learning the importance weights of quality maps at different scales to generate a comprehensive quality representation F enhanced ; the specific implementation is as follows:

[0043] The graph attention fusion module first performs statistical pooling on the edge attributes e and global quality representations g of the three scales, respectively, to calculate the maximum value, minimum value, mean value, and standard deviation and concatenate them to obtain the scale map fusion feature representation h s :

[0044] h s =[max(e s ),min(e s ),mean(e s ),std(e s ),g s ]

[0045] where s represents different scales, s = {fine, medium, coarse}.

[0046] The statistical features are input into a fully connected network to generate attention weights a s for each scale.

[0047] According to the learned attention weights, the quality map features of the three scales are weighted and fused to obtain a full-scale map fusion feature F enhanced :

[0048]

[0049] where represents a concatenation operation, a s represents the weight corresponding to scale s, e s represents the edge attribute corresponding to scale s, and g s represents the global quality representation corresponding to scale s.

[0050] In step 4, the multi-layer perception regression head is composed of three fully connected layers, the first two layers are followed by a ReLU activation function and a Dropout layer, and the quality score q is calculated.

[0051] The multi-target joint loss function is represented as:

[0052]

[0053] Wherein, is the subjective quality regression loss; is the ranking loss, and λ1 is a balance coefficient.

[0054] Another aspect of the present application is also a no-reference image quality assessment system based on a multi-scale graph structure, which comprises the following modules:

[0055] The multi-dimensional quality feature extraction module: for the input image, a multi-branch convolutional neural network is used to extract multi-dimensional quality feature representations.

[0056] The multi-scale quality graph construction module: based on the multi-dimensional quality feature representation, quality relationship graphs are constructed at different spatial granularities, including fine-grained local quality graphs, medium-grained regional quality graphs, and coarse-grained global quality graphs. The quality similarity between different spatial positions is calculated as the edge attribute in the three quality graphs, and the quality relationship graphs at different spatial granularities are globally pooled to obtain the corresponding global quality representation.

[0057] The graph attention fusion module: the graph attention fusion module is constructed, and the edge features and global features of the global quality representation at different spatial granularities are respectively statistically pooled. By learning the importance weight of the quality graph at different scales, the multi-scale graph features are adaptively fused to generate a comprehensive quality representation.

[0058] The quality score prediction module: the comprehensive quality feature is input into the multi-layer perception regression head to output the final image quality score; and the training is carried out.

[0059] Compared with the prior art, the present application has the following advantages:

[0060] (1) The present application proposes a multi-dimensional quality feature extraction framework, which can consider multiple key factors of image quality by designing special clarity branch, texture branch, noise branch and color branch, avoid the one-sidedness of single-dimensional evaluation, and make the quality evaluation more comprehensive and accurate.

[0061] (2) The multi-scale quality relationship graph constructed by the present application can represent the spatial relationship of image quality at different spatial granularities. The fine-grained graph captures local quality details, the medium-grained graph captures regional quality consistency, and the coarse-grained graph captures global quality distribution, achieving more comprehensive quality representation than single-scale methods.

[0062] (3) The graph attention fusion module designed by the present application can adaptively learn the importance weights of quality graphs of different scales. For different types of distortion, the contribution of each scale can be automatically adjusted, improving the flexibility and robustness of the evaluation, so that the evaluation process can be dynamically adjusted according to the image content and distortion characteristics.

[0063] (4) The multi-objective joint loss function used by the present application not only optimizes the regression accuracy of the quality score, but also ensures the monotonicity of the quality prediction through the ranking loss, significantly improving the accuracy and consistency of the evaluation.

[0064] In summary, the present application combines multi-dimensional quality feature extraction and multi-scale graph structure modeling, and applies a graph attention adaptive fusion mechanism to achieve comprehensive and accurate evaluation of image quality without reference images, improving the accuracy and generalization ability of no-reference image quality evaluation, and constructing an efficient, stable and adaptable image quality evaluation system. The no-reference image quality evaluation method proposed by the present application has wide application prospects, including image compression optimization, image enhancement processing, medical image analysis, video communication quality control and other fields. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 The flowchart of the method proposed by the present application is shown in the figure.

[0066] Figure 2 The network model structure diagram of the method proposed by the present application is shown in the figure. DETAILED DESCRIPTION

[0067] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings.

[0068] As shown in Figure 1 and Figure 2 , in one aspect of the present application, a no-reference image quality evaluation method based on multi-scale graph structure is provided, including the following steps:

[0069] Step 1: Multi-dimensional quality feature extraction

[0070] A multi-branch convolutional neural network is used to extract multi-dimensional quality features from the input image, obtaining a multi-dimensional quality feature representation.

[0071] The multi-branch convolutional neural network comprises four special feature extraction branches, each branch adopts MobileNetV2 as a backbone network, and extracts features for different quality dimensions. MobileNetV2 is selected as the backbone network because it has lower computational complexity while maintaining high feature extraction capability, which is suitable for real-time application scenarios.

[0072] 1.1 Sharpness feature extraction

[0073] Sharpness is an important dimension of image quality and directly affects human perception of images. The sharpness branch designed in the present application extracts sharpness features through a frequency domain analysis module combined with discrete cosine transform (DCT) and gradient statistics.

[0074] Discrete cosine transform can convert an image from spatial domain to frequency domain, and the energy distribution of high-frequency components reflects the sharpness of the image. For a sharp image, the high-frequency component energy is strong; while for a blurred image, the high-frequency component energy is weak. Gradient statistics directly reflects the sharpness of the image edge, and a sharp image has a strong gradient response.

[0075] The calculation formula of the sharpness feature is:

[0076]

[0077] wherein, represents a frequency domain feature encoding neural network, DCT represents a discrete cosine transform function, represents a gradient feature encoding neural network, represents a gradient calculation function, and Φ represents a feature fusion function, which can use a vector splicing operation.

[0078] By fusing the frequency domain features and the spatial domain gradient features, the sharpness of the image can be more comprehensively represented.

[0079] 1.2 Texture feature extraction

[0080] Texture is another important quality dimension of an image, reflecting the structural and regularity of the image content. The present application uses a multi-direction Gabor filter bank to extract texture features.

[0081] Gabor filters can perform localized analysis in both spatial and frequency domains, and have good response characteristics for texture patterns of different directions and scales. By applying a Gabor filter bank at multiple scales s and multiple directions θ, rich texture information in the image can be captured.

[0082] The calculation formula of the texture feature is:

[0083]

[0084] where G s,θ denotes the Gabor filter with scale s and orientation θ, and * denotes the convolution operation. The proposed method employs a Gabor filter bank with 4 scales and 8 orientations, totally 32 filters, to comprehensively capture texture patterns with different granularities and orientations.

[0085] 1.3 Noise feature extraction

[0086] Noise is one of the main factors that degrade image quality. The noise branch of the proposed method extracts noise features through high-pass filtering and statistical feature analysis.

[0087] High-pass filtering can highlight the high-frequency noise components in the image and suppress the low-frequency content signals. By analyzing the statistical features of the high-pass filtered image, such as variance, skewness, kurtosis, etc., the type and intensity of the noise can be effectively estimated.

[0088] The proposed method evaluates the image noise level by constructing three detection models for typical noise types: for Gaussian noise, a detection method that analyzes the normal characteristics of pixel value distribution is used; for salt and pepper noise, the number and spatial distribution of extreme value pixels are used for identification; for JPEG compression noise, the quantization error pattern in DCT coefficients is analyzed to evaluate the noise level. By extracting these three noise features simultaneously, the noise level of the image can be comprehensively evaluated, and the accuracy of quality assessment can be improved. Specifically, it is expressed as:

[0089] F noise = Ψ(Var(H(I)), Kurt(H(I)), Skew(H(I)), Ent(H(I)))

[0090] where H(I) denotes the high-pass filtered residual image, Var, Kurt, Skew, and Ent denote the functions for calculating variance, kurtosis, skewness, and entropy, respectively, and Ψ denotes the feature fusion function, which can be a simple concatenation operation here.

[0091] 1.4 Color feature extraction

[0092] Color quality directly affects the visual appeal and realism of an image. The proposed method extracts color features in multiple color spaces to comprehensively represent color quality.

[0093] Different color spaces have different advantages, and the complementary advantages of RGB, HSV and Lab three color spaces are fused to comprehensively evaluate the image color quality: the color distribution characteristics are analyzed in the device native RGB space to reflect the original color richness, the hue and saturation are analyzed in the HSV space conforming to the psychological perception to evaluate the color vividness, and the color contrast is measured in the Lab space to capture the human eye perception difference. By fusing the features of multiple color spaces, the color quality of the image can be more accurately evaluated. Specifically:

[0094] F color dist contrast saturation

[0095] Wherein, Φ3 represents a feature fusion function;

[0096]

[0097] c represents the channel of each color space, is expectation, and sigma is standard deviation.

[0098] Indicates gradient calculation in Lab color space.

[0099]

[0100] Wherein I S Indicates the saturation channel in HSV space.

[0101] Then the clarity feature, texture feature, noise feature and color feature are respectively extracted through the MobileNetV2 model, and then the feature splicing and fusion are carried out to obtain the multi-dimensional quality feature representation.

[0102] The application constructs a comprehensive quality feature representation through the above multi-dimensional feature extraction. The output feature dimension of each branch is 512, and the features of the four branches are obtained after the feature fusion layer, and the multi-dimensional quality feature vector with a dimension of 2048 is obtained.

[0103] Step 2: Multi-scale quality map construction

[0104] Based on the quality features extracted in step 1, the quality relationship graph is constructed on different space granularities, including fine-grained local quality graph, medium-grained regional quality graph and coarse-grained global quality graph.

[0105] ​​​​The human visual system pays attention to both local details and global impression when evaluating image quality. Local quality defects (e.g. noise or blur in small regions) and global quality distribution (e.g. overall contrast and color balance) both affect the final quality judgment. To simulate this multi-scale perception mechanism, the invention constructs quality graphs at three different spatial resolutions.

[0106] 2.1 Fine-grained local quality graph construction

[0107] The fine-grained quality graph is constructed at a spatial resolution of 7x7, to capture local quality details. The multi-dimensional quality features are first down-sampled to a spatial size of 7x7 through spatial pyramid pooling, resulting in 49 local quality feature vectors.

[0108] For any two spatial locations i and j, the structural similarity between them is computed as the edge weight of the graph, i.e. the fine-grained graph edge attribute

[0109]

[0110] where f i and f j are the quality feature vectors of locations i and j, μ is the mean, σ is the standard deviation, and σ ij is the covariance. C1 = (0.01 x L) 2 and C2 = (0.03 x L) 2 are stability constants, and L is the dynamic range of feature values.

[0111] The SSIM index considers the similarity in brightness, contrast, and structure, and can effectively measure the quality consistency of two locations. A high SSIM value indicates that the quality features of two locations are similar, and they should have strong connections in the quality graph; a low SSIM value indicates that the quality features are quite different, and the connections are weak.

[0112] 2.2 Medium-grained regional quality graph construction

[0113] The medium-grained quality graph is constructed at a spatial resolution of 14x14, to capture the quality consistency between neighboring regions. The multi-dimensional quality features are down-sampled to a spatial size of 14x14, resulting in 196 regional quality feature vectors.

[0114] The perceptual distance is used to measure the edge attribute:

[0115]

[0116] where ||f i -f j||2 represents the L2 distance, and σ is a learnable scale parameter with an initial value of 1.0. This similarity measure in the form of a Gaussian kernel function has good smoothness for feature distances and can naturally map distances to the [0, 1] interval.

[0117] The medium-grained quality map mainly focuses on the quality transition at the region level, such as the gradual transition from a clear region to a blurred region, the change from a low-noise region to a high-noise region, and the like. This medium-grained representation is very important for understanding the spatial distribution pattern of quality.

[0118] 2.3 Construction of coarse-grained global quality map

[0119] The coarse-grained quality map is constructed at a spatial resolution of 28x28, which is used to describe the quality correlation between large-scale regions. The multi-dimensional quality features are down-sampled to a spatial size of 28x28, obtaining 784 global quality feature vectors.

[0120] The cosine similarity is used to measure the edge attribute:

[0121]

[0122] The cosine similarity focuses on the direction of the feature vector rather than the amplitude, and is invariant to the scale change of the feature. This measurement method is particularly suitable for the global quality map, because the global quality is more concerned about whether the quality patterns of different regions are similar, rather than the difference in absolute values.

[0123] The coarse-grained map captures the overall quality distribution, such as whether the overall contrast is uniform and whether the color balance is coordinated. This macroscopic quality representation plays an important role in forming the final overall quality impression.

[0124] 2.4 Extraction of global quality features

[0125] In addition to constructing the quality relationship map at three scales, the global quality representation at each scale also needs to be extracted for subsequent attention weight calculation.

[0126] The global average pooling is performed on the quality features at each scale:

[0127]

[0128] where N is the number of spatial positions corresponding to the scale (49 for fine-grained, 196 for medium-grained, and 784 for coarse-grained). The global quality representation g scale reflects the overall quality level of the image at this scale, providing global context information for the attention mechanism.

[0129] By constructing the quality relationship map and the global quality representation at three scales, the present application can comprehensively represent the spatial relationship of image quality at multiple spatial granularities.

[0130] Step 3: Graph Attention Adaptive Fusion

[0131] The graph attention fusion module is designed to learn the importance weights of quality maps at different scales, adaptively fuse multi-scale feature maps, and generate comprehensive quality representations.

[0132] For different types of image distortion, the importance of quality information at different scales varies. For example, for noise distortion with strong locality, fine-grained quality maps are more important; for global blur distortion, coarse-grained quality maps are more critical. The graph attention mechanism designed in this invention can adaptively adjust the weights of each scale according to the specific image content.

[0133] 3.1 Multi-scale feature statistical pooling

[0134] First, the edge features and global features at three scales are respectively subjected to statistical pooling to extract comprehensive descriptive statistics:

[0135] h s =[max(e s ),min(e s ),mean(e s ),std(e s ),g s ]

[0136] Where: max(e s ) refers to the maximum edge weight, reflecting the strongest quality association at this scale; min(e s ) refers to the minimum edge weight, reflecting the weakest quality association; mean(e s ) refers to the mean edge weight, reflecting the average quality consistency level; std(e s ) refers to the standard deviation of the edge weight, reflecting the dispersion degree of the quality distribution; g s refers to the global quality representation, providing the overall quality context.

[0137] These five statistics describe the characteristics of the quality map at this scale from different angles, providing sufficient information for attention weight calculation.

[0138] 3.2 Attention weight generation

[0139] The statistical features are input into a double-layer fully connected network to generate attention weights at each scale:

[0140] α s =Sigmoid(W2·ReLU(W1·h s +b1)+b2)

[0141] where, and where W is a learnable weight matrix, and is a bias term.

[0142] The first fully connected layer maps the 517-dimensional statistical features to a 256-dimensional hidden space, introducing a nonlinear transformation through ReLU activation. The second fully connected layer maps the hidden features to scalar weight values, normalizing the weights to the [0, 1] interval through a Sigmoid function.

[0143] This two-layer network structure can learn complex nonlinear mapping relationships and adaptively generate attention weights based on different statistical feature patterns. Three scales generate three attention weights α fine , α medium , and α coarse , respectively.

[0144] 3.3 Multi-scale feature weighted fusion

[0145] According to the learned attention weights, the quality map features of the three scales are weighted and fused:

[0146]

[0147] where denotes the concatenation operation, α s denotes the weight corresponding to scale s, e s denotes the edge attribute corresponding to scale s, and g s denotes the global quality representation corresponding to scale s. For each scale, the edge features and global features are concatenated and then weighted. This concatenation method can preserve both local correlation information and global context information.

[0148] The edge features e scale are obtained through global average pooling, resulting in a unified feature representation with a dimension of 512. The global features g scale also have a dimension of 512. After concatenation, the feature dimension of each scale is 1024, and after weighted fusion of the three scales, the dimension of F enhanced is 1024.

[0149] Step 4: Quality score prediction and model training

[0150] The integrated quality features are fed into a multi-layer perceptron regression head to output the final image quality score. A multi-objective joint loss function is used for model training to ensure the accuracy and consistency of the evaluation results.

[0151] 4.1 Multi-layer perceptron regression head

[0152] The multi-layer perceptron regression head consists of three fully connected layers, with the following structure design:

[0153] The input layer is configured to receive an input vector with 1024 features; the first hidden layer is configured to map the 1024-dimensional input to a 512-dimensional feature space and apply a ReLU activation function with a dropout rate of 0.3; the second hidden layer is configured to map the 512-dimensional features to a 256-dimensional feature space and also apply a ReLU activation function with a dropout rate of 0.3; and the output layer is configured to map the 256-dimensional features to a one-dimensional scalar as the final quality score. The formula for calculating the quality score is:

[0154] q = W3·Dropout(ReLU(W2·Dropout(ReLU(W1·F enhanced + b1))+b2))+b3

[0155] The Dropout layer is used to prevent overfitting by randomly dropping 30% of the neurons during training. The ReLU activation function introduces nonlinearity and enhances the network's expressive power. The final output quality score q is usually in the range of 0 to 100, with higher values indicating better quality.

[0156] 4.2 Multi-objective joint loss function

[0157] The present application adopts a multi-objective joint loss function for model training, which comprehensively considers regression accuracy and ranking consistency:

[0158]

[0159] Subjective quality regression loss:

[0160]

[0161] where q i is the predicted quality score by the model, is the true quality score labeled by humans, and B is the batch size (set to 32). The MSE loss optimizes the absolute error between the predicted value and the true value, which is the basic objective of quality regression.

[0162] Ranking loss:

[0163]

[0164] where q i , q j is the predicted quality score by the multi-layer perceptron regression head, is the true quality score, B is the batch size, λ is the balance coefficient, ∈ is the boundary parameter, and the sign function returns the sign of the difference between the true quality score. The ranking loss ensures that when the true quality, , the predicted quality also satisfies q i>q j , ensuring the monotonicity and ranking consistency of quality prediction.

[0165] The balance coefficient is set as λ = 0.1. These hyperparameters are obtained by grid search on the validation set.

[0166] 4.3 Model training strategy

[0167] The present application adopts a two-stage training strategy to gradually optimize model performance:

[0168] First stage: Branch pre-training Pre-train the clarity branch, texture branch, noise branch, and color branch respectively. Supervised learning is performed using corresponding quality dimension labeled data, and each branch is trained for 50 epochs.

[0169] The pre-training stage of the model is based on a multi-attribute labeled data set, and the construction method of the data set includes: for the clarity attribute, generating image pairs with gradient blur degree and labeling their clarity scores; for the texture attribute, collecting images with diversified texture complexity and labeling their texture quality scores; for the noise attribute, applying different types and intensities of noise to the image and labeling its noise level; for the color attribute, selecting images with differences in color performance and labeling their color quality scores. Wherein, the labeling scores of each attribute jointly constitute the multi-task supervision signal for model pre-training.

[0170] Each branch attribute is trained using the MSE loss function, with a learning rate of 1 × 10 -3 , using the Adam optimizer.

[0171] Second stage: End-to-end fine-tuning Freeze the backbone network parameters (MobileNetV2) of each branch, and only train the multi-scale graph construction module, graph attention fusion module, and quality regression head. This strategy preserves the quality dimension specific knowledge learned in the pre-training stage, while learning multi-scale fusion and overall quality evaluation capabilities.

[0172] End-to-end training for 100 epochs, with an initial learning rate of 3 × 10 -4 , using the cosine annealing strategy for learning rate decay:

[0173] Wherein,

[0174] η max = 3 × 10 -4 , η min = 1 × 10 -6

[0175] T is the total number of training steps.

[0176] The Adam optimizer is used, with the parameter settings of β1=0.9, β2=0.999, and a weight decay of 1×10 -4 The batch size is set to 32.

[0177] 4.4 Model inference

[0178] After inputting the image I to be evaluated into the trained model in the inference stage, the multi-dimensional quality features are extracted synchronously through the four-branch architecture, then the quality relationship graphs representing local and global correlations are constructed at three different scales, and then the adaptive fusion of multi-scale features is realized through the graph attention mechanism, and finally the comprehensive quality score q is output through the multi-layer perceptron regression head. The final output quality score q reflects the overall quality level of the image, which can be directly used for quality evaluation, image sorting or quality control and other applications.

[0179] Another aspect of the application is also a no-reference image quality assessment system based on a multi-scale graph structure, which includes the following modules:

[0180] Multi-dimensional quality feature extraction module: for the input image, a multi-branch convolutional neural network is used to extract multi-dimensional quality feature representations.

[0181] Multi-scale quality graph construction module: based on the multi-dimensional quality feature representations, quality relationship graphs are constructed at different spatial granularities, including fine-grained local quality graphs, medium-grained regional quality graphs, and coarse-grained global quality graphs; the quality similarity between different spatial positions is calculated as the edge attribute in the three quality graphs, and the quality relationship graphs at different spatial granularities are globally pooled to obtain the corresponding global quality representations.

[0182] Graph attention fusion module: the graph attention fusion module is constructed, and the edge features and global features of the global quality representations at different spatial granularities are respectively statistically pooled, the importance weights of the quality graphs at different scales are learned, the multi-scale graph features are adaptively fused, and the comprehensive quality representation is generated.

[0183] Quality score prediction module: the comprehensive quality features are input into the multi-layer perceptron regression head to output the final image quality score; and the training is performed.

[0184] Experiments on the standard dataset show that, compared with existing no-reference image quality assessment methods, the present method has achieved significant improvement in SRCC (Spearman Rank Order Correlation Coefficient) and PLCC (Pearson Linear Correlation Coefficient) indicators, proving the effectiveness of the proposed scheme.

[0185] Simulation experiment:

[0186] The present application is verified by using standard image quality evaluation datasets, including the LIVE dataset and the TID2013 dataset. These datasets contain images of various distortion types and distortion levels, as well as corresponding subjective quality scores (MOS).

[0187] According to the standard evaluation protocol, 80% of the data is used for training and 20% of the data is used for testing. In order to ensure the reliability of the results, 10 random partition experiments are conducted and the average performance indicators are reported.

[0188] The evaluation indicators include: SRCC (Spearman Rank Order Correlation Coefficient): measures the ordering consistency of the predicted quality score and the true quality score. PLCC (Pearson Linear Correlation Coefficient): measures the linear correlation between the predicted quality score and the true quality score.

[0189] Table 1 The performance of the method of the present application (denoted as MSGQ) and the performance of representative existing methods are as follows:

[0190]

[0191] As can be seen from Table 1, the proposed multi-scale graph structure-based no-reference image quality evaluation method of the present application achieves the best performance on the above standard datasets.

[0192] These significant performance improvements demonstrate the effectiveness of the multi-dimensional quality feature extraction, multi-scale graph structure modeling and graph attention adaptive fusion mechanism proposed in the present application. In particular, on the TID2013 and LIVE datasets containing complex mixed distortions, the performance improvement is more obvious, indicating that the method has stronger generalization ability and robustness.

Claims

1. A no-reference image quality assessment method based on multi-scale graph structure, characterized in that, Includes the following steps: Step 1: For the input image, use a multi-branch convolutional neural network to extract multi-dimensional quality feature representations; Step 2: Based on the multi-dimensional quality feature representation, construct quality relationship graphs at different spatial granularities, and perform global pooling to obtain global quality representations at different spatial granularities; Step 3: Construct a graph attention fusion module to perform weighted fusion of global quality representations at different spatial granularities to generate a comprehensive quality representation; Step 4: Feed the comprehensive quality features into the multilayer perceptron regression head to output the final image quality score; and then perform training.

2. The no-reference image quality assessment method based on multi-scale graph structure according to claim 1, characterized in that, The specific implementation process of step 1 is as follows: The input image I is fed into a multi-branch convolutional neural network. Different branches of the network process the sharpness feature F of the input image. sharp Texture features F texture Noise characteristics F noise and color characteristics F color Extraction; each branch in a multi-branch convolutional neural network includes a feature preprocessing module and a MobileNetV2 model in sequence; Sharpness features, texture features, noise features, and color features are extracted using the MobileNetV2 model, and then the features are concatenated and fused to obtain a multi-dimensional quality feature representation.

3. The no-reference image quality assessment method based on multi-scale graph structure according to claim 2, characterized in that, The feature preprocessing module includes the following four implementation methods: Method 1, Sharpness Feature Extraction: Through the frequency domain analysis module, combined with discrete cosine transform and gradient statistics extraction, it is expressed as: in, This represents a frequency domain feature encoding neural network, where DCT stands for Discrete Cosine Transform function. This represents a gradient feature encoding neural network. Φ1 represents the gradient calculation function and the feature fusion function. Method 2, Texture Feature Extraction: Employing a multi-directional Gabor filter bank to extract features at different scales and directions, as shown below: Among them, G s,θ The Gabor filter represents the scale s and direction θ, and * denotes the convolution operation; Method 3, Noise Feature Extraction: This method extracts noise features through high-pass filtering and statistical feature analysis, including Gaussian noise, salt-and-pepper noise, and JPEG compression noise features, as follows: F noise =Φ2(Var(H(I)),Kurt(H(I)),Skew(H(I)),Ent(H(I))) Where H(I) represents the residual image after high-pass filtering, Var, Kurt, Skew, and Ent represent functions for calculating variance, kurtosis, skewness, and entropy, respectively, and Φ2 represents the feature fusion function. Method 4, Color Feature Extraction: Extracting color distribution, color contrast, and color saturation features from multiple color spaces, the output of which is represented as follows: F color =Φ3(F dist ,F contrast ,F saturation ) Where Φ3 represents the feature fusion function; 'c' represents the channel of each color space. Let σ be the expected value and σ be the standard deviation. This represents gradient calculation in the Lab color space; Where I S This represents the saturation channel in the HSV color space.

4. The no-reference image quality assessment method based on multi-scale graph structure according to claim 3, characterized in that, Step 2 is specifically implemented as follows: Based on multi-dimensional quality feature representation, a quality relationship graph is constructed at different spatial granularities, including a fine-grained local quality graph, a medium-grained regional quality graph, and a coarse-grained global quality graph; the quality similarity between different spatial locations is calculated as the edge attribute in the three quality graphs, and global pooling is performed on the quality relationship graphs constructed at different spatial granularities to obtain the corresponding global quality representation; The specific steps for constructing mass relationship diagrams at different spatial granularities are as follows: The fine-grained local quality map is constructed at a spatial resolution of a×a. For each spatial location i and j, the structural similarity of its quality features is calculated as an edge attribute. The medium-grained region quality map is constructed at a spatial resolution of 2a×2a, and uses perceptual distance metric as edge attribute. The coarse-grained global quality graph is constructed at a spatial resolution of 4a×4a, and cosine similarity is used as the edge attribute.

5. The no-reference image quality assessment method based on multi-scale graph structure according to claim 4, characterized in that, Step 3 is specifically implemented as follows: A graph attention fusion module is constructed, which performs statistical pooling on edge attributes e and global quality representation g at different spatial granularities. By learning the importance weights of quality graphs at different scales, multi-scale graph features are adaptively fused to generate a comprehensive quality representation F. enhanced The details are as follows: The graph attention fusion module first performs statistical pooling on the edge attributes e and the global quality representation g at the three scales, calculates the maximum, minimum, mean, and standard deviation, and concatenates them to obtain the graph fusion feature representation h at each scale. s : h s =[max(e s ),min(e s ),mean(e s ),std(e s ),g s ] Where s represents different scales, s = {fine, medium, coarse} Statistical features are input into a fully connected network to generate attention weights α at various scales. s : α s =Sigmoid(W2·ReLU(W1·h s +b1)+b2) Where W1, W2, b1, and b2 are learnable parameters; Based on the learned attention weights, the quality map features at the three scales are weighted and fused to obtain the full-scale map fusion feature F. enhanced : F enhanced =∑α s ·(e s ⊕g s ) Where ⊕ represents the concatenation operation, α s e represents the weight corresponding to scale s. s G represents the edge attribute corresponding to scale s. s This represents the global quality representation corresponding to scale s.

6. The no-reference image quality assessment method based on multi-scale graph structure according to claim 5, characterized in that, In step 4, the loss function used for training is a multi-objective joint loss function, expressed as: The subjective quality regression loss is: The sorting loss is: Where, q i q j To predict quality scores for the multilayer perceptron regression head. λ is the true quality fraction, B is the batch size, λ is the balance coefficient, and ε is the boundary parameter.

7. A no-reference image quality assessment system based on a multi-scale graph structure, used to implement the no-reference image quality assessment method according to any one of claims 1 to 6, characterized in that, Includes the following modules: Multi-dimensional quality feature extraction module: For the input image, a multi-branch convolutional neural network is used to extract multi-dimensional quality feature representations respectively; Multi-scale quality graph construction module: Based on multi-dimensional quality feature representation, it constructs quality relationship graphs at different spatial granularities, including fine-grained local quality graphs, medium-grained regional quality graphs, and coarse-grained global quality graphs; it calculates the quality similarity between different spatial locations as the edge attributes of the three quality graphs, and performs global pooling on the quality relationship graphs constructed at different spatial granularities to obtain the corresponding global quality representation; Graph Attention Fusion Module: Construct a graph attention fusion module, which performs statistical pooling on the edge features and global features of the global quality representation at different spatial granularities. By learning the importance weights of quality graphs at different scales, it adaptively fuses multi-scale graph features to generate a comprehensive quality representation. Quality score prediction module: It feeds the comprehensive quality features into the multilayer perceptron regression head, outputs the final image quality score, and performs training.

Images