Image quality evaluation method based on texture and color harmony perception

Abstract

The invention discloses an image quality evaluation method based on texture and color harmony perception, and the method comprises the steps: firstly converting an input image into an HSV color space and an RGB color space, obtaining color features through an attention module, and calculating the color disharmony degree; secondly, multi-scale edge features of the input image are extracted by adopting an overall nested edge detection algorithm, and spatial texture features are obtained through a feature extraction module; and then extracting multi-scale features of the input image by using a lightweight MobileNet network, and carrying out feature fusion to form image content features. And finally, jointly sending the color features, the spatial texture features and the image content features into a gating fusion network, outputting an image quality score through softmax, and performing training in combination with color disharmony. According to the method, more accurate modeling of color composition is realized, the perception of color harmony is more in line with human visual characteristics, and quality evaluation is accurately and efficiently completed.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to an image quality assessment method based on texture and color harmony perception. Background Technology

[0002] Image quality assessment is an important research direction in the field of image processing. Its goal is to quantitatively analyze and evaluate the visual quality of images through computational models, in order to simulate the consistent human perception of image quality. With the rapid development of multimedia communication and digital image processing technologies, automated quality assessment technology is playing an increasingly important role in applications such as image coding, image enhancement, and image reconstruction.

[0003] Currently, most deep learning-based quality assessment methods adopt an end-to-end approach, learning the mapping relationship between visual features and quality scores from scoring data, treating quality assessment as a classification or regression task. Although these methods have made some progress in score prediction, they typically only output numerical scores or rough grades, failing to reveal the inherent logic of quality judgment and exhibiting significant "black box" characteristics. This deficiency limits the practical application value of assessment results in professional image processing systems that require clear directions for quality improvement. To improve model interpretability, some studies have attempted to introduce interpretable artificial intelligence techniques, such as visualizing the image regions focused on by the model through attention mechanisms. However, such methods usually only provide local visual cues, making it difficult to form systematic quality analysis conclusions and failing to meet the need for a comprehensive understanding of image quality composition. Furthermore, existing methods generally lack the ability to effectively model key elements affecting visual quality, such as texture complexity, color harmony, and structural integrity. These core quality elements are difficult to quantify and represent, resulting in assessment results that mostly remain at the level of surface visual features, failing to accurately reflect the perceptual characteristics of the human visual system. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides an image quality assessment method based on texture and color harmony perception, which solves the issues of insufficient interpretability, single assessment dimensions, lack of theoretical support for quality, and limited generalization ability in existing technologies. The method integrates image feature analysis and visual semantic understanding, and achieves multi-dimensional quantitative assessment of image quality by constructing color analysis networks, image texture analysis networks, and image content analysis networks. This invention aims to improve the accuracy of score prediction while focusing on providing a theoretical basis and coherent logic for quality judgment based on texture and color harmony perception, thereby enabling automated assessment to meet the stringent requirements of professional quality assessment in terms of credibility and practicality.

[0005] To achieve the above-mentioned objectives of this invention, the technical solution adopted is as follows:

[0006] An image quality assessment method based on texture and color harmony perception includes the following steps:

[0007] S1: First, the input image is converted to HSV and RGB color spaces respectively. By calculating the wavelet entropy of each color channel in the HSV and RGB color spaces, the two channels with the richest information are selected. Then, these two channel images are input into the attention module to establish a relationship model between color blocks and obtain color features. At the same time, the color incongruity of the input image is calculated as an auxiliary supervision signal for subsequent processing.

[0008] S2: Next, the overall nested edge detection HED algorithm is used to extract multi-scale edge features of the input image. By fusing the shallow output of HED, the overall layout and moderate details of the image are preserved. Then, the feature extraction module further processes the image to obtain spatial texture features.

[0009] S3: Subsequently, the lightweight MobileNet network is used to extract multi-scale features of the input image. High-level semantic features and low-level detail features are progressively fused through a cascaded multi-scale feature aggregation module to form image content features.

[0010] S4: Finally, the color features of S1, the spatial texture features of S2, and the image content features of S3 are fed into the induction fusion network. The image quality score is output through softmax and trained by combining EMD and color incongruity.

[0011] The steps for extracting color features from an image and calculating color incongruity specifically include:

[0012] For color feature extraction, the original RGB image is input and first converted to HSV and RGB color spaces respectively. The wavelet entropy of each color channel in the HSV and RGB color spaces is calculated to select the two channels with the richest information. These two channels are then segmented into k×k non-overlapping image blocks. The average pixel value of each block is extracted as a block feature and input into a self-attention module to model the relationships between color blocks, resulting in an attention score matrix. Finally, color features are obtained through a fully connected layer. Simultaneously, based on the HSV color space, the hue channel of the input image is extracted and discretized and quantized. The pixel distribution density of each hue level is statistically analyzed to obtain the hue distribution statistical features of the image. Then, the minimum distance between the hue value of each pixel and the boundaries of each set rotation template is calculated. Combined with pixel saturation information, the deviation of the image color distribution from the ideal template distribution is evaluated to obtain the comprehensive quality score under the current template configuration. Finally, a global search is performed to traverse all templates and their rotation angles to find the optimal template configuration that minimizes the comprehensive quality score. This minimum score value is the color incongruity of the image. This incongruity will be used as a color incongruity loss term, providing a direct supervisory signal for color harmony in the total model loss.

[0013] The specific methods for extracting spatial texture features of the image in step two include:

[0014] For spatial texture feature extraction, the overall nested edge detection (HED) algorithm is used to extract multi-scale edge features. The multi-scale edge features are fused with the first and second layers of HED to retain the overall layout and moderate details of the image. The fused edge features are then input into the feature extraction module to extract high-dimensional features that can reflect the spatial texture of the image, i.e., spatial texture features.

[0015] The specific methods for extracting image content features in step three include:

[0016] For image content feature extraction, a pre-trained MobileNet backbone network is used to extract multi-scale features. The multi-scale features are unified in size and number of channels through bilinear interpolation and 1×1 convolution. Then, they are input into a cascaded multi-scale feature aggregation module to progressively fuse high-level semantic features and low-level detail features. After concatenation, the image content features are obtained through 1×1 convolution.

[0017] The specific methods for feature fusion and quality evaluation in step four include:

[0018] Color features, spatial texture features, and image content features are jointly input into a gated fusion network. First, the color features, spatial texture features, and image content features are each passed through three independent fully connected layers and a sigmoid activation function, generating weight coefficients q1, q2, and q3 for each feature. Then, weights q1, q2, and q3 are used to weight the color features, spatial texture features, and image content features, and the weighted features are concatenated along the channel dimension to form a fused feature. The fused feature is then processed sequentially through a global average pooling layer, a fully connected layer, and a dropout layer, ultimately outputting an image quality score. During model training, a joint optimization using a weighted combination of EMD loss and color inconsistency is employed to improve the model's quality assessment performance.

[0019] This invention achieves a breakthrough in image quality perception and color harmony assessment by utilizing image quality evaluation methods based on color composition and spatial texture, and by introducing a multi-path feature extraction architecture and a multi-scale feature fusion mechanism. Compared to existing technologies, it has the following significant advantages:

[0020] In terms of color relationship modeling, this invention uses wavelet entropy channel selection and self-attention mechanism to deeply explore the complex relationships between color blocks inside an image, achieving more accurate modeling of color composition. Combined with color disharmony calculation, the model's perception of color harmony is more in line with human visual characteristics.

[0021] In terms of multi-feature fusion, this invention effectively combines color, spatial texture, and content features through a three-path parallel architecture and cascaded multi-scale feature aggregation modules. This solves the problem of insufficient feature representation in complex scenarios by traditional methods. Its adaptive feature fusion strategy ensures that the model obtains a more comprehensive quality perception capability and completes quality assessment accurately and efficiently. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the image quality assessment method based on texture and color harmony perception described in the embodiments of the present invention;

[0023] Figure 2 This is a schematic diagram of the image quality assessment method framework based on texture and color harmony perception described in the embodiments of the present invention;

[0024] Figure 3 This is a schematic diagram of the aggregation module of the image quality assessment method based on texture and color harmony perception described in the embodiments of the present invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] like Figure 1 As shown and Figure 2 The method of this invention includes three parallel feature extraction paths: a color composition path, a spatial texture path, and an image content path. The input image is preprocessed and then fed into each of the three paths. The extracted features are fused to output a quality score distribution.

[0027] Step 1: This step aims to deeply explore the complex relationships between the color distribution and internal color blocks of an image, and to calculate the color disharmony of the image. The specific implementation is as follows:

[0028] First, the input image I is simultaneously converted to the HSV color space while retaining its original RGB color space. Then, the information abundance of each channel in both color spaces is quantitatively evaluated. For a single channel image C in any color space, a Discrete Wavelet Transform (DWT) is performed, as follows: LL,LH,HL,HH=DWT(C), where LL are approximation coefficients, and LH,HL,HH are detail coefficients. Subsequently, the energy values ​​of these detail coefficients are calculated, and this energy value is defined as the information abundance representation value E for that channel. detail The formula is as follows: E detail =∑|LH| 2 +|HL| 2 +|HH| 2 This energy value is an approximate representation of the wavelet entropy of that channel. Next, in both the HSV and RGB color spaces, the two channels with the highest information abundance values ​​are selected as the key information channels for that color space. Then, the selected channel image I... channel Divide into k×k non-overlapping image blocks of equal size {P 11 ,P 12 ,...,P kk For each image block P ij The average value of all pixels within the image patch is calculated as the representation value v of that patch. ij : Where, N p is the total number of pixels within an image patch, where p is the pixel value. The representation values ​​of all k×k image patches constitute a feature matrix. It can be regarded as k 2 Each "color block center point" is identified. Then, this feature matrix M is flattened into a sequence. And input an attention module: first through a learnable weight matrix W Q W K and W V The input sequence X is mapped to a query matrix Q, a key matrix K, and a value matrix V, respectively, using the following formula: Q = XW Q K = XW K V = XW VNext, the attention score matrix A is calculated: Where d k It is the dimension of the key vector. The scaling factor is used. After the attention matrix A is flattened, the two matrices are concatenated, and then transformed and reduced in dimensionality using a fully connected layer L and a ReLU activation function to obtain the high-level color relation features f. attn The formula is as follows: f attn =ReLU(W L Flatten(A)+b L Ultimately, color constitutes the output feature f of the path. color That is, the high-level color relationship features f learned through the attention mechanism. attn .

[0029] Meanwhile, to provide a more direct measure of color harmony, this invention calculates the overall color disharmony D of the image. inharmony First, a standardized color template set is constructed, defining eight hue distribution templates: T = {i, V, L, mirror_L, I, T, Y, X}, where each template consists of a set of standardized sector regions (c, w), where c represents the normalized sector center position parameter (value range [0, 1], corresponding to the actual angle [0°, 360°]), and w represents the normalized sector width parameter (value range [0, 1], corresponding to the actual angle [0°, 360°]). Specific configurations include: template i is configured as a single sector region (0.00, 0.05), template V is configured as a single sector region (0.00, 0.26), and template... Template L is configured as a double-sector region [(0.00,0.05),(0.25,0.22)], template mirror_L is configured as a double-sector region [(0.00,0.05),(-0.25,0.22)], template I is configured as a double-sector region [(0.00,0.05),(0.50,0.05)], template T is configured as a single-sector region (0.25,0.50), template Y is configured as a double-sector region [(0.00,0.26),(0.50,0.05)], and template X is configured as a double-sector region [(0.00,0.26),(0.50,0.26)]. To enhance computational adaptability, for any template m∈T, a rotation angle α∈[0°,360°] is applied to generate a rotation-adaptive template: Here, α represents the template rotation angle parameter, enabling 360-degree omnidirectional adaptive positioning of the template on the color wheel, enhancing the system's adaptability to different color distributions. Next, after acquiring the input RGB image, an image color space conversion is first performed, converting the input RGB format image X_rgb to the HSV color space through a non-linear transformation: X_hsv =

[0030] RGB_to_HSV(X_rgb), where X_hsv contains three feature channels: hue channel H, saturation channel S, and lightness channel V. Next, the hue channel H is discretized: H quantized =floor(H×2)mod 360, where floor(·) represents the precise quantization of continuous hue values ​​into 360 discrete levels through modulo operation by rounding down, establishing a standard quantization system for hue values. Based on the quantization results, the pixel distribution density of each quantization level is calculated: histo[k]=count({p∈X∣H quantized (p)=k}), k=0,1,...,359, where histo[k] represents the number of pixels at hue level k, forming a complete statistical representation of hue distribution. To quantify the fit between image color and theoretical distribution, a hue ring distance metric is first defined to calculate the shortest arc distance between any two points on the hue ring: D(a,b)=min(∣ab∣,360-∣ab∣), where a and b represent two angle values ​​on the hue ring. This metric ensures the continuity of the toroidal surface in distance calculation and eliminates the 360-degree boundary effect. Based on this, a template boundary distance calculation model is established:

[0031]

[0032] Where h represents the hue value to be evaluated, c represents the center position of the sector, and w represents the sector width parameter. Furthermore, a single sector boundary distance function is defined:

[0033]

[0034] Finally, construct the boundary distance function for the multi-sector template: in This represents the minimum angular distance from the hue value h to the boundaries of all sectors of the rotated template T_m(α). This function returns zero when the hue is within any sector and the arc distance to the nearest boundary when it is outside the sector. Based on this, a comprehensive image quality scoring function is constructed: Where X represents the entire set of pixels in the input image, p represents a single pixel in the image, H(p) represents the quantized hue value of pixel p, and S(p) represents the normalized saturation value of pixel p. F(X,(m,α)) represents the minimum distance between the pixel hue and the template boundary. F(X,(m,α)) represents the overall quality score of image X relative to template m at a rotation angle α. The smaller the score, the higher the degree of fit between the image color distribution and the ideal distribution of the template.

[0035] Finally, the optimal template configuration is determined through a global search: (m, α) = argmin{m∈T, α∈

[0036] [0°, 360°]}, where m represents the optimal matching template, α represents the optimal rotation angle, and the system outputs the triplet evaluation result (m, α, D). inharmony ), where D inharmony =F(X,(m,α)) represents the minimum quality score, serving as a quantitative evaluation metric for image color quality. This incoherence will be treated as an additional loss term. The total loss is used to constrain the model's perception of color harmony.

[0037] Step 2: Extract edge information from the image using the Holistic Nested Edge Detection (HED) algorithm. The specific implementation is as follows:

[0038] The HED algorithm can output multiple side output layers, each capturing edge features at different scales and levels of abstraction. To preserve the overall layout while avoiding excessive detail interference, this step fuses the first and second layers of the HED algorithm with the final output. The fused edge map E fused It can be represented as: E fused =Stack(E side1 E side2 , E), where E side1 E side2 E represents the edge maps of the first, second, and final output layers of HED, respectively, and Stack(·) represents the stacking operation of feature maps. The resulting fused edge maps are then fed into a feature extraction network. The first layer of this network uses a 3×3 convolutional kernel for basic feature extraction, the second layer also uses a 3×3 convolutional kernel to further combine features, the third layer uses a 5×5 convolutional kernel to expand the receptive field, the fourth layer uses a 7×7 convolutional kernel to capture a more macroscopic distribution, and finally the fifth layer integrates global texture information through a 9×9 convolutional kernel. This step extracts features f that reflect the spatial texture of the image. space .

[0039] Step 3: This step aims to extract and fuse multi-scale visual features of the image, containing rich low-level details and high-level semantic information. The specific implementation is as follows:

[0040] First, the input image I is fed into a pre-trained lightweight backbone network, MobileNet, to obtain feature maps at different levels. Let the set of multi-scale feature maps output by the backbone network be {F1, F2, ..., F...}. L}, where L is the selected layer number, F l This represents the feature map output from layer l. Then, all feature maps F are processed using bilinear interpolation. l Adjust them to the same spatial size H×W, and use 1×1 convolutional layers to uniformly reduce their channel dimensions to 128, resulting in standardized feature maps. Then, these standardized multi-scale feature maps are input into a cascaded multi-scale feature aggregation module, such as... Figure 3 As shown, it includes parallel branches, using 1×1 convolution, 3×3 convolution, and max pooling (with a stride of 1) to perform multi-scale convolution operations to achieve non-linear feature aggregation. This process can be formalized as: F content =Concat[Conv 1×1 (F in ),Conv 3×3 (F in ),MaxPool(F in )], where F in These are the input features of the aggregation module, Conv k×k This represents a k×k convolution operation, MaxPool represents a max pooling operation, and Concat represents a concatenation operation along the channel dimension. Cascaded aggregation modules can progressively and effectively aggregate high-level and low-level features at different scales, enhancing feature discriminative power and suppressing noise interference; finally, F... content Image content features f are obtained through 1×1 convolution. content .

[0041] Step 4: Extract the feature vector f from the three paths mentioned above. color ,f space ,f content The weight coefficients of each feature are calculated using three independent weight generation networks:

[0042] w i =σ(W i f i +b i )

[0043] Where σ represents the Sigmoid activation function, f i W represents the various features of the image. i and b i These are the learnable parameters corresponding to each path. Then, the original features are weighted and concatenated:

[0044] f fused =Concat(w1·f color ,w2·f space ,w3·f content )

[0045] Finally, image quality is scored using the following formula:

[0046] q=Softmax(W5·Dropout(ReLU(W4·GAP(f fused )+b4))+b5)

[0047] Where GAP represents global average pooling, the Dropout rate is 0.5, and the final output q is the image quality score.

[0048] The model training employs a comprehensive loss function, including EMD loss and color inconsistency loss. The total loss function is defined as: in The main loss for predicting the score distribution is calculated as follows: Where d is the true rating distribution. This is the rating distribution predicted by the model, N=10, and CDF is the cumulative distribution function, r=2. The color disharmony loss is calculated by step S1 and is used to constrain the model to learn color harmony. λ is a hyperparameter used to balance the contributions of the two losses.

[0049] The model was trained using the Adam optimizer with an initial learning rate of 1×10⁻⁶. -5 Finally, the model's performance was evaluated using metrics such as the Pearson linear correlation coefficient (PLCC) and the Spearman rank correlation coefficient (SRCC). The formulas for calculating PLCC and SRCC are as follows:

[0050]

[0051] Where, x i and y i These are the predicted rating and the actual rating. and These are their average values, the ranking difference between the test score and the actual score, respectively.

[0052] This invention was validated on the LIVEC (live in the wild image quality challenge) and TID2013 open-source datasets. The experimental results are shown in Table 1 below:

[0053] Table 1

[0054]

[0055] The above results demonstrate that this invention effectively improves the accuracy of image quality assessment by constructing a multi-path feature extraction architecture and a color harmony metric evaluation mechanism, while also enhancing the interpretability of the model. This provides an innovative solution to the problems of insufficient interpretability and single feature expression in traditional evaluation methods.

[0056] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. An image quality assessment method based on texture and color harmony perception, characterized in that, Includes the following steps: S1: Convert the input image to HSV and RGB color spaces respectively, obtain color features after passing through the attention module, and calculate color dissonance; S2: The HED algorithm for global nested edge detection is used to extract multi-scale edge features from the input image, and then spatial texture features are obtained through the feature extraction module. S3: Utilize the lightweight MobileNet network to extract multi-scale features from the input image and perform progressive feature fusion to form image content features; S4: Color features, spatial texture features, and image content features are fed into the induction fusion network, and the image quality score is output through softmax. The network is then trained by combining color inconsistency.

2. The image quality assessment method based on texture and color harmony perception according to claim 1, characterized in that, The specific implementation of step S1 is to convert the input image to HSV and RGB color spaces respectively, and select the two channels with the richest information by calculating the wavelet entropy of each color channel in HSV and RGB color spaces respectively. The two channel images are then input into the attention module to obtain color features; at the same time, the color incongruity is calculated based on the input images.

3. The image quality assessment method based on texture and color harmony perception according to claim 2, characterized in that, The specific process for obtaining color features is as follows: For color feature extraction, the original RGB image is input. First, the image is converted to HSV and RGB color spaces respectively. By calculating the wavelet entropy of each color channel in the HSV and RGB color spaces, the two channels with the richest information are selected respectively. Then, the images of these two channels are divided into k×k non-overlapping image blocks. The average pixel value of each image block is extracted as the block feature. The self-attention module is input to model the relationship between color blocks to obtain the attention score matrix. Finally, the color features are obtained through a fully connected layer.

4. The image quality assessment method based on texture and color harmony perception according to claim 2, characterized in that, The specific process for calculating color incongruity is as follows: Simultaneously, based on the HSV color space, the hue channel is extracted and discretized and quantized, and the pixel distribution density of each hue level is statistically analyzed to obtain the hue distribution statistical features of the image. Then, the minimum distance between the hue value of each pixel and the boundary of each set rotating template is calculated, and combined with the pixel saturation information, the degree of deviation between the image color distribution and the ideal distribution of the template is evaluated to obtain the comprehensive quality score under the current template configuration. Finally, by traversing all templates and their rotation angles through a global search, the optimal template configuration that minimizes the comprehensive quality score is found. This minimum score value is the color incongruity of the image.

5. The image quality assessment method based on texture and color harmony perception according to claim 1, characterized in that, The specific implementation process of step S2 is as follows: use the overall nested edge detection HED algorithm to extract multi-scale edge features, fuse the multi-scale edge features with the output of the first and second layers of HED, and input the fused edge features into the feature extraction module to obtain spatial texture features.

6. The image quality assessment method based on texture and color harmony perception according to claim 1, characterized in that, The specific implementation process of step S3 is as follows: multi-scale features are extracted using a pre-trained MobileNet backbone network. The multi-scale features are unified in size and number of channels through bilinear interpolation and 1×1 convolution. Then, they are input into a cascaded multi-scale feature aggregation module for progressive fusion. After splicing, the image content features are obtained through 1×1 convolution.

7. The image quality assessment method based on texture and color harmony perception according to any one of claims 1 to 6, characterized in that, The specific implementation process of step S4 is as follows: color features, spatial texture features, and image content features are input into a gated fusion network. First, color features, spatial texture features, and image content features are respectively passed through three independent fully connected layers and a Sigmoid activation function to generate weights q1, q2, and q3 corresponding to the three features. Then, weights q1, q2, and q3 are used to weight the color features, spatial texture features, and image content features, and the weighted features are concatenated in the channel dimension to form fused features. The fused features are processed sequentially through a global average pooling layer, a fully connected layer, and a Dropout layer, and the final output image quality score is obtained. During training, EMD loss and color inconsistency are weighted and combined for joint optimization.

Images