An image quality evaluation method based on saliency information fusion

Abstract

The application belongs to the technical field of deep learning, and particularly relates to an image quality evaluation method based on saliency information fusion, which comprises the following steps: firstly, extracting a saliency target mask of an image; then, learning the interaction relationship between each component by encoding the features of the salient region and the non-salient region through an attention mechanism; then, enhancing the features extracted by the network and improving the global context perception ability through a channel decoupling spatial attention module; finally, designing a saliency target guided Transformer network, realizing the deep fusion of saliency information and spatial domain features through a self-attention mechanism and a cross-attention mechanism, so as to more effectively improve the performance of image quality evaluation. The salient target region and the non-salient target region are divided in the form of components, and the self-attention is calculated between different components and in the same component, so that the salient target information is fully utilized, and more effective and more human visual perception conforming feature fusion is realized.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, specifically relating to an image quality evaluation method based on saliency information fusion. Background Technology

[0002] With the widespread adoption of visual media applications and portable devices, massive amounts of images circulate on the internet, making effective monitoring and improvement of image quality a key challenge. Traditional subjective evaluation methods rely on manual observation and scoring, which, while yielding highly reliable results, require strict laboratory environments and are time-consuming, expensive, and difficult to apply on a large scale. However, image quality control is not only a core requirement for ensuring user experience and security but also a crucial basis for optimizing image processing algorithms (such as denoising and super-resolution). Therefore, researchers have turned to objective evaluation methods that do not require human intervention, namely, automatically assessing image quality by calculating visual features.

[0003] Objective evaluation methods are categorized into three types based on the availability of reference images: full-reference (relying on a complete reference image to achieve high-precision evaluation through similarity comparison), semi-reference (utilizing partial reference information), and no-reference (completely relying on the features of the distorted image itself). While full-reference methods offer superior performance, reference images are often unavailable in real-world scenarios. Therefore, no-reference methods have become a research hotspot, and breakthroughs in their application have greater practical value for real-world applications (such as social media and medical image analysis).

[0004] In recent years, deep learning-based image quality assessment (IQA) methods have developed rapidly. Early methods attempted to use shallow CNNs, but their performance was limited by insufficient receptive field and global information. To learn better feature representations, some deep CNN-based NR-IQA models have attempted to use multi-feature fusion learning strategies, mainly manifested in the fusion learning of local and global features. Leveraging these different but closely related features in images, these models can learn more discriminative feature representations from the raw input data, thereby effectively improving quality prediction performance. For example, Yang et al. designed an NR-IOA model based on a multi-task learning framework. They first extracted distortion type and distortion level features of SCI separately, and then combined these two types of features for quality score prediction, fusing distortion features as local and global features. However, a large number of internet images captured by real cameras are often affected by complex mixtures of various real-world distortions, and the distortions generated by the limited algorithms in traditional IQA datasets cannot well simulate these distortions. Therefore, this distortion recognition subtask cannot accurately identify the complex distortion mixtures present in real-world distorted images, and may lead to performance degradation when applied to evaluate real-world images with various real-world distortions.

[0005] To address the aforementioned limitations, researchers used visual saliency prediction as a local feature. Compared to local features in distortion recognition, visual saliency always exists when viewing each image, regardless of the type of distortion. Furthermore, since humans tend to focus on visually salient regions when evaluating image quality, visual saliency is closely related to image quality. However, existing saliency-based NR-IQA methods still have certain shortcomings in utilizing saliency information. Most methods simply estimate the saliency map as a local weighting function, assigning weights to different regions of the image to guide quality prediction, without deeply integrating saliency information into the network's feature representation learning process. Moreover, these methods often employ a uniform feature extraction approach when dealing with salient and non-salient regions, neglecting to differentiate between these two types of regions, thus failing to fully explore the potential differential features between salient and non-salient regions and the interaction information between them. Therefore, in order to achieve a more ideal image quality assessment result, it is necessary to design a deep learning model that can make fuller and more effective use of visual saliency information and distinguish between salient and non-salient regions of an image, so as to overcome the limitations of existing methods and further improve the performance of image quality assessment. Summary of the Invention

[0006] To address the technical problems of existing saliency-based NR-IQA methods, this invention provides an image quality evaluation method based on saliency information fusion.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] An image quality assessment method based on saliency information fusion includes the following steps:

[0009] S1. An adaptive preprocessing method combining full-reference and no-reference tasks is adopted to extract the salient target mask of the input image for subsequent feature extraction.

[0010] S2. Encode salient and non-salient regions as components, and learn the interaction relationship between different components by calculating inter-component attention and intra-component attention. The salient target embedding network realizes differentiated feature processing through the salient target component encoder module.

[0011] S3. Use channel decoupling spatial attention to enhance the features extracted by the feature extraction network;

[0012] S4. A Transformer network guided by saliency targets is adopted. This network utilizes self-attention and cross-attention mechanisms to achieve deep interaction and fusion between saliency features and spatial domain features, thereby more effectively utilizing saliency information to improve the accuracy of image quality assessment.

[0013] The method for extracting the salient target mask of the input image in S1 is as follows: For a full-reference image quality assessment task, a reference image and a distorted image are input; for a no-reference image quality assessment task, a distorted image is input; the salient target mask of the input image is extracted. For the full-reference task, the salient target mask of the reference image is extracted and appended to the corresponding distorted image. For the no-reference task, the salient target mask is directly extracted from the distorted image, and the RGB image and the salient target mask are resized and cropped; the salient target mask is implemented through a pre-trained salient target detection algorithm, and a binary mask image is output, where white areas represent salient areas in the image and black areas represent non-salient areas.

[0014] The method for differential feature processing in the salient target embedding network in S2 through the salient target component encoder module is as follows: First, the primary feature extraction network extracts features from salient and non-salient regions. Then, independent convolutional layers are used to process foreground and background features respectively, and the convolutional kernel parameters are optimized through non-shared weights. Then, through component partitioning and tokenization, the two types of features are divided into multiple local components, and transformed into a feature vector sequence through linear mapping, which serves as the input to the subsequent Transformer network. The Transformer network captures the internal relationships and interdependencies between salient and non-salient regions through attention calculations within and between components, thereby achieving feature enhancement of salient information.

[0015] The method for enhancing the feature extraction network using channel decoupling and spatial attention in S3 is as follows: First, through the channel attention submodule, a multilayer perceptron structure is used to learn channel weights, achieving adaptive adjustment of the importance of features from different channels; then, a channel shuffling operation is introduced to promote information interaction and fusion between channel groups, achieving channel decoupling; based on this, the spatial attention submodule is further used to learn spatial weights through a convolutional neural network, achieving adaptive adjustment of the importance of features from different spatial locations.

[0016] X out =Shuffle((X*σ(A)) channel ))*σ(A spatial ))

[0017] Where: X represents the input feature, σ represents the sigmoid function, and A channel A represents the channel attention result. spatialThis represents the spatial attention result, and Shuffle represents the channel shuffling algorithm.

[0018] The method for deep fusion of saliency information and spatial features in the salient target-guided Transformer network in S4 is as follows: First, the spatial domain features Fs' weighted by the channel decoupling spatial attention module are received, aiming to enhance the feature representation of salient regions; then, this module utilizes the self-attention mechanism in the Transformer network to capture long-distance dependencies between different locations in the image, thereby achieving modeling of the global structure of the image; to further fuse salient features, this module introduces a cross-attention mechanism, the core idea of ​​which is to use salient features to guide the learning of spatial features; the salient features Fm' processed by the salient feature enhancement module are used as Query(Q), and the spatial domain features Fs' are used as Key(K) and Value(V) for cross-attention calculation.

[0019]

[0020] Where: Fm' represents the salient features after processing by the salient feature enhancement module, and Fs' represents the spatial domain features. This represents the scaling factor.

[0021] The primary feature extraction network is constructed as follows: the primary feature extraction network implements dual-branch feature separation based on an improved ResNet50 backbone. The spatial domain feature extraction branch generates multi-scale feature maps with 256, 512, 1024, and 2048 channels through four stages, ultimately outputting a 2048-dimensional feature map F containing texture, edge, and color information. s Simultaneously, the salient target extraction branch uses a pre-trained salient detection algorithm to generate a binary mask, which is then used to extract the F... s Decomposed into foreground features F fg =F s ⊙M fg and background features F bg =F s ⊙M bg The element-wise multiplication of the mask isolates salient and non-salient region information.

[0022] The method for constructing the salient object embedding network is as follows: the salient object embedding network differentially encodes two types of features through a salient object component encoder (SCE) module. First, it uses parameter-isolated independent 1×1 convolutional layers to process F separately. fg and F bg The features are then divided into N along the spatial dimension. t ×H t ×W tOverlapping local components, each containing multiple adjacent image patches to preserve local context, and a 512-dimensional token sequence is generated through linear mapping, with an output shape of B,N. f ′,H′W′,C′.

[0023] Compared with the prior art, the beneficial effects of this invention are:

[0024] This invention divides salient and non-salient target regions into components and calculates self-attention both between different components and within the same component, fully utilizing salient target information to achieve more effective feature fusion that better aligns with human visual perception. The proposed method has achieved excellent performance metrics on several publicly available image quality assessment benchmark datasets. Specifically, on the TID2013 dataset, the model achieves SRCC and PLCC metrics as high as 0.9642 and 0.9731, respectively; on the CSIQ dataset, it reaches outstanding levels of 0.9821 and 0.9886, respectively; and on the KONIQ-10k real-world image quality assessment dataset, it also achieves excellent results with SRCC of 0.9181 and PLCC of 0.9240. Attached Figure Description

[0025] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0026] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0027] Figure 1 This is a diagram of the overall architecture of the present invention;

[0028] Figure 2 This is a structural diagram of the salient target component encoder module and the salient feature enhancement module of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. These descriptions are only for further illustrating the features and advantages of the present invention, and not for limiting the claims of the present invention. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0031] This invention is implemented in the PyTorch deep learning framework. It provides an image quality assessment method based on saliency information fusion, specifically including the following steps:

[0032] 1. Data Preparation

[0033] The image quality assessment model training and performance evaluation in this embodiment rely on three widely used public datasets: TID2013, CSIQ, and KONIQ-10k. The TID2013 dataset contains 25 reference images, each with 24 different types of distortion applied, each with five levels, totaling 3000 distorted images covering various traditional distortion types. The CSIQ dataset contains 30 original images, each undergoing six distortion types and varying degrees of distortion, totaling 866 distorted images, also including various typical distortions. The KONIQ-10k dataset is a large-scale real-world image quality assessment dataset, containing 10073 real distorted images from the internet, with complex and diverse distortion types, more closely resembling real-world application scenarios. To ensure sufficient model training and objective evaluation, 70% of the images in each dataset are randomly selected for model training, and the remaining 30% are used for performance testing.

[0034] In the data preprocessing stage, this embodiment adopts different strategies for two image quality assessment tasks: full-reference (FR) and no-reference (NR). For the full-reference task, considering that humans often focus on the salient regions of the reference image to form an initial impression, which can influence the judgment of the distorted image, this embodiment pre-extracts the salient object mask of the reference image and uses a pre-trained BiRefNet model to perform mask extraction. Subsequently, the RGB image of the reference image and its salient object mask are cropped to the same size. For the no-reference task, the salient object mask of the input distorted image itself is directly extracted, also using the BiRefNet model. After the salient object mask extraction is completed, regardless of whether it is a full-reference or no-reference task, all images and their salient object masks are uniformly cropped to a size of 384×384 pixels to eliminate the influence of image size differences, and random horizontal flipping is used to increase the diversity of training samples.

[0035] 2. Model Building

[0036] The constructed model consists of four sub-networks: a primary feature extraction network, a salient object embedding network, a salient object-guided transformer network, and a quality score regression network. The specific network model structure is as follows: Figure 1 As shown, the primary feature extraction network achieves dual-branch feature separation based on an improved ResNet50 backbone. The spatial domain feature extraction branch generates multi-scale feature maps with 256, 512, 1024, and 2048 channels through four stages (stage 1 to stage 4), ultimately outputting a 2048-dimensional feature map F containing texture, edge, and color information. s Simultaneously, the salient target extraction branch uses a pre-trained salientity detection algorithm to generate a binary mask, which is then used to extract the F... s Decomposed into foreground features M fg =F s ⊙M fg and background features F bg =F s ⊙M bg The element-wise multiplication of the mask isolates salient and non-salient region information; such as Figure 2 As shown, the salient object embedding network differentially encodes two types of features through the SCE module. First, it uses parameter-isolated independent 1×1 convolutional layers to process F... fg and F bg The features are then divided into N along the spatial dimension. t ×H t ×W t Overlapping local blocks (components) are generated, each containing multiple adjacent image blocks to preserve local context. A 512-dimensional token sequence is generated through linear mapping, with an output shape of B,N.f ′,H′W′,C′; The salient target-guided Transformer network achieves feature interaction through Local-FC and Cross-FC attention mechanisms, where Local-FC computes self-attention within each component (formula is given). Focusing on local detail relationships, Cross-FC achieves global cross-attention between components through dimension permutation, and enhances nonlinear expressive power by combining a feedforward network (containing two fully connected layers and GELU activation). The quality score regression network first optimizes features through a Global Channel Decoupled Spatial Attention Module (GCSA). Channel attention uses fully connected layers to compress and restore channel weight matrices, and spatial attention generates spatial weight matrices through 7×7 convolutions. The output feature is F. out =F⊙A c +F⊙A s Finally, the feature weights w and quality scores f are predicted by the two-branch regression head, and then weighted and summed by Q. pred =∑(f i ·w i ) / (∑w i The +∈) output quality score assigns higher weights to salient regions to align with human subjective evaluation characteristics. The entire model achieves efficient and accurate image quality evaluation through bi-branch feature separation, overlapping component partitioning, cross-attention fusion, and channel decoupling spatial collaborative optimization.

[0037] 3. Model Training

[0038] During training, the constructed model achieves end-to-end learning by jointly optimizing the Huber loss and ranking loss. The specific process is as follows:

[0039] Huber loss: The absolute error between the supervised predicted quality score and the actual MOS value, expressed as follows:

[0040]

[0041] Where: δ=1.0, f(x) i ) represents the model's predicted score, y i This is the actual MOS value.

[0042] Ranking loss: forces the model to maintain consistency in the quality ranking among samples, and is defined as:

[0043]

[0044] Where C is the number of valid sample pairs, y i >y j This indicates the actual quality ranking.

[0045] Total loss: The weighted sum of the two losses, with a weighting coefficient λ = 0.5, i.e.:

[0046] L total =L Huber +λL rank

[0047] 4. Test Results

[0048] During testing, the input image was cropped and adjusted to a uniform size of 384*384 using the conditional center, and then normalized (mean [0.485, 0.456, 0.406], standard deviation [0.229, 0.224, 0.225]) to fit the model input. During testing, the model outputs a predicted quality score, and the Spearman rank correlation coefficient (SROCC) and Pearson linear correlation coefficient (PLCC) are calculated against the actual MOS value as core evaluation indicators.

[0049] 5. Model Evaluation

[0050] The Spearman rank correlation coefficient (SROCC) and Pearson linear correlation coefficient (PLCC) were calculated using the predicted quality score and the true MOS value to evaluate the performance of the model. The best model is shown in bold.

[0051] Table 1 Comparison results of different methods on open datasets.

[0052]

[0053] TOPIQA's core innovation lies in its cross-scale attention mechanism, leveraging a high-level semantically guided model to focus on important local distortion regions. On real-world datasets like Koniq-10k, TOPIQA's cross-scale feature fusion strategy effectively captures the spatial distribution of complex distortions, thus achieving the best performance. However, its drawback lies in insufficient sensitivity to structured distortions, as high-level semantic guidance may mask low-frequency distortion details. VCRNET generates restored images through a visual compensation restoration network and predicts quality scores based on restoration features. Although it performs reasonably well on the LIVE dataset, its reliance on a non-adversarial restoration module results in insufficient restoration capabilities for severely distorted images, making it difficult to accurately establish the quality mapping between degraded and restored images. MANIQA introduces Transposed Attention Blocks (TABs) and Scale Rotation Transformers (SSTBs) to enhance global-local interactions, achieving superior performance on the LIVE dataset. However, its multi-dimensional attention mechanism does not explicitly distinguish between salient and non-salient regions, leading to oversensitivity to background noise and large fluctuations in prediction scores; therefore, it was not tested on Koniq-10k. Both SGDNET and this study use salient targets as local features, but SGDNET only performs a simple weighted fusion of salient and insalient regions, ignoring the complex interaction between these two regions by the human eye, resulting in poor performance. In contrast, the model proposed in this study divides salient and insalient target regions into components and calculates self-attention separately between different components and within the same component. This fully utilizes salient target information, achieving more effective feature fusion that better aligns with human visual perception, thus demonstrating the superiority of our approach.

[0054] The above description only illustrates the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention, and all such changes should be included within the protection scope of the present invention.

Claims

1. An image quality assessment method based on saliency information fusion, characterized in that, Includes the following steps: S1. An adaptive preprocessing method combining full-reference and no-reference tasks is adopted to extract the salient target mask of the input image for subsequent feature extraction. S2. Encode salient and non-salient regions as components, and learn the interaction relationship between different components by calculating inter-component attention and intra-component attention. The salient target embedding network realizes differentiated feature processing through the salient target component encoder module. First, a primary feature extraction network extracts features from salient and non-salient regions. Then, independent convolutional layers are used to process foreground and background features separately, and the convolutional kernel parameters are optimized through non-shared weights. Next, through component partitioning and tokenization, the two types of features are divided into multiple local components, and transformed into a sequence of feature vectors through linear mapping, which serves as the input to the subsequent Transformer network. The Transformer network captures the internal relationships and interdependencies between salient and non-salient regions through intra-component and inter-component attention calculations, thereby achieving feature enhancement of salient information. The primary feature extraction network is constructed as follows: the primary feature extraction network implements dual-branch feature separation based on an improved ResNet50 backbone. The spatial domain feature extraction branch generates multi-scale feature maps with 256, 512, 1024, and 2048 channels through four stages, and finally outputs a 2048-dimensional feature map containing texture, edge, and color information. Simultaneously, the salient target extraction branch uses a pre-trained salientity detection algorithm to generate a binary mask, which... Decomposed into foreground features and background features The element-wise multiplication of the mask isolates salient and non-salient region information; The method for constructing the salient object embedding network is as follows: the salient object embedding network differentially encodes two types of features through a salient object component encoder (SCE) module, and firstly, it processes them separately using parameter-isolated independent 1×1 convolutional layers. and Then the features are divided along the spatial dimension into Overlapping local block components, each containing multiple adjacent image blocks to preserve local context, and generating a 512-dimensional token sequence through linear mapping, with an output shape of... ; S3. Use channel decoupling spatial attention to enhance the features extracted by the feature extraction network; S4. A Transformer network guided by saliency targets is adopted. This network utilizes self-attention and cross-attention mechanisms to achieve deep interaction and fusion between saliency features and spatial domain features, thereby more effectively utilizing saliency information to improve the accuracy of image quality assessment. First, the spatial domain features Fs', weighted by the channel-decoupled spatial attention module, are received to enhance the feature representation of salient regions. Then, this module utilizes the self-attention mechanism in the Transformer network to capture long-distance dependencies between different locations in the image, thereby achieving modeling of the global structure of the image. To further integrate salient features, this module introduces a cross-attention mechanism, the core idea of ​​which is to use salient features to guide the learning of spatial features. The salient feature Fm' processed by the salient feature enhancement module is used as Query (Q), and the spatial domain feature Fs' is used as Key (K) and Value (V) for cross-attention calculation. Where: Fm' represents the salient features after processing by the salient feature enhancement module, and Fs' represents the spatial domain features. This represents the scaling factor.

2. The image quality assessment method based on saliency information fusion according to claim 1, characterized in that, The method for extracting the salient target mask of the input image in S1 is as follows: For the full reference image quality assessment task, the input is a reference image and a distorted image; For the no-reference image quality assessment task, the input is a distorted image; the salient object mask of the input image is extracted. The full-reference task extracts the salient object mask of the reference image and appends it to the corresponding distorted image. The no-reference task directly extracts the salient object mask from the distorted image and resizes and crops the RGB image and the salient object mask. The salient object mask is implemented through a pre-trained salient object detection algorithm, and the output is a binary mask image, where white areas represent salient regions in the image and black areas represent non-salient regions.

3. The image quality assessment method based on saliency information fusion according to claim 1, characterized in that, The method for extracting features using the channel decoupling spatial attention enhanced feature extraction network in S3 is as follows: First, through the channel attention submodule, the channel weights are learned using a multilayer perceptron structure to achieve adaptive adjustment of the importance of features in different channels; Subsequently, a channel shuffling operation is introduced to promote information interaction and fusion between channel groups, achieving channel decoupling. Building upon this, a spatial attention submodule is further utilized to learn spatial weights through a convolutional neural network, enabling adaptive adjustment of the importance of features at different spatial locations. Where: X represents the input feature. This represents the sigmoid function. This indicates the channel attention result. This represents the spatial attention result. This indicates the channel shuffling algorithm.

Images