Image quality evaluation method based on saliency information fusion

Abstract

The invention belongs to the technical field of deep learning, and particularly relates to a saliency information fusion-based image quality evaluation method, which comprises the following steps of: firstly, extracting a saliency target mask of an image, and further learning an interaction relationship among components by utilizing an attention mechanism through coding the characteristics of a salient region and a non-salient region; a space attention module is decoupled through a channel, features extracted by the network are enhanced, and the global context sensing capability is improved; and finally, designing a saliency target guided Transform network, and realizing deep fusion of saliency information and spatial domain features through a self-attention mechanism and a cross attention mechanism, thereby more effectively improving the performance of image quality evaluation. According to the method, the salient target region and the non-salient target region are divided in a component form, and self-attention is calculated between different components and in the same component, so that salient target information is fully utilized, and feature fusion which is more effective and more in line with human visual perception is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and particularly relates to an image quality evaluation method based on saliency information fusion. Background Art

[0002] With the popularization of visual media applications and portable devices, a large number of images are circulating on the Internet. How to effectively monitor and improve image quality has become a key challenge. Traditional subjective evaluation methods rely on manual observation and scoring. Although high-confidence results can be obtained, they require a strict laboratory environment and are time-consuming and expensive, making it difficult to be applied on a large scale. However, image quality control is not only a core requirement for ensuring user experience and safety, but also an important basis for optimizing image processing algorithms (such as denoising, super-resolution, etc.). Therefore, researchers have turned to objective evaluation methods that do not require manual intervention, that is, automatically evaluate image quality by calculating visual features of images.

[0003] Objective evaluation methods are divided into three categories according to the availability of reference images: full-reference (relying on a complete reference image to achieve high-precision evaluation through similarity comparison), semi-reference (utilizing partial reference information), and no-reference (completely relying on the characteristics of the distorted image itself). Although full-reference methods have excellent performance, reference images are often not available in actual scenarios. Therefore, no-reference methods have become a research hotspot, and their technological breakthroughs have higher practical value for real-world applications (such as social media, medical image analysis).

[0004] In recent years, image quality assessment (IQA) methods based on deep learning have developed rapidly. Early methods tried to use shallow CNNs, but were limited by insufficient receptive fields and global information, resulting in limited performance. To learn better feature representations, some deep-CNN-based NR-IQA models attempt to use a learning strategy of multi-feature fusion, mainly manifested as the fusion of local features and global features. By leveraging these different but closely related features in images, these models can learn more discriminative feature representations from the input raw data, thereby effectively improving quality prediction performance. For example, Yang et al. designed an NR-IOA model based on a multi-task learning framework. First, the distortion type and distortion level features of SCI were extracted separately, and then these two types of features were combined for quality score prediction. They fused the distortion features as local features and global features. However, a large number of Internet images captured by real cameras are usually affected by a complex mixture of various real distortions, and the limited algorithms-generated distortions in traditional IQA datasets cannot well simulate these distortions. Therefore, this distortion recognition sub-task cannot accurately identify the complex distortion mixtures existing in real distorted images and may lead to performance degradation when applied to evaluate real-world images with various real distortions.

[0005] To address the above existing limitations, researchers use visual saliency prediction as a local feature. Compared with the local features for distortion recognition, visual saliency always exists when viewing each image regardless of the type of distortion. And since humans tend to focus on visually salient regions when evaluating image quality, visual saliency is closely related to image quality. However, existing saliency-based NR-IQA methods still have certain deficiencies in utilizing saliency information. Most methods simply estimate the saliency map as a local weighted function, applying weights to different regions of the image through the saliency map to guide quality prediction, but fail to more deeply integrate the saliency information into the process of learning the feature representation of the network. In addition, when dealing with salient regions and non-salient regions, these methods often adopt a unified feature extraction method, ignoring the differential processing of these two types of regions, thus unable to fully exploit the possible differential features of salient regions and non-salient regions and the interaction information between them. Therefore, in order to obtain a more ideal image quality evaluation effect, it is necessary to design a deep learning model that can more fully and effectively utilize visual saliency information and can distinguish between salient regions and non-salient regions of the image to overcome the limitations of existing methods and further improve the performance of image quality evaluation. Summary of the Invention

[0006] In view of the above technical problems existing in the existing saliency-based NR-IQA methods, the present invention provides an image quality evaluation method based on saliency information fusion.

[0007] To solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0008] An image quality evaluation method based on saliency information fusion, comprising the following steps:

[0009] S1. Adopt an adaptive preprocessing method that combines full-reference and no-reference tasks to extract the saliency object mask of the input image for subsequent feature extraction;

[0010] S2. Encode the salient region and the non-salient region into components, and let the model learn the interaction relationship between different components by calculating the attention between components and the attention within components. The salient object embedding network realizes differential feature processing through the salient object component encoder module;

[0011] S3. Use channel decoupled spatial attention to enhance the features extracted by the feature extraction network;

[0012] S4. Adopt a Transformer network guided by significant objects. This network utilizes the self-attention mechanism and the cross-attention mechanism to achieve deep interaction and fusion between significant features and spatial domain features, thereby more effectively using significant information to improve the accuracy of image quality assessment.

[0013] The method for extracting the significant object mask of the input image in S1 is as follows: For the full-reference image quality evaluation task, input the reference image and the distorted image; for the no-reference image quality evaluation task, input the distorted image; extract the significant object mask of the input image. In the full-reference task, extract the significant object mask of the reference image and attach it to the corresponding distorted image. In the no-reference task, directly extract the significant object mask from the distorted image, and perform size adjustment and cropping on the RGB image and the significant object mask. The significant object mask is implemented through a pre-trained significant object detection algorithm, and a binary mask image is output, where the white area represents the significant region in the image, and the black area represents the non-significant region.

[0014] The method for the significant object embedding network to achieve differential feature processing through the significant object component encoder module in S2 is as follows: First, the primary feature extraction network extracts the features of the significant and non-significant regions. Then, use independent convolutional layers to process the foreground and background features respectively, and optimize the convolutional kernel parameters through non-shared weights. Then, through component division and Tokenization, divide the two types of features into multiple local components, and convert them into a sequence of feature vectors through linear mapping as the input of the subsequent Transformer network. The Transformer network captures the internal relationships and interdependencies between the significant region and the non-significant region through intra-component and inter-component attention calculations, thereby realizing feature enhancement of significant information.

[0015] The method for using channel decoupled spatial attention to strengthen the features extracted by the feature extraction network in S3 is as follows: First, through the channel attention sub-module, use a multi-layer perceptron structure to learn channel weights to achieve adaptive adjustment of the importance of different channel features. Subsequently, introduce a channel shuffle operation to promote information interaction and fusion between channel groups and achieve channel decoupling. On this basis, further use the spatial attention sub-module to learn spatial weights through a convolutional neural network to achieve adaptive adjustment of the importance of features at different spatial positions:

[0016] X out =Shuffle((X*σ(A channel ))*σ(A spatial ))

[0017] Where: X represents the input feature, σ represents the sigmoid function, A channel represents the channel attention result, A spatialIt represents the spatial attention result, and Shuffle represents the channel shuffle algorithm.

[0018] The method for the significant object-guided Transformer network in S4 to achieve the deep fusion of saliency information and spatial features is as follows: First, it receives the spatial domain feature Fs' weighted by the channel decoupled spatial attention module, aiming to enhance the feature expression of the salient region; then, this module uses the self-attention mechanism in the Transformer network to capture the long-range dependencies between different positions in the image, thereby realizing the modeling of the global structure of the image; in order to further fuse the saliency features, this module introduces the cross-attention mechanism, and its core idea is to use the saliency features to guide the learning of spatial features; taking the saliency feature Fm' processed by the saliency feature enhancement module as Query (Q), taking the spatial domain feature Fs' as Key (K) and Value (V), and performing cross-attention calculation:

[0019]

[0020] where: Fm' represents the saliency feature processed by the saliency feature enhancement module, and Fs' represents the spatial domain feature, represents the scaling coefficient.

[0021] The construction method of the primary feature extraction network is as follows: The primary feature extraction network realizes the separation of dual-branch features based on the improved ResNet50 backbone. Among them, the spatial domain feature extraction branch gradually generates multi-scale feature maps with 256, 512, 1024, and 2048 channels through four stages, and finally outputs a 2048-dimensional feature map F containing texture, edge, and color information s ; at the same time, the significant object extraction branch uses the pre-trained saliency detection algorithm to generate a binary mask, and decomposes F s into foreground feature F fg =F s ⊙M fg and background feature F bg =F s ⊙M bg , and isolates the significant and non-significant region information through the element-wise multiplication of the mask.

[0022] The construction method of the significant object embedding network is as follows: The significant object embedding network differentially encodes the two types of features through the significant object component encoder (SCE) module. First, it uses independent 1×1 convolutional layers with parameter isolation to process F fg and F bg , and then divides the features along the spatial dimension into N t ×H t ×W tOverlapping local components, each component contains multiple adjacent image patches to preserve local context, and generates a 512-dimensional token sequence through linear mapping, with the output shape being B, N f ′, H′W′, C′.

[0023] The beneficial effects of the present invention compared with the prior art are as follows:

[0024] The present invention divides the significant target area and the non-significant target area in the form of components, and calculates self-attention between different components and within the same component respectively, making full use of the significant target information and achieving more effective and more human vision perception-compliant feature fusion. The method proposed by the present invention has achieved excellent performance indicators on multiple publicly available image quality evaluation benchmark datasets. Specifically, on the TID2013 dataset, the SRCC and PLCC indicators of the model are as high as 0.9642 and 0.9731 respectively; on the CSIQ dataset, they reach the excellent levels of 0.9821 and 0.9886 respectively; on the KONIQ-10k real-world image quality evaluation dataset, excellent results of SRCC 0.9181 and PLCC 0.9240 are also obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only exemplary, and for those of ordinary skill in the art, without creative efforts, other implementation drawings can be obtained according to the provided drawings.

[0026] The structures, ratios, sizes, etc. illustrated in this specification are only used to cooperate with the content disclosed in the specification for those familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention. Therefore, they do not have technical essence. Any modification of the structure, change of the ratio relationship or adjustment of the size, without affecting the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope covered by the technical content disclosed in the present invention.

[0027] Figure 1 It is the overall architecture diagram of the present invention;

[0028] Figure 2 It is the structure diagram of the significant target component encoder module and the significant feature enhancement module of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. These descriptions are only for further explaining the features and advantages of the present invention, rather than limiting the claims of the present invention; based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0030] The following will further describe in detail the specific implementation manners of the present invention in conjunction with the accompanying drawings and embodiments. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0031] The present invention is implemented under the pytorch deep learning framework. The present invention provides an image quality evaluation method based on saliency information fusion, which specifically includes the following steps:

[0032] 1. Data preparation

[0033] The training and performance evaluation of the image quality evaluation model in this embodiment rely on three widely used public datasets: TID2013, CSIQ, and KONIQ-10k. Among them, the TID2013 dataset contains 25 reference images, and 24 different types of distortions are imposed on each reference image. Each distortion is divided into five levels, with a total of 3,000 distorted images, covering a variety of traditional distortion types; the CSIQ dataset contains 30 original images, and each original image undergoes 6 types of distortions and different degrees of distortion, with a total of 866 distorted images, also including a variety of typical distortions; the KONIQ-10k dataset is a large-scale real-world image quality evaluation dataset, which includes 10,073 real distorted images from the Internet. Its distortion types are complex and diverse, and it is closer to the actual application scenario. To ensure sufficient training of the model and objective evaluation, 70% of the images are randomly selected from each dataset for model training, and the remaining 30% is used for performance testing.

[0034] In the data preprocessing stage, this embodiment adopts different strategies for two image quality assessment tasks: full-reference (FR) and no-reference (NR). For the full-reference task, considering that when humans perform full-reference quality assessment, they often first focus on the significant regions of the reference image to form a preliminary impression, which in turn affects the judgment of the distorted image. Therefore, this embodiment pre-extracts the saliency object mask of the reference image and uses the pre-trained BiRefNet model to achieve mask extraction. Subsequently, the RGB image of the reference image and its saliency object mask are cropped to the same size. For the no-reference task, the saliency object mask of the input distorted image itself is directly extracted, and the BiRefNet model is also used. After the saliency object mask extraction is completed, for both the full-reference and no-reference tasks, all images and their saliency object masks are uniformly cropped to a size of 384×384 pixels to eliminate the influence of image size differences, and random horizontal flipping is adopted to increase the diversity of training samples.

[0035] 2. Model Construction

[0036] The constructed model consists of four sub-networks: a primary feature extraction network, a saliency object embedding network, a saliency object-guided transformer network, and a quality score regression network. The specific network model structure is as Figure 1 shown. The primary feature extraction network realizes dual-branch feature separation based on the improved ResNet50 backbone. Among them, the spatial domain feature extraction branch gradually generates multi-scale feature maps with 256, 512, 1024, and 2048 channels through four stages (stage1 to stage4), and finally outputs a 2048-dimensional feature map F containing texture, edge, and color information s . At the same time, the saliency object extraction branch uses the pre-trained saliency detection algorithm to generate a binary mask, and decomposes F s into foreground feature M fg = F s ⊙ M fg and background feature F bg = F s ⊙ M bg , isolating the significant and non-significant region information through element-wise multiplication of the mask; as Figure 2 shown, the saliency object embedding network differentially encodes the two types of features through the SCE module. First, it uses independent 1×1 convolutional layers with parameter isolation to process F fg and F bg respectively. Subsequently, the features are divided into overlapping local blocks (components) of N t × H t × W t along the spatial dimension. Each component contains multiple adjacent image patches to retain local context, and generates a 512-dimensional Token sequence through linear mapping, with the output shape of B, Nf ′, H′, W′, C′; The Transformer network guided by significant targets achieves feature interaction through Local-FC and Cross-FC attention mechanisms. Among them, Local-FC calculates self-attention within each component (the formula is focusing on local detail associations, while Cross-FC realizes global cross-attention between components by permuting dimensions, and combines a feed-forward network (including two fully connected layers and GELU activation) to enhance the non-linear expression ability; The quality score regression network first optimizes features through a global channel decoupled spatial attention module (GCSA). Channel attention uses a fully connected layer to compress and recover to generate a channel weight matrix, and spatial attention generates a spatial weight matrix through a 7×7 convolution. The output feature is F out = F⊙A c + F⊙A s , and finally, the dual-branch regression head predicts the feature weight w and the quality score f, and through weighted summation Q pred = ∑(f i ·w i ) / (∑w i + ∈) outputs the quality score, assigning higher weights to significant regions to conform to the characteristics of human subjective evaluation. The entire model realizes efficient and accurate image quality evaluation through dual-branch feature separation, overlapping component division, cross-attention fusion, and channel decoupled spatial collaborative optimization.

[0037] 3. Model Training

[0038] During the training process, the constructed model realizes end-to-end learning by jointly optimizing the Huber loss and the ranking loss. The specific process is as follows:

[0039] Huber loss: The absolute error between the supervised predicted quality score and the true MOS value. The formula is:

[0040]

[0041] where: δ = 1.0, f(x i ) is the model prediction score, and y i is the true MOS value.

[0042] Ranking loss: Forces the model to maintain the consistency of the quality ranking among samples, defined as:

[0043]

[0044] where C is the number of valid sample pairs, and y i > y j represents the true quality ranking.

[0045] Total loss: The weighted sum of the two, with the weight coefficient λ = 0.5, that is:

[0046] L total = L Huber + λL rank

[0047] 4. Test results

[0048] During the test, the input image was adjusted to a unified size of 384*384 by conditional central cropping and adapted to the model input through normalization (mean [0.485, 0.456, 0.406], standard deviation [0.229, 0.224, 0.225]). During the test, the model outputs the predicted quality score, and the Spearman rank correlation coefficient (SROCC) and Pearson linear correlation coefficient (PLCC) are calculated with the true MOS value as the core evaluation indicators.

[0049] 5. Model evaluation

[0050] The Spearman rank correlation coefficient (SROCC) and Pearson linear correlation coefficient (PLCC) are calculated using the predicted quality score and the true MOS value to evaluate the performance of the model. The best model is shown in bold.

[0051] Table 1 Comparison results of different methods on the open dataset

[0052]

[0053] The core innovation of TOPIQA is to use a cross-scale attention mechanism to guide the model to focus on important local distortion regions with high-level semantics. In real-scene datasets such as koniq-10k, TOPIQA's cross-scale feature fusion strategy can effectively capture the spatial distribution of complex distortions, so it performs best. However, its drawback is the insufficient sensitivity to structured distortions, because high-level semantic guidance may obscure low-frequency distortion details. VCRNET generates a restored image through a visual compensation restoration network and predicts the quality score based on the restored features. Although it performs moderately well in LIVE, its reliance on a non-adversarial restoration module results in insufficient restoration ability for severely distorted images and difficulty in accurately establishing the quality mapping relationship between the degraded image and the restored image. MANIQA introduces a transposed attention block (TAB) and a scale rotation Transformer (SSTB) to enhance global-local interaction and achieves relatively excellent performance in the LIVE dataset. However, its multi-dimensional attention mechanism does not explicitly distinguish between significant regions and non-significant regions, leading to excessive sensitivity to background noise and large fluctuations in the predicted scores, so it was not tested in koniq-10k. SGDNET and this study also use salient objects as local features, but SGDNET only performs simple weighted fusion on salient regions and non-salient regions, ignoring the complex interaction of the human eye in these two regions, resulting in poor performance. The model proposed in this study divides the salient object region and the non-salient object region in the form of components, and calculates self-attention between different components and within the same component respectively, making full use of the salient object information, achieving more effective and more human-vision-perception-compliant feature fusion, fully demonstrating the superiority of the approach in this study.

[0054] The above only elaborates on the preferred embodiments of the present invention in detail. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those of ordinary skill in the art, various changes can be made without departing from the gist of the present invention, and all such changes should be included within the protection scope of the present invention.

Claims

1. An image quality evaluation method based on saliency information fusion, characterized in that, It includes the following steps: S1. An adaptive preprocessing method combining full-reference and no-reference tasks is adopted to extract the saliency object mask of the input image for subsequent feature extraction; S2. The salient region and non-salient region are encoded as components, and the model learns the interaction relationships between different components by calculating the attention between components and within components. The saliency object embedding network realizes differential feature processing through the saliency object component encoder module; S3. Channel decoupled spatial attention is used to enhance the features extracted by the feature extraction network; S4. A saliency object-guided Transformer network is adopted. This network utilizes the self-attention mechanism and cross-attention mechanism to achieve deep interaction and fusion between saliency features and spatial domain features, thereby more effectively using saliency information to improve the accuracy of image quality assessment.

2. The image quality evaluation method based on saliency information fusion according to claim 1, wherein The method for extracting the saliency object mask of the input image in S1 is as follows: For the full-reference image quality assessment task, the reference image and the distorted image are input; For the no-reference image quality assessment task, the distorted image is input; the saliency object mask of the input image is extracted. The saliency object mask of the reference image is extracted in the full-reference task and attached to the corresponding distorted image. In the no-reference task, the saliency object mask is directly extracted from the distorted image, and the RGB image and the saliency object mask are resized and cropped; the saliency object mask is implemented through a pre-trained saliency object detection algorithm, and a binary mask image is output, where the white area represents the salient region in the image and the black area represents the non-salient region.

3. The image quality evaluation method based on saliency information fusion according to claim 1, wherein The method for the saliency object embedding network to realize differential feature processing through the saliency object component encoder module in S2 is as follows: First, the primary feature extraction network extracts the features of the salient and non-salient regions, and then the foreground and background features are processed separately using independent convolutional layers, and the convolutional kernel parameters are optimized through non-shared weights; Then, through component partitioning and Tokenization, the two types of features are divided into multiple local components and transformed into a sequence of feature vectors through linear mapping as the input of the subsequent Transformer network; The Transformer network captures the internal relationships and mutual dependencies between the salient region and the non-salient region through the attention calculation within and between components, thereby realizing the feature enhancement of saliency information.

4. The image quality evaluation method based on saliency information fusion according to claim 1, wherein, The method for using channel decoupled spatial attention to enhance the features extracted by the feature extraction network in S3 is as follows: First, through the channel attention sub-module, a multi-layer perceptron structure is used to learn the channel weights to achieve adaptive adjustment of the importance of different channel features; Subsequently, a channel shuffle operation is introduced to promote the information interaction and fusion between channel groups to achieve channel decoupling; On this basis, the spatial attention sub-module is further utilized to learn the spatial weights through a convolutional neural network to achieve adaptive adjustment of the importance of features at different spatial positions: X out = Shuffle((X * σ(A channel )) * σ(A spatial )) Where: X represents the input feature, σ represents the sigmoid function, A channel represents the channel attention result, A spatial represents the spatial attention result, and Shuffle represents the channel shuffle algorithm.

5. The image quality evaluation method based on saliency information fusion according to claim 1, characterized in that, The method for the Transformer network guided by significant targets in S4 to achieve deep fusion of saliency information and spatial features is as follows: First, it receives the spatial domain feature Fs' weighted by the channel decoupled spatial attention module, aiming to enhance the feature expression of the salient region; Then, this module uses the self-attention mechanism in the Transformer network to capture the long-range dependencies between different positions in the image, thereby realizing the modeling of the global structure of the image; To further fuse the saliency features, this module introduces a cross-attention mechanism, whose core idea is to use the saliency features to guide the learning of spatial features; The saliency feature Fm' processed by the saliency feature enhancement module is used as Query (Q), and the spatial domain feature Fs' is used as Key (K) and Value (V) for cross-attention calculation: Among them: Fm' represents the salient feature after being processed by the salient feature enhancement module, and Fs' represents the spatial domain feature. represents the scaling factor.

6. The image quality evaluation method based on saliency information fusion according to claim 3, wherein The method for constructing the primary feature extraction network is as follows: The primary feature extraction network realizes dual-branch feature separation based on the improved ResNet50 backbone. Among them, the spatial domain feature extraction branch gradually generates multi-scale feature maps with 256, 512, 1024, and 2048 channels through four stages, and finally outputs a 2048-dimensional feature map F containing texture, edge, and color information. s ; At the same time, the saliency target extraction branch uses a pre-trained saliency detection algorithm to generate a binary mask, and decomposes F s into foreground feature F fg = F s ⊙ M fg and background feature F bg = F s ⊙ M bg , isolating the information of significant and non-significant regions through element-wise multiplication of the mask.

7. The image quality evaluation method based on saliency information fusion according to claim 3, characterized in that The construction method of the salient object embedding network is as follows: The salient object embedding network differentially encodes two types of features through the Salient Object Component Encoder (SCE) module. First, independent 1×1 convolutional layers with parameter isolation are used to process F fg and F bg respectively. Subsequently, the features are divided into overlapping local block components of N t ×H t ×W t along the spatial dimension. Each component contains multiple adjacent image patches to preserve local context and generates a 512-dimensional Token sequence through linear mapping. The output shape is B, N f ′, H′W′, C′.

Images