No-Reference Image Quality Assessment Method Based on Siamese Network and Feature Fusion

Abstract

The present invention belongs to the field of multimedia processing technology, and aims to propose a reference-free objective image quality evaluation method. To this end, the technical solution adopted by the present invention is a reference-free image quality evaluation method based on a twin network and feature fusion, and the steps are as follows: using a synthetic distortion perception network pre-trained based on a twin network structure to extract synthetic distortion features; using a real distortion perception network pre-trained on the ImageNet dataset to extract real distortion features; performing feature fusion on the outputs of the above two networks and adaptively adjusting the weights of the two network outputs to form a reference-free image quality evaluation network for image quality evaluation, and training the reference-free image quality evaluation network to obtain an optimized reference-free image quality evaluation network. The present invention is mainly used in multimedia processing scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of multimedia processing technology, and specifically designs a reference-free image quality evaluation method based on twin network pre-training strategy and feature fusion. Background Art

[0002] With the rapid development of digital technology, high-quality visual information has attracted increasing attention. Because visual information suffers from quality loss during compression and transmission, image quality assessment has become crucial for accurately acquiring high-quality visual information. Image quality assessment can be categorized into subjective and objective methods. Subjective quality assessment methods rely on human perception to assess image quality. However, subjective image quality assessment is susceptible to subjective perception and is labor-intensive and time-consuming. To address the challenges of subjective image quality assessment, objective image quality assessment has emerged, aiming to design computer algorithms to automatically predict image quality. Depending on the use of a reference image, it can be categorized into full-reference image quality assessment, semi-reference image quality assessment, and no-reference image quality assessment. Because distorted images often lack a reference image, no-reference image quality assessment methods are more suitable for practical applications and play an irreplaceable role in military, journalism, education, medicine, and social applications.

[0003] Kang et al. [Kang L, Ye P, Li Y, et al. Convolutional neural networks for no-reference image quality assessment [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2014: 1733-1740.] proposed an image quality assessment method based on convolutional neural networks, which uses a combination of convolution and pooling to extract distortion features, and then uses a fully connected layer to regress the features to a quality score. Ma et al. [Ma K, Liu W, Zhang K, et al. End-to-end blind image quality assessment using deep neural networks [J]. IEEE Transactions on Image Processing, 2017, 27 (3): 1202-1213.] designed a multi-task convolutional neural network that not only predicts image quality but also identifies the distortion type, thereby constraining the image quality assessment task and enhancing the network's ability to assess image quality. Wu et al. [Wu J, Ma J, Liang F, et al. End-to-end blind image quality prediction with cascaded deep neural network [J]. IEEE Transactions on ImageProcessing, 2020, 29: 7414-7426.] designed a hierarchical feature fusion quality assessment network for end-to-end joint optimization based on the hierarchical processing mechanism of the human visual system from local features to global semantics. Su et al. [Su S, YanQ, Zhu Y, et al. Blindly assess image quality in the wild guided by a self-adaptive hyper network [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 3667-3676.] proposed a HyperIQA algorithm for real distortion. The algorithm first extracts semantic features from ResNet50 and then uses fully connected layers to adaptively learn network parameters to perform quality prediction.Shi Ping et al. [Shi Ping, Panda, Ying Zefeng, et al. A no-reference objective image quality assessment method based on a multi-scale generative adversarial network] used a multi-scale generative adversarial network to generate similarity quality maps corresponding to distorted images. These similarity quality maps at different scales were then regressed through a convolutional neural network to obtain image quality scores. Although these networks have achieved some success, the significant difference between synthetic and real distortions, and the fact that these networks are typically designed for either type of distortion, make it difficult for existing models to simultaneously evaluate datasets with both types of distortion, thus limiting their practical applications. Summary of the Invention

[0004] To overcome the shortcomings of the existing technology, the present invention aims to propose a no-reference objective image quality assessment method. To this end, the technical solution adopted by the present invention is a no-reference image quality assessment method based on a twin network and feature fusion, which includes the following steps: extracting synthetic distortion features using a synthetic distortion perception network pre-trained based on the twin network structure; extracting real distortion features using a real distortion perception network pre-trained on the ImageNet dataset; performing feature fusion on the outputs of the above two networks and adaptively adjusting the weights of the two network outputs to form a no-reference image quality assessment network for image quality assessment; and training the no-reference image quality assessment network to obtain an optimized no-reference image quality assessment network.

[0005] The synthetic distortion-aware network structure pre-trained based on the twin network structure is a twin network structure built with two EfficientNet-B0 sub-networks. The two EfficientNet-B0s share parameters. The residuals of the paired features are then calculated and globally averaged pooled. Finally, the fully connected layer outputs the final score, and the optimal model parameters are saved and applied to the synthetic distortion-aware network.

[0006] Among them, the EfficientNet-B0 classification network is selected to build a real distortion-aware network, the image classification task is used as the pre-training strategy, and the network parameters are frozen.

[0007] Fusing the outputs of the two networks and adaptively adjusting the weights of the two network outputs means introducing a weight-adaptive fusion network based on a multi-scale convolution block and a gating mechanism to adjust the weights of the synthetic distortion features and the real distortion features. The weight-adaptive fusion network first inputs the features extracted by the synthetic distortion perception network and the real distortion perception network into the corresponding multi-scale convolution block, perceives the features at different receptive fields and fuses them as the final extracted synthetic distortion features and real distortion features. Finally, the Sigmoid function is used as a gating mechanism to calculate the weights of the synthetic distortion and real distortion features, and the features are further enhanced through the multi-scale convolution block.

[0008] The synthetic distortion-aware network adopts a twin network structure and inputs the distorted image and its corresponding reference image as a pair of data during the pre-training process. The predicted value is obtained by regressing the residual between the distorted image features and the reference image features. At the same time, the mean absolute error loss function is used to calculate the loss between the predicted value and the label, and then the parameters are updated through backpropagation until the training loss reaches the minimum.

[0009] The multi-scale feature module used in the weight adaptive fusion network consists of three branches. One branch consists of a 3×3 convolution with a stride of 2; one branch consists of a 3×3 convolution with a stride of 2 and a 3×3 convolution with a stride of 1 in series; and one branch consists of a 1×1 convolution with a stride of 2.

[0010] The weight-adaptive fusion network first extracts features of the input at different scales through a multi-scale feature module to enhance its generalization. The multi-scale feature module is built on three sets of convolutional layers with different receptive fields, including 1×1, 3×3, and 5×5 convolutions. To achieve the same receptive field as the 5×5 convolutional layer and reduce the network computational complexity, two 3×3 convolutional layers are used instead of the 5×5 convolutional layer. This process is modeled as follows:

[0011]

[0012] Among them, M() is the output of the multi-scale feature module, I is the input, C1() represents a 3×3 convolution with a step size of 1, C2() represents a 3×3 convolution with a step size of 2, and C3() represents a 1×1 convolution with a step size of 1. Represents batch normalization and parameter correction linear unit PReLU (Parametric Rectified Linear Unit) activation function, For splicing operation;

[0013] Then, the splicing operation is used to fuse the feature maps extracted by the multi-scale feature module, and the weight of each modality is calculated using two independent 3×3 convolutions and sigmoid activation functions. Finally, the two weight values are multiplied with the corresponding synthetic distortion features and real distortion features respectively. Here, the extracted synthetic distortion W s and true distortion W a Defined as

[0014]

[0015] Among them, f s and f a Represent the feature maps of the synthetic distorted image and the real distorted image respectively, is the concatenation operation, ⊙ is the multiplication operation;

[0016] Finally, the output of the weight-adjustable network is defined as:

[0017]

[0018] The multi-scale features extracted from the MB2_Conv2, MB2_Conv4, MB2_Conv6, and Conv9 layers of EfficientNet-B0 are used as the input of the weight adaptive fusion network. Therefore, a total of four weight-adjustable adaptive fusion networks are used to process distortion features at different scales. The outputs of the four weight-adjustable network models are then concatenated according to the channel dimension and finally connected to a three-layer fully connected network to output the final prediction value.

[0019] During training, the no-reference image quality assessment network uses the mean absolute error loss function to calculate the loss between the predicted value and the label. The network parameters are then updated through backpropagation to minimize the training loss. The SROCC and PLCC indicators of the training set and the SROCC and PLCC indicators of the test set continue to increase, while the indicators of the test set are not much different from those of the training set.

[0020] The characteristics and beneficial effects of the present invention are:

[0021] By adopting the method of the present invention, good evaluation capability can be obtained not only when dealing with synthetic distortion images, but also when dealing with real distortion images. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a structural diagram of a synthetic distortion pre-training network based on a twin network pre-training strategy according to the present invention;

[0023] Figure 2 This is a diagram of the weight adaptive fusion network structure based on multi-scale convolution and gating mechanism described in the present invention;

[0024] Figure 3 The distortion map in the test set and the corresponding distortion feature map extracted by the weighted adaptive adjustment network described in the present invention;

[0025] Figure 4 This is the structural diagram of the reference-free image quality evaluation network based on the twin network pre-training strategy and feature fusion described in the present invention. DETAILED DESCRIPTION

[0026] To address the shortcomings of the existing technology, the purpose of the present invention is to provide a reference-free objective image quality assessment method based on a twin network pre-training strategy and feature fusion. This method uses a synthetic distortion-aware network pre-trained on the twin network structure to extract synthetic distortion features, and a real distortion-aware network pre-trained on the ImageNet dataset to extract real distortion features. In addition, a feature fusion module is designed to fuse the outputs of the two networks and adaptively adjust the weights of both. This solution improves the network's ability to perceive distortion features, enabling the network to achieve good results.

[0027] To achieve the above objectives, the technical solution adopted by the present invention is: a reference-free image quality assessment method based on twin pre-training strategy and weight adaptive fusion, comprising the following steps:

[0028] (1) Select and construct a synthetic distortion pre-training dataset;

[0029] (2) Use two EfficientNet-B0 sub-networks to build a twin network structure as a pre-training network. The network structure is as shown in the attached figure. Figure 1 As shown, it is trained on the synthetic distortion pre-training dataset constructed in (1). A pair of distorted pictures and reference pictures are input and features are extracted using EfficientNet-B0 respectively, and the two EfficientNet-B0 parameters are shared. Then, the residuals of the paired features are calculated and global average pooling is performed. Finally, the fully connected layer is connected to output the final score, the optimal model parameters are saved, and the optimal model parameters are applied to the synthetic distortion perception network.

[0030] (3) Considering that ImageNet, as a dataset of real distorted images, lacks subjective ratings, the real distortion-aware network uses image classification tasks as a pre-training strategy. Since EfficientNet-B0 achieves the best classification accuracy on ImageNet, this classification network is selected as the real distortion-aware network and the network parameters are frozen.

[0031] (4) Based on the multi-scale convolution block and the gating mechanism, a weight adaptive fusion network is introduced to adjust the weights of the synthetic distortion features and the real distortion features. The network first inputs the features extracted by the synthetic distortion perception network and the real distortion perception network into the corresponding multi-scale feature module, perceives the features in different receptive fields and fuses them as the final extracted synthetic distortion features and real distortion features. Finally, the Sigmoid function is used as the gating mechanism to calculate the weights of the synthetic distortion and real distortion features, and the features are further enhanced through the multi-scale feature module.

[0032] (5) A no-reference image quality assessment network is constructed by connecting the pre-trained synthetic distortion perception network, the real distortion perception network and the weight adaptive fusion network through the dual-link idea;

[0033] (6) Select and process the quality assessment dataset for validating the no-reference image quality assessment network in (5);

[0034] (7) The no-reference image quality assessment network is trained to obtain an optimized no-reference image quality assessment network.

[0035] Furthermore, under the condition of ensuring the aspect ratio of the images in the image quality assessment dataset, the present invention appropriately scales the oversized images to reduce the computational complexity, and then randomly divides each adjusted image into 25 image blocks with a resolution of 224 × 224.

[0036] Furthermore, the synthetic distortion-aware network adopts a twin network structure and inputs the distorted image and its corresponding reference image as a pair of data during the pre-training process. The predicted value is obtained by regressing the residual between the distorted image features and the reference image features. At the same time, the mean absolute error loss function is used to calculate the loss between the predicted value and the label, and then the parameters are updated through backpropagation until the training loss reaches the minimum.

[0037] Furthermore, the multi-scale feature module used in the weight adaptive fusion network includes three branches in total. One branch consists of a 3×3 convolution with a stride of 2; one branch consists of a 3×3 convolution with a stride of 2 and a 3×3 convolution with a stride of 1 in series; and one branch consists of a 1×1 convolution with a stride of 2.

[0038] Furthermore, the present invention uses four weight-adaptive fusion networks to perform feature fusion operations on features of different scales of the synthetic distortion perception network and the real distortion perception network to simulate the process of human visual system's perception of image quality from global to local.

[0039] Furthermore, the no-reference image quality assessment network described in the present invention uses the mean absolute error loss function to calculate the loss between the predicted value and the label during training, and then updates the network parameters through backpropagation to guide the training loss to the minimum. The SROCC and PLCC indicators of the training set and the SROCC and PLCC indicators of the test set continue to increase, while the indicators of the test set are not much different from those of the training set.

[0040] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0041] Step 1: Collect and process the pre-training dataset.

[0042] To enable the synthetic distortion-aware network to extract features from diverse distortions, the pre-training dataset must include a wide variety of distortion types and levels of distortion. Therefore, the Large-Scale Quality-Annotated (LSQA) dataset was selected. This dataset contains 10,000 reference images, which have been subjected to 105 different distortions, resulting in 105 × 10,000 = 1,050,000 distorted images. To achieve uniform image size, the proposed method first resizes the distorted images and their corresponding reference images to 256 × 256 resolution. Then, a random 224 × 224 image block is randomly partitioned from each resized distorted image and its corresponding reference image. Finally, the image blocks are divided into training and test sets in a 4:1 ratio.

[0043] Step 2: Use the quality assessment results to pre-train the synthetic distortion perception network.

[0044] The pre-training framework of synthetic distortion-aware network is as follows Figure 1 As shown, a pair of distorted and reference images are input and features are extracted using EfficientNet-B0, respectively. The two EfficientNet-B0s share parameters. The paired features are then subjected to residuals and global average pooling. Finally, a fully connected layer is connected to output the final score. Furthermore, the training and test sets used in this step are randomly split into a 4:1 ratio from the dataset processed in step 1. The pre-trained network uses the mean absolute error loss function to calculate the loss between the predicted value and the label. Adam is used as the network optimizer with a learning rate set to 0.0001. The network parameters are then updated through backpropagation. The quality of the model is assessed using the Pearson linear correlation coefficient (PLCC) and the Spearman rank-order correlation coefficient (SROCC). During training, the present invention selects SROCC as the verification indicator of the model. The training results of the model are evaluated by observing the SROCC indicator and the convergence of the loss. The model is trained for 3000 rounds and the model with the best SROCC indicator in this process is saved. This model will be used to extract the synthetic distortion in the image.

[0045] Step 3: Construct a real distortion-aware network.

[0046] Since image quality evaluation labels for real distortion are difficult to obtain, and considering that most images in ImageNet are captured by cameras and thus contain a large amount of real distortion, this paper selects EfficientNet-B0, pre-trained on the ImageNet dataset, as the real distortion-aware network and freezes its parameters.

[0047] Step 4: Construct a weight adaptive fusion network.

[0048] Weight adaptive fusion network model such as Figure 2 As shown in the figure, the network first extracts features of the input at different scales through a multi-scale feature module to enhance its generalization. The multi-scale feature module is built on three sets of convolutional layers with different receptive fields, including 1×1, 3×3, and 5×5 convolutions. In order to achieve the same receptive field as the 5×5 convolutional layer and reduce the network computational complexity, the present invention uses two 3×3 convolutional layers instead of the 5×5 convolutional layer. This process is modeled as follows:

[0049]

[0050] Among them, M() is the output of the multi-scale feature module, and I is the input. C1() represents a 3×3 convolution with a stride of 1, C2() represents a 3×3 convolution with a stride of 2, and C3() represents a 1×1 convolution with a stride of 1. represents batch normalization and Parametric Rectified Linear Unit (PReLU) activation function, For splicing operation.

[0051] Then, the feature maps extracted by the multi-scale feature module are fused using a splicing operation, and the weights of each modality are calculated using two independent 3×3 convolutions and sigmoid activation functions. Finally, the two weight values are multiplied by the corresponding synthetic distortion features and real distortion features respectively. Here, the extracted synthetic distortion W s and true distortion W a Defined as

[0052]

[0053] Among them, f s and f a Represent the feature maps of the synthetic distorted image and the real distorted image respectively, is the concatenation operation, and ⊙ is the multiplication operation.

[0054] Finally, the output of the weight-adjustable network can be defined as:

[0055]

[0056] Figure 3 The distortion map and the corresponding distortion feature map extracted using the weighted adaptive fusion network are shown. This figure shows that the synthetic distortion-aware network and the real distortion-aware network are sensitive to synthetic distortion features and real distortion features, respectively. The distortion map, the feature map extracted by the synthetic distortion-aware network, the feature map extracted by the real distortion-aware network, and the feature map extracted by the weighted adaptive fusion network clearly demonstrate the network's ability to effectively fuse learned synthetic and real distortion features.

[0057] Step 5: Construct a no-reference quality evaluation network.

[0058] The idea of dual links is used to connect the synthetic distortion perception network and the real distortion perception network in parallel, and their features are input into the weight adaptive fusion network for feature fusion. Since the human visual system will consider both the low-level features (such as local distortion) and high-level features (such as scene type, content information of different objects) of an image when humans perceive a distorted image, and the EfficientNet-B0 used by the synthetic distortion perception network and the real distortion perception network in the above steps can perceive features of different scales, which is in line with the perception law of the human visual system, the present invention uses the multi-scale features extracted from the MB2_Conv2, MB2_Conv4, MB2_Conv6 and Conv9 layers of EfficientNet-B0 as the input of the weight adaptive fusion network, and its feature map size is shown in Table 1. Therefore, a total of 4 weight-adjustable adaptive fusion networks are used to process distortion features at different scales. The outputs of the 4 weight-adjustable network models are then spliced together according to the channel dimension, and finally connected to a 3-layer fully connected network to output the final prediction value. The final constructed reference-free image quality evaluation network based on the twin network pre-training strategy and feature fusion is shown in FIG. Figure 4 shown.

[0059] Table 1. Feature map size of each module in EfficientNet-B0, where "W×H×C" represents the width,

[0060] High and channel.

[0061] Step 6: Collect and process image quality assessment dataset.

[0062] To verify the ability of the proposed method to handle both synthetic and real distortion, the no-reference quality assessment network was evaluated using four synthetic distortion datasets: LIVE, CSIQ, TID2013, and KADID-10k; and four real distortion datasets: LIVEC, CID2013, KonIQ-10K, and SPAQ. The method then appropriately scaled oversized images to reduce computational complexity while maintaining the aspect ratio of the original datasets. Each resized image was then randomly partitioned into 25 image blocks with a resolution of 224×224.

[0063] Step 7: Use the quality assessment results to train the no-reference image quality assessment network.

[0064] The no-reference image quality assessment network is trained using mean absolute error as its loss function to calculate the loss between the predicted value and the label. Backpropagation is then used to update the network parameters until the test set's SROCC reaches its highest value. The PLCC and SROCC metrics of the training set and the test set continue to increase, while the test set metrics remain similar to those of the training set. The training and test sets are randomly divided into a 4:1 ratio using the processed dataset from step 6. This process is repeated 10 times to eliminate bias, and the network training results are evaluated by calculating the average PLCC and SROCC values from these 10 runs. Adam is used as the optimizer for this distortion assessment network, and training is performed for 30 epochs. The initial learning rate is 0.0001, and the learning rate is adjusted to 0.9 times the original learning rate every 10 epochs.

[0065] The quality assessment model developed in this example was tested using the LIVE, CSIQ, TID2013, KADID-10k, LIVEC, CID2013, KonIQ-10k, and SPQA datasets. The test hardware platform was an NVIDIA 2080Ti GPU with 12GB of video memory, running in Ubuntu 20.04LTS. The image quality metrics SROCC and PLCC are shown in Tables 2 and 3:

[0066] Table 2 Comparison of SROCC and PLCC of different schemes on four synthetic distortion datasets

[0067]

[0068] The two best results are in bold

[0069] Table 3 Comparison of SROCC and PLCC of different schemes on four real distortion datasets

[0070]

[0071] The two best results are in bold

[0072] In the above table, the comparison scheme 1 is the scheme proposed by Bosse et al. [Bosse S, Maniry D, Müller KR, et al. Deep neural networks for no-reference and full-reference imagequality assessment[J]. IEEE Transactions on image processing, 2017, 27(1): 206-219.], the comparison scheme 2 is the scheme proposed by Yang et al. [Yan Q, Gong D, Zhang Y. Two-stream convolutional networks for blind image quality assessment[J]. IEEE Transactions on Image Processing, 2018, 28(5): 2200-2211.], the comparison scheme 3 is the scheme proposed by Wu et al. [Wu J, Ma J, Liang F, et al. End-to-end blind image quality prediction with cascaded deep neural network[J]. IEEE Transactions on Image Processing, 2020, 29: 7414-7426.], and the comparison scheme 4 is the scheme proposed by Zhang et al. [Zhang W, Ma K, Yan J, et al. Blindimage quality assessment using a deep bilinear convolutional neural network[J].IEEE Transactions on Circuits and Systems for Video Technology, 2018,30(1):36-47.], Comparative Scheme 5 is the scheme proposed by Su et al. PatternRecognition.2020:3667-3676.].Table 2 shows that, for SROCC, the present invention achieves the best performance on the LIVE, TID2013, and KADID-10k datasets, and second only to Comparative Solution 4 on CSIQ. For PLCC, the present invention achieves the best performance on the TID2013 and KADID-10k datasets, and second only to Comparative Solution 2 and Comparative Solution 4 on LIVE and CSIQ, respectively. Table 3 shows that, for both SROCC and PLCC, the present invention achieves the best performance on the LIVEC, CID2013, and SPAQ datasets, and second only to Comparative Solution 5 on KonIQ-10k. In summary, the present invention achieves state-of-the-art performance on both synthetic and real distorted images, demonstrating its ability to effectively learn from different types of distortion and its high ability to fuse them. The excellent performance of the proposed no-reference image quality assessment method based on the Siamese network pre-training strategy and feature fusion is due to the following reasons: (1) the synthetic and real distortion features extracted by the distortion perception module can well reflect the different distortions in the image. (2) Compared with other fusion operations such as the bilinear pooling used in the comparison scheme 4, the proposed weight-adjustable fusion module can better fuse different distortion features.

[0073] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A no-reference image quality assessment method based on Siamese network and feature fusion, characterized by the following steps: As follows: a synthetic distortion perception network pre-trained based on a twin network structure is used to extract synthetic distortion features; a real distortion perception network pre-trained on the ImageNet dataset is used to extract real distortion features; the outputs of the above two networks are feature-fused and the weights of the two network outputs are adaptively adjusted to form a no-reference image quality assessment network for image quality assessment, and the no-reference image quality assessment network is trained to obtain an optimized no-reference image quality assessment network; wherein, the outputs of the two networks are feature-fused and the weights of the two network outputs are adaptively adjusted, referring to introducing a weight adaptive fusion network based on a multi-scale convolution block and a gating mechanism to adjust the weights of the synthetic distortion features and the real distortion features. The weight adaptive fusion network first inputs the features extracted by the synthetic distortion perception network and the real distortion perception network into the corresponding multi-scale convolution block, perceives the features on different receptive fields and fuses them as the final extracted synthetic distortion features and real distortion features. Finally, the Sigmoid function is used as a gating mechanism to calculate the weights of the synthetic distortion and real distortion features, and the multi-scale convolution block is used to further enhance the features.

2. The no-reference image quality assessment method based on Siamese network and feature fusion according to claim 1, characterized in that: in, The synthetic distortion-aware network structure pre-trained based on the twin network structure is a twin network structure built with two EfficientNet-B0 sub-networks. The two EfficientNet-B0s share parameters. The residuals of the paired features are then calculated and global average pooling is performed. Finally, the fully connected layer is connected to output the final score, and the optimal model parameters are saved and applied to the synthetic distortion-aware network. Among them, the EfficientNet-B0 classification network is selected to build a real distortion-aware network, the image classification task is used as the pre-training strategy, and the network parameters are frozen.

3. The no-reference image quality assessment method based on Siamese network and feature fusion according to claim 1, characterized in that: The synthetic distortion-aware network adopts a twin network structure and inputs the distorted image and its corresponding reference image as a pair of data during the pre-training process. The predicted value is obtained by regressing the residual between the distorted image features and the reference image features. At the same time, the mean absolute error loss function is used to calculate the loss between the predicted value and the label, and then the parameters are updated through backpropagation until the training loss reaches the minimum.

4. The no-reference image quality assessment method based on Siamese network and feature fusion according to claim 1, characterized in that: The multi-scale feature module used in the weight adaptive fusion network consists of three branches. One branch consists of a 3×3 convolution with a stride of 2; one branch consists of a 3×3 convolution with a stride of 2 and a 3×3 convolution with a stride of 1 in series; and one branch consists of a 1×1 convolution with a stride of 2.

5. The no-reference image quality assessment method based on Siamese network and feature fusion according to claim 4 is characterized in that: The weight-adaptive fusion network first extracts features of the input at different scales through a multi-scale feature module to enhance its generalization. The multi-scale feature module is built on three sets of convolutional layers with different receptive fields, including 1×1, 3×3, and 5×5 convolutions. To achieve the same receptive field as the 5×5 convolutional layer and reduce the network computational complexity, two 3×3 convolutional layers are used instead of the 5×5 convolutional layer. This process is modeled as follows: Among them, M() is the output of the multi-scale feature module, I is the input, C1() represents a 3×3 convolution with a step size of 1, C2() represents a 3×3 convolution with a step size of 2, and C3() represents a 1×1 convolution with a step size of 1. Represents batch normalization and parameter correction linear unit PReLU (Parametric Rectified Linear Unit) activation function, For splicing operation; Then, the splicing operation is used to fuse the feature maps extracted by the multi-scale feature module, and the weight of each modality is calculated using two independent 3×3 convolutions and sigmoid activation functions. Finally, the two weight values are multiplied with the corresponding synthetic distortion features and real distortion features respectively. Here, the extracted synthetic distortion W s and true distortion W a Defined as Among them, f s and f a Represent the feature maps of the synthetic distorted image and the real distorted image respectively, is the concatenation operation, ⊙ is the multiplication operation; Finally, the output of the weight-adjustable network is defined as: The multi-scale features extracted from the MB2_Conv2, MB2_Conv4, MB2_Conv6, and Conv9 layers of EfficientNet-B0 are used as the input of the weight adaptive fusion network. Therefore, a total of four weight-adjustable adaptive fusion networks are used to process distortion features at different scales. The outputs of the four weight-adjustable network models are then concatenated according to the channel dimension and finally connected to a three-layer fully connected network to output the final prediction value.

6. The no-reference image quality assessment method based on Siamese network and feature fusion according to claim 4, characterized in that: The no-reference image quality assessment network uses the mean absolute error loss function to calculate the loss between the predicted value and the label during training, and then updates the network parameters through backpropagation to guide the training loss to the minimum. The SROCC and PLCC indicators of the training set and the SROCC and PLCC indicators of the test set continue to increase, while the indicators of the test set are not much different from those of the training set.

Images