No-reference image quality assessment method based on multi-granularity network

Abstract

A no-reference image quality evaluation method based on a multi-granularity network, comprising the following steps: Step 1, obtaining a training sample set B and a test sample set C; Step 2, constructing a no-reference image quality evaluation network model S based on a multi-granularity network; Step 3, performing iterative training on the no-reference image quality evaluation network model S based on a multi-granularity network to obtain a trained no-reference image quality evaluation network model S<supgt;*< / supgt>; Step 4, obtaining the no-reference quality evaluation result of an image: taking the test sample set C as the input of the trained no-reference image quality evaluation network model S<supgt;*< / supgt> based on a multi-granularity network for forward inference to obtain the quality prediction score of each test sample to verify the model effect. The present invention realizes the no-reference quality evaluation and analysis evaluation of an image by analyzing the rich local features contained in image blocks of different granularities.

Description

Technical Field

[0001] The present invention belongs to the technical field of image quality assessment, and in particular relates to a reference-free image quality assessment method based on a multi-granularity network. Background Art

[0002] At present, there are many natural scene images generated and uploaded by users on the Internet. Since most users do not have professional knowledge of image shooting, from the perspective of image quality, these images may be distorted, resulting in poor quality. Moreover, the distortion in these images is not artificially synthesized, but natural distortion. Therefore, the quality evaluation of such images is significantly different from the quality evaluation of synthetically distorted images that has been widely studied, so higher requirements are put forward for image quality evaluation.

[0003] There are many factors that affect image quality, including distortion type, degree of distortion, image content and scene, etc., especially for non-synthetic real distorted images, the situation is more complicated. For the quality evaluation of real distorted images, the type, degree of distortion and the scene and content of the image are ever-changing. Unlike ordinary synthetic distortions (such as Gaussian blur, JPEG compression) that are evenly distributed throughout the image, real distorted images have not only global uniform distortion (such as defocus, low illumination), but also local non-uniform distortion (such as object movement, over-illumination, ghosting), etc. Therefore, the challenge facing the algorithm is to accurately capture global and local distortions, extract their corresponding features, and merge these features to form an appropriate quality prediction score.

[0004] The key to image quality assessment is how to effectively extract features that can reflect the quality of the image. Traditional image quality assessment methods are limited to hand-crafted features, most of which cannot fully characterize complex image structures and distortions. Moreover, researchers have invested a lot of effort in designing features, but the performance of the methods has improved quite slowly, indicating that these methods based on hand-crafted features have limitations and are not perfect. On the other hand, deep learning-based methods have excellent performance in many computer vision tasks due to their strong representation ability, and have also been applied to a large number of image quality assessment studies. Deep learning methods mainly use convolutional layers to extract image features, and then use fully connected layers to map features to quality scores. Since the extracted image features can be automatically trained without time-consuming and labor-intensive manual design, deep learning methods can extract more suitable features with higher efficiency. However, even though many image quality assessment works based on deep learning have improved the quality assessment performance to a certain extent, there is still room for further improvement in their evaluation effects.

[0005] The patent application with application publication number CN115272203A and name “A no-reference image quality assessment method based on deep learning” discloses a no-reference image quality assessment method based on deep learning. This method utilizes multi-level feature fusion to enhance the expression ability of image content and improves the performance of no-reference image quality assessment tasks. However, the shortcoming of this method is that the multi-level features extracted are all global features, ignoring the attention to local areas, and failing to consider that richer local features can be obtained in partitions of different granularities of the image, resulting in inconsistency between the model prediction quality and the human visual perception quality.

[0006] The patent application with application publication number CN115082756A and name “No-reference image quality assessment method based on visual saliency and gradient features” discloses a no-reference image quality assessment method based on visual saliency and gradient features. The method divides the image into image blocks and weights the local quality scores, making good use of local features and improving the performance of the no-reference image quality assessment task. However, the shortcoming of this method is that the granularity of the extracted image block features is single, and it does not take into account that richer local features can be obtained in partitions of different granularities of the image, resulting in inconsistency between the model prediction quality and the human visual perception quality.

[0007] Most existing image quality assessment methods ignore the focus on local areas, or fail to consider that richer local features can be obtained in partitions of different granularities of the image, resulting in inconsistency between model prediction quality and human visual perception quality. Summary of the invention

[0008] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a reference-free image quality evaluation method based on a multi-granularity network, which utilizes the multi-granularity network to perform feature extraction and analysis and evaluation on image blocks of various granularities, and realizes reference-free quality evaluation of the image by analyzing the rich local features contained in image blocks of different granularities, and generates a predicted quality score for the image, which is used to solve the problem of inconsistency between model prediction quality and human visual perception quality caused by ignoring the rich local features contained in image blocks of different granularities in the prior art.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] The method for evaluating image quality without reference based on multi-granularity network comprises the following steps:

[0011] Step 1: Get the training sample set B and the test sample set C:

[0012] Step 2: Construct a reference-free image quality assessment network model S based on a multi-granularity network:

[0013] Step 3, iteratively train the no-reference image quality assessment network model S based on the multi-granularity network to obtain the trained no-reference image quality assessment network model S based on the multi-granularity network * ;

[0014] Step 4: Obtain the image's no-reference quality evaluation results:

[0015] The test sample set C is used as the trained no-reference image quality assessment network model S based on a multi-granularity network * The input is used for forward reasoning to obtain the quality prediction score of each test sample to verify the model effect.

[0016] In step 1, images that account for a certain proportion of all images in the public image quality assessment dataset are arbitrarily selected as the training sample set B, and the remaining images are used as the test sample set C, and the quality score labels of the dataset are mapped to the interval [0, 1].

[0017] The step 2 specifically includes the following steps:

[0018] Step 2.1, construct a multi-granularity feature extraction module E, using Resnet-50 as the backbone network, after the residual block of res_conv4_1 (the first block of the 4th layer of ResNet-50); the part of Resnet-50 after the residual block is copied into three independent isomorphic copies, each of which is modified to a certain extent (whether the downsampling operation is used in the res_conv5_1 module), and then connected to pooling normalized activation dimensionality reduction (global branch), stripe 2 division, each stripe has separate pooling normalized activation dimensionality reduction (local Part-2 branch), stripe 3 division, each stripe has separate pooling normalized activation dimensionality reduction (local Part-3 branch), and three slightly different parallel branches are obtained. The input of the three branches is the output of res_conv4_1;

[0019] The Resnet-50 backbone network is divided into three parallel branches: global branch, local Part-2 branch, and local Part-3 branch;

[0020] The global branch uses a convolutional layer with a step size of 2 for downsampling in the res_conv5_1 module, performs a global maximum pooling operation on the part of Resnet-50 after the res_conv4_1 block in the branch (specifically, the output feature map of the last residual block of res_conv5 corresponding to the branch), and uses a special fully connected layer Y (a convolutional layer with a convolution kernel of 1×1) with batch normalization and ReLU activation function to reduce the 2048-dimensional features to 256-dimensional features, which are global features.

[0021] The local Part-2 branch is different from the global branch in that no downsampling operation is used in the res_conv5_1 module, and the corresponding output feature map is evenly divided into two strips in the horizontal direction. The global maximum pooling operation is performed separately on each strip, and then a convolution layer with a convolution kernel of 1×1 and batch normalization and ReLU activation function is used to reduce the dimensionality of the 2048-dimensional features to obtain two 256-dimensional features, which are local features with a granularity of 2. and The number of partitions of this branch is 2, so the feature representation with a granularity of 2 is learned;

[0022] The local Part-3 branch is different from the local Part-2 branch in that the output feature map of the res_conv5_1 module is evenly divided into three strips in the horizontal direction; after the global maximum pooling and convolution are performed separately on each strip, three 256-dimensional features are obtained, which are local features with a granularity of 3. and The number of partitions of this branch is 3, so the feature representation with a granularity of 3 is learned;

[0023] Step 2.2: Construct a feature regression module P to extract the features output by the multi-granularity feature extraction module E. and The multi-granularity features F are concatenated and regressed using a special fully connected layer Y to obtain the quality prediction score.

[0024] The calculation formula of the multi-granularity feature F is as follows

[0025]

[0026] in, Represents the global features,

[0027] F: Multi-granularity features, The global features extracted by the global branch, The 0th local feature extracted from the local Part-2 branch, The first local feature extracted from the local Part-2 branch, The 0th local feature extracted from the local Part-3 branch, The first local feature extracted from the local Part-3 branch, The second local feature extracted from the local Part-3 branch, f superscript: G: the feature comes from the global branch, P2: the feature comes from the local Part-2 branch, P3: the feature comes from the local Part-3 branch, f subscript: g: the feature is the global feature, p0: the feature is the 0th local feature, p1: the feature is the 1st local feature, p2: the feature is the 2nd local feature.

[0028] The step 3 is specifically as follows:

[0029] Step 3.1, initialize the number of iterations to t, the maximum number of iterations to T, and the current reference-free image quality assessment network model based on multi-granularity network is S t , and let t = 1, S t =S;

[0030] Step 3.2: randomly select b training samples from the training sample set B in step 1 without replacement as the no-reference image quality assessment network model S t The multi-granularity feature extraction module E extracts multi-granularity features from each training sample, and the feature regression module P concatenates the global features of each training sample with the multi-granularity local features to obtain the multi-granularity features. A special fully connected layer is then used to regress the multi-granularity features to obtain the quality prediction score corresponding to each training sample.

[0031] Step 3.3: First, calculate the mean square error between the quality prediction score corresponding to each training sample and the quality score label corresponding to the training sample as the model S t The loss value is then optimized using the Adam algorithm to optimize the network model S t Update the weight parameters of each layer;

[0032] The formula for calculating the mean square error between the quality prediction score corresponding to each training sample and the quality score label corresponding to the training sample is:

[0033]

[0034] Using the Adam optimization algorithm, the learning rate is updated according to the following formula

[0035]

[0036]

[0037] In formula (3-1), b represents the number of training samples randomly selected from the training sample set B without replacement when iteratively training the reference-free image quality assessment network model S based on a multi-granularity network, and q g represents the quality score label corresponding to the g-th training sample among the b training samples, represents the quality prediction score corresponding to the g-th training sample among the b training samples;

[0038] In formulas (3-2) and (3-3), t represents the current number of iterations, s represents the step size for updating the learning rate, that is, the learning rate is updated once every s iterations, d represents the decay coefficient of the learning rate, a regularization term is added to the loss function, and weight decay weight_decay is added during the training process;

[0039] Step 3.4, determine whether the training sample set B has been traversed, if so, execute step 3.5, otherwise, execute steps 3.2 and 3.3;

[0040] Step 3.5, determine whether t = T is established. If so, obtain the trained multi-granularity network-based reference-free image quality assessment network model S * =S t , otherwise, let t = t + 1 and execute steps 3.3 and 3.4.

[0041] Beneficial effects of the present invention:

[0042] The present invention uses a multi-granularity network to extract features from images on image blocks of various granularities, and realizes reference-free quality evaluation and analytical evaluation of images by analyzing the rich local features contained in image blocks of different granularities. Since the rich local features contained in image blocks of different granularities in the image are fully utilized, the present invention has the advantage of highly consistent model prediction quality and human visual perception quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0045] See attached Figure 1 , the present invention comprises the following steps:

[0046] Step 1: Get the training sample set B and the test sample set C:

[0047] Randomly select 80% of all images from the public image quality assessment dataset as the training sample set B, and the remaining images as the test sample set C, and map the quality score labels of the dataset to the [0,1] interval;

[0048] In this embodiment, the TID2013 dataset is used, and 80% of the distorted images of the reference images are used as the training set, and the remaining 20% ​​of the distorted images of the reference images are used as the test set. This operation ensures that there is no image data with the same image scene and content in the training set and the test set;

[0049] Step 2: Construct a reference-free image quality assessment network model S based on a multi-granularity network:

[0050] Step 2.1, construct a multi-granularity feature extraction module E, using Resnet-50 as the backbone network. After the residual block res_conv4_1 (the first block of the fourth layer of ResNet-50), the Resnet-50 backbone network is divided into three parallel branches: global branch, local Part-2 branch, and local Part-3 branch;

[0051] The global branch uses a convolutional layer with a step size of 2 for downsampling in the res_conv5_1 module, performs a global maximum pooling operation on the corresponding output feature map, and uses a convolutional layer with a convolution kernel of 1×1 with batch normalization and ReLU activation function to reduce the 2048-dimensional feature to a 256-dimensional feature, which is the global feature. This branch learns global feature representation without any partition information, so this branch is called the global branch.

[0052] The network structure of the local Part-2 branch is similar to that of the global branch. The difference is that no downsampling operation is used in the res_conv5_1 module to reserve enough space to extract local features. The corresponding output feature map is evenly divided into 2 strips in the horizontal direction, and the global maximum pooling operation is performed separately on each strip. Then, a convolution layer with a convolution kernel of 1×1 and batch normalization and ReLU activation function is used to reduce the dimensionality of the 2048-dimensional features to obtain 2 256-dimensional features, which are local features with a granularity of 2. and The number of partitions of this branch is 2, so the feature representation with a granularity of 2 is learned.

[0053] The network structure of the local Part-3 branch is similar to that of the local Part-2 branch. The difference is that the output feature map of the res_conv5_1 module is evenly divided into three strips in the horizontal direction. Therefore, after the global maximum pooling and convolution are performed separately on each strip, three 256-dimensional features are obtained, which are local features with a granularity of 3. and The number of partitions of this branch is 3, so the feature representation with a granularity of 3 is learned.

[0054] Step 2.2: Construct a feature regression module P to extract the features output by the multi-granularity feature extraction module E. and The multi-granularity features F are concatenated and regressed using a special fully connected layer Y to obtain the quality prediction score.

[0055] The multi-granularity feature F is obtained by concatenating the global features extracted by the multi-granularity feature extraction network and the multi-granularity local features. It can combine global information with local information to improve the comprehensiveness and richness of the learned features. The calculation formula of the multi-granularity feature F is as follows:

[0056]

[0057] in, Represents the global features, represents the i-th local feature extracted in the local Part-2 branch, Represents the jth local feature extracted in the local Part-3 branch.

[0058] The special fully connected layer Y is a convolution layer with a convolution kernel of 1×1. The ordinary fully connected layer will destroy the spatial structure information of the image, while the special fully connected layer will not destroy the spatial structure information of the image. Moreover, once the network structure of the ordinary fully connected layer is fixed, the number of parameters to be learned and the input size are also fixed, but for the special fully connected layer, no matter how the input scale changes, the convolution kernel size and parameters remain unchanged, which is more flexible and convenient.

[0059] Step 3, iteratively train the no-reference image quality assessment network model S based on the multi-granularity network:

[0060] Step 3.1, initialize the number of iterations to t, the maximum number of iterations to T, and the current reference-free image quality assessment network model based on multi-granularity network is S t , and let t = 1, S t =S;

[0061] In this embodiment, T=80;

[0062] Step 3.2: randomly select b training samples from the training sample set B without replacement as the no-reference image quality assessment network model S t The multi-granularity feature extraction module E extracts multi-granularity features from each training sample, and the feature regression module P concatenates the global features of each training sample with the multi-granularity local features to obtain the multi-granularity features. A special fully connected layer is then used to regress the multi-granularity features to obtain the quality prediction score corresponding to each training sample.

[0063] In this embodiment, b=16;

[0064] Step 3.3: First, calculate the mean square error between the quality prediction score corresponding to each training sample and the quality score label corresponding to the training sample as the model S tThe loss value is then optimized using the Adam algorithm to optimize the network model S t Update the weight parameters of each layer;

[0065] In this embodiment, the formula for calculating the mean square error between the quality prediction score corresponding to each training sample and the quality score label corresponding to the training sample is:

[0066]

[0067] In this embodiment, the Adam optimization algorithm is used. Except for the learning rate, all other parameters use the default values. In order to ensure the convergence speed of the model and the accuracy of learning, the learning rate is updated according to the following formula

[0068]

[0069]

[0070] In formula (3-1), b represents the number of training samples randomly selected from the training sample set B without replacement when iteratively training the reference-free image quality assessment network model S based on a multi-granularity network, and q g represents the quality score label corresponding to the g-th training sample among the b training samples, represents the quality prediction score corresponding to the g-th training sample among the b training samples;

[0071] In formulas (3-2) and (3-3), t represents the current number of iterations, s represents the step size for updating the learning rate, that is, the learning rate is updated once every s iterations, and d represents the decay coefficient of the learning rate. At the same time, in order to prevent the model from overfitting during training, a regularization term is added to the loss function, and weight decay weight_decay is added during training;

[0072] In this embodiment, b=16, s=1, d=0.5, weight_decay=5×10 -4 ;

[0073] Step 3.4, determine whether the training sample set B has been traversed, if so, execute step 3.5, otherwise, execute steps 3.2 and 3.3;

[0074] Step 3.5, determine whether t = T is established. If so, obtain the trained multi-granularity network-based reference-free image quality assessment network model S * =S t , otherwise, let t = t + 1, and execute steps 3.3 and 3.4;

[0075] Step 4: Obtain the image's no-reference quality evaluation results:

[0076] The test sample set C is used as the trained no-reference image quality assessment network model S based on a multi-granularity network * The input is used for forward reasoning to obtain the quality prediction score of each test sample to verify the model effect.

[0077] The following is a description of the technical effects of the present invention in combination with simulation experiments:

[0078] 1. Simulation conditions and contents:

[0079] The hardware platform of the simulation experiment of the present invention is: the processor is Intel (R) Core (TM) i9-7900X CPU, the main frequency is 3.30GH, the memory is 32GB, and the graphics card is NVIDIA GeForce GTX 1080Ti.

[0080] The software platform for the simulation experiment of the present invention is: Ubuntu 16.04, Pytorch 1.6.0, Python 3.7.

[0081] 2. Simulation Experiment

[0082] The input images used in the simulation experiment of the present invention are from the image quality assessment databases LIVE, TID2013 and CSIQ, wherein the CSIQ data set is an image quality assessment database proposed by E.C. Larson et al. in “Most apparent distortion: full-reference image quality assessment and the role of strategy. Journal of Electronic Imaging, 19(1): 011006, 2010”, the LIVE data set is an image quality assessment database proposed by D. Ghadiyaram et al. in “Massive online crowdsourced study of subjective and objective picture quality. IEEE Transactions on Image Processing, 25(1): 372–387, 2016”, and the TID2013 data set is an image quality assessment database proposed by N. Ponomarenko et al. in “Color image database TID2013: Peculiarities and preliminary results. In European Workshop on Visual Information Processing (EUVIP), 106–111, 2013”.

[0083] The simulation experiment of the present invention adopts two indicators, Spearman rank-order correlation coefficient SROCC (Spearman rank-order correlation coefficient) and Pearson linear correlation coefficient PLCC (Pearson linear correlation coefficient), to judge the image quality evaluation effects of the present invention and the prior art respectively. Specifically, the prior art and the present invention respectively select n samples from the test sample set C for image quality evaluation, output the quality prediction score, and calculate the values ​​of SROCC and PLCC by the quality prediction score of the sample and the quality label score corresponding to the sample. Among them, the two indicators are calculated according to the following formulas, SROCC∈[-1,1], the higher the value, the more accurately the evaluation result of the no-reference image quality evaluation method being judged can reflect the quality of the image, and PLCC∈[-1,1], the higher the value, the closer the evaluation result of the no-reference image quality evaluation method being judged is to the subjective evaluation score of humans.

[0084]

[0085]

[0086] Where n represents the total number of images, b e represents the difference between the sequence number of the quality prediction score of the e-th distorted image in the quality prediction score ranking of n distorted images and the sequence number of its quality label score, s r represents the quality prediction score of the rth distorted image, represents the average quality prediction score of all distorted images, p r represents the subjective evaluation score of the rth distorted image, Represents the average of the subjective evaluation scores of all distorted images.

[0087] The simulation results are shown in Table 1.

[0088] Table 1. Comparison of evaluation results of the present invention and prior art

[0089]

[0090] As can be seen from Table 1, compared with the prior art, the Spearman rank correlation coefficient SROCC and the Pearson linear correlation coefficient PLCC of the evaluation results of the present invention on the CSIQ, TID2013, and LIVE datasets are both higher, proving that the accuracy of the reference-free image quality evaluation of the present invention is higher.

Claims

1. A no-reference image quality assessment method based on a multi-granularity network, It is characterized in that The steps include: Step 1: Get the training sample set B and the test sample set C: Step 2: Construct a reference-free image quality assessment network model S based on a multi-granularity network: Step 3, iteratively train the no-reference image quality assessment network model S based on the multi-granularity network to obtain the trained no-reference image quality assessment network model S based on the multi-granularity network * ; Step 4: Obtain the image's no-reference quality evaluation results: The test sample set C is used as the trained no-reference image quality assessment network model S based on a multi-granularity network * Perform forward reasoning on the input to obtain the quality prediction score of each test sample to verify the model effect; The step 2 specifically includes the following steps: Step 2.1, construct a multi-granularity feature extraction module E, using Resnet-50 as the backbone network. After the res_conv4_1 residual block, the Resnet-50 backbone network is divided into three parallel branches: global branch, local Part-2 branch, and local Part-3 branch; The global branch uses a convolutional layer with a step size of 2 for downsampling in the res_conv5_1 module, performs a global maximum pooling operation on the output feature map of the last residual block of res_conv5 corresponding to the branch, and uses a special fully connected layer Y with batch normalization and ReLU activation function to reduce the 2048-dimensional feature to a 256-dimensional feature, which is the global feature The local Part-2 branch is different from the global branch in that no downsampling operation is used in the res_conv5_1 module, and the corresponding output feature map is evenly divided into two strips in the horizontal direction. The global maximum pooling operation is performed separately on each strip, and then a convolution layer with a convolution kernel of 1×1 and batch normalization and ReLU activation function is used to reduce the 2048-dimensional feature dimension to obtain two 256-dimensional features, which are local features with a granularity of 2. and The local Part-3 branch is different from the local Part-2 branch in that the output feature map of the res_conv5_1 module is evenly divided into three strips in the horizontal direction; after the global maximum pooling and convolution are performed separately on each strip, three 256-dimensional features are obtained, which are local features with a granularity of 3. and Step 2.2: Construct a feature regression module P to extract the features output by the multi-granularity feature extraction module E. and The multi-granularity features F are concatenated and regressed using a special fully connected layer Y to obtain the quality prediction score.

2. The method for evaluating image quality without reference based on a multi-granularity network according to claim 1, It is characterized in that In step 1, images that account for a certain proportion of all images in the public image quality assessment dataset are arbitrarily selected as the training sample set B, and the remaining images are used as the test sample set C, and the quality score labels of the dataset are mapped to the interval [0, 1].

3. The method for evaluating image quality without reference based on a multi-granularity network according to claim 1, It is characterized in that The calculation formula of the multi-granularity feature F is as follows in, Represents the global features, F: Multi-granularity features, The global features extracted by the global branch, The 0th local feature extracted from the local Part-2 branch, The first local feature extracted from the local Part-2 branch, The 0th local feature extracted from the local Part-3 branch, The first local feature extracted from the local Part-3 branch, The second local feature extracted from the local Part-3 branch, f superscript: G: the feature comes from the global branch, P2: the feature comes from the local Part-2 branch, P3: the feature comes from the local Part-3 branch, f subscript: g: the feature is the global feature, p0: the feature is the 0th local feature, p1: the feature is the 1st local feature, p2: the feature is the 2nd local feature.

4. The method for evaluating image quality without reference based on a multi-granularity network according to claim 1, It is characterized in that The step 3 is specifically as follows: Step 3.1, initialize the number of iterations to t, the maximum number of iterations to T, and the current reference-free image quality assessment network model based on multi-granularity network is S t , and let t = 1, S t =S; Step 3.2: randomly select b training samples from the training sample set B in step 1 without replacement as the no-reference image quality assessment network model S t Input; The multi-granularity feature extraction module E extracts multi-granularity features from each training sample. The feature regression module P concatenates the global features of each training sample with the multi-granularity local features to obtain the multi-granularity features. A special fully connected layer is then used to regress the multi-granularity features to obtain the quality prediction score corresponding to each training sample. Step 3.3: First, calculate the mean square error between the quality prediction score corresponding to each training sample and the quality score label corresponding to the training sample as the model S t The loss value is then optimized using the Adam algorithm to optimize the network model S t Update the weight parameters of each layer; The formula for calculating the mean square error between the quality prediction score corresponding to each training sample and the quality score label corresponding to the training sample is: Using the Adam optimization algorithm, update the learning rate according to the following formula In formula (3-1), b represents the number of training samples randomly selected from the training sample set B without replacement when iteratively training the reference-free image quality assessment network model S based on a multi-granularity network, and q g represents the quality score label corresponding to the g-th training sample among the b training samples, represents the quality prediction score corresponding to the g-th training sample among the b training samples; In formulas (3-2) and (3-3), t represents the current number of iterations, s represents the step size for updating the learning rate, that is, the learning rate is updated once every s iterations, d represents the decay coefficient of the learning rate, a regularization term is added to the loss function, and weight decay weight_decay is added during the training process; Step 3.4, determine whether the training sample set B has been traversed, if so, execute step 3.5, otherwise, execute steps 3.2 and 3.3; Step 3.5, determine whether t = T is established. If so, obtain the trained multi-granularity network-based reference-free image quality assessment network model S * =S t , otherwise, let t = t + 1 and execute steps 3.3 and 3.4.

Images