Blind image quality assessment method based on multi-scale features and long-range dependencies

Abstract

The application discloses a blind image quality evaluation method based on multi-scale features and long-distance dependence, and comprises the following steps: step 1, acquiring a training sample set and a test sample set; step 2, constructing a network model of a no-reference image quality evaluation method based on multi-scale features and long-distance dependence; the network model is used for extracting and fusing multi-scale features and long-distance dependence of an image to regress to an image quality score; step 3, iteratively training the network model of the no-reference image quality evaluation method based on multi-scale features and long-distance dependence; and step 4, acquiring a no-reference quality evaluation result of the image. The application is used for solving the problems of large parameter quantity and low calculation efficiency of an existing method and the problem that the existing method neglects the fusion of long-distance dependence between multi-scale features and local quality features, thereby leading to low consistency between a model prediction result and human visual perception.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image quality evaluation, and particularly relates to a blind image quality evaluation method based on multi-scale features and long-distance dependence. BACKGROUND

[0002] Image quality evaluation methods can be divided into subjective evaluation methods and objective evaluation methods from the perspective of whether a person is involved. The former relies on the subjective perception of experimental personnel to evaluate the quality of the object, and the latter measures image quality by simulating the human visual system perception mechanism through the establishment of a mathematical model. According to the degree of dependence on reference images, objective quality evaluation methods can be divided into three categories: full-reference image quality evaluation, partial-reference image quality evaluation and no-reference image quality evaluation. Full-reference image quality evaluation needs to compare the complete reference image with the image to be measured to evaluate the image quality, partial-reference image quality evaluation only needs to rely on part of the features of the reference image to evaluate the quality of the image to be measured, and no-reference image quality evaluation can complete image quality evaluation without any reference image information. As a kind of objective image quality evaluation method, no-reference image quality evaluation method has the advantages of high efficiency and low cost, avoids the influence of the physical environment of experimental personnel compared with subjective methods, and eliminates the need for reference images compared with full-reference and partial-reference methods. It is the most practical and widely used image quality evaluation method. An excellent image quality evaluation method can be applied to the fields of evaluating and guiding image processing algorithms, dynamically detecting and adjusting image quality, and screening and recommending high-quality images.

[0003] The patent application with the application publication number CN115272203A and the name of "a no-reference image quality evaluation method based on deep learning" uses multi-level feature fusion to enhance the expression ability of image content and improve the performance of the no-reference image quality evaluation task. However, the method has the disadvantage that the multi-level features extracted are all global features, ignoring the attention to local regions and not considering the long-distance dependence relationship between local region features, resulting in inconsistency between the model prediction quality and human visual perception quality.

[0004] The patent application with the application publication number CN115222996A and the name of "content-guided and self-attention real distortion blind reference image quality evaluation method" solves the problem of difficult modeling of the relationship between image content and image quality. However, the method has the disadvantage that the EfficientNetB0 network has a large number of parameters, the network training and inference calculation efficiency is low, and the method only considers the influence of different regions of the image on the image quality, ignoring the influence of multi-scale features of the image on the image quality, resulting in inconsistency between the model prediction quality and human visual perception quality.

[0005] The existing image quality evaluation method uses global features or local features to obtain a quality score by weighting, ignores the influence of long-distance dependence between multi-scale features and local features on image quality, and thus cannot accurately predict the image quality. SUMMARY

[0006] In order to overcome the problems existing in the prior art, the purpose of the present application is to provide a blind image quality evaluation method based on multi-scale features and long-distance dependence, which uses a parallel lightweight convolutional neural network MobileNet and a Transformer as a feature extractor, uses the MobileNet to extract multi-scale quality features of the image, uses the Transformer to model the long-distance dependence between local quality features, and uses a lightweight cross-attention as a bridge between the MobileNet and the Transformer to fuse the multi-scale quality features and the long-distance dependence, so as to solve the problems of large parameter quantity and low computational efficiency of the existing method and the problem of low consistency between the model prediction result and human visual perception caused by the existing method ignoring the fusion of the long-distance dependence between the multi-scale features and the local quality features.

[0007] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0008] The blind image quality evaluation method based on multi-scale features and long-distance dependence comprises the following steps:

[0009] Step 1: extracting image groups from an image quality dataset to form a training sample set B and a test sample set C; the training sample set B is used for parameter training and optimization of the network model, so that the network model establishes a mapping from the image to the quality score, and the test sample set C is used for testing and verifying the prediction performance of the network model on the image quality score;

[0010] Step 2: constructing a network model S of a no-reference image quality evaluation method based on multi-scale features and long-distance dependence; the network model S is used for extracting and fusing the multi-scale features and the long-distance dependence of the image to regress to the image quality score;

[0011] Step 3: iteratively training the network model S of the no-reference image quality evaluation method based on multi-scale features and long-distance dependence;

[0012] Step 4: obtaining a no-reference quality evaluation result of the image, and obtaining a model meeting the requirements, and using the model to evaluate the image quality.

[0013] The step 1 is specifically:

[0014] A part of the images in the public image quality dataset is randomly selected as the training sample set B, and the remaining images are used as the test sample set C, ensuring that there is no repeated image data in the training sample set and the test sample set, and the quality score label of the data set is mapped to the interval [0, 1], ensuring that the prediction result of the model is within the standard unified interval.

[0015] The data set uses six public image quality data sets, including artificial synthetic distortion data sets: LIVE, CSIQ, TID2013 and real distortion data sets: koniq_10k, BID, LIVEC; all reference images in the artificial synthetic distortion data sets LIVE, CSIQ and TID2013 are randomly divided into two subsets R1 and R2, the distortion images corresponding to R1 are used as the training set B, and the distortion images corresponding to R2 are used as the test set C; all image data in the real distortion data sets koniq_10k, BID and LIVEC are randomly divided into a training sample set B and a test sample set C.

[0016] The step 2 is specifically: constructing a parallel MobileNet and Transformer as a feature extractor MT (step 2.1), constructing a quality regression network R (step 2.2), and connecting the feature extractor MT and the quality regression network R in series to form a network model S of the no-reference image quality evaluation method based on multi-scale features and long-distance dependencies.

[0017] Further:

[0018] Step 2.1, constructing a parallel MobileNet and Transformer as a feature extractor MT;

[0019] The feature extractor MT is composed of N feature extractors MT modules connected in sequence, and each MT module is composed of four sub-modules, namely Mobile block, M2T block, Trans block and T2M block, wherein the Mobile block is the basic module of MobileNet, MobileNet is composed of multiple Mobile blocks connected in series, the Trans block is the basic module of Transformer, and Transformer is composed of multiple Trans blocks connected in series, the Trans block and the Mobile block are parallel, the M2T block and the T2M block are connected with the Trans block and the Mobile block of the same level based on the cross attention mechanism, and serve as the bridge of the parallel MobileNet and Transformer, and fuse the multi-scale features extracted by the MobileNet and the long-distance dependencies extracted by the Transformer.

[0020] The Mobile block performs five-step operations on the input multi-scale feature x, which are convolution, ReLU activation, down-sampling convolution, ReLU activation and convolution in turn, to obtain the output feature x of the Mobile block hidden The purpose of this step is to extract semantic information in the low-level multi-scale feature and convert the low-level multi-scale feature into a high-level multi-scale feature.

[0021] The input feature x of the first feature extractor MT module is input into the reverse bottle-neck block I-B from the sample image in the training sample set B or the test sample set C obtained in step 1, and is obtained in turn through convolution, depth separation convolution and convolution, wherein x is a low-level multi-scale feature, and the low-level multi-scale feature contains rich geometric information but lacks semantic information.

[0022] The M2T block uses a lightweight cross-attention mechanism to divide the multi-scale feature x and the global token z into h parts, performs multi-head attention calculation, maps the multi-scale feature x to the global token z, and the calculation formula is:

[0023]

[0024] Wherein is the feature after the input multi-scale feature x of the Mobile block in the same level feature extractor MT module is divided into h parts, is the token after the global token z output by the Trans block in the previous level feature extractor MT module is divided into h parts, and the global token z of the first level feature extractor MT module is a randomly initialized learnable parameter, and W i Q is the query projection matrix of the i-th head, W O is a combination function, which is used to combine multiple heads together; Attention(Q, K, V) is a standard attention function of query Q, key K and value V, and the calculation formula is Compared with the standard attention mechanism, the lightweight cross-attention mechanism used by the M2T block removes the key projection matrix W K and the value projection matrix W V to improve the calculation efficiency. The purpose of this step is to map the multi-scale feature to the dimension of the global token, so as to perform feature fusion in the next step.

[0025] After the multi-scale feature x is mapped to the global token z, the feature A x→z is added to the global token to obtain the output z hidden of the M2T block. The purpose of this step is to fuse the multi-scale feature and the global token to obtain a multi-scale feature carrying long-distance dependency relationship.

[0026] The Trans block is a standard Transformer block, including a multi-head attention module (MHA) and a feed-forward network (FFN), which maps the output z of the M2T block hidden to the output z' of the Trans block through the multi-head attention calculation in the MHA and the multi-layer perceptron and normalization calculation in the FFN.

[0027] The T2M block uses cross-attention in the opposite direction to the M2T block, which maps the global tokens z to the multi-scale features x, and the calculation formula is:

[0028]

[0029] wherein is the feature after the output x of the Mobile block is divided into h parts, hidden is the token after the output z' of the Trans block is divided into h parts, and W i K and W i V are the projection matrices of the keys K and the values V, and compared with the standard attention mechanism, the lightweight cross-attention mechanism used by the T2M block removes the projection matrix W i Q of the query Q to improve the calculation efficiency. The role of this step is to map the global tokens z to the dimension of the multi-scale features x, as the input of the next-level multi-scale feature extractor MT module.

[0030] Step 2.2, construct the quality regression network R;

[0031] concatenate the output x of the Nth MT module and z into a feature vector, which is the image quality feature extracted by the feature extractor MT from the original input image, which is the fused multi-scale feature and long-distance dependent relationship; wherein x is the multi-scale feature obtained by adding the output of the Mobile block in the Nth MT module and the output of the T2M block, and z is the global token z obtained by adding the output of the Trans block in the Nth MT module and the input of the M2T block;

[0032] input the image quality feature into the multi-layer perceptron MLP to regress to the image quality score, wherein the MLP is composed of an output layer network, a hidden layer network and an output layer network, the first layer network of the MLP is the input layer network, the middle layer network is the hidden layer network, which can have multiple layers, and the last layer network is the output layer network;

[0033] At this point, the network model S is constructed, and the network model S can be represented as

[0034] S(x,z)=[MT(x,z)] N R ​

[0035] wherein x, z are inputs of the network model S, x is a multi-scale feature, z is a global token, MT is a feature extractor MT module, N represents a cascade of N feature extractor MT modules, and R represents a quality regression network.

[0036] The step 3 is specifically:

[0037] Step 3.1, initializing the iteration number as t, the maximum iteration number as T, and the current multi-granularity network-based no-reference image quality assessment network model as S t , and letting t = 1, S t = S.

[0038] Step 3.2, randomly selecting b training samples from the training sample set B as inputs of the no-reference image quality assessment network model S t without replacement;

[0039] Step 3.3, first calculating the mean square error between the quality prediction score corresponding to each training sample and the quality score label corresponding to the training sample as the loss value of the model S t , and then updating the weight parameters of each layer of the network model S t using the Adam optimization algorithm;

[0040] The formula for calculating the mean square error between the quality prediction score corresponding to each training sample and the quality score label corresponding to the training sample is

[0041]

[0042] Using the Adam optimization algorithm, except for the learning rate, the remaining parameters all use default values. In order to ensure the convergence speed and learning accuracy of the model, the learning rate is updated according to the following formula

[0043]

[0044]

[0045] b represents the number of training samples randomly selected from the training sample set B without replacement when the model S based on the Transformer encoder is iteratively trained, q g represents the quality score label corresponding to the gth training sample in the b training samples, represents the quality prediction score corresponding to the gth training sample in the b training samples.

[0046] T represents the current number of iterations, S represents the step of updating the learning rate, that is, the learning rate is updated once every S iterations, d represents the attenuation coefficient of the learning rate, at the same time, in order to prevent the model from overfitting in the training process, a regularization term is added to the loss function, and weight decay is added in the training process;

[0047] Step 3.4, judging whether the training sample set B has been traversed, if yes, executing step 3.5, otherwise, executing steps 3.2 and 3.3;

[0048] Step 3.5, judging whether t=T is true, if yes, obtaining the trained multi-granularity network-based no-reference image quality evaluation network model S * t , otherwise, let t=t+1, and execute steps 3.3 and 3.4.

[0049] The step 4 is specifically:

[0050] The test sample set C is input into the trained Transformer encoder-based no-reference image quality evaluation method model S * for forward inference to obtain the quality prediction score of each test sample, so as to verify the model effect.

[0051] The beneficial effects of the present application are:

[0052] The present application uses the lightweight convolutional neural network MobileNet to extract image multi-scale features, uses the Transformer to model long-distance dependency relationships, and uses the lightweight cross-attention module to fuse multi-scale features and long-distance dependency relationships. By using the lightweight convolutional neural network MobileNet and the lightweight cross-attention module, the network model complexity is reduced, the training and inference speed is improved, the long-distance dependency relationship between the distortion information in the multi-scale features and the local area is analyzed, the no-reference quality evaluation of the image is realized, and the image quality score highly consistent with the human visual perception quality is generated. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 It is a network model schematic diagram of the present application.

[0054] Figure 2 It is an MT module diagram in the network model of the present application. DETAILED DESCRIPTION

[0055] The present application will be further described in detail below in combination with the drawings.

[0056] Referring to the drawings Figure 1 , the present application includes the following steps: ​

[0057] Step 1, obtain the training sample set B and the test sample set C.

[0058] Randomly extract 80% of the images from the public image quality dataset as the training sample set B, and the remaining 20% of the images as the test sample set C, ensure that there is no repeated image data in the training sample set and the test sample set, and map the quality score label of the data set to the interval [0, 1] to ensure that the prediction result of the model is within the standard unified interval.

[0059] In this embodiment, six public image quality datasets are used, including artificial synthetic distortion datasets: LIVE, CSIQ, TID2013 and real distortion datasets: koniq_10k, BID, LIVEC. All reference images in the artificial synthetic distortion datasets LIVE, CSIQ, TID2013 are randomly divided into two subsets R1 and R2 in a ratio of 8:2, and the distortion images corresponding to R1 are used as the training set B, and the distortion images corresponding to R2 are used as the test set C. All image data in the real distortion datasets koniq_10k, BID, LIVEC are randomly divided into training sample set B and test sample set C in a ratio of 8:2.

[0060] Step 2, construct a network model S of a no-reference image quality evaluation method based on multi-scale features and long-distance dependency relationships; as shown in Figure 2 ;

[0061] Step 2.1, construct a parallel MobileNet and Transformer as a feature extractor MT.

[0062] The feature extractor MT is composed of N MT modules connected in sequence, and each MT module is composed of four sub-modules, which are Mobile block, M2T block, Trans block and T2M block. Among them, the M2T block and the T2M block are based on the cross-attention mechanism, which serves as a bridge between the parallel MobileNet and the Transformer, and fuses the multi-scale features extracted by the MobileNet and the long-distance dependency relationships extracted by the Transformer.

[0063] The Mobile block performs five-step operations on the input multi-scale feature x, which are 1*1 convolution, ReLU activation, 3*3 down-sampling convolution, ReLU activation and 1*1 convolution in turn, to obtain the output feature x hidden of the Mobile block. The input x of the first MT module is obtained by the original image input reverse bottle neck block I-B, which is obtained by 1*1 convolution, 3*3 depth separation convolution and 1*1 convolution in turn.

[0064] The M2T block uses a lightweight cross-attention mechanism to divide the multi-scale feature x and the global token z into h parts, performs multi-head attention calculation, and fuses the multi-scale feature x into the global token z, and the calculation formula is:

[0065]

[0066] wherein is the feature after the multi-scale feature x is divided into h parts, is the token after the global token z is divided into h parts, the global token z carries the long-distance dependency relationship of the local feature, W i Q is the query projection matrix of the i-th head, W O is a combination function, which is used to combine multiple heads together. Attention(Q, K, V) is a standard attention function of query Q, key K and value V, and the calculation formula is Compared with the standard attention mechanism, the lightweight cross-attention mechanism used by the M2T block removes the key projection matrix W K and the value projection matrix W V to improve the calculation efficiency.

[0067] After the multi-scale feature x is fused into the global token z, the feature A x→z is obtained by adding the global token to obtain the output z hidden of the M2T block.

[0068] The Trans block is a standard Transformer block, including a multi-head attention module (MHA) and a feed-forward network (FFN). The output z hidden of the M2T block is input into the Trans block, and sequentially passes through the multi-head attention calculation in the MHA and the multi-layer perceptron and normalization calculation in the FFN to obtain the output z' of the Trans block.

[0069] The T2M block uses cross-attention in the opposite direction of the M2T block. It fuses the global token z into the multi-scale feature x, and the calculation formula is:

[0070]

[0071] wherein is the feature after the output x hidden of the Mobile block is divided into h parts, is the token after the output z' of the Trans block is divided into h parts, W i K and W i V are the projection matrices of the key K and the value V. Compared with the standard attention mechanism, the lightweight cross-attention mechanism used by the T2M block removes the projection matrix Wi Q to improve the computational efficiency.

[0072] In this embodiment, N is 11 and h is 16.

[0073] Step 2.2, constructing a quality regression network R.

[0074] The output x and z of the Nth MT module are concatenated into a feature vector, which is the image quality feature extracted by the feature extractor MT from the original input image, and the image quality feature is fused with multi-scale features and long-distance dependencies.

[0075] The image quality feature is input into a multi-layer perception MLP to regress to an image quality score.

[0076] In this embodiment, the multi-layer perception MLP is 3 layers.

[0077] Step 3, iteratively training the multi-scale feature and long-distance dependency based no-reference image quality assessment method network model S.

[0078] Step 3.1, initializing the iteration number as t, the maximum iteration number as T, and the current multi-granularity network based no-reference image quality assessment network model as S t , and let t = 1, S t = S.

[0079] In this embodiment, T = 20;

[0080] Step 3.2, randomly selecting b training samples from the training sample set B as the input of the no-reference image quality assessment network model S t

[0081] In this embodiment, b = 16;

[0082] Step 3.3, first calculating the mean square error between the quality prediction score corresponding to each training sample and the quality score label corresponding to the training sample as the loss value of the model S t , and then updating the weight parameters of each layer of the network model S t using the Adam optimization algorithm.

[0083] In this embodiment, the formula for calculating the mean square error between the quality prediction score corresponding to each training sample and the quality score label corresponding to the training sample is

[0084]

[0085] In this embodiment, the Adam optimization algorithm is used, and the default values are used for the remaining parameters except the learning rate. In order to ensure the convergence speed and learning accuracy of the model, the learning rate is updated according to the following formula ​

[0086]

[0087]

[0088] b represents the number of training samples randomly selected from the training sample set B without replacement when the Transformer encoder-based no-reference image quality evaluation method model S is iteratively trained, q g represents the quality score label corresponding to the gth training sample in the b training samples, represents the quality prediction score corresponding to the gth training sample in the b training samples;

[0089] T represents the current iteration number, S represents the step of updating the learning rate, i.e. the learning rate is updated once every S iterations, d represents the decay coefficient of the learning rate, and at the same time, in order to prevent overfitting of the model during training, a regularization term is added to the loss function, and weight decay is added during training;

[0090] In this embodiment, b = 16, s = 1, d = 0.5, and weight_decay = 5x10 -4 ;

[0091] Step 3.4, determine whether the training sample set B has been traversed, if yes, execute step 3.5, otherwise, execute steps 3.2 and 3.3;

[0092] Step 3.5, determine whether t = T is true, if yes, obtain the trained multi-granularity network-based no-reference image quality evaluation network model S * = S t , otherwise, let t = t + 1, and execute steps 3.3 and 3.4;

[0093] Step 4, obtain the no-reference quality evaluation result of the image:

[0094] The test sample set C is input into the trained Transformer encoder-based no-reference image quality evaluation method model S * for forward inference to obtain the quality prediction score of each test sample to verify the model effect.

[0095] The technical effects of the present application are described below in combination with a simulation experiment:

[0096] 1. Simulation conditions and content:

[0097] The hardware platform of the simulation experiment of the application is: the processor is Intel(R) Core(TM) i9-7900X CPU, the frequency is 3.30GH, the memory is 32GB, and the graphics card is NVIDIA GeForce GTX 1080Ti.

[0098] The software platform of the simulation experiment of the application is: Ubuntu 16.04, Pytorch 1.6.0, and Python 3.7.

[0099] 2. Simulation experiment

[0100] The input image used in the simulation experiment of the present application is derived from the image quality data set disclosed in the image quality evaluation database, including artificial synthetic distortion data set: LIVE, CSIQ, TID2013 and real distortion data set: koniq_10k, BID, LIVEC, wherein, the LIVE data set is an image quality evaluation data set proposed by D. Ghadiyaram et al. in “Massive online crowdsourced study of subjective and objective picture quality. IEEE Transactions on Image Processing, 25(1): 372-387, 2016”, the CSIQ data set is an image quality evaluation data set proposed by E. C. Larson et al. in “Most apparent distortion: full-reference image quality assessment and the role of strategy. Journal of Electronic Imaging, 19(1): 011006, 2010”, the TID2013 data set is an image quality evaluation data set proposed by N. Ponomarenko et al. in “Color image database TID2013: Peculiarities and preliminary results. In European Workshop on Visual Information Processing (EUVIP), 106-111, 2013.”, the koniq_10k is an image quality evaluation data set proposed by V. Hosu et al. in “KonIQ-10k: An Ecologically Valid Database for Deep Learning of Blind Image Quality Assessment. in IEEE Transactions on Image Processing, vol. 29, pp. 4041-4056, 2020”, the BID is an image quality evaluation data set proposed by A. Ciancio et al. in “No-reference blur assessment of digital pictures based on multifeature classiflers. IEEE Transactions on Image Processing, 20(1): 64-75, Jan. 2010.”, and the LIVEC is an image quality evaluation data set proposed by D.The image quality evaluation dataset proposed by Ghadiyaram et al. in "Massive online crowd sourced study of subjective and objective picture quality. IEEE Transactions on Image Processing, 25(1):372-387, Jan. 2016."

[0101] In the simulation experiment of the present application, two indexes, the Spearman rank-order correlation coefficient (SROCC) and the Pearson linear correlation coefficient (PLCC), are used to evaluate the image quality evaluation effect of the present application and the prior art. Specifically, the prior art and the present application each select n samples from the test sample set C for image quality evaluation, and output quality prediction scores. The values of SROCC and PLCC are calculated from the quality prediction scores of the samples and the quality label scores corresponding to the samples. The two indexes are calculated according to the following formula, SROCC ∈ [-1, 1], the higher the value, the more correct the evaluation result of the evaluated no-reference image quality evaluation method reflects the quality of the image, and PLCC ∈ [-1, 1], the higher the value, the closer the evaluation result of the evaluated no-reference image quality evaluation method is to the subjective evaluation score of a human.

[0102]

[0103]

[0104] wherein n represents the total number of images, b e represents the difference between the order number of the quality prediction score of the e-th distorted image in the order of the quality prediction scores of the n distorted images and the order number of the quality label score, s r represents the quality prediction score of the r-th distorted image, represents the average value of the quality prediction scores of all distorted images, p r represents the subjective evaluation score of the r-th distorted image, represents the average value of the subjective evaluation scores of all distorted images.

[0105] The simulation results are shown in Table 1.

[0106] Table 1. Comparison of evaluation results of the present application and the prior art

[0107]

[0108]

[0109] From table 1, compared with the prior art, the SROCC and PLCC of the evaluation results on the CSIQ, TID2013 and LIVE data sets are all higher, proving that the precision of the no-reference image quality evaluation of the application is higher.

[0110] The application extracts image multi-scale features and long-distance dependency relationships between local regions by using a lightweight convolutional neural network MobileNet and a Transformer, fuses the multi-scale features and long-distance dependency relationships by adding a lightweight cross-attention module between the parallel MobileNet and Transformer, improves the model training and inference efficiency by reducing network parameters, and realizes no-reference quality evaluation of the image by analyzing distortion information in the multi-scale features and the long-distance dependency relationships between the local regions, to generate a predicted quality score of the image.

[0111] The above content only illustrates the technical idea of the application, and cannot limit the protection scope of the application, and any modification made according to the technical idea of the application on the basis of the technical scheme falls within the protection scope of the claims of the application.

Claims

1. A blind image quality assessment method based on multi-scale features and long-range dependencies, characterized in that, The method comprises the following steps: Step 1, extracting image composition training sample set B and test sample set C from the image quality data set; The training sample set B is used for parameter training and optimization of the network model, so that the network model establishes a mapping from the image to the quality score, and the test sample set C is used for testing and verifying the prediction performance of the network model on the image quality score; Step 2, constructing a network model S of a no-reference image quality evaluation method based on multi-scale features and long-distance dependencies; for extracting and fusing multi-scale features and long-distance dependencies of the image to regress to the image quality score; Step 3, iteratively training the network model S of the no-reference image quality evaluation method based on multi-scale features and long-distance dependencies; Step 4, obtaining the no-reference quality evaluation result of the image, obtaining the model meeting the requirements, and using the model to evaluate the image quality; The step 2 is specifically: constructing a parallel MobileNet and Transformer as a feature extractor MT, constructing a quality regression network R, and connecting the feature extractor MT and the quality regression network R in series to form the network model S of the no-reference image quality evaluation method based on multi-scale features and long-distance dependencies; Step 2.1, constructing a parallel MobileNet and Transformer as a feature extractor MT; The feature extractor MT is composed of N feature extractors MT modules connected in sequence, each MT module is composed of four sub-modules, which are Mobile block, M2T block, Trans block and T2M block, wherein the Mobile block is the basic module of MobileNet, MobileNet is composed of multiple Mobile blocks in cascade, the Trans block is the basic module of Transformer, Transformer is composed of multiple Trans blocks in cascade, the Trans block and the Mobile block are parallel, the M2T block and the T2M block are based on the cross attention mechanism, and are connected with the Trans block and the Mobile block at the same level as the bridge of the parallel MobileNet and Transformer, and fuse the multi-scale features extracted by the MobileNet and the long-distance dependencies extracted by the Transformer; Step 2.2, constructing a quality regression network R; The output x and z of the Nth MT module are connected into a feature vector, which is the image quality feature of the fused multi-scale features and long-distance dependencies extracted from the original input image by the feature extractor MT; wherein x is the multi-scale feature obtained by adding the output of the Mobile block in the Nth MT module and the output of the T2M block, and z is the global token z obtained by adding the output of the Trans block in the Nth MT module and the input of the M2T block; Input the image quality feature into the multi-layer perception MLP to regress to the image quality score, wherein the MLP is composed of an output layer network, a hidden layer network and an output layer network, the first layer network of the MLP is an input layer network, the middle layer network is a hidden layer network, which can have multiple layers, and the last layer network is an output layer network; Up to now, the network model S is constructed, and the network model S is represented as S(x,z) = [MT(x,z)] N R Wherein, x and z are inputs of the network model S, x is a multi-scale feature, z is a global token, MT is a feature extractor MT module, N represents N feature extractor MT module cascades, and R represents a quality regression network. The five-step operation on the input multi-scale feature x in the Mobile block is, in sequence, convolution, ReLU activation, down-sampling convolution, ReLU activation, and convolution, to obtain the output feature x of the Mobile block hidden ; The input feature x of the first feature extractor MT module is input into the reverse bottleneck block I-B from the sample image in the training sample set B or the test sample set C obtained in step 1, and is obtained in turn through convolution, depth separation convolution and convolution, wherein x is a low-level multi-scale feature, and the low-level multi-scale feature contains rich geometric information but lacks semantic information.

2. The blind image quality assessment method based on multi-scale features and long-range dependencies according to claim 1, characterized in that, The step 1 is specifically: A part of images is randomly extracted from the public image quality dataset as the training sample set B, and the remaining images are used as the test sample set C, so that there is no repeated image data in the training sample set and the test sample set, the quality score label of the dataset is mapped to the interval [0, 1], and the prediction result of the model is ensured to be in the standard unified interval; The dataset uses six public image quality datasets, including artificial synthetic distortion datasets: LIVE, CSIQ, TID2013 and real distortion datasets: koniq_10k, BID, LIVEC; all reference images in the artificial synthetic distortion datasets LIVE, CSIQ and TID2013 are randomly divided into two subsets R1 and R2, the distortion images corresponding to R1 are used as the training set B, and the distortion images corresponding to R2 are used as the test set C; all image data in the real distortion datasets koniq_10k, BID and LIVEC are randomly divided into the training sample set B and the test sample set C. 3.The blind image quality assessment method based on multi-scale features and long-range dependencies according to claim 1, characterized in that, The M2T block uses a lightweight cross-attention mechanism to divide the multi-scale feature x and the global token z into h parts, perform multi-head attention calculation, and map the multi-scale feature x to the global token z, and the calculation formula is: wherein is the feature after dividing the input multi-scale feature x of the Mobile block in the same level feature extractor MT module into h parts, is the token after dividing the global token z output by the Trans block in the upper level feature extractor MT module, the global token z of the first level feature extractor MT module is a randomly initialized learnable parameter, W i Q is the query projection matrix of the i-th head, W O is a combination function, which is used to combine multiple heads together; Attention(Q, K, V) is a standard attention function of query Q, key K and value V, and the calculation formula is A feature A after mapping the multi-scale feature x to the global token z x→z Adding the global token to get the output z of M2T fast hidden .

4. The blind image quality assessment method based on multi-scale features and long-range dependencies according to claim 1, characterized in that, The Trans block is a standard Transformer block, including a multi-head attention module (MHA) and a feed-forward network (FFN), which takes the output z of the M2T block hidden The input Trans block goes through the multi-head attention computation in MHA and the multi-layer perceptron and normalization computation in FFN in sequence to obtain the output z' of the Trans block.

5. The blind image quality assessment method based on multi-scale features and long-range dependencies according to claim 1, characterized in that, The cross-attention used by the T2M block is opposite to that of the M2T block, which maps the global token z to the multi-scale feature x, and the calculation formula is: wherein is a feature of the Mobile block's output x hidden split into h parts, is a token of the Trans block's output z′ split into h parts, W i K and W i V is a projection matrix of keys K and values V.

6. The blind image quality assessment method based on multi-scale features and long-range dependencies according to claim 1, characterized in that, The step 3 is specifically: Step 3.1, initialize the iteration number t, the maximum iteration number T, and the current multi-granularity network based no-reference image quality assessment network model S t , and let t = 1, S t = S; Step 3.

2. Randomly select b training samples from the training sample set B as the input of the no-reference image quality evaluation network model S without replacement. t Step 3.

2. Randomly select b training samples from the training sample set B as the input of the no-reference image quality evaluation network model S without replacement. Step 3.3, first calculate the mean square error between the quality prediction score corresponding to each training sample and the quality score label corresponding to the training sample as the loss value of the model S t , and then update the weight parameters of each layer of the network model S t using the Adam optimization algorithm. The formula for calculating the mean square error of the quality prediction score corresponding to each training sample and the quality score label corresponding to the training sample is The Adam optimization algorithm is used, and the default value is used for the remaining parameters except the learning rate. In order to ensure the convergence speed and learning accuracy of the model, the learning rate is updated according to the following formula: b represents the number of training samples randomly selected without replacement from the training sample set B when iteratively training the no-reference image quality evaluation method model S based on the Transformer encoder, q g represents the quality score label corresponding to the gth training sample in the b training samples, represents the quality prediction score corresponding to the gth training sample in the b training samples; T represents the current iteration number, S represents the step of updating the learning rate, that is, the learning rate is updated once every S iterations, and d represents the attenuation coefficient of the learning rate. Step 3.4, if the training sample set B has been traversed, step 3.5 is executed, otherwise, steps 3.2 and 3.3 are executed; Step 3.5, judge whether t=T is true, if yes, get the trained multi-granularity network based no-reference image quality assessment network model S * =S t , otherwise, let t=t+1, and execute step 3.3 and 3.

4.

7. The blind image quality assessment method based on multi-scale features and long-range dependencies according to claim 6, characterized in that, The step 4 is specifically: Forward inference is performed on the test sample set C as the input of the trained Transformer encoder-based no-reference image quality evaluation method model S to obtain the quality prediction score of each test sample to verify the model effect. * Forward inference is performed on the test sample set C as the input of the trained Transformer encoder-based no-reference image quality evaluation method model S to obtain the quality prediction score of each test sample to verify the model effect.

Images