A no-reference image quality assessment method, device and computer-readable storage medium based on image feature fusion

Abstract

The present invention belongs to the field of computer vision, and specifically relates to a no-reference image quality evaluation method, device, and computer-readable storage medium based on image feature fusion, including: obtaining a naturally distorted image to be evaluated, generating a gradient image according to the naturally distorted image, and inputting the naturally distorted image and the gradient image into a trained no-reference image quality evaluation model based on image feature fusion to obtain a quality evaluation score; wherein, the no-reference image quality evaluation model based on image feature fusion includes: a backbone network, a cross-domain feature fusion model, and a cross-scale feature fusion model. The present invention not only evaluates images, but also fully considers the global semantic information and local semantic information in the domain of naturally distorted images, and can more accurately evaluate the quality of naturally distorted images.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and particularly relates to a no-reference image quality evaluation method, device, and computer-readable storage medium based on image feature fusion. Background Art

[0002] No-reference image quality assessment refers to evaluating the quality of a distorted image itself only through the distorted image. Different from the full-reference image quality assessment method, no-reference image quality assessment does not require a reference image.

[0003] With the in-depth research on backbone networks, the methods for extracting image features have been very rich. However, how to apply the extracted features to image quality evaluation is still a challenging task, which is also one of the factors hindering the development of image quality evaluation.

[0004] Existing image quality evaluation technologies only use a kind of feature information of the image itself to evaluate the quality of the image. For example, in patent CN201810759247.X, a generative adversarial network is mainly used to construct an image training model. Then, a high-definition lossless image is used as a training data set and sent into the image training model for training and learning to obtain a no-reference image quality evaluation model with a trained complete discriminative network. Finally, the image to be evaluated is sent into the trained complete discriminative network, and the final evaluation result is obtained through scoring and weighting. This method mainly evaluates the quality of the image without a reference image according to the features of the image itself, does not pay attention to the role of other images in the quality evaluation of the original image, and cannot take into account the global semantic information and local information of the image, resulting in inaccurate evaluation of the image quality. Summary of the Invention

[0005] In order to solve the problem that the existing technology does not pay attention to the role of other images in the quality evaluation of the original image, cannot take into account the global semantic information and local information of the image, and results in inaccurate evaluation of the image quality, the present invention proposes a no-reference image quality evaluation method, device, and computer-readable storage medium based on image feature fusion, which is used to fuse the features of the original image quality and the features of the original image gradient map, so as to evaluate the quality of the original image, fully consider the global semantic information and local information of the original image, and improve the accuracy of image quality evaluation.

[0006] The present invention adopts the following technical solutions:

[0007] A no-reference image quality evaluation method based on image feature fusion includes the following steps:

[0008] Obtain the natural distorted image to be evaluated, generate a gradient image according to the natural distorted image, and input the natural distorted image and the gradient image into the trained no-reference image quality evaluation model based on image feature fusion to obtain a quality evaluation score; wherein, the no-reference image quality evaluation model based on image feature fusion includes: a backbone network, a cross-domain feature fusion model, a cross-scale feature fusion model, and two linear regression layers;

[0009] The process of training the no-reference image quality evaluation model based on image feature fusion includes:

[0010] S1: Obtain the natural distorted image domain with true labels, where the true labels represent the true scores of the natural distorted images in the natural distorted image domain;

[0011] S2: Generate a gradient image domain according to the natural distorted image domain;

[0012] S3: Input the natural distorted image domain and its corresponding gradient image domain into the backbone network to extract the hierarchical features of the natural distorted image domain and the hierarchical features of the gradient image domain;

[0013] S4: Input the hierarchical features of the natural distorted image domain and the hierarchical features of the gradient image domain into the cross-domain feature fusion model for fusion, and calculate the cross-domain fusion hierarchical features of the natural distorted image domain;

[0014] S5: Input the cross-domain fusion hierarchical features into the cross-scale feature fusion model for fusion, and calculate the cross-scale fusion features of the natural distorted image domain;

[0015] S6: Input the cross-scale fusion features into two linear regression layers for regression processing, and calculate the quality evaluation scores of the natural distorted image domain;

[0016] S7: Calculate the loss function of the no-reference image quality evaluation model based on image feature fusion according to the true labels and quality evaluation scores of the natural distorted image domain;

[0017] S8: Continuously adjust the parameters of the model, and complete the training of the model when the loss function is less than the set threshold.

[0018] To achieve the above object, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned no-reference image quality evaluation method based on image feature fusion is implemented.

[0019] To achieve the above object, the present invention also provides a no-reference image quality evaluation device based on image feature fusion, including a processor and a memory; the memory is used for storing a computer program; the processor is connected to the memory and is used for executing the computer program stored in the memory, so that the no-reference image quality evaluation device based on image feature fusion executes the above-mentioned no-reference image quality evaluation method based on image feature fusion.

[0020] The present invention has at least the following beneficial effects:

[0021] The present invention extracts features from the natural distortion image domain and the gradient image domain through a backbone network, and cross-domain fuses the features between different levels of the gradient image domain and the natural distortion image domain through a cross-domain feature fusion model to obtain cross-domain fusion hierarchical features with local semantic information. The cross-scale feature fusion module can perform cross-scale feature fusion on the cross-domain fusion hierarchical features, so as to obtain natural distortion image domain features with both global semantic information and local semantic information. The natural distortion image domain features with global semantic information and local semantic information are subjected to regression processing to calculate the quality evaluation score of the natural distortion image domain, so that the obtained quality evaluation score can better reflect the quality information of the natural distortion image domain. Through the present invention, accurate image quality information can be obtained without a reference image. Compared with the traditional no-reference image quality evaluation method, the present invention fully considers the gradient image domain as an auxiliary image domain of the natural distortion image domain, and can obtain richer prior information and local information by fusing different image domains, which is beneficial to improving the accuracy of quality evaluation. By performing cross-scale fusion on the features of different levels in the cross-domain fusion hierarchical features, image features with global and local information can be obtained, and the evaluation score of the image quality is obtained by performing linear regression on the image features with global and local information, which improves the accuracy of the image quality evaluation. The no-reference image quality evaluation method based on image feature fusion provided by the present invention can be widely applied to fields such as artificial intelligence, image recognition, photographing, and target detection, and improves the quality evaluation ability of natural distortion images. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 : is a block diagram of the implementation system of the present invention;

[0023] Figure 2 : is a flowchart of the model training of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention. Based on the embodiments of the present invention, other embodiments obtained by those of ordinary skill in the art without creative efforts all belong to the protection scope of the present invention.

[0025] Please refer to Figure 1 , the present invention proposes a no-reference image quality evaluation method based on image feature fusion, which specifically includes the following steps:

[0026] Obtain the natural distorted image to be evaluated, generate a gradient image according to the natural distorted image, and input the natural distorted image and the gradient image into the trained no-reference image quality evaluation model based on image feature fusion to obtain a quality evaluation score; wherein, the no-reference image quality evaluation model based on image feature fusion includes: a backbone network, a cross-domain feature fusion model, a cross-scale feature fusion model, and two linear regression layers. Among them, the natural distorted image to be evaluated can be an image taken by any camera, or an image downloaded from the network, or an image obtained from other storage devices, and can also be a static image or a moving image, etc. The natural distorted image to be evaluated may include one or more objects. For example, when taking a picture of a person, the environment around the person, such as a car, a tree, etc., may be captured. Then, the person, the car, the electronics, the tree, etc. are all objects included in the image to be processed.

[0027] Please refer to Figure 2 , the process of training the no-reference image quality evaluation model based on image feature fusion includes:

[0028] S1: Obtain the natural distorted image domain with real labels, where the real label represents the real score of the natural distorted image in the natural distorted image domain.

[0029] In view of the fact that most traditional NR-IQA methods only use RGB images as the input of the model, there are problems such as whether it is beneficial to use multi-feature fusion or whether one image is sufficient. Image gradients also play a crucial role in many visual tasks. Image gradients acutely reflect the structural components of an image, such as image edges. The gradient image can robustly reflect the details of the image structure under changes in image intensity and color. Therefore, using the gradient map as the data input to supplement the natural distorted images and assist in the feature extraction of natural distorted images. This design assists in the feature extraction of the texture quality of natural distorted images and reduces the difficulty of extracting features from a single natural distorted image. Among them, in the present invention, the natural distorted image domain mainly uses the LIVE-Challenge dataset. The LIVE-Challenge dataset contains 1,169 natural distorted images without reference images. The image size is 500x500, and the true label is the subjective evaluation score (MOS value) of the natural distorted images in the natural distorted image domain. The size range of MOS is [0, 100], which is obtained by testing 8,100 testers. The higher the score, the better the picture quality. Among them, the natural distorted image domain is represented as an image set containing multiple natural distorted images.

[0030] S2: Generate a gradient image domain according to the natural distorted image domain.

[0031] A specific embodiment of a no-reference image quality evaluation method based on image feature fusion. The specific implementation method of processing the natural distorted image into a gradient image includes:

[0032]

[0033]

[0034]

[0035]

[0036] Among them, (x, y) is the pixel coordinate of the natural distorted image I Z of, represents the partial derivative of the natural distorted image I Z in the X direction; represents the partial derivative of the natural distorted image I Z in the Y direction; represents the total partial derivative of the natural distorted image I Z ; G(I Z ) represents the gradient magnitude of the natural distorted image I Z .

[0037] S3: Input the natural distortion image domain and its corresponding gradient image domain into the backbone network to extract the hierarchical features of the natural distortion image domain and the hierarchical features of the gradient image domain.

[0038] A specific embodiment of a no-reference image quality assessment method based on image feature fusion is as Figure 2 shown:

[0039] Input the processed natural distortion image domain and gradient image domain into the Resnet50 backbone network to extract L hierarchical features Stage1-L of the natural distortion image domain and the gradient image domain respectively. Figure 2 The square blocks containing D represent the L hierarchical features of the natural distortion image domain (there are actually L hierarchical features, only one hierarchical feature is drawn), and the square blocks containing G represent the L hierarchical features of the gradient image domain (there are actually L hierarchical features, only one hierarchical feature is drawn). It should be noted that in this embodiment, in order to reduce the computational complexity, only 4 hierarchical features (L = 4) extracted by the backbone network are selected, which does not mean that only 4 hierarchical features can be processed in the present invention. The reason for selecting 4 hierarchical features is that through a large number of experimental studies in the existing technology and this field, it is found that extracting 4 hierarchical features has the best cost performance and the best processing effect.

[0040] S4: Correspondingly input the hierarchical features of the natural distortion image domain and the hierarchical features of the gradient image domain into the cross-domain feature fusion model for fusion, and calculate the cross-domain fusion hierarchical features of the natural distortion image domain.

[0041] A specific embodiment of a no-reference image quality assessment method based on image feature fusion is as Figure 2 shown. The cross-domain feature fusion model (CDFM) includes: a fully connected layer, a self-attention mechanism layer, and a multi-layer perceptron MLP.

[0042] A specific embodiment of a no-reference image quality assessment method based on image feature fusion is as Figure 2 shown. The calculation method of the cross-domain fusion hierarchical features includes:

[0043] S41: Generate a query Q 1n , a key K 1n , and a value V 1n through the fully connected layer for the feature of the nth layer of the natural distortion image domain, and process Q 1n , K 1n , and V 1n according to the self-attention mechanism to calculate the first global semantic hierarchical feature X n of the nth layer of the natural distortion image domain;

[0044] S42: Generate a query Q through the fully connected layer for the feature of the nth layer of the gradient image domain2n and key K 2n and value V 2n , and process Q according to the self-attention mechanism 2n , K 2n and V 2n , and calculate the second global semantic level feature Y of the nth level in the gradient image domain n ;

[0045] S43: Process Q 1n , K 2n and V 2n according to the self-attention mechanism, and calculate the fusion feature Z of the nth level in the natural distortion image domain and the gradient image domain n ;

[0046] S44: Input X n , Y n and Z n into the multi-layer perceptron MLP, and calculate the cross-domain fusion level feature F of the nth level in the natural distortion image domain n , where by fusing the features between different levels of the natural distortion image domain and the gradient image domain, the obtained cross-domain fusion level feature has rich prior information and local information, which is beneficial to improving the accuracy of quality evaluation.

[0047] S5: Input the cross-domain fusion level feature into the cross-scale feature fusion model for fusion, and calculate the cross-scale fusion feature of the natural distortion image domain.

[0048] A specific embodiment of a no-reference image quality evaluation method based on image feature fusion, as Figure 2 shown, the cross-scale feature fusion model includes: STB model (Swin-Transformer Block) and global average pooling layer (GAP);

[0049] A specific embodiment of a no-reference image quality evaluation method based on image feature fusion, as Figure 2 shown, the calculation method of the cross-scale fusion feature includes:

[0050] S51: Input the cross-domain fusion level feature F of the nth level in the natural distortion image domain n into the STB model to calculate the feature map A n :

[0051] S52: Concatenate the cross-domain fusion level feature F of the n+1th level in the natural distortion image domain n+1 with the feature map A n using the Cat function and input them into the STB model to calculate the feature map A n+1 ;

[0052] S53: Repeat step S52 to obtain feature map A L ;

[0053] S54: Input feature map A L into the GAP layer to output cross-scale fusion features;

[0054] where A L represents the feature map at the L-th level in the natural distortion image domain, n = 1, 2, 3... L - 1, L represents the number of hierarchical features extracted by the backbone network. By fusing features at different levels in the cross-domain fusion hierarchical features across scales, rich global and local image information can be obtained, which is beneficial to improving the accuracy of quality evaluation. In this embodiment, only 4 hierarchical features extracted by the backbone network are selected above, so L is 4.

[0055] A specific embodiment of a no-reference image quality evaluation method based on image feature fusion. The specific calculation method of cross-scale fusion features includes:

[0056] The cross-domain fusion hierarchical features output by the cross-domain feature fusion model CDFM are sequentially fed into the Swin-Transformer Block (STB). STB module: First, the input feature block is fed into the Window Multi-head Self-attention (W-MSA) module through the LayerNorm layer for window-based self-attention, and then fused through another LayerNorm layer and MLP layer; then fed into the Shift Window Multi-head Self-attention (SW-MSA) module through the LayerNorm layer, followed by a LayerNorm layer and MLP layer, and finally a Conv (convolution) operation is performed to reduce the feature scale.

[0057] S6: Input the cross-scale fusion features into two linear regression layers (FC1 and FC2) for regression processing to calculate the quality evaluation score in the natural distortion image domain.

[0058] S7: Calculate the loss function of the no-reference image quality evaluation model based on image feature fusion according to the true label and quality evaluation score in the natural distortion image domain.

[0059] The loss function of the no-reference image quality evaluation model based on image feature fusion includes:

[0060]

[0061] where N represents the number of natural distortion images in the natural distortion image domain, F i (I D,I G ) represents the quality evaluation score of the \(i\)-th natural distortion image domain, \(I\) D represents the natural distortion image domain, \(I\) G represents the gradient image domain, \(Q\) i represents the true label of the \(i\)-th natural distortion image, and loss represents the loss function.

[0062] S8: Continuously adjust the parameters of the model. When the loss function is less than the set threshold, the training of the model is completed, where the threshold is set by those skilled in the art according to the actual situation.

[0063] In this design, ViT-B / 8 is selected as the pre-trained model. This model is trained on ImageNet-21k and fine-tuned on ImageNet 1k. The patch size \(P\) is set to 8. Following the standard training strategy of existing IQA algorithms, the learning rate of the pre-trained model is set to \(1\times10^{-5}\), the batch size \(Batch-size (B)\) is set to 32, and the Adaptive Moment Estimation (ADAM) optimizer with a weight decay of \(1\times10^{-5}\) is used to optimize the model, and the learning rate of the model is optimized using the cosine annealing learning rate (CosineannealingLR) strategy.

[0064] In an embodiment of the present invention, the present invention further includes a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements any one of the above-mentioned no-reference image quality evaluation methods based on image feature fusion.

[0065] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk, or optical disc and other various media that can store program codes.

[0066] A no-reference image quality evaluation device based on image feature fusion includes a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the no-reference image quality evaluation device based on image feature fusion executes any one of the above-mentioned no-reference image quality evaluations based on image feature fusion.

[0067] Specifically, the memory includes: ROM, RAM, magnetic disk, USB flash drive, memory card, or optical disc and other various media that can store program codes.

[0068] Preferably, the processor may be a general-purpose processor, including a Central Processing Unit (CPU for short), a Network Processor (NP for short), etc.; it may also be a Digital Signal Processor (DSP for short), an Application Specific Integrated Circuit (ASIC for short), a Field Programmable Gate Array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0069] The method designed by the present invention can improve the accuracy and generalization ability of natural image quality evaluation. Specifically: the designed model can further utilize the local information and global semantic information of the picture, so as to give an image quality evaluation result closer to the human visual system, so as to realize the quality evaluation task of various types of images without a reference image. For example, according to the quality evaluation score of the natural distorted image, assist the natural distorted image for image restoration; when applied to artificial intelligence, the robot judges the target through the quality evaluation score; when used for taking pictures, automatically adjust parameters such as the resolution of the lens according to the quality evaluation score.

[0070] In addition, the above-mentioned drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present invention, rather than for limiting purposes. It is easy to understand that the processes shown in the above-mentioned drawings do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously in, for example, multiple modules.

[0071] Those skilled in the art will readily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the claims.

[0072] It should be understood that the present disclosure is not limited to the exact structures already described and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A no-reference image quality assessment method based on image feature fusion, characterized in that Including: Obtain a natural distorted image to be evaluated, generate a gradient image according to the natural distorted image, and input the natural distorted image and the gradient image into a trained no-reference image quality evaluation model based on image feature fusion to obtain a quality evaluation score; wherein, the no-reference image quality evaluation model based on image feature fusion includes: a backbone network, a cross-domain feature fusion model, a cross-scale feature fusion model, and two linear regression layers. The process of training the no-reference image quality evaluation model based on image feature fusion includes: S1: Obtain a natural distorted image domain with true labels, where the true labels represent the true scores of the natural distorted images in the natural distorted image domain. S2: Generate a gradient image domain according to the natural distorted image domain. S3: Input the natural distorted image domain and its corresponding gradient image domain into the backbone network to extract the hierarchical features of the natural distorted image domain and the hierarchical features of the gradient image domain. S4: Input the hierarchical features of the natural distorted image domain and the hierarchical features of the gradient image domain into the cross-domain feature fusion model for fusion, and calculate the cross-domain fusion hierarchical features of the natural distorted image domain. S5: Input the cross-domain fusion hierarchical features into the cross-scale feature fusion model for fusion, and calculate the cross-scale fusion features of the natural distorted image domain. S6: Input the cross-scale fusion features into two linear regression layers for regression processing, and calculate the quality evaluation scores of the natural distorted image domain. S7: Calculate the loss function of the no-reference image quality evaluation model based on image feature fusion according to the true labels and quality evaluation scores of the natural distorted image domain. S8: Continuously adjust the parameters of the model, and complete the training of the model when the loss function is less than the set threshold.

2. The no-reference image quality assessment method based on image feature fusion according to claim 1, wherein The cross-domain feature fusion model includes: a fully connected layer, a self-attention mechanism layer, and a multi-layer perceptron MLP.

3. A no-reference image quality assessment method based on image feature fusion according to claim 2, characterized in that, The calculation method of the cross-domain fusion hierarchical features includes: S41: Generate a query, a key, and a value from the features of the n-th level in the natural distortion image domain through a fully connected layer, and process them according to the self-attention mechanism to calculate the first global semantic level feature of the n-th level in the natural distortion image domain , key , and value , and process according to the self-attention mechanism , , and , and calculate the first global semantic level feature of the n-th level in the natural distortion image domain ; S42: Generate queries for the features of the n-th level in the gradient image domain through a fully connected layer , keys and values , and process them according to the self-attention mechanism , and , and calculate the second global semantic level features of the n-th level in the gradient image domain ; S43: Process according to the self-attention mechanism , and , calculate the fusion features of the nth layer in the natural distortion image domain and the gradient image domain ; S44: Input , and into a multi-layer perceptron (MLP) to calculate the cross-domain fusion hierarchical features of the n-th level in the natural distortion image domain .

4. A no-reference image quality assessment method based on image feature fusion according to claim 1, characterized in that The cross-scale feature fusion model includes: an STB model and a GAP layer.

5. A no-reference image quality assessment method based on image feature fusion according to claim 4, characterized in that The calculation method of the cross-scale fusion features includes: S51: Input the cross-domain fusion hierarchical features of the th layer in the natural distortion image domain into the STB model to calculate the feature map ; ​ S52: Take the cross-domain fusion hierarchical feature of the th level in the natural distortion image domain and the feature map concatenate them together using the Cat function and input them into the STB model to calculate the feature map ; S53: Repeat step S52 to obtain the feature map ; S54: Input the feature map into the GAP layer to output cross-scale fusion features; Among them, represents the feature map of the L-th layer in the natural distortion image domain, , where L represents the number of hierarchical features extracted by the backbone network.

6. A no-reference image quality assessment method based on image feature fusion according to claim 1, characterized in that The loss function of the no-reference image quality evaluation model based on image feature fusion includes: Among them, represents the number of natural distortion images, represents the quality evaluation score of the i-th natural distortion image, represents the natural distortion image domain, represents the gradient image domain, represents the true label of the i-th natural distortion image, represents the loss function.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor to implement a no-reference image quality evaluation method according to any one of claims 1 to 6.

8. An image quality evaluation device without reference based on image feature fusion, characterized in that Including a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that a no-reference image quality evaluation device based on image feature fusion executes a no-reference image quality evaluation method according to any one of claims 1 to 6.

Images