A no-reference image quality assessment method combining Transformer and convolutional network

Abstract

The present invention discloses a reference-free image quality assessment method that combines a Transformer and a convolutional network. The method comprises: downsampling and position encoding a distorted image to obtain a feature sequence, which is input into a Transformer-based generative network to obtain multi-layer features of the distorted image; linearly inverse-averaging the third-layer features of the distorted image to obtain a pseudo-reference image; transforming the multi-layer features of the distorted image through convolution and normalization operations, and then weighted addition to obtain the global features of the distorted image; inputting the distorted image and the pseudo-reference image into a convolutional neural network to extract convolution features, respectively, and concatenating them to obtain local features of the distorted image; concatenating the global and local features of the distorted image to obtain fused features, which are input into a linear regression network to obtain the quality assessment result of the distorted image. By combining the Transformer-based global features of the distorted image with the local convolutional features of the distorted image and the pseudo-reference image, the method achieves better quality prediction accuracy.

Description

Technical Field

[0001] The present invention relates to a no-reference image quality assessment method combining a Transformer and a convolutional network, and specifically to a no-reference image quality assessment method, device, and storage medium combining global Transformer features and local convolutional features, belonging to the technical field of image quality assessment. Background Art

[0002] Image quality assessment is crucial for image processing and computer vision tasks. Image quality assessment methods can assess distortion, as raw images often become distorted during compression, transmission, and storage. Furthermore, when designing objective functions for tasks like image enhancement and reconstruction, effective image quality assessment methods can help achieve higher-quality image processing results.

[0003] Image quality assessment methods are generally divided into three categories: full-reference, semi-reference, and no-reference image quality assessment. Full-reference image quality assessment requires the original image as a reference and judges image quality based on the difference between the original and distorted images. Semi-reference image quality assessment only requires partial original image information, or certain statistical characteristics of the original and distorted images to evaluate image quality. No-reference image quality assessment does not require the original image. Currently, mainstream methods include those based on the statistical characteristics of distorted images, those based on the human visual system, and those based on deep learning.

[0004] Among the no-reference image quality assessment methods based on deep learning, the traditional CNN network framework has achieved great success, but the CNN network's ability to extract global features is relatively weak.

[0005] To this end, it is necessary to improve the performance of no-reference image quality assessment methods. Summary of the Invention

[0006] In order to overcome the defects of the existing technology, the present invention proposes a reference-free image quality assessment method that combines Transformer and convolutional network. The generative network is used to restore the distorted image to obtain a pseudo-reference image. At the same time, the model based on the Transformer structure can better utilize the global information of the distorted image, thereby improving the accuracy of image quality assessment.

[0007] In order to solve the above technical problems, the present invention adopts the following technical means:

[0008] In a first aspect, the present invention proposes a method for evaluating image quality without reference, comprising:

[0009] Downsample and position encode the distorted image to obtain a feature sequence;

[0010] Inputting the feature sequence into a Transformer-based generative network to obtain multi-layer features of the distorted image, wherein the multi-layer features of the distorted image include first-layer features of the distorted image, second-layer features of the distorted image, and third-layer features of the distorted image;

[0011] The third layer features of the distorted image are linearly inverse amortized to obtain a pseudo reference image;

[0012] The multi-layer features of the distorted image are transformed through convolution and normalization operations respectively, and the transformed features are weighted added to obtain the global features of the distorted image;

[0013] The distorted image and the pseudo reference image are input into the convolutional neural network to extract the convolution features of the distorted image and the pseudo reference image respectively, and the local features of the distorted image are obtained by performing channel-dimensional connection;

[0014] The global features and local features of the distorted image are connected to obtain fused features of the distorted image, and the fused features of the distorted image are input into a linear regression network to obtain a quality evaluation result of the distorted image.

[0015] In some embodiments, downsampling and position encoding the distorted image to obtain a feature sequence includes:

[0016] The distorted image is downsampled four times, the downsampled image is divided into 64 blocks, and each block is position-encoded to obtain a feature sequence.

[0017] In some embodiments, the feature sequence is input into a Transformer-based generative network to obtain multi-layer features of the distorted image, including:

[0018] The Transformer-based generation network includes a first Transformer module, a second Transformer module, and a third Transformer module;

[0019] Input the feature sequence into the first Transformer module to extract the first layer features of the distorted image;

[0020] The first layer features of the distorted image are upsampled twice by bilinear interpolation and then input into the second Transformer module to extract the second layer features of the distorted image;

[0021] The second layer features of the distorted image are upsampled twice by bilinear interpolation and then input into the third Transformer module to extract the third layer features of the distorted image.

[0022] Furthermore, in some embodiments, the feature sequence is input into a Transformer-based generative network to obtain multi-layer features of the distorted image, including:

[0023] Convert the input feature sequence into a tensor and capture the contextual information of the distorted image through a multi-head attention mechanism:

[0024]

[0025] Q=W q X,K=W k X,V=W v X

[0026] Where X represents the input tensor, Q, K, and V represent query, key, and value respectively, and W q 、W k 、W v Represents the weight matrix of query, key, and value respectively, d k represents the dimension of the attention head;

[0027] Each layer of feature Attention (Q, K, V) is fully connected and fused:

[0028] MultiHead(Q,K,V)=Concat(head1,…,head n )W O

[0029] head i =Attention(QW i Q ,KW i K ,VW i V )

[0030] Among them, W i Q 、W i K 、W i V Indicates the i-th head i The linear transformation weight matrix, W P Represents the weight matrix, Concat represents the connection operation of the tensor;

[0031] The output features of the multi-head attention are fed into a feedforward neural network to obtain a high-dimensional feature tensor.

[0032] In some embodiments, multiple layers of features of the distorted image are transformed by convolution and normalization respectively, and the transformed features are weighted added to obtain the global features of the distorted image, including:

[0033] The first layer features of the distorted image, the second layer features of the distorted image, and the third layer features of the distorted image are transformed by convolution and normalization operations respectively, and the global features are obtained by weighted addition;

[0034] F Global =w1×T1+w2×T2+w3×T3

[0035] k1+k2+k3=1

[0036] Among them, T1, T2, and T3 represent the first layer features of the distorted image, the second layer features of the distorted image, and the third layer features of the distorted image, respectively. K1, K2, and K3 represent the weights of the three feature tensors. F Global Represents the global features of the distorted image.

[0037] In some embodiments, the no-reference image quality assessment method further includes discriminating the pseudo-reference image using a discriminant network:

[0038] The pseudo reference image and the distorted image are divided into image blocks, which are then converted into one-dimensional feature tensors through linear flattening. The feature tensors of the pseudo reference image and the distorted image are merged according to the number of channels. The feature tensors are positionally encoded, and classification tags are added to the head of the feature tensor. The feature tensor with classification tags is input into the Transformer module, and finally the discrimination result of the pseudo reference image is output through a multi-layer perceptron.

[0039] Furthermore, the method for obtaining a pseudo reference image is:

[0040] The input distorted image is resized to 256*256 and downsampled fourfold using pooling, resulting in a 64*64 image. The downsampled image is then divided into 64 blocks, each 8*8 in size, and positional encoding is performed on the 64 blocks. The positionally encoded sequence is fed into the first Transformer module to extract global features from the distorted image, resulting in an 8*8 sequence. This sequence is then upsampled twice using bilinear interpolation to a 16*16 sequence, which is then fed into the second Transformer module to obtain a 16*16 sequence. This sequence is then upsampled again using bilinear interpolation to a 32*32 sequence, which is then fed into the third Transformer module to obtain a 32*32 sequence. The feature sequence is then linearly inverse amortized to generate a pseudo reference image of 256*256*3.

[0041] Furthermore, the method for obtaining the local features of the distorted image is:

[0042] Input the distorted image and the pseudo reference image into the convolutional neural network to extract the image convolution features of the two;

[0043] Among them, the convolutional neural network consists of four layers of convolution and pooling layers. The first layer uses 7*7 convolution and maximum pooling, the second layer uses 5*5 convolution and maximum pooling, the third layer uses 3*3 convolution and maximum pooling, and the fourth layer uses 3*3 convolution and maximum pooling.

[0044] Connect the local features of the distorted image with the local features of the pseudo reference image to obtain the local features of the distorted image for quality evaluation:

[0045] F Local =Concat(D Local ,G Local )

[0046] Among them D Local Represents the local features of the distorted image, G Local represents the local features of the pseudo reference image, F Local Represents local features of distorted images for quality assessment.

[0047] In a second aspect, the present invention provides a device for evaluating image quality without reference, comprising:

[0048] The pseudo reference image generation module is configured to: downsample and position-encode the distorted image to obtain a feature sequence; input the feature sequence into a Transformer-based generation network to obtain multi-layer features of the distorted image, wherein the multi-layer features of the distorted image include first-layer features of the distorted image, second-layer features of the distorted image, and third-layer features of the distorted image; and obtain a pseudo reference image by linear inverse amortization of the third-layer features of the distorted image.

[0049] The global feature extraction module is used to transform the multi-layer features of the distorted image through convolution and normalization operations, and perform weighted addition on the transformed features to obtain the global features of the distorted image;

[0050] The local feature extraction module is used to: input the distorted image and the pseudo reference image into the convolutional neural network to extract the convolution features of the distorted image and the convolution features of the pseudo reference image respectively, and perform channel-dimensional connection to obtain the local features of the distorted image;

[0051] The quality score regression module is used to connect the global features and local features of the distorted image to obtain a fused feature of the distorted image, and input the fused feature of the distorted image into a linear regression network to obtain a quality evaluation result of the distorted image.

[0052] In some embodiments, the no-reference image quality assessment device further includes a pseudo-reference image discrimination module, which is used to: divide the pseudo-reference image and the distorted image into image blocks, and then convert them into one-dimensional feature tensors through linear amortization, merge the feature tensors of the pseudo-reference image and the distorted image according to the number of channels, perform position encoding on the feature tensors, and add a classification mark to the head of the feature tensor, input the feature tensor with the classification mark into the Transformer module, and finally output the discrimination result of the pseudo-reference image through a multi-layer perceptron.

[0053] In a third aspect, the present invention provides a device including a processor and a storage medium;

[0054] The storage medium is used to store instructions;

[0055] The processor is configured to operate according to the instructions to execute the steps of the method of the first aspect.

[0056] In a fourth aspect, the present invention proposes a storage medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when executed by a processor.

[0057] The above technical means can achieve the following advantages in image quality evaluation:

[0058] This paper proposes a no-reference image quality assessment method that combines the Transformer and convolutional networks. By generating high-quality pseudo-reference images based on the Transformer generative network, this method assists in no-reference quality assessment of distorted images. Compared with conventional no-reference image quality assessment methods, this method better simulates human perception of image quality. By using the Transformer network to extract global features from the distorted image and fusing them with local features, this method overcomes the limitation of conventional no-reference image quality assessment networks in their single feature utilization, enabling better representation of image features and achieving higher image quality assessment accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Schematic diagram of the process of a no-reference image quality assessment method according to an embodiment of the present invention;

[0060] Figure 2 This is a schematic diagram of the overall network framework in an embodiment of the present invention;

[0061] Figure 3 is a schematic diagram of a device for evaluating image quality without reference according to an embodiment of the present invention;

[0062] In the figure, 1 is the pseudo reference image generation module, 2 is the pseudo reference image discrimination module, 3 is the global feature extraction module, 4 is the local feature extraction module, and 5 is the quality score regression module. DETAILED DESCRIPTION

[0063] The present invention will be further described below in conjunction with the accompanying drawings and examples. The following examples are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0064] In the description of the present invention, "several" means more than one, "plurality" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0065] In the description of the present invention, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the exemplary expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0066] Example 1

[0067] Currently, existing image quality assessment models built using traditional neural networks often focus only on local features of distorted images and do not pay sufficient attention to the global features of the image. To address this issue, this embodiment proposes a no-reference image quality assessment method that can solve this problem by focusing on global features.

[0068] A no-reference image quality assessment method, comprising:

[0069] Downsample and position encode the distorted image to obtain a feature sequence;

[0070] Inputting the feature sequence into a Transformer-based generative network to obtain multi-layer features of the distorted image, wherein the multi-layer features of the distorted image include first-layer features of the distorted image, second-layer features of the distorted image, and third-layer features of the distorted image;

[0071] The third layer features of the distorted image are linearly inverse amortized to obtain a pseudo reference image;

[0072] The multi-layer features of the distorted image are transformed through convolution and normalization operations respectively, and the transformed features are weighted added to obtain the global features of the distorted image;

[0073] The distorted image and the pseudo reference image are input into the convolutional neural network to extract the convolution features of the distorted image and the pseudo reference image respectively, and the local features of the distorted image are obtained by performing channel-dimensional connection;

[0074] The global features and local features of the distorted image are connected to obtain fused features of the distorted image, and the fused features of the distorted image are input into a linear regression network to obtain a quality evaluation result of the distorted image.

[0075] In some embodiments, downsampling and position encoding the distorted image to obtain a feature sequence includes:

[0076] The distorted image is downsampled four times, the downsampled image is divided into 64 blocks, and each block is position-encoded to obtain a feature sequence.

[0077] In some embodiments, the feature sequence is input into a Transformer-based generative network to obtain multi-layer features of the distorted image, including:

[0078] The Transformer-based generation network includes a first Transformer module, a second Transformer module, and a third Transformer module;

[0079] Input the feature sequence into the first Transformer module to extract the first layer features of the distorted image;

[0080] The first layer features of the distorted image are upsampled twice by bilinear interpolation and then input into the second Transformer module to extract the second layer features of the distorted image;

[0081] The second layer features of the distorted image are upsampled twice by bilinear interpolation and then input into the third Transformer module to extract the third layer features of the distorted image.

[0082] Furthermore, in some embodiments, the feature sequence is input into a Transformer-based generative network to obtain multi-layer features of the distorted image, including:

[0083] Convert the input feature sequence into a tensor and capture the contextual information of the distorted image through a multi-head attention mechanism:

[0084]

[0085] Q=W q X,K=W k X,V=W v X

[0086] Where X represents the input tensor, Q, K, and V represent query, key, and value respectively, and W q 、W k 、W v Represents the weight matrix of query, key, and value respectively, d k represents the dimension of the attention head;

[0087] Each layer of feature Attention (Q, K, V) is fully connected and fused:

[0088] MultiHead(Q,K,V)=Concat(head1,…,head n )W O

[0089] head i =Attention(QW i Q ,KW i K ,VW i V )

[0090] Among them, W i Q 、W i K 、W i V Indicates the i-th head i The linear transformation weight matrix, W O Represents the weight matrix, Concat represents the connection operation of the tensor;

[0091] The output features of the multi-head attention are fed into a feedforward neural network to obtain a high-dimensional feature tensor.

[0092] In some embodiments, multiple layers of features of the distorted image are transformed by convolution and normalization respectively, and the transformed features are weighted added to obtain the global features of the distorted image, including:

[0093] The first layer features of the distorted image, the second layer features of the distorted image, and the third layer features of the distorted image are transformed by convolution and normalization operations respectively, and the global features are obtained by weighted addition;

[0094] F Global =w1×T1+w2×T2+w3×T3

[0095] k1+k2+k3=1

[0096] Among them, T1, T2, and T3 represent the first layer features of the distorted image, the second layer features of the distorted image, and the third layer features of the distorted image, respectively. K1, K2, and K3 represent the weights of the three feature tensors. FGlobal Represents the global features of the distorted image.

[0097] In some embodiments, the no-reference image quality assessment method further includes discriminating the pseudo-reference image using a discriminant network:

[0098] The pseudo reference image and the distorted image are divided into image blocks, which are then converted into one-dimensional feature tensors through linear flattening. The feature tensors of the pseudo reference image and the distorted image are merged according to the number of channels. The feature tensors are positionally encoded, and classification tags are added to the head of the feature tensor. The feature tensor with classification tags is input into the Transformer module, and finally the discrimination result of the pseudo reference image is output through a multi-layer perceptron.

[0099] Furthermore, the method for obtaining a pseudo reference image is:

[0100] The input distorted image is resized to 256*256 and downsampled fourfold using pooling, resulting in a 64*64 image. The downsampled image is then divided into 64 blocks, each 8*8 in size, and positional encoding is performed on the 64 blocks. The positionally encoded sequence is fed into the first Transformer module to extract global features from the distorted image, resulting in an 8*8 sequence. This sequence is then upsampled twice using bilinear interpolation to a 16*16 sequence, which is then fed into the second Transformer module to obtain a 16*16 sequence. This sequence is then upsampled again using bilinear interpolation to a 32*32 sequence, which is then fed into the third Transformer module to obtain a 32*32 sequence. The feature sequence is then linearly inverse amortized to generate a pseudo reference image of 256*256*3.

[0101] Furthermore, the method for obtaining the local features of the distorted image is:

[0102] Input the distorted image and the pseudo reference image into the convolutional neural network to extract the image convolution features of the two;

[0103] Among them, the convolutional neural network consists of four layers of convolution and pooling layers. The first layer uses 7*7 convolution and maximum pooling, the second layer uses 5*5 convolution and maximum pooling, the third layer uses 3*3 convolution and maximum pooling, and the fourth layer uses 3*3 convolution and maximum pooling.

[0104] Connect the local features of the distorted image with the local features of the pseudo reference image to obtain the local features of the distorted image for quality evaluation:

[0105] F Local =Concat(D Local ,G Local )

[0106] Among them D Local Represents the local features of the distorted image, G Local represents the local features of the pseudo reference image, F Local Represents local features of distorted images for quality assessment.

[0107] like Figure 2 As shown, the method of the present invention specifically comprises the following steps:

[0108] Step A01: Obtain a distorted image D and adjust the size of the distorted image to a uniform size of 256*256 pixels;

[0109] Step A02: Downsample the adjusted distorted image by four times. The downsampled image size is 64*64 pixels. Divide it into 64 blocks of 8*8 pixels. Position-encode each block to obtain a feature tensor of size (64, 8*8*3, C), where C=384 is the dimension of the feature tensor.

[0110] Step B01: Input the feature tensor of size (64, 8*8*3, C) into the first Transformer module for feature extraction, and the output is T1(64, 8*8*3, C).

[0111] Step B02: T1 is upsampled twice by bilinear interpolation to obtain a feature tensor of size (64, 16*16, C / 4).

[0112] Step B03: Input the feature tensor of size (64, 16*16, C / 4) into the second Transformer module for feature extraction, and the output is T2(64, 16*16, C / 4).

[0113] Step B04: T2 is upsampled twice by bilinear interpolation to obtain a feature tensor of size (64, 32*32, C / 16).

[0114] Step B05: Input the feature tensor of size (64, 32*32, C / 16) into the third Transformer module for feature extraction, and the output is T3(64, 32*32, C / 16).

[0115] Step B06: perform linear inverse amortization on T3 and combine them in sequence order to obtain a pseudo reference image G with a size of (256*256,3).

[0116] Step C01: Divide the pseudo reference image G and the distorted image D, and convert them into a one-dimensional feature tensor G of size 32*32*3 by linear amortization.F and D F , G F and D F Merge the number of channels and add classification marks to the header.

[0117] Step C02: Input the feature tensor with the classification label into the Transformer module, input the output of the classification label into the fully connected layer, and then judge the quality of the pseudo reference image based on the value output by the activation function.

[0118] Step D01: Extract global image features. Pass 1*1 convolution and normalization layers on T1, T2, and T3 in step B respectively to obtain three feature tensors of size 256*256.

[0119] Step D02: Perform weighted addition operation on three feature tensors of size 256*256 to obtain the image global feature F of size 65536. Global .

[0120] Step E01: Input the distorted image into the convolutional neural network to obtain the local feature D of the distorted image with a size of 32768. Local ,

[0121] Step E02: Input the pseudo reference image into the convolutional neural network to obtain the pseudo reference image convolution feature G with a size of 32768. Local ,

[0122] Step E03, D Local and G Local Connect and get the image local feature F of size 65536 Local .

[0123] Step F01, F Global and F Local Connect and get the fusion feature F1 with a size of 131072.

[0124] Step F02: Pass F1 through the fully connected layer to output the feature tensor F2 with a size of 32768.

[0125] Step F03: pass F2 through the fully connected layer to output the feature tensor F3 with a size of 2048.

[0126] Step F04: pass F3 through the fully connected layer to output the feature tensor F4 with a size of 512.

[0127] Step F04: pass F4 through the fully connected layer to output the quality evaluation score Q of the distorted image D.

[0128] Example 2:

[0129] A non-reference image quality assessment device, comprising:

[0130] The pseudo reference image generation module is configured to: downsample and position-encode the distorted image to obtain a feature sequence; input the feature sequence into a Transformer-based generation network to obtain multi-layer features of the distorted image, wherein the multi-layer features of the distorted image include first-layer features of the distorted image, second-layer features of the distorted image, and third-layer features of the distorted image; and obtain a pseudo reference image by linear inverse amortization of the third-layer features of the distorted image.

[0131] The global feature extraction module is used to transform the multi-layer features of the distorted image through convolution and normalization operations, and perform weighted addition on the transformed features to obtain the global features of the distorted image;

[0132] The local feature extraction module is used to: input the distorted image and the pseudo reference image into the convolutional neural network to extract the convolution features of the distorted image and the convolution features of the pseudo reference image respectively, and perform channel-dimensional connection to obtain the local features of the distorted image;

[0133] The quality score regression module is used to connect the global features and local features of the distorted image to obtain a fused feature of the distorted image, and input the fused feature of the distorted image into a linear regression network to obtain a quality evaluation result of the distorted image.

[0134] In some embodiments, the no-reference image quality assessment device further includes a pseudo-reference image discrimination module, which is used to: divide the pseudo-reference image and the distorted image into image blocks, and then convert them into one-dimensional feature tensors through linear amortization, merge the feature tensors of the pseudo-reference image and the distorted image according to the number of channels, perform position encoding on the feature tensors, and add a classification mark to the head of the feature tensor, input the feature tensor with the classification mark into the Transformer module, and finally output the discrimination result of the pseudo-reference image through a multi-layer perceptron.

[0135] In some embodiments, a no-reference image quality assessment device combining global Transformer features and local convolution features, such as Figure 3 As shown, it mainly includes: a pseudo reference image generation module 1, a pseudo reference image discrimination module 2, a global feature extraction module 3, a local feature extraction module 4, and a quality score regression module 5.

[0136] The pseudo reference image generation module mainly inputs the distorted image into the Transformer-based generation network to obtain the pseudo reference image. The specific method is as follows:

[0137] The distorted image is uniformly resized to 256*256 pixels and downsampled four times. The image size after downsampling is 64*64, divided into blocks of the same size, and position encoding is performed on each block. The block is input into the first Transformer module, and the output is upsampled twice and input into the second Transformer module. The output is upsampled twice and input into the third Transformer module to obtain deeper global features of the distorted image, which are linearly inverse-amortized to obtain the pseudo-reference image.

[0138] The pseudo reference image discrimination module mainly judges the quality of pseudo reference images. The specific method is as follows:

[0139] The pseudo reference image and the distorted image are divided and converted into a one-dimensional feature tensor through linear flattening. The feature tensor of the pseudo reference image and the feature tensor of the distorted image are merged according to the number of channels, and a classification mark is added to the head. The feature tensor with the added classification mark is input into the Transformer module, and the output of the classification mark is input into the fully connected layer. The quality of the pseudo reference image is then judged by the value output by the activation function.

[0140] The global feature extraction module extracts the features processed by multiple Transformer modules in the pseudo reference image generation module, and obtains the global features of the image through 1*1 convolution and normalization, and then weighted addition.

[0141] The local feature extraction module mainly extracts the local features of the distorted image and the pseudo reference image through a convolutional neural network, and performs channel-level connections to obtain the local features of the image.

[0142] The quality score regression module mainly connects the global features and local features to obtain the fusion features of the image, and finally obtains the quality evaluation score of the distorted image through a linear regression network.

[0143] Example 3:

[0144] The present invention also proposes a device comprising a processor and a storage medium; wherein the storage medium is used to store instructions; and the processor is used to operate according to the instructions to execute the steps of the no-reference image quality assessment method in Example 1.

[0145] Example 4:

[0146] The present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the method for evaluating quality of a non-reference image in embodiment 1.

[0147] Compared with the existing technology, the present invention makes better use of the global features in the image when constructing the image quality assessment model, making up for the shortcomings of traditional reference-free image quality assessment methods that focus on local features of the image and ignore global features. It fuses local features and global features, enhances the expression of image context information, and improves the accuracy of the model.

[0148] Compared with full-reference image quality evaluation and semi-reference image quality evaluation, the present invention does not need to rely on the original image of the distorted image and is more applicable. Compared with traditional no-reference image quality evaluation, the present invention can generate a pseudo-reference image based on the distorted image to assist the network in judging the quality of the distorted image and obtain better quality evaluation results.

[0149] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0150] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0153] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for evaluating image quality without reference, characterized in that: include: Downsample and position encode the distorted image to obtain a feature sequence; Inputting the feature sequence into a Transformer-based generative network to obtain multi-layer features of the distorted image, wherein the multi-layer features of the distorted image include first-layer features of the distorted image, second-layer features of the distorted image, and third-layer features of the distorted image; specifically comprising: the Transformer-based generative network includes a first Transformer module, a second Transformer module, and a third Transformer module; inputting the feature sequence into the first Transformer module to extract the first-layer features of the distorted image; the first-layer features of the distorted image are upsampled twice by bilinear interpolation and then input into the second Transformer module to extract the second-layer features of the distorted image; the second-layer features of the distorted image are upsampled twice by bilinear interpolation and then input into the third Transformer module to extract the third-layer features of the distorted image; The third layer features of the distorted image are linearly inverse amortized to obtain a pseudo reference image; The multi-layer features of the distorted image are transformed through convolution and normalization operations respectively, and the transformed features are weighted added to obtain the global features of the distorted image; The distorted image and the pseudo reference image are input into the convolutional neural network to extract the convolution features of the distorted image and the pseudo reference image respectively, and the local features of the distorted image are obtained by performing channel-dimensional connection; The global features and local features of the distorted image are connected to obtain fused features of the distorted image, and the fused features of the distorted image are input into a linear regression network to obtain a quality evaluation result of the distorted image.

2. The no-reference image quality assessment method according to claim 1, wherein: The distorted image is downsampled and position-encoded to obtain a feature sequence, including: The distorted image is downsampled four times, the downsampled image is divided into 64 blocks, and each block is position-encoded to obtain a feature sequence.

3. The no-reference image quality assessment method according to claim 1, wherein: The feature sequence is input into the Transformer-based generative network to obtain multi-layer features of the distorted image, including: Convert the input feature sequence into a tensor and capture the contextual information of the distorted image through a multi-head attention mechanism: Q=W q X,K=W k X,V=W v X Where X represents the input tensor, Q, K, and V represent query, key, and value respectively, and W q 、W k 、W v Represents the weight matrix of query, key, and value respectively, d k represents the dimension of the attention head; Each layer of feature Attention (Q, K, V) is fully connected and fused: MultiHead(Q,K,V)=Concat(head1,…,head n )W O in, Indicates the i-th head i The linear transformation weight matrix, W O Represents the weight matrix, Concat represents the connection operation of the tensor; The output features of the multi-head attention are fed into a feedforward neural network to obtain a high-dimensional feature tensor.

4. The no-reference image quality assessment method according to claim 1, wherein: The multi-layer features of the distorted image are transformed through convolution and normalization operations respectively, and the transformed features are weighted added to obtain the global features of the distorted image, including: The first layer features of the distorted image, the second layer features of the distorted image, and the third layer features of the distorted image are transformed by convolution and normalization operations respectively, and the global features are obtained by weighted addition; F Global =w1×T1+w2×T2+w3×T3 k1+k2+k3=1 Among them, T1, T2, and T3 represent the first layer features of the distorted image, the second layer features of the distorted image, and the third layer features of the distorted image, respectively. K1, K2, and K3 represent the weights of the three feature tensors. F Global Represents the global features of the distorted image.

5. The no-reference image quality assessment method according to claim 1, wherein: It also includes discriminating the pseudo reference image through the discriminant network: The pseudo reference image and the distorted image are divided into image blocks, which are then converted into one-dimensional feature tensors through linear flattening. The feature tensors of the pseudo reference image and the distorted image are merged according to the number of channels. The feature tensors are positionally encoded, and classification tags are added to the head of the feature tensor. The feature tensor with classification tags is input into the Transformer module, and finally the discrimination result of the pseudo reference image is output through a multi-layer perceptron.

6. A non-reference image quality assessment device, characterized in that: include: The pseudo reference image generation module is used to: downsample and position encode the distorted image to obtain a feature sequence; input the feature sequence into a Transformer-based generation network to obtain multi-layer features of the distorted image, wherein the multi-layer features of the distorted image include first-layer features of the distorted image, second-layer features of the distorted image, and third-layer features of the distorted image; the third-layer features of the distorted image are linearly inversely amortized to obtain a pseudo reference image; input the feature sequence into a Transformer-based generation network to obtain multi-layer features of the distorted image, specifically comprising: the Transformer-based generation network includes a first Transformer module, a second Transformer module, and a third Transformer module; input the feature sequence into the first Transformer module to extract the first-layer features of the distorted image; the first-layer features of the distorted image are upsampled twice by bilinear interpolation and then input into the second Transformer module to extract the second-layer features of the distorted image; the second-layer features of the distorted image are upsampled twice by bilinear interpolation and then input into the third Transformer module to extract the third-layer features of the distorted image; The global feature extraction module is used to transform the multi-layer features of the distorted image through convolution and normalization operations, and perform weighted addition on the transformed features to obtain the global features of the distorted image; The local feature extraction module is used to: input the distorted image and the pseudo reference image into the convolutional neural network to extract the convolution features of the distorted image and the convolution features of the pseudo reference image respectively, and perform channel-dimensional connection to obtain the local features of the distorted image; The quality score regression module is used to connect the global features and local features of the distorted image to obtain a fused feature of the distorted image, and input the fused feature of the distorted image into a linear regression network to obtain a quality evaluation result of the distorted image.

7. The non-reference image quality assessment device according to claim 6, wherein: It also includes a pseudo-reference image discrimination module, which is used to: divide the pseudo-reference image and the distorted image into image blocks, then convert them into one-dimensional feature tensors through linear amortization, merge the feature tensors of the pseudo-reference image and the distorted image according to the number of channels, perform position encoding on the feature tensors, add classification tags to the head of the feature tensor, input the feature tensor with the classification tags into the Transformer module, and finally output the discrimination result of the pseudo-reference image through a multi-layer perceptron.

8. A computer device, characterized in that: including processors and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 5.

9. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images