No-reference image quality evaluation method, device and equipment

Abstract

The invention relates to a no-reference image quality evaluation method, device and equipment. The method comprises the steps of obtaining a to-be-evaluated image; the to-be-evaluated image is input into a feature extraction structure of the target quality evaluation model, target features output by the feature extraction structure are obtained, and the feature extraction structure comprises a global convolution layer and a down-sampling multiplication layer; inputting the target features into a feature mapping structure of the target quality evaluation model to obtain a quality evaluation result output by the feature mapping structure; by adopting the method, the real-time performance of non-reference image quality evaluation can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device and equipment for evaluating image quality without reference. Background Art

[0002] In order to alleviate the growing pressure on transmission and storage of images and videos, image and video processing algorithms are committed to maximizing the compression of images and videos while maintaining or improving the perceived quality of images. No-reference image quality evaluation methods can provide image quality reference indicators for the above-mentioned image and video processing algorithms and guide their optimization strategies.

[0003] The no-reference image quality evaluation method in the related art is to capture various image quality information under different image contents through a convolutional neural network (CNN) model.

[0004] However, in the above-mentioned method of using a convolutional neural network model, in order to achieve better generalization ability, a deeper and wider feature extraction structure is generally required in the model, which has a high computational complexity and results in poor real-time performance of no-reference image quality evaluation. Summary of the invention

[0005] Based on this, it is necessary to provide a reference-free image quality evaluation method, device and equipment that improves real-time performance in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for evaluating image quality without reference, the method comprising:

[0007] Obtain the image to be evaluated;

[0008] The image to be evaluated is input into the feature extraction structure of the target quality evaluation model to obtain the target features output by the feature extraction structure, wherein the feature extraction structure includes a global convolution layer and a downsampling multiplication layer;

[0009] The target features are input into the feature mapping structure of the target quality evaluation model to obtain the quality evaluation result output by the feature mapping structure.

[0010] In a second aspect, the present application also provides a non-reference image quality assessment device, comprising:

[0011] An acquisition module, used for acquiring an image to be evaluated;

[0012] An extraction module, used for inputting the image to be evaluated into the feature extraction structure of the target quality evaluation model to obtain the target features output by the feature extraction structure, wherein the feature extraction structure includes a global convolution layer and a downsampling multiplication layer;

[0013] The mapping module is used to input the target features into the feature mapping structure of the target quality evaluation model to obtain the quality evaluation result output by the feature mapping structure.

[0014] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0015] The above-mentioned method, device and equipment for evaluating quality of reference-free images obtain an image to be evaluated; input the image to be evaluated into the feature extraction structure of the target quality evaluation model to obtain the target feature output by the feature extraction structure, wherein the feature extraction structure includes a global convolution layer and a downsampling multiplication layer; input the target feature into the feature mapping structure of the target quality evaluation model to obtain the quality evaluation result output by the feature mapping structure; in this way, the feature extraction structure includes a downsampling multiplication layer, and the nonlinear fitting ability of the feature extraction structure is improved by multiplication processing, which can implicitly improve the network dimension and increase the characteristics of the network expression ability, while maintaining the prediction and reasoning performance of the network, maintaining a small model size and model operation amount, reducing the computational complexity, and improving the real-time performance of the quality evaluation of reference-free images. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] FIG1 is a diagram of an application environment of a method for evaluating image quality without reference in one embodiment;

[0018] Figure 2 is a flow chart of a method for evaluating quality of a reference-free image in one embodiment;

[0019] Figure 3 is a schematic diagram of the structure of a target quality evaluation model in one embodiment;

[0020] Figure 4 is a schematic diagram of the structure of a multiplication unit in one embodiment;

[0021] Figure 5 is a structural schematic diagram of a feature mapping structure in an embodiment;

[0022] Figure 6 A schematic diagram of a process for obtaining a target quality evaluation model in one embodiment;

[0023] Figure 7is a flow chart of a method for evaluating quality of a reference-free image in another embodiment;

[0024] FIG. 8 is a diagram of the internal layers of a computer device in one embodiment. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0026] The reference-free image quality assessment method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0027] The server 104 obtains an image to be evaluated; inputs the image to be evaluated into a feature extraction structure of a target quality evaluation model to obtain target features output by the feature extraction structure, wherein the feature extraction structure includes a global convolution layer and a downsampling multiplication layer; inputs the target features into a feature mapping structure of the target quality evaluation model to obtain a quality evaluation result output by the feature mapping structure.

[0028] In an exemplary embodiment, Figure 2 As shown, a method for evaluating image quality without reference is provided, and the method is described by taking the application of the method to the server 104 in FIG. 1 as an example, including the following steps 202 to 206. Among them:

[0029] Step 202: Obtain an image to be evaluated.

[0030] The image to be evaluated refers to an image used to evaluate the image quality. For example, the image to be evaluated may be a frame of a compressed video, used to evaluate the image quality corresponding to the compression processing algorithm corresponding to the video; the image to be evaluated may also be the image processing result of a certain image processing or image compression algorithm.

[0031] Step 204: Input the image to be evaluated into the feature extraction structure of the target quality evaluation model to obtain the target features output by the feature extraction structure. The feature extraction structure includes a global convolution layer and a downsampling multiplication layer.

[0032] Among them, the global convolution layer is used to completely extract the underlying features of the input image, which is called the global features of the image in this embodiment. The downsampling multiplication layer is a quality and content extractor of the image, which downsamples and multiplies the global features of the image extracted by the global convolution layer. The nonlinear fitting ability of the feature extraction structure is improved through multiplication processing, which can invisibly improve the network dimension and increase the network expression ability, but does not increase the actual dimension of the network, and can maintain a small network model scale and amount of calculation.

[0033] Step 206: Input the target features into the feature mapping structure of the target quality assessment model to obtain the quality assessment result output by the feature mapping structure.

[0034] Among them, the feature mapping structure includes a pooling layer, a fully connected layer, a normalization layer and an activation function layer. By combining the above layers, the target features obtained by the downsampling multiplication layer are mapped to the final quality evaluation result.

[0035] For example, the quality evaluation result may be an image quality score. For example, the quality evaluation result may adopt a Mean Opinion Score (MOS) system.

[0036] Among them, the quality evaluation results can be used to guide the optimization of image processing algorithms, automatically detect and repair quality problems in images, and can also be used to provide real-time image quality indicators for image and video processing algorithms, helping to reduce the storage and transmission bandwidth costs of cloud service providers in subsequent online video and real-time communication scenarios.

[0037] The method for evaluating quality of a reference-free image provided in the above-mentioned embodiment obtains an image to be evaluated; inputs the image to be evaluated into a feature extraction structure of a target quality evaluation model to obtain a target feature output by the feature extraction structure, wherein the feature extraction structure includes a global convolution layer and a downsampling multiplication layer; inputs the target feature into a feature mapping structure of the target quality evaluation model to obtain a quality evaluation result output by the feature mapping structure; in this way, the feature extraction structure includes a downsampling multiplication layer, and the nonlinear fitting ability of the feature extraction structure is improved by multiplication processing, which can implicitly improve the network dimension and increase the characteristics of the network's expression ability, while maintaining the prediction and reasoning performance of the network, maintaining a small model size and model operation amount, reducing the computational complexity, and improving the real-time performance of the reference-free image quality evaluation.

[0038] The above-mentioned no-reference image quality evaluation method can be applied to scenarios such as image quality analysis before CDN (Content Delivery Network) streaming, codec optimization evaluation in on-demand or live broadcast scenarios, and detection of flowery or solid-color screens in monitoring scenarios.

[0039] In an exemplary embodiment, in the non-reference image quality assessment method provided by this embodiment, the feature extraction structure 300 in the target quality assessment model includes a global convolution layer 310 and multiple downsampling multiplication layers 320. Please refer to Figure 3 ,The target quality assessment model also includes a feature mapping structure 400, denoted as projector in the figure.

[0040] In a possible implementation, the output data dimensions of the multiple downsampling multiplication layers are increased successively; in this way, more complex feature representation requirements can be gradually adapted through the multiple downsampling multiplication layers.

[0041] In this embodiment, the process of inputting the image to be evaluated into the feature extraction structure of the target quality evaluation model and obtaining the target features output by the feature extraction structure includes steps A2 to A6, wherein:

[0042] Step A2: input the image to be evaluated into the global convolution layer to obtain the global features of the image output by the global convolution layer.

[0043] Among them, the global convolution layer is a non-grouped convolution module, which is used to completely extract the underlying features of the image.

[0044] For example, the global convolution layer is a convolution layer with a convolution kernel of 3×3, 32 channels, a stride of 2, a padding of 1, and a group number of 1. The global convolution layer can be expressed as Conv2d(3,32,3,2,1,1), such as Figure 3 shown.

[0045] Step A4: input the global image features to the first downsampling multiplication layer 320 among the multiple downsampling multiplication layers to obtain intermediate features.

[0046] In one possible implementation, please refer to Figure 3 , the downsampling multiplication layer includes a downsampling grouped convolution sublayer and a multiplication unit StarBlock; where, please refer to Figure 4 , a schematic diagram of the network architecture of the multiplication unit is given as an example; Figure 5 As shown, the multiplication unit StarBlock includes a first grouped convolution sublayer 321, a second grouped convolution sublayer 322, a grouped multiplication sublayer 323, a third grouped convolution sublayer 324, a fourth grouped convolution sublayer 325 and a residual sublayer 326.

[0047] In this embodiment, step A4 further includes steps B2 to B14, wherein:

[0048] Step B2: input the global features of the image into the downsampling grouped convolution sublayer to obtain the intermediate downsampling features output by the downsampling grouped convolution sublayer.

[0049] For example, the downsampled grouped convolution sublayer is a grouped convolution layer with a convolution kernel of 3×3, 32 input channels, 64 output channels, a stride of 2, a padding of 1, and a grouping number of 32. The downsampled grouped convolution sublayer can be represented as Conv2d(32,64,3,2,1,32), as shown in Figure 4 shown.

[0050] Step B4, input the intermediate down-sampled features into the first grouped convolution sublayer to obtain the intermediate first grouped features output by the first grouped convolution sublayer, and the number of groups of the first grouped convolution sublayer is the same as the number of input channels of the first grouped convolution sublayer.

[0051] Exemplarily, the first grouped convolution sublayer is a deep convolution layer with a convolution kernel of 7×7, the number of input channels being the same as the number of output channels of the downsampled grouped convolution sublayer, the number of output channels of the first grouped convolution sublayer being the same as its own number of input channels, the stride being 2, the padding being 3, and the number of groups being the same as the number of input channels. The first grouped convolution sublayer can be represented as Conv2d(c,c,7,1,3,c), where c represents the number of output channels of the downsampled grouped convolution sublayer in the current downsampled multiplication layer, such as Figure 4 shown.

[0052] Step B6, input the intermediate first grouping feature into the second grouping convolution sublayer to obtain the intermediate second grouping feature output by the second grouping convolution sublayer, and the number of groups in the second grouping convolution sublayer is 2.

[0053] Exemplarily, the second grouped convolution sublayer is a grouped convolution layer with a convolution kernel of 1×1, the number of input channels is the number of output channels of the first grouped convolution sublayer, the number of output channels of the second grouped convolution sublayer is 6 times the number of input channels, the step size is 1, the padding is 0, and the number of groups is 2. The second grouped convolution sublayer can be expressed as Conv2d(c,c*6,1,1,0,2), such as Figure 4 shown.

[0054] Step B8, input the intermediate second grouping feature into the group multiplication sublayer to obtain the intermediate product feature output by the group multiplication sublayer, the intermediate product feature is based on dividing the intermediate second grouping feature into a first segmentation feature and a second segmentation feature, activating the first segmentation feature to obtain a first activation feature, and multiplying the first activation feature by the second segmentation feature.

[0055] The second intermediate grouping feature is divided into two parts, one of which is activated and then multiplied with the other to obtain the output data of the group multiplication sublayer. The group multiplication sublayer maps the first segmentation feature and the second segmentation feature from the relative feature space to a more complex latent space, which can capture more potential information while maintaining the feature dimension, and improve the feature fitting ability and expression ability of the feature extraction layer.

[0056] For example, please refer to Figure 4 , according to the channel dimension, the intermediate second grouping feature is divided into the first segmentation feature split[0] and the second segmentation feature split[1], the first segmentation feature split[0] is activated by the ReLU activation function to obtain the first activation feature, the first activation feature and the second segmentation feature are passed through a multiplication layer to obtain the intermediate product feature.

[0057] Step B10, input the intermediate product feature into the third grouped convolution sublayer to obtain the intermediate third grouped feature output by the third grouped convolution sublayer, and the number of groups of the third grouped convolution sublayer is the same as the number of input channels of the third grouped convolution sublayer.

[0058] Exemplarily, the third grouped convolution sublayer is a convolution kernel of 1×1, the number of input channels is half of the number of output channels of the second grouped convolution sublayer, the number of output channels of the third grouped convolution sublayer is the number of output channels of the downsampling grouped convolution sublayer of the current downsampling multiplication layer, that is, the convolution layer is restored to c, the step size is 1, the padding is 0, and the number of groups is the number of output channels. The third grouped convolution sublayer can be expressed as Conv2d(c*3,c,1,1,0,c), such as Figure 4 shown.

[0059] Step B12, input the intermediate third grouping features into the fourth grouping convolution sublayer to obtain the intermediate fourth grouping features output by the fourth grouping convolution sublayer, and the number of groups of the fourth grouping convolution sublayer is the same as the number of input channels of the fourth grouping convolution sublayer.

[0060] Exemplarily, the fourth grouped convolution sublayer is a convolution layer with a convolution kernel of 7×7, the number of input channels is the number of output channels of the third grouped convolution sublayer, the number of output channels of the third grouped convolution sublayer is the same as its own input channel number, the stride is 1, the padding is 3, and the number of groups is the number of output channels. The fourth grouped convolution sublayer can be expressed as Conv2d(c,c,7,1,3,c).

[0061] Step B14, input the fourth grouping feature and the intermediate down-sampling feature into the residual sub-layer to obtain the intermediate feature output by the residual sub-layer.

[0062] Among them, the fourth group feature output by the fourth group convolution sublayer and the intermediate downsampling feature output by the downsampling group convolution sublayer are added through the residual sublayer, so that the overall output of the downsampling multiplication layer can retain the information of the input part.

[0063] In this embodiment, each convolution layer in the downsampling multiplication layer adopts the form of grouped convolution. In the convolution operation, the input data and the convolution kernel are divided into multiple groups, each group performs a convolution operation independently, and finally the output of each group is merged. This can reduce the amount of computation for each convolution operation and increase the flexibility of feature learning.

[0064] Step A6: input the intermediate feature into the second downsampling multiplication layer among the multiple downsampling multiplication layers until the target feature is obtained, where the target feature is the feature extraction result output by the last downsampling multiplication layer among the multiple downsampling multiplication layers.

[0065] Exemplary, reference Figure 4 The feature extraction structure includes four downsampling multiplication layers. The corresponding c in the first downsampling multiplication layer is 64, the corresponding c in the second downsampling multiplication layer is 128, the corresponding c in the third downsampling multiplication layer is 256, and the corresponding c in the fourth downsampling multiplication layer is 512. The feature dimension of each downsampling multiplication layer gradually increases, and the high-order features in the image are gradually learned to improve the fitting ability of the model.

[0066] In an exemplary embodiment, please refer to Figure 5 The feature mapping structure 400 in the target quality assessment model used in the provided no-reference image quality assessment method includes a pooling layer 401, a first fully connected layer 402, a first normalization layer 403, an activation layer 404, a second fully connected layer 405, a second normalization layer 406 and a third fully connected layer 407.

[0067] In this embodiment, the process of inputting the target features into the feature mapping structure of the target quality assessment model and obtaining the quality assessment result output by the feature mapping structure includes steps C2 to C14, wherein:

[0068] Step C2: input the target features into the pooling layer to obtain the pooled data output by the pooling layer.

[0069] For example, the pooling layer uses a two-bit adaptive mean pooling layer, the pooling window size is (1,1), and the pooling layer can be expressed as Avgpool2d ((1,1)), as shown in Figure 5 shown.

[0070] Step C4: input the pooled data into the first fully connected layer to obtain the first fully connected data output by the first fully connected layer.

[0071] For example, the number of input nodes and the number of output nodes of the first fully connected layer are both 512. The first fully connected layer can be represented as Linear (512, 512), such as Figure 5 shown.

[0072] Step C6: input the first fully connected data into the first normalization layer to obtain the first normalized data output by the first normalization layer.

[0073] Exemplarily, the first normalization layer is a one-dimensional batch normalization layer with a length of 512. The first normalization layer can be represented as BatchNorm1d(512), such as Figure 5 shown.

[0074] Step C8, input the first normalized data into the activation layer to obtain the activation data output by the activation layer.

[0075] Exemplarily, the activation layer adopts the ReLU activation function.

[0076] Step C10: input the activation data into the second fully connected layer to obtain second fully connected data input by the second fully connected layer.

[0077] For example, the number of input nodes of the second fully connected layer is 512, and the number of output nodes is 128. The second fully connected layer can be represented as Linear (512, 128), such as Figure 5 shown.

[0078] Step C12: input the second fully connected data into the second normalization layer to obtain the second normalized data output by the second normalization layer.

[0079] Exemplarily, the second normalization layer is a one-dimensional batch normalization layer with a length of 128. The second normalization layer can be represented as BatchNorm1d(128), such as Figure 5 shown.

[0080] Step C14, inputting the second normalized data into the third fully connected layer to obtain the quality evaluation result output by the third fully connected layer.

[0081] For example, the number of input nodes of the second fully connected layer is 128, and the number of output nodes is 1. The second fully connected layer can be represented as Linear(128,1), such as Figure 5 shown.

[0082] In this embodiment, through the pooling layer, the first fully connected layer, the first normalization layer, the activation layer, the second fully connected layer, the second normalization layer and the third fully connected layer, feature redundancy is gradually reduced, useful information in the features is focused, and the generalization ability of the feature mapping structure is improved.

[0083] In an exemplary embodiment, the provided no-reference image quality assessment method further includes a process of obtaining a target quality assessment model. Figure 6 As shown, the process includes steps 602 to 604, wherein:

[0084] Step 602: Perform unsupervised iterative learning on the feature extraction structure in the initial quality assessment model to obtain an intermediate quality assessment model.

[0085] In a possible implementation, step 602 further includes steps D2 to D4, wherein:

[0086] Step D2, obtaining a plurality of sample original images and a set of sample degraded images corresponding to each sample original image, wherein the set of sample degraded images includes a plurality of sample degraded images corresponding to the sample original images, and the sample degraded images are obtained by performing distortion and degradation processing on the sample original images.

[0087] For example, the sample original images in the open source dataset are subjected to degradation processing with various distortion types, including image compression, white noise, blur, etc., and each distortion type corresponds to multiple degradation degrees. Through the processing, a large number of sample degraded images can be obtained from the dataset, and randomly form unsupervised data pairs with the original images.

[0088] As another example, the distortion type may also include random construction and random scaling of the sample original image by 0.5 to 1 to further expand the image content, so that a richer image content distribution can be learned in the unsupervised iterative learning process, and image quality information that is independent of the image scaling scale can be learned.

[0089] Step D4, using each sample original image and the sample degraded image set corresponding to each sample original image to perform unsupervised iterative learning on the feature extraction structure in the initial quality evaluation model to obtain an intermediate quality evaluation model. The loss function in the unsupervised learning process includes a contrast loss function and a feature alignment loss function.

[0090] Exemplary, contrastive loss function ,in, It indicates that the i-th sample original image belongs to the same set of distortion types, N is the batch data size, z is the unsupervised network output feature, The vector dot product operation is to calculate the similarity of two vectors. is the temperature parameter, is the indicator function.

[0091] Exemplary, feature contrast loss function ,in, , , where d is the feature dimension of z, T is the vector transpose, is a vector whose elements are all 1.

[0092] The overall loss function for the unsupervised learning process .

[0093] Exemplarily, the learning rate in the unsupervised iterative learning process is set to 0.1, and the optimizer is selected as Adam.

[0094] In the above implementation, by using different sample original images and corresponding sample degraded image sets as training samples, the trained sample feature extraction structure can have a certain generalization ability for images with different contents, different distortion types and different distortion levels; at the same time, feature contrast loss calculation is used to accelerate the unsupervised learning training process.

[0095] Step 604: Perform supervised iterative learning on the intermediate quality evaluation model to obtain a target quality evaluation model.

[0096] In a possible implementation, step 604 further includes steps E2 to E4, wherein:

[0097] Step E2, obtaining a plurality of sample training images, wherein the sample training images have quality evaluation data labels.

[0098] The quality evaluation data label refers to the true MOS value of the image corresponding to the sample training image.

[0099] Step E4, using each sample training image to perform supervised iterative learning on the intermediate quality evaluation model to obtain a target quality evaluation model.

[0100] Example of loss function in supervised iterative learning process ;in, is the model prediction output, is the real MOS value of the image, is the L1 loss function.

[0101] Among them, in the supervised iterative learning process, the learning rate corresponding to the feature extraction structure is smaller than the learning rate corresponding to the feature mapping structure; in this way, the feature extraction structure not only retains the generalization ability of image distortion type and degree, but also can learn the data distribution of specific no-reference image quality evaluation sample data.

[0102] Exemplarily, in the supervised iterative learning process, the learning rate corresponding to the feature mapping module is set to 0.1, that of the feature extraction module is set to 0.001, and the optimizer is Adam.

[0103] In this embodiment, sample training images with quality evaluation data labels are used to perform supervised iterative learning on the entire model, so that the entire model can learn more accurate image quality information and give a reasonable image quality score prediction.

[0104] In an exemplary embodiment, the initial weights of the convolutional layers in the initial quality assessment model are determined based on the number of input channels and the number of output channels of the convolutional layers, and the initial weights of the fully connected layers in the initial quality assessment model are determined based on the number of input nodes and the number of output nodes of the fully connected layers.

[0105] For example, the initial weights of the convolutional layer and the initial weights of the fully connected layer (w, b) are made to have a mean of 0 and a variance of The distribution of ,in, Indicates the number of input channels of the convolutional layer or the number of input nodes of the fully connected layer. Represents the number of output channels of the convolutional layer or the number of output hidden nodes of the fully connected layer.

[0106] Exemplarily, the downsampling grouped convolution sublayer, the first grouped convolution sublayer, the second grouped convolution sublayer, the grouped multiplication sublayer, the third grouped convolution sublayer, the first fully connected layer, the second fully connected layer and the third fully connected layer all use the above method to determine the corresponding initial weights respectively.

[0107] In a possible implementation, the initial weights of the batch normalization layer are determined by random initialization.

[0108] In an exemplary embodiment, Figure 7 As shown, a method for evaluating image quality without reference is provided, and the method is applied to the server 104 in FIG. 1 as an example for description, including the following steps 702 to 710. Among them:

[0109] Step 702: Obtain an image to be evaluated.

[0110] Step 704: input the image to be evaluated into the global convolution layer in the feature extraction structure in the target quality evaluation model to obtain the global features of the image output by the global convolution layer.

[0111] Step 706: input the global image feature to the first down-sampling multiplication layer of the multiple down-sampling multiplication layers in the feature extraction structure to obtain the intermediate feature.

[0112] Optionally, each downsampling multiplication layer includes a downsampling grouped convolution sublayer, a first grouped convolution sublayer, a second grouped convolution sublayer, a grouped multiplication sublayer, a third grouped convolution sublayer, a fourth grouped convolution sublayer and a residual sublayer; the corresponding step 706 includes: inputting the global image feature into the downsampling grouped convolution sublayer to obtain an intermediate downsampling feature output by the downsampling grouped convolution sublayer; inputting the intermediate downsampling feature into the first grouped convolution sublayer to obtain an intermediate first grouped feature output by the first grouped convolution sublayer, the number of groups in the first grouped convolution sublayer is the same as the number of input channels of the first grouped convolution sublayer; inputting the intermediate first grouped feature into the second grouped convolution sublayer to obtain an intermediate second grouped feature output by the second grouped convolution sublayer, the number of groups in the second grouped convolution sublayer is 2; inputting the intermediate second grouped feature into the grouped multiplication sublayer. , the intermediate product feature output by the group multiplication sublayer is obtained, the intermediate product feature is based on dividing the intermediate second group feature into the first segmentation feature and the second segmentation feature, activating the first segmentation feature to obtain the first activation feature, and multiplying the first activation feature by the second segmentation feature; the intermediate product feature is input into the third group convolution sublayer to obtain the intermediate third group feature output by the third group convolution sublayer, the number of groups in the third group convolution sublayer is the same as the number of input channels of the third group convolution sublayer; the intermediate third group feature is input into the fourth group convolution sublayer to obtain the intermediate fourth group feature output by the fourth group convolution sublayer, the number of groups in the fourth group convolution sublayer is the same as the number of input channels of the fourth group convolution sublayer; the fourth group feature and the intermediate down-sampling feature are input into the residual sublayer to obtain the intermediate feature output by the residual sublayer.

[0113] Step 708, input the intermediate feature into the second downsampling multiplication layer among the multiple downsampling multiplication layers until the target feature is obtained, where the target feature is the feature extraction result output by the last downsampling multiplication layer among the multiple downsampling multiplication layers.

[0114] Step 710: Input the target features into the feature mapping structure of the target quality assessment model to obtain the quality assessment result output by the feature mapping structure.

[0115] Optionally, the feature mapping structure includes a pooling layer, a first fully connected layer, a first normalization layer, an activation layer, a second fully connected layer, a second normalization layer, and a third fully connected layer. Correspondingly, step 710 includes: inputting the target feature into the pooling layer to obtain the pooling data output by the pooling layer; inputting the pooling data into the first fully connected layer to obtain the first fully connected data output by the first fully connected layer; inputting the first fully connected data into the first normalization layer to obtain the first normalized data output by the first normalization layer; inputting the first normalized data into the activation layer to obtain the activation data output by the activation layer; inputting the activation data into the second fully connected layer to obtain the second fully connected data input by the second fully connected layer; inputting the second fully connected data into the second normalization layer to obtain the second normalized data output by the second normalization layer; inputting the second normalized data into the third fully connected layer to obtain the quality evaluation result output by the third fully connected layer.

[0116] Optionally, the training process of the target detection model includes: performing unsupervised iterative learning on the feature extraction structure in the initial quality assessment model to obtain an intermediate quality assessment model; performing supervised iterative learning on the intermediate quality assessment model to obtain a target quality assessment model; wherein, the initial weights of the convolutional layer in the initial quality assessment model are determined based on the number of input channels and the number of output channels of the convolutional layer; and the initial weights of the fully connected layer in the initial quality assessment model are determined based on the number of input nodes and the number of output nodes of the fully connected layer.

[0117] Optionally, the unsupervised iterative learning process includes: obtaining multiple sample original images and a set of sample degraded images corresponding to each sample original image, the sample degraded image set including multiple sample degraded images corresponding to the sample original image, and the sample degraded images are obtained by distortion and degradation processing based on the sample original images; using each sample original image and the set of sample degraded images corresponding to each sample original image to perform unsupervised iterative learning on the feature extraction structure in the initial quality evaluation model to obtain an intermediate quality evaluation model; wherein the loss function in the unsupervised learning process includes a contrast loss function and a feature alignment loss function.

[0118] Optionally, the supervised iterative learning process includes: obtaining multiple sample training images, the sample training images having quality evaluation data labels; using each sample training image to perform supervised iterative learning on an intermediate quality evaluation model to obtain a target quality evaluation model; wherein, in the supervised iterative learning process, the learning rate corresponding to the feature extraction structure is less than the learning rate corresponding to the feature mapping structure.

[0119] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0120] It is understood that the term "based on" used in this application is used to describe one or more factors that affect the determination, and does not exclude other factors that may affect the determination. For example, the phrase "determine A based on B" means that the determination of A can be based entirely or at least partially on factor B, that is, B is a factor that affects the determination of A, but does not exclude that the determination of A is also based on C.

[0121] Based on the same inventive concept, the embodiment of the present application also provides a non-reference image quality assessment device for implementing the non-reference image quality assessment method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more non-reference image quality assessment device embodiments provided below can refer to the limitations of the non-reference image quality assessment method above, and will not be repeated here.

[0122] In an exemplary embodiment, a non-reference image quality assessment device is provided, comprising: an image acquisition module, a feature extraction module and a feature mapping module, wherein:

[0123] An acquisition module, used for acquiring an image to be evaluated;

[0124] An extraction module, used for inputting the image to be evaluated into the feature extraction structure of the target quality evaluation model to obtain the target features output by the feature extraction structure, wherein the feature extraction structure includes a global convolution layer and a downsampling multiplication layer;

[0125] The prediction module is used to input the target features into the feature mapping structure of the target quality evaluation model to obtain the quality evaluation result output by the feature mapping structure.

[0126] In an exemplary embodiment, the feature extraction structure includes a global convolution layer and multiple downsampling multiplication layers; the extraction module is used to input the image to be evaluated into the global convolution layer to obtain the global features of the image output by the global convolution layer, and to input the global features of the image into the first downsampling multiplication layer among the multiple downsampling multiplication layers to obtain intermediate features, and to input the intermediate features into the second downsampling multiplication layer among the multiple downsampling multiplication layers until the target features are obtained, and the target features are the feature extraction results output by the last downsampling multiplication layer among the multiple downsampling multiplication layers.

[0127] In an exemplary embodiment, the downsampling multiplication layer includes a downsampling grouped convolution sublayer, a first grouped convolution sublayer, a second grouped convolution sublayer, a grouped multiplication sublayer, a third grouped convolution sublayer, a fourth grouped convolution sublayer and a residual sublayer; the extraction module is used to input the global features of the image into the downsampling grouped convolution sublayer to obtain the intermediate downsampling features output by the downsampling grouped convolution sublayer, input the intermediate downsampling features into the first grouped convolution sublayer to obtain the intermediate first grouped features output by the first grouped convolution sublayer, the number of groups in the first grouped convolution sublayer is the same as the number of input channels of the first grouped convolution sublayer, input the intermediate first grouped features into the second grouped convolution sublayer to obtain the intermediate second grouped features output by the second grouped convolution sublayer, the number of groups in the second grouped convolution sublayer is 2, and input the intermediate second grouped features into the grouped multiplication sublayer. In the process, the intermediate product feature of the group multiplication sublayer output is obtained. The intermediate product feature is based on dividing the intermediate second group feature into the first segmentation feature and the second segmentation feature, activating the first segmentation feature to obtain the first activation feature, multiplying the first activation feature by the second segmentation feature, inputting the intermediate product feature into the third group convolution sublayer, and obtaining the intermediate third group feature output by the third group convolution sublayer. The number of groups in the third group convolution sublayer is the same as the number of input channels of the third group convolution sublayer. The intermediate third group feature is input into the fourth group convolution sublayer, and the intermediate fourth group feature output by the fourth group convolution sublayer is obtained. The number of groups in the fourth group convolution sublayer is the same as the number of input channels of the fourth group convolution sublayer. The fourth group feature and the intermediate down-sampling feature are input into the residual sublayer to obtain the intermediate feature output by the residual sublayer.

[0128] In an exemplary embodiment, the feature mapping structure includes a pooling layer, a first fully connected layer, a first normalization layer, an activation layer, a second fully connected layer, a second normalization layer and a third fully connected layer; the prediction module is used to input the target feature into the pooling layer to obtain the pooled data output by the pooling layer; the pooling data is input into the first fully connected layer to obtain the first fully connected data output by the first fully connected layer; the first fully connected data is input into the first normalization layer to obtain the first normalized data output by the first normalization layer; the first normalized data is input into the activation layer to obtain the activation data output by the activation layer; the activation data is input into the second fully connected layer to obtain the second fully connected data input by the second fully connected layer; the second fully connected data is input into the second normalization layer to obtain the second normalized data output by the second normalization layer; the second normalized data is input into the third fully connected layer to obtain the quality evaluation result output by the third fully connected layer.

[0129] In an exemplary embodiment, the no-reference image quality assessment device also includes a training module for performing unsupervised iterative learning on the feature extraction structure in the initial quality assessment model to obtain an intermediate quality assessment model; and performing supervised iterative learning on the intermediate quality assessment model to obtain a target quality assessment model.

[0130] In an exemplary embodiment, the training module also includes an unsupervised training unit, which is used to obtain multiple sample original images and a set of sample degraded images corresponding to each sample original image, the sample degraded image set includes multiple sample degraded images corresponding to the sample original image, and the sample degraded images are obtained by distortion and degradation processing based on the sample original image; the feature extraction structure in the initial quality evaluation model is unsupervised iteratively learned using each sample original image and the set of sample degraded images corresponding to each sample original image to obtain an intermediate quality evaluation model; wherein the loss function in the unsupervised learning process includes a contrast loss function and a feature alignment loss function.

[0131] In an exemplary embodiment, the training module also includes a supervised training unit for acquiring multiple sample training images, each of which has a quality evaluation data label; using each sample training image to perform supervised iterative learning on an intermediate quality evaluation model to obtain a target quality evaluation model; wherein, during the supervised iterative learning process, the learning rate corresponding to the feature extraction structure is less than the learning rate corresponding to the feature mapping structure.

[0132] In an exemplary embodiment, the initial weight of the convolutional layer in the initial quality assessment model is determined based on the number of input channels and the number of output channels of the convolutional layer; the initial weight of the fully connected layer in the initial quality assessment model is determined based on the number of input nodes and the number of output nodes of the fully connected layer.

[0133] Each module in the above-mentioned non-reference image quality assessment device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0134] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal layer diagram may be shown in FIG8. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store parameter data in the target quality assessment model and data that needs to be called in the process of implementing the non-reference image quality assessment method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a non-reference image quality assessment method is implemented.

[0135] Those skilled in the art will understand that the layers shown in FIG. 8 are merely block diagrams of some layers related to the scheme of the present application, and do not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0136] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0137] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0138] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0139] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0140] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0141] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for evaluating image quality without reference, characterized in that: The method comprises: Obtain the image to be evaluated; Inputting the image to be evaluated into a feature extraction structure of a target quality evaluation model to obtain target features output by the feature extraction structure, wherein the feature extraction structure includes a global convolution layer and a downsampling multiplication layer; The target feature is input into the feature mapping structure of the target quality assessment model to obtain the quality assessment result output by the feature mapping structure.

2. The method according to claim 1, characterized in that The feature extraction structure includes the global convolution layer and a plurality of downsampling multiplication layers; The step of inputting the image to be evaluated into a feature extraction structure of a target quality evaluation model to obtain target features output by the feature extraction structure includes: Inputting the image to be evaluated into the global convolution layer to obtain the global features of the image output by the global convolution layer; Inputting the global image feature into a first downsampling multiplication layer among the multiple downsampling multiplication layers to obtain an intermediate feature; The intermediate feature is input into a second down-sampling multiplication layer among the multiple down-sampling multiplication layers until the target feature is obtained, where the target feature is a feature extraction result output by a last down-sampling multiplication layer among the multiple down-sampling multiplication layers.

3. The method according to claim 2, characterized in that The downsampling multiplication layer includes a downsampling grouped convolution sublayer, a first grouped convolution sublayer, a second grouped convolution sublayer, a grouped multiplication sublayer, a third grouped convolution sublayer, a fourth grouped convolution sublayer and a residual sublayer; Inputting the global image feature into the first downsampling multiplication layer of the multiple downsampling multiplication layers to obtain an intermediate feature includes: Inputting the global image features into a downsampling grouped convolutional sublayer to obtain an intermediate downsampling feature output by the downsampling grouped convolutional sublayer; Inputting the intermediate down-sampled features into the first grouped convolutional sublayer to obtain an intermediate first grouped feature output by the first grouped convolutional sublayer, wherein the number of groups of the first grouped convolutional sublayer is the same as the number of input channels of the first grouped convolutional sublayer; Inputting the intermediate first grouping feature into the second grouping convolution sublayer to obtain the intermediate second grouping feature output by the second grouping convolution sublayer, the number of groups of the second grouping convolution sublayer being 2; Inputting the intermediate second grouping feature into the group multiplication sublayer to obtain an intermediate product feature output by the group multiplication sublayer, wherein the intermediate product feature is based on dividing the intermediate second grouping feature into a first segmentation feature and a second segmentation feature, performing an activation operation on the first segmentation feature to obtain a first activation feature, and multiplying the first activation feature by the second segmentation feature; Inputting the intermediate product feature into the third grouped convolution sublayer to obtain an intermediate third grouped feature output by the third grouped convolution sublayer, wherein the number of groups of the third grouped convolution sublayer is the same as the number of input channels of the third grouped convolution sublayer; Inputting the intermediate third grouping feature into the fourth grouping convolution sublayer to obtain the intermediate fourth grouping feature output by the fourth grouping convolution sublayer, the number of groups of the fourth grouping convolution sublayer being the same as the number of input channels of the fourth grouping convolution sublayer; The fourth grouping feature and the intermediate down-sampling feature are input into the residual sub-layer to obtain the intermediate feature output by the residual sub-layer.

4. The method according to claim 1, characterized in that: The feature mapping structure includes a pooling layer, a first fully connected layer, a first normalization layer, an activation layer, a second fully connected layer, a second normalization layer and a third fully connected layer; Inputting the target feature into the feature mapping structure of the target quality evaluation model to obtain the quality evaluation result output by the feature mapping structure includes: Inputting the target feature into the pooling layer to obtain pooled data output by the pooling layer; Inputting the pooled data into the first fully connected layer to obtain first fully connected data output by the first fully connected layer; Inputting the first fully connected data into the first normalization layer to obtain first normalized data output by the first normalization layer; Inputting the first normalized data into the activation layer to obtain activation data output by the activation layer; Inputting the activation data into the second fully connected layer to obtain second fully connected data input by the second fully connected layer; Inputting the second fully connected data into a second normalization layer to obtain second normalized data output by the second normalization layer; The second normalized data is input into the third fully connected layer to obtain the quality evaluation result output by the third fully connected layer.

5. The method according to claim 1, characterized in that The method further comprises: Perform unsupervised iterative learning on the feature extraction structure in the initial quality evaluation model to obtain an intermediate quality evaluation model; The intermediate quality evaluation model is subjected to supervised iterative learning to obtain the target quality evaluation model.

6. The method according to claim 5, characterized in that The unsupervised iterative learning of the feature extraction structure in the initial quality evaluation model to obtain the intermediate quality evaluation model includes: Acquire a plurality of sample original images and a set of sample degraded images corresponding to each of the sample original images, wherein the set of sample degraded images includes a plurality of sample degraded images corresponding to the sample original images, and the sample degraded images are obtained by performing distortion and degradation processing on the sample original images; Using each of the sample original images and a set of sample degraded images corresponding to each of the sample original images, unsupervised iterative learning is performed on the feature extraction structure in the initial quality assessment model to obtain an intermediate quality assessment model; Among them, the loss functions in the unsupervised learning process include contrast loss function and feature alignment loss function.

7. The method according to claim 5, characterized in that The supervised iterative learning of the intermediate quality evaluation model to obtain the target quality evaluation model includes: Acquire a plurality of sample training images, wherein the sample training images have quality assessment data labels; Using each of the sample training images to perform supervised iterative learning on the intermediate quality assessment model to obtain the target quality assessment model; Among them, in the supervised iterative learning process, the learning rate corresponding to the feature extraction structure is smaller than the learning rate corresponding to the feature mapping structure.

8. The method according to claim 5, characterized in that The initial weight of the convolutional layer in the initial quality assessment model is determined based on the number of input channels and the number of output channels of the convolutional layer; The initial weight of the fully connected layer in the initial quality assessment model is determined based on the number of input nodes and the number of output nodes of the fully connected layer.

9. A non-reference image quality assessment device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be evaluated; An extraction module, used for inputting the image to be evaluated into a feature extraction structure of a target quality evaluation model to obtain target features output by the feature extraction structure, wherein the feature extraction structure includes a global convolution layer and a downsampling multiplication layer; A mapping module is used to input the target feature into the feature mapping structure of the target quality evaluation model to obtain the quality evaluation result output by the feature mapping structure.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images