Image quality determination method and device

Abstract

The invention discloses an image quality determination method and device, relates to the field of image processing, and aims to solve the problem that the image quality cannot be evaluated. The method comprises the steps of obtaining first image information, performing feature extraction on the first image information to obtain first feature information, performing convolution processing on the first feature information to obtain first weight information, performing feature enhancement on the first feature information by using the first weight information to obtain second feature information, and determining the image quality of the first image information according to the second feature information. Through the scheme, the quality of the image can be evaluated.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and device for determining image quality. Background Art

[0002] With the rapid development of network and multimedia technology, images have become deeply integrated into people's daily lives. For example, in the fields of remote sensing, medical imaging and monitoring systems, high-quality images with bright colors, rich textures and good contrast are an indispensable source of information. For another example, various applications and social media continue to generate massive images for users to use and share. Today, images play a vital role as an indispensable carrier and widely used data medium in the information age.

[0003] Therefore, how to evaluate the quality of images is an urgent problem that needs to be solved. Summary of the invention

[0004] The present invention provides an image quality determination method and device for evaluating image quality.

[0005] In a first aspect, the present invention provides a method for determining image quality, the method comprising: acquiring first image information, performing feature extraction on the first image information to obtain first feature information, performing convolution processing on the first feature information to obtain first weight information, using the first weight information to perform feature enhancement on the first feature information to obtain second feature information, and determining the image quality of the first image information based on the second feature information.

[0006] Through the above scheme, the first image information can be firstly extracted to obtain the first feature information, and the first weight of the first feature information can be obtained. The spatial position importance of the first feature information can be learned through the first weight to obtain the second feature information that enhances the first feature information. Then, the image quality can be determined through the second feature information. In this way, the image quality can be evaluated.

[0007] Optionally, there are multiple first feature information, different first feature information corresponds to different spatial position features, there are multiple first weight information, and the first weight information is used to enhance the first feature information to obtain the second feature information, including: using the first weight information to perform weighted summation on the first feature information to obtain the third feature information, calculating the sum of the first weights, dividing the third feature information by the sum, and obtaining the second feature information.

[0008] Through the above scheme, the weights are multiplied element-by-element by the input features through the attention mechanism and the sum is taken for each channel, and then divided by the sum of the weights for each channel. In this way, the obtained feature vector can be directly used for the regression of a single fully connected layer, effectively reducing network parameters and computational burden; and, by integrating the attention of two dimensions, that is, paying attention to the feature information of different channels and spatial positions at the same time, the expressiveness and generalization capabilities of the model can be improved.

[0009] Optionally, feature extraction is performed on the first image information to obtain first feature information of the first image information, including: performing convolution processing on the first image information, performing residual processing on the image information after the convolution processing, and performing convolution processing on the image information after the residual processing again to obtain the first feature information of the first image information.

[0010] Through the above scheme, compared with the feature extraction of image information in the prior art, the processing steps of the pooling layer and the network layer are removed, the steps of extracting the first image information are simplified, and the parameters and the amount of calculation are reduced.

[0011] Optionally, feature extraction is performed on the first image information to obtain first feature information of the first image information, including: inputting the first image information into a first model to obtain first feature information output by the first model, wherein the first model includes a first convolutional network module, a second convolutional network module and a bottleneck module, the bottleneck module is located between the first convolutional network module and the second convolutional network module, the first convolutional network module includes a 3*3 convolutional layer, a first normalization layer and a first activation function layer connected in sequence, the bottleneck module includes a plurality of bottleneck units with an inverted residual structure connected in sequence, and the second convolutional network module includes a 1*1 convolutional layer, a second normalization layer and a second activation function layer connected in sequence.

[0012] Through the above scheme, the feature information of the first image can be deeply extracted through the first model. Compared with the traditional quality evaluation method used in the prior art, this scheme has strong feature representation ability and good generalization performance.

[0013] Optionally, convolution processing is performed on the first feature information to obtain first weight information, and the first weight information is used to perform feature enhancement on the first feature information to obtain second feature information, including: inputting the first feature information into the second model to obtain second feature information output by the second model, wherein the second model includes a third convolutional network module, two groups of channels and a data processing module, wherein the input end of one group of channels is connected to the output end of the third convolutional network module and the input end of the second model, the input end of the other group of channels is connected to the output end of the third convolutional network module, and the output ends of both groups of channels are connected to the data processing module.

[0014] Through the above scheme, the second model can integrate the two dimensions of space and channel through the third convolution module and two groups of channels. That is, the second model can capture both channel differences and spatial differences. This can help the model identify key areas in the image and effectively improve the expressiveness of features without excessively increasing the computational complexity.

[0015] Optionally, the second model is determined in the following manner: obtaining reference image information, the reference image information including a reference quality value, performing feature extraction on the reference image information to obtain first feature information of the reference image information, and inputting the first feature information of the reference image information into the second model to obtain second feature information of the reference image information output by the second model, determining a first quality value of the reference image information based on the second feature information of the reference image information, determining a model loss function based on the first quality value and the reference quality value, and using the model loss function to adjust parameters of the second model so that the second model converges.

[0016] Through the above scheme, by calculating the loss function to adjust the parameters of the second model, the second model can quickly converge to the optimal solution, minimize the difference between the model prediction and the reference quality score, and improve the accuracy of the predicted image quality score.

[0017] Optionally, a model loss function is determined based on the first quality value and the reference quality value, including: calculating a Huber loss function based on the difference between the first quality value and the reference quality value, calculating a normalized loss function based on the first quality value, the reference quality value, the statistics of the first quality value, and the statistics of the reference quality value, flipping the reference image and obtaining a second quality value of the quality of the flipped reference image, calculating a flip consistency loss function based on the first quality value and the second quality value, and taking a weighted average of the Huber loss function, the normalized loss function, and the flip consistency loss function to determine the model loss function.

[0018] Through the above scheme, the Huber loss function is used to constrain the consistency between the predicted quality score and the subjective quality score, the normalization loss function is used to further constrain the standardized result, and the generalization ability of the model is enhanced by flipping the consistency loss function to effectively prevent overfitting. In this way, the second model converges quickly through multiple loss functions.

[0019] In a second aspect, the present invention provides an image quality determination device, the device comprising:

[0020] An acquisition module, used for acquiring first image information;

[0021] A feature extraction module, used to extract features from the first image information to obtain first feature information;

[0022] A feature enhancement module, configured to perform convolution processing on the first feature information to obtain first weight information, and perform feature enhancement on the first feature information using the first weight information to obtain second feature information;

[0023] The determination module is used to determine the image quality of the first image information according to the second characteristic information.

[0024] In one possible implementation, there are multiple first feature information, different first feature information corresponds to different spatial position features, there are multiple first weight information, and the feature enhancement module is specifically used to: use the first weight information to weighted sum the first feature information to obtain the third feature information, calculate the sum of the first weights, divide the third feature information by the sum, and obtain the second feature information.

[0025] In a possible implementation, the feature extraction module is specifically used to: perform convolution processing on the first image information, perform residual processing on the image information after the convolution processing, and perform convolution processing on the image information after the residual processing again to obtain first feature information of the first image information.

[0026] In a possible implementation, the feature extraction module is specifically used to: input the first image information into the first model, and obtain the first feature information output by the first model, wherein the first model includes a first convolutional network module, a second convolutional network module and a bottleneck module, the bottleneck module is located between the first convolutional network module and the second convolutional network module, the first convolutional network module includes a 3*3 convolutional layer, a first normalization layer and a first activation function layer connected in sequence, the bottleneck module includes a plurality of bottleneck units with an inverted residual structure connected in sequence, and the second convolutional network module includes a 1*1 convolutional layer, a second normalization layer and a second activation function layer connected in sequence.

[0027] In one possible implementation, the feature enhancement module is specifically used to: input the first feature information into the second model to obtain the second feature information output by the second model, wherein the second model includes a third convolutional network module, two groups of channels and a data processing module, wherein the input end of one group of channels is connected to the output end of the third convolutional network module and the input end of the second model, the input end of the other group of channels is connected to the output end of the third convolutional network module, and the output ends of the two groups of channels are connected to the data processing module.

[0028] In one possible implementation, the feature enhancement module is also used to: obtain reference image information, the reference image information includes a reference quality value, perform feature extraction on the reference image information to obtain first feature information of the reference image information, and input the first feature information of the reference image information into the second model to obtain second feature information of the reference image information output by the second model, determine the first quality value of the reference image information according to the second feature information of the reference image information, determine the model loss function according to the first quality value and the reference quality value, and use the model loss function to adjust the parameters of the second model so that the second model converges.

[0029] In a possible implementation, it also includes a loss function calculation module, which is used to: calculate the Huber loss function according to the difference between the first quality value and the reference quality value, calculate the normalized loss function according to the first quality value, the reference quality value, the statistics of the first quality value and the statistics of the reference quality value, flip the reference image and obtain a second quality value of the quality of the reference image after flipping, calculate the flipping consistency loss function according to the first quality value and the second quality value, perform weighted averaging on the Huber loss function, the normalized loss function and the flipping consistency loss function, and determine the model loss function.

[0030] In a third aspect, the present invention also provides an image quality determination device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method described in the various possible designs of the first aspect.

[0031] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program or instructions stored thereon. When the computer program or instructions are executed by a processor, the method described in the various possible designs of the first aspect is implemented.

[0032] In a fifth aspect, the present invention further provides a computer program product, which, when executed on a computer, enables the computer to execute any of the methods described in the first aspect above.

[0033] These implementations or other implementations of the present application will be more concise and understandable in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1A flowchart of a method for determining image quality provided by an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of the structure of a first model provided by an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of a process of extracting features from first image information provided by an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of the structure of a bottleneck module provided by an embodiment of the present invention;

[0039] Figure 5 A schematic diagram of the structure of a second model provided by an embodiment of the present invention;

[0040] Figure 6 A schematic diagram of the structure of another second model provided by an embodiment of the present invention;

[0041] Figure 7 A schematic diagram of a process of enhancing the first feature information according to the first weight information provided in an embodiment of the present invention;

[0042] Figure 8 A schematic diagram of the structure of an image quality assessment system provided by an embodiment of the present invention;

[0043] Fig. 9 A schematic diagram of a process for determining a second model provided by an embodiment of the present invention;

[0044] Fig.10 A schematic diagram of an image quality determination device provided by an embodiment of the present invention;

[0045] Fig.11 A schematic diagram of another image quality determination device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and beneficial effects of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] Below, some terms in this application are explained. It should be noted that these explanations are for the convenience of understanding by those skilled in the art and do not constitute a limitation on the scope of protection claimed in this application.

[0048] 1. Image quality assessment (IQA).

[0049] IQA is a basic technology in image processing. It mainly analyzes the characteristics of images, evaluates the quality of images (i.e., the degree of image distortion), and gives corresponding quality scores.

[0050] 2. Loss function.

[0051] The loss function is a function that maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. In applications, the loss function is usually associated with the optimization problem as a learning criterion, that is, the model is solved and evaluated by minimizing the loss function.

[0052] As mentioned in the background technology, images play an important role in the information age, and it is particularly important to evaluate image quality. For example, by evaluating the degree of image distortion and giving a quality score, it can be determined whether the image meets people's visual perception and needs. At present, IQA methods can be divided into two types of evaluation methods: reference and non-reference. However, in the reference evaluation method, the reference image is usually difficult to obtain. Therefore, the reference evaluation method is less practical. In contrast, the non-reference evaluation method is more suitable for actual application scenarios.

[0053] However, the current non-reference evaluation methods are relatively simple, that is, mathematical formulas or models are designed and established based on prior knowledge to extract features and evaluate image quality. For example, a non-reference IQA method currently provided in the industry is to extract two types of complementary low-level image feature distributions and combine shallow machine learning as a regression model to evaluate image quality. However, this method focuses on processing features at the pixel level, that is, it mainly focuses on pixel differences and cannot obtain deep features. Therefore, the feature representation ability is weak, and the accuracy and generalization of the evaluation are limited.

[0054] For the above problems, the industry has proposed another solution, which is to use deep neural networks and large pre-trained models for IQA evaluation. For example, the current solution provided by the industry is: determine the multi-scale features of the image based on the deep network, fuse all the multi-scale features into the feature interaction network, and regress the quality score of the panoramic image through the interactive features. However, this method has high model complexity, many parameters, and large calculations, resulting in high model operation time and space costs, making it difficult to deploy on mobile devices, making it difficult to lightweight the model.

[0055] In addition, both of the above methods use simple loss functions, which leads to slow convergence of the model and poor constraint ability.

[0056] Based on this, the present invention provides an image quality assessment method, which accurately assesses image quality by performing feature extraction and feature enhancement on image information.

[0057] The present application scheme is described in detail below in conjunction with the accompanying drawings.

[0058] See also Figure 1 , a flow chart of a method for determining image quality is shown, which specifically includes the following steps:

[0059] Step 110: Acquire first image information.

[0060] Here, the first image information may be obtained from a preset database, or an image file may be read from a designated storage location without limitation.

[0061] Optionally, after acquiring the first image information, the image may be preprocessed, such as resizing, color space conversion, etc., or the first image information may be converted into a set data format.

[0062] Step 120: extract features from the first image information to obtain first feature information.

[0063] Optionally, the first image information can be input into the first model to obtain the first feature information output by the first model. In this way, the feature information of the first image can be deeply extracted through the first model. Compared with the traditional quality evaluation method used in the prior art, this solution has strong feature representation ability and good generalization performance.

[0064] Among them, the first model can be any model with image processing capabilities, for example, it can be an EfficientNet model, it can be a MobileNet-v2 model, it can also be a model that improves the above models, and so on, without limitation.

[0065] Optionally, the first model includes a first convolutional network module, a second convolutional network module and a bottleneck module, the bottleneck module is located between the first convolutional network module and the second convolutional network module, the first convolutional network module includes a 3*3 convolutional layer, a first normalization layer and a first activation function layer connected in sequence, the bottleneck module includes a plurality of bottleneck units with an inverted residual structure connected in sequence, and the second convolutional network module includes a 1*1 convolutional layer, a second normalization layer and a second activation function layer connected in sequence.

[0066] See also Figure 2, showing a schematic diagram of the structure of the first model, which can be improved on the basis of the existing MobileNet-v2 model, removing the pooling layer and network layer of the original model, retaining its convolutional network module and bottleneck module, that is, the three modules on the far left (3*3 convolutional layer, the first normalization layer and the first activation function layer) are together the first convolutional network module, and the three modules on the far right (1*1 convolutional layer, the second normalization layer and the second activation function layer) are together the second convolutional network module, and there are several bottleneck modules in the middle. In this way, the first image processing model can not only realize image feature extraction, but also simplify the structure of the MobileNet-v2 model, reduce calculation parameters and simplify calculations.

[0067] Through the above scheme, by adopting the MobileNet-v2 model as the lightweight basic backbone network, in order to further streamline the network structure, the original MobileNet-v2 model was modified, that is, the pooling layer and network layer in the original model were removed, so that the number of parameters and the amount of calculation of the model can be reduced.

[0068] See also Figure 3 , showing a schematic diagram of a process of extracting features from the first image information, step 120 may specifically include the following steps 310 to 330:

[0069] Step 310: Perform convolution processing on the first image information.

[0070] Optionally, the first image information may be input into a first convolutional network module ConvNormActivation of a MobileNet-v2 model for convolution processing.

[0071] Optionally, before step 310, the MobileNet-v2 model may be pre-trained on a dataset, such as an ImageNet dataset, so that the MobileNet-v2 model can learn some common image features that can be applied to new image quality assessment tasks.

[0072] Please continue reading Figure 2The first convolutional network module ConvNormActivation includes: 1*1 convolutional layer, normalization layer and rectified linear unit (ReLU) activation function. Inputting the first image information into the first convolutional network module includes: inputting the first image information into the convolutional layer to extract local features of the image, such as edges, textures, etc., and then normalizing the features through the normalization layer to improve the stability and convergence speed of the module, and finally inputting into the ReLU activation function to introduce nonlinearity to enhance the expression ability. In this way, after being processed by the ConvNormActivation module, the surface features of the image can be obtained, which lays the foundation for subsequent deep feature extraction and image quality assessment.

[0073] Step 320: Perform residual processing on the image information after convolution processing.

[0074] Optionally, residual processing can be performed through the bottleneck module of the MobileNet-v2 model.

[0075] Among them, the MobileNet-v2 model has several bottleneck modules. For details, please refer to the above Figure 2 . See Figure 4 , shows a schematic diagram of the structure of a bottleneck module. The bottleneck module adopts an inverted residual structure, that is, it first performs a dimensionality increase operation and then a dimensionality reduction operation. In this way, it can capture richer feature information in a high-dimensional space while maintaining computational efficiency. The specific steps of the bottleneck module to extract deep features are as follows:

[0076] Step 1: For the input feature f∈H*W*K, the main branch of each residual module first uses 1x1 convolution to perform a dimensionality increase operation to map the feature from the low-dimensional space to the high-dimensional space, obtain the feature of H*W*tk, and increase the expressiveness of the feature.

[0077] Step 2: Use 3x3 depth-separable convolution for feature extraction, and then use 1×1 convolution to reduce the number of channels to obtain the reduced-dimensional features. In this way, the number of model parameters can be reduced while maintaining the execution efficiency of the model.

[0078] Step 3: Use residual connection to add the input and the features obtained by the main branch to output the final feature f 1 The use of residual connections can alleviate the gradient vanishing problem, making deep networks easier to train, while also improving the expressiveness of features.

[0079] Step 330: performing convolution processing on the image information after the residual processing to obtain first feature information of the first image information.

[0080] Optionally, the first image information may be input to the second convolutional network module ConvNormActivation of the MobileNet-v2 model for convolution processing. Figure 3 It can be seen that the second convolutional network module is similar to the first convolutional network module, but the convolutional layer of the second convolutional network module is different from that of the first convolutional network module. This step can help the network model better learn and extract higher-order feature representations and obtain feature f i , providing a basis for the next step of feature enhancement.

[0081] Through the above steps 310 to 330, compared with the prior art of extracting features from image information, the processing steps of the pooling layer and the network layer are removed, the steps of extracting the first image information are simplified, and the parameters and the amount of calculation are reduced.

[0082] Step 130: Perform convolution processing on the first feature information to obtain first weight information, and use the first weight information to perform feature enhancement on the first feature information to obtain second feature information.

[0083] Optionally, the first feature information may be input into the second model to obtain the second feature information output by the second model, wherein the second model may be any model having image processing capability.

[0084] Optionally, the second model may include a third convolutional network module, two sets of channels and a data processing module, see Figure 5 , showing a schematic diagram of the structure of the second model, wherein the input end of one group of channels is connected to the output end of the third convolutional network module and the input end of the second model, the input end of another group of channels is connected to the output end of the third convolutional network module, and the output end of each group of channels is connected to the data processing module. Through the above scheme, the second model can integrate the two dimensions of space and channel through the third convolutional module and two groups of channels, that is, the second model can capture both channel differences and spatial differences. In this way, it can help the model identify key areas in the image and effectively improve the expressiveness of features without excessively increasing the computational complexity.

[0085] For details, please refer to Figure 6 , showing another schematic diagram of the structure of the second model. It can be seen from the figure that the third convolution module of the second model includes a convolution layer, a batch normalization BatchNorm layer and a ReLU activation function layer. The input feature f iAfter passing through the convolution layer, BatchNorm layer, and ReLU activation function layer, the attention weight w can be obtained. After that, it passes through different channels. As shown in the figure, the upper channel connects the output of the third convolution module and the input of the second model, and the lower channel connects the output of the third convolution module. After passing through different channels, the attention weight w enters the data processing module, that is, the element-by-element division processing module in the figure, and finally the output f is obtained. e . The second model can integrate the attention mechanisms of both space and channel dimensions, that is, spatial attention is used to focus on the importance of different spatial positions in the feature map to help the model identify key areas in the image, and channel attention is used to focus on the importance of different feature channels to help the model identify which types of features are more important. The present invention combines two attention mechanisms to simultaneously consider the importance of features in the spatial and channel dimensions, thereby improving the expressiveness and generalization capabilities of the model.

[0086] See also Figure 7 , showing a schematic diagram of a process of enhancing the first feature information according to the first weight information, step 130 specifically includes the following steps 710 to 740:

[0087] Step 710: Perform convolution processing on the first feature information to obtain first weight information.

[0088] Optionally, the second model may be used to enhance the first feature information.

[0089] Please continue reading Figure 6 , the first feature information f obtained in step 330 i The input is sent to the second model, and after passing through the 1x1 convolution layer, BatchNorm layer, and ReLU activation function layer, an attention weight w with the same size as the input feature is obtained. The attention weight w allows the second model to learn the importance of each channel feature at different spatial positions, that is, to identify which spatial regions are more important for each channel. In this way, the first weight of the channel feature of each channel at different spatial positions can be obtained.

[0090] This step uses a simplified network structure (1x1 convolution layer, BatchNorm layer, ReLU activation function layer) to achieve network lightweight and reduce calculation parameters and amount of calculation.

[0091] Step 720: Use the first weight information to perform weighted summation on the first feature information to obtain a third feature information.

[0092] That is, the first weight is combined with the input feature f i Multiply element-wise and sum the results channel-wise.

[0093] Step 730: Calculate the sum of the first weights.

[0094] That is, the first weight is summed across all channels.

[0095] Step 740: Divide the third feature information by the sum value to obtain the second feature information.

[0096] That is, the result of step 720 is divided by the result of step 730, so that the first weight can be adjusted according to the channel to further strengthen the focus on important features, and the resulting output feature vector f e It can be directly used for the next step of calculation (i.e., fully connected regression calculation), reducing parameters and computational burden.

[0097] Through steps 710 to 740, the feature vector f of each channel can be obtained. e , specifically the following formula:

[0098]

[0099] in, The feature f i The value corresponding to the position (x, y), w k,l f i The first weight corresponds to the position (x, y), ε = 1*10 -8 is a constant used to avoid division by zero in the above formula.

[0100] Through the above steps 710 to 740, the weights are multiplied element-by-element by the input features through the attention mechanism and the sum is taken for each channel, and then divided by the sum of the weights for each channel. In this way, the obtained feature vector can be directly used for the regression of a single fully connected layer, effectively reducing network parameters and computational burden; and, by integrating the attention of two dimensions, that is, paying attention to the feature information of different channels and spatial positions at the same time, the expression ability and generalization ability of the model can be improved.

[0101] Step 140: Determine the image quality of the first image information according to the second feature information.

[0102] Optionally, after obtaining the second feature information, a fully connected layer may be used to regress the second feature information, and finally the image quality score may be obtained. A fully connected layer is a type of layer in a neural network, that is, each neuron is connected to all neurons in the previous layer, and is used to map the features extracted by the previous layer to the final output for final prediction.

[0103] Through the above steps 110 to 140, the first image information can be firstly subjected to feature extraction to obtain the first feature information, and the first weight of the first feature information can be obtained. The spatial importance of the first feature can be learned through the first weight to obtain the second feature information that enhances the first feature information. The image quality can then be determined through the second feature information. In this way, the image quality can be evaluated.

[0104] See also Figure 8 , showing a schematic diagram of the structure of an image quality assessment system provided by the present application, the system can realize the functions of the above steps 110 to 140. The system includes a feature extraction module, a feature enhancement module, and a fully connected FC module. The image I is input into the system, and the system extracts features from the image to obtain the first feature information f i (Refer to the above step 120), enhance the image features to obtain the second feature information f e (Refer to the above step 130), perform full connection regression on the image, and finally obtain the prediction score q of the image, so that the image quality can be evaluated.

[0105] Optional, see Fig. 9 , showing a schematic diagram of a process for determining the second model, the solution of the present application may further include the following steps:

[0106] Step 910: Obtain reference image information.

[0107] The reference image information includes a reference quality value. The reference image information can be obtained from a preset database or the image file can be read from a specified storage location without restriction.

[0108] Step 920: Determine a first quality value of the reference image information.

[0109] That is, feature extraction is performed on the reference image information to obtain first feature information of the reference image information, the first feature information of the reference image information is input into the second model to obtain second feature information output by the second model, and the first quality value of the reference image information is determined according to the second feature information. For details, reference may be made to the above steps 110 to 140, which will not be repeated here.

[0110] Step 930: Calculate a model loss function according to the first quality value and the reference quality value.

[0111] Optionally, the loss function includes one or more of the following: Huber loss function, normalization loss function, and flip consistency loss function. The normalization loss function can further constrain the normalization result to achieve rapid convergence of the model, and the flip consistency loss function can enhance the generalization ability of the model and prevent the second model from overfitting. The calculation methods of the three loss functions are as follows:

[0112] 1) Huber loss function:

[0113] The Huber loss function can be calculated based on the difference between the first quality value and the reference quality value. The Huber loss function can be used to measure the difference between the image quality score predicted by the model and the reference quality score. The formula is as follows:

[0114]

[0115] Among them, δ is the hyperparameter of the Huber loss function, s is the image quality reference value of the reference image, and q is the quality score predicted by the model.

[0116] 2) Normalized loss function:

[0117] A normalized loss function may be calculated based on the first quality value, the reference quality value, the statistics of the first quality value, and the statistics of the reference quality value. Specifically, the normalized loss function may be calculated according to the following steps:

[0118] 1. Calculate statistics of the first value and the reference value respectively, where the statistics may include a mean and an L2 norm.

[0119] 2. Normalize the data based on step 1, that is, subtract the mean from the first value and the reference value and divide by the norm.

[0120] 3. Calculate the difference between the two results.

[0121] Combining steps 1 to 3, we can get the following formula for calculating the normalized loss function:

[0122]

[0123] Where N is the batch size, s i is the image quality reference value of the reference image, is the mean of the reference values, q i is the first value obtained according to the model, is the mean of the first values, ‖·‖ 2 is the L2 norm, ε=1*10 -8 , setting ε can avoid division by zero during the calculation process.

[0124] 3) Flip consistency loss function:

[0125] The reference image is flipped, and the second quality value of the reference image quality after flipping is obtained. The flip consistency loss function is calculated based on the first quality value and the second quality value. In this way, by calculating the first value and the second value and minimizing the error between the two, the image quality scores before and after flipping can be made as close as possible, thereby reducing the risk of model overfitting and enhancing the stability of the model. The formula of the flip consistency loss function is as follows:

[0126] L flip (qqf)=|qq f |

[0127] Among them, q is the first value, q f is the second value.

[0128] Optionally, the Huber loss function, the normalized loss function, and the flip consistency loss function are finally used to calculate the model loss function according to the second weight.

[0129] Through the above scheme, the Huber loss function is used to constrain the consistency between the predicted quality score and the subjective quality score, the normalization loss function is used to further constrain the standardized result, and the generalization ability of the model is enhanced by flipping the consistency loss function to effectively prevent overfitting. In this way, the second model converges quickly through multiple loss functions.

[0130] Step 940: Adjust the parameters of the second model according to the model loss function so that the second model converges.

[0131] Through the above steps 810 to 840, the second model can be converged, and then the converged second model can be used to determine the image quality of the image information (ie, the above steps 110 to 140).

[0132] Through the above steps, the parameters of the second model are adjusted by calculating the loss function, so that the second model can quickly converge to the optimal solution, minimize the difference between the model prediction and the reference quality score, and improve the accuracy of the predicted image quality score.

[0133] The above steps 910 to 940 introduce a method for determining the second model. Another method for determining the second model is described below:

[0134] Step 1: Initialize model parameters, batch size, and training rounds.

[0135] First, initialize the model parameters, such as setting initial values ​​for the weights and biases in the neural network, and setting the number of samples and iterations used for calculation in each iteration.

[0136] Step 2: Input the reference image and obtain primary features through the feature extraction module.

[0137] Step 3: Optimize the features through the feature enhancement module to obtain features suitable for quality regression.

[0138] Step 4: Use a fully connected layer to regress the final features and map them to the quality score of the reference image.

[0139] The above steps 2-4 may refer to the above steps 110-140 to obtain the quality score of the reference image.

[0140] Step 5: Repeat steps 2-4 for each batch, calculate the loss function, and update the parameters using the Stochastic Gradient Descent (SGD) algorithm.

[0141] SGD is an optimization algorithm used to train machine learning models. After each batch of training, the loss function is calculated through the model, and the model parameters are adjusted according to the calculated loss according to the SGD algorithm to reduce the loss and improve the model performance.

[0142] Step 6: Repeat iterative steps 2 to 5 until the model converges or the set number of iterations is reached.

[0143] Through the above scheme, the first image information can be firstly extracted to obtain the first feature information, and the first weight of the first feature information can be obtained. The spatial position importance of the first feature can be learned through the first weight to obtain the second feature information that enhances the first feature information. Then, the image quality can be determined through the second feature information. In this way, the image quality can be evaluated.

[0144] Based on the same concept, an embodiment of the present application further provides an image quality determination device, which can execute the prediction method introduced in the above content.

[0145] See also Fig.10 , a schematic diagram of the structure of an image quality determination device provided in an embodiment of the present application is given, such as Fig.10 As shown, the prediction device comprises:

[0146] An acquisition module 1001 is used to acquire first image information;

[0147] The feature extraction module 1002 is used to extract features from the first image information to obtain first feature information;

[0148] The feature enhancement module 1003 is used to perform convolution processing on the first feature information to obtain first weight information, and use the first weight information to perform feature enhancement on the first feature information to obtain second feature information;

[0149] The determination module 1004 is configured to determine the image quality of the first image information according to the second feature information.

[0150] In one possible implementation, there are multiple first feature information, different first feature information corresponds to different spatial position features, there are multiple first weight information, and the feature enhancement module 1003 is specifically used to: use the first weight information to weighted sum the first feature information to obtain the third feature information, calculate the sum of the first weights, divide the third feature information by the sum, and obtain the second feature information.

[0151] In a possible implementation, the feature extraction module 1002 is specifically used to: perform convolution processing on the first image information, perform residual processing on the image information after the convolution processing, and perform convolution processing on the image information after the residual processing again to obtain first feature information of the first image information.

[0152] In a possible implementation, the feature extraction module 1002 is specifically used to: input the first image information into the first model, and obtain the first feature information output by the first model, wherein the first model includes a first convolutional network module, a second convolutional network module and a bottleneck module, the bottleneck module is located between the first convolutional network module and the second convolutional network module, the first convolutional network module includes a 3*3 convolutional layer, a first normalization layer and a first activation function layer connected in sequence, the bottleneck module includes a plurality of bottleneck units with an inverted residual structure connected in sequence, and the second convolutional network module includes a 1*1 convolutional layer, a second normalization layer and a second activation function layer connected in sequence.

[0153] In one possible implementation, the feature enhancement module 1003 is specifically used to: input the first feature information into the second model to obtain the second feature information output by the second model, wherein the second model includes a third convolutional network module, two groups of channels and a data processing module, wherein the input end of one group of channels is connected to the output end of the third convolutional network module and the input end of the second model, the input end of the other group of channels is connected to the output end of the third convolutional network module, and the output ends of the two groups of channels are connected to the data processing module.

[0154] In one possible implementation, the feature enhancement module 1003 is also used to: obtain reference image information, the reference image information includes a reference quality value, perform feature extraction on the reference image information to obtain first feature information of the reference image information, and input the first feature information of the reference image information into the second model to obtain second feature information of the reference image information output by the second model, determine the first quality value of the reference image information according to the second feature information of the reference image information, determine a calculation model loss function according to the first quality value and the reference quality value, and use the model loss function to adjust the parameters of the second model so that the second model converges.

[0155] In a possible implementation, it also includes a loss function calculation module 1005, which is used to: calculate the Huber loss function according to the difference between the first quality value and the reference quality value, calculate the normalized loss function according to the first quality value, the reference quality value, the statistics of the first quality value and the statistics of the reference quality value, flip the reference image and obtain a second quality value of the quality of the flipped reference image, calculate the flip consistency loss function according to the first quality value and the second quality value, perform weighted averaging on the Huber loss function, the normalized loss function and the flip consistency loss function, and determine the model loss function.

[0156] See also Fig.11 , showing a schematic diagram of the structure of another image quality determination device provided in an embodiment of the present application, such as Fig.11 As shown, the image quality determination device includes: a memory 1101 and a processor 1102, and the processor 1102 is coupled to the memory 1101. The memory 1101 is used to store program instructions, and the processor 1102 is used to call the program instructions stored in the memory 1101 and execute the above-mentioned image quality determination method according to the obtained program.

[0157] Optionally, the data processing device may further include an interface circuit 1103, which may be a transceiver or an input / output interface. The input / output interface is used to input and / or output information, where output may be understood as sending and input may be understood as receiving. The processor 1102 may communicate with other devices in the test device or other devices other than the test device through the interface circuit 1103 to obtain information required for executing the above image quality determination method.

[0158] When the image quality determination device 1100 is used to implement Figure 1 When the method is shown, the processor 1102 is used to implement the functions of the above-mentioned acquisition module 1001, feature extraction module 1002, feature enhancement module 1003, determination module 1004, and loss function calculation module 1005.

[0159] It is understandable that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0160] The memory in the embodiments of the present application can be a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, a register, a hard disk, a mobile hard disk, a compact disc read-only memory (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. The storage medium can also be a component of the processor.

[0161] Based on the same technical concept, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the computer executes the above-mentioned image quality determination method.

[0162] Based on the same technical concept, an embodiment of the present invention further provides a computer-readable program product. When the computer-readable program product is executed, the computer executes the above-mentioned image quality determination method.

[0163] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0164] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0167] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for determining image quality, characterized in that: The method comprises: Acquiring first image information; Performing feature extraction on the first image information to obtain first feature information; Performing convolution processing on the first feature information to obtain first weight information, and using the first weight information to perform feature enhancement on the first feature information to obtain second feature information; The image quality of the first image information is determined according to the second feature information.

2. The method according to claim 1, characterized in that There are multiple first feature information, different first feature information corresponds to different spatial position features, there are multiple first weight information, and using the first weight information to enhance the first feature information to obtain the second feature information includes: Using the first weight information to perform weighted summation on the first feature information to obtain third feature information; Calculating the sum of the first weights; The third characteristic information is divided by the sum to obtain the second characteristic information.

3. The method according to claim 1, characterized in that The extracting features from the first image information to obtain first feature information of the first image information includes: The first image information is subjected to convolution processing, and the image information subjected to convolution processing is subjected to residual processing, and the image information subjected to residual processing is subjected to convolution processing again to obtain first feature information of the first image information.

4. The method according to claim 1, characterized in that The extracting features from the first image information to obtain first feature information of the first image information includes: Inputting the first image information into a first model to obtain first feature information output by the first model; Among them, the first model includes a first convolutional network module, a second convolutional network module and a bottleneck module, and the bottleneck module is located between the first convolutional network module and the second convolutional network module; the first convolutional network module includes a 3*3 convolutional layer, a first normalization layer and a first activation function layer connected in sequence, the bottleneck module includes a plurality of bottleneck units with an inverted residual structure connected in sequence, and the second convolutional network module includes a 1*1 convolutional layer, a second normalization layer and a second activation function layer connected in sequence.

5. The method according to claim 1, characterized in that The step of performing convolution processing on the first feature information to obtain first weight information, and performing feature enhancement on the first feature information using the first weight information to obtain second feature information includes: Inputting the first feature information into a second model to obtain second feature information output by the second model; Among them, the second model includes a third convolutional network module, two groups of channels and a data processing module, wherein the input end of one group of channels is connected to the output end of the third convolutional network module and the input end of the second model, the input end of the other group of channels is connected to the output end of the third convolutional network module, and the output ends of both groups of channels are connected to the data processing module.

6. The method according to claim 5, characterized in that The second model is determined by: Acquiring reference image information, wherein the reference image information includes a reference quality value; performing feature extraction on the reference image information to obtain first feature information of the reference image information, inputting the first feature information of the reference image information into the second model to obtain second feature information of the reference image information output by the second model, and determining a first quality value of the reference image information according to the second feature information of the reference image information; Determining a model loss function according to the first quality value and the reference quality value; The parameters of the second model are adjusted using the model loss function so that the second model converges.

7. The method according to claim 6, characterized in that The determining of a model loss function according to the first quality value and the reference quality value includes: Calculating a Huber loss function according to a difference between the first mass value and the reference mass value; calculating a normalized loss function according to the first quality value, the reference quality value, a statistic of the first quality value, and a statistic of the reference quality value; Flipping the reference image, obtaining a second quality value of the quality of the flipped reference image, and calculating a flip consistency loss function according to the first quality value and the second quality value; The Huber loss function, the normalized loss function, and the flip consistency loss function are weighted averaged to determine the model loss function.

8. An image quality determination device, characterized in that: include: An acquisition module, used for acquiring first image information; A feature extraction module, used to extract features from the first image information to obtain first feature information; a feature enhancement module, configured to perform convolution processing on the first feature information to obtain first weight information, and perform feature enhancement on the first feature information using the first weight information to obtain second feature information; A determination module is used to determine the image quality of the first image information according to the second characteristic information.

9. An image quality determination device, characterized in that: include: A processor, wherein the processor is coupled to a memory, the memory is used to store a computer program or an instruction, and the processor is used to execute the computer program or the instruction to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device, and when the program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 7.

11. A computer program product, characterized in that When the method is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 7.

Images