Image quality evaluation method and apparatus, terminal device, and computer readable medium

Abstract

The application is suitable for the technical field of image processing, and provides an image quality evaluation method and device, terminal equipment and computer readable medium. The method comprises the following steps: extracting a plurality of distortion samples from a distorted image, and extracting a plurality of reference samples from a reference image corresponding to the distorted image, the plurality of distortion samples and the plurality of reference samples corresponding to each other; determining a sample score of each distortion sample according to the plurality of distortion samples and the plurality of reference samples; and determining a prediction score of the distorted image according to the sample score of each distortion sample. Through the above method, the result of image quality evaluation can be more consistent with the subjective feeling of human eyes.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to an image quality evaluation method, apparatus, terminal equipment and computer-readable medium. Background Technology

[0002] Image quality can be used to represent the degree of error a test image produces relative to a standard image in the human visual system. Image quality can be evaluated using a full-reference quality assessment method. Full-reference image quality assessment involves comparing the test image with the reference image, selecting an ideal image as the reference image, analyzing the degree of distortion in the test image, and thus obtaining a quality assessment of the test image.

[0003] Existing full-reference image quality assessment methods include Mean Square Error (MSE), Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity (SSIM). However, these methods only measure pixel differences or structural differences between the reference and distorted images, without considering the filtering effects of the human visual system. Therefore, the image quality assessment results obtained using these methods differ from subjective human perception.

[0004] Alternatively, image quality can be evaluated using no-reference quality assessment methods, such as the Natural Image Quality Evaluator (NIQE). However, no-reference quality assessment methods have very poor generalization ability. Summary of the Invention

[0005] This application provides an image quality evaluation method, apparatus, terminal device, and computer-readable medium, which can solve the problem that there is a large discrepancy between existing image quality evaluation results and subjective perception.

[0006] In a first aspect, embodiments of this application provide an image quality evaluation method, including:

[0007] Multiple distorted samples are extracted from the distorted image, and multiple reference samples are extracted from the reference image corresponding to the distorted image, with each distorted sample corresponding to one of the multiple reference samples;

[0008] The sample score for each distorted sample is determined based on multiple distorted samples and multiple reference samples;

[0009] The predicted score of the distorted image is determined based on the sample score of each distorted sample.

[0010] Secondly, embodiments of this application provide an image quality evaluation device, comprising:

[0011] The sample extraction module is used to extract multiple distorted samples from the distorted image and multiple reference samples from the reference image corresponding to the distorted image, with each distorted sample corresponding to one of the multiple reference samples.

[0012] The sample score determination module is used to determine the sample score of each distorted sample based on multiple distorted samples and multiple reference samples.

[0013] The prediction score determination module is used to determine the prediction score of the distorted image based on the sample score of each distorted sample.

[0014] Thirdly, embodiments of this application provide a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method described in the first aspect above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0016] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the method described in the first aspect.

[0017] The beneficial effects of this application's embodiments compared to existing technologies are as follows: When evaluating the quality of distorted images, distorted sample images of different scales at various locations within the distorted image are used. These are compared with corresponding reference samples of the same size at the corresponding locations in the corresponding reference image. The sample score for each distorted sample is determined, and then the predicted score for the entire distorted image is calculated based on the sample scores of multiple distorted samples. This application evaluates image quality based on distorted samples at different spatial locations and scales within the distorted image. During the evaluation process, distorted samples with greater distortion are highlighted. Since the human eye is more sensitive to areas of significant image distortion, the image quality evaluation results are more consistent with the subjective perception of the human eye. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an image quality evaluation method provided in Embodiment 1 of this application;

[0020] Figure 2 This is a flowchart illustrating an image quality evaluation method provided in Embodiment 2 of this application;

[0021] Figure 3 This is a flowchart illustrating an image quality evaluation method provided in Embodiment 3 of this application;

[0022] Figure 4 This is a schematic diagram of the model structure used in an image quality evaluation system provided in Embodiment 3 of this application;

[0023] Figure 5 This is a schematic diagram of the structure of an image quality evaluation system provided in Embodiment 3 of this application;

[0024] Figure 6 This is a schematic diagram of a test result provided in Embodiment 3 of this application;

[0025] Figure 7 This is a schematic diagram of the structure of an image quality evaluation device provided in Embodiment 4 of this application;

[0026] Figure 8 This is a schematic diagram of the structure of a terminal device provided in Embodiment 5 of this application. Detailed Implementation

[0027] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0028] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0029] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0030] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0031] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0032] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0033] Figure 1 This is a flowchart illustrating an image quality evaluation method provided in Embodiment 1 of this application, as shown below. Figure 1 As shown, the method includes:

[0034] S101, the terminal device extracts multiple distorted samples from the distorted image and multiple reference samples from the reference image corresponding to the distorted image, with each distorted sample corresponding to one of the multiple reference samples.

[0035] The method in this embodiment can be applied to tasks that guide the improvement of image quality or video restoration, such as image denoising, image deblurring, and video restoration; it can also be used to evaluate the degree of image distortion during image compression, image processing, and image transmission.

[0036] Specifically, the aforementioned distorted image refers to an image that requires quality evaluation, and the distorted image can correspond to a clear reference image. In this embodiment, the evaluation of the distorted image is based on the reference image; that is, the reference image is used as a benchmark to evaluate the degree of distortion of the distorted image relative to the reference image.

[0037] Specifically, distorted samples are extracted from different regions of the distorted image, and the resolution of these distorted samples is greater than or equal to a preset minimum resolution. Following the same method as extracting distorted samples from the distorted image, multiple reference samples are extracted from the reference image, so that the extracted reference samples and distorted samples can correspond one-to-one.

[0038] Specifically, 32 distorted samples and 32 reference samples can be extracted to form 32 sample pairs, which can be used to evaluate the quality of distorted images.

[0039] S102, the terminal device determines the sample score of each distorted sample based on multiple distorted samples and multiple reference samples;

[0040] Specifically, the terminal device can compare each distorted sample with its corresponding reference sample to determine the score of each distorted sample.

[0041] Specifically, features can be extracted for each distorted sample and features for each reference sample. These features are then compared, and the difference between the features of each distorted sample and its corresponding reference sample is calculated. The score of the distorted sample is determined based on this difference. The score of the distorted sample is directly proportional to the difference between the distorted sample and its corresponding reference sample; the larger the difference, the higher the score.

[0042] S103, the terminal device determines the predicted score of the distorted image based on the sample score of each distorted sample.

[0043] Specifically, the terminal device can calculate the predicted score of the distorted image based on the score of each distorted sample using a preset calculation method.

[0044] Because distorted samples are located at different positions within a distorted image and vary in size, their degree of distortion may also differ. Based on prior knowledge of the human visual system, the human eye is more sensitive to areas of significant image distortion. Therefore, in image evaluation, samples with higher degrees of distortion can be magnified, making the evaluation results more consistent with subjective human perception. Specifically, when calculating the predicted score of a distorted image, the scores of the distorted samples can be squared. Squaring amplifies the differences in magnitude between data points; therefore, using the squared values ​​of the distorted sample scores to calculate the predicted value of the distorted image allows samples with larger scores to have a greater impact on the predicted score.

[0045] The method in this embodiment can be used to evaluate image restoration models. Specifically, the same image to be restored can be input into different image restoration models, and the restored image obtained by each model can be considered as the distorted image. Using the method in this embodiment, with the same reference image as the comparison object, the prediction score of the distorted image output by each image restoration model is calculated. Comparing the prediction scores of the distorted images corresponding to each image restoration model can determine the image restoration quality of the model. In this embodiment, the smaller the prediction score, the closer the distorted image is to the reference image, meaning the corresponding image restoration model has a better image restoration function.

[0046] The method described in this embodiment can also be used to construct loss functions for tasks such as image inpainting and video super-resolution. In this embodiment, the method evaluates image quality based on human perception. While some images may have similar data, the perceived differences can be significant. Constructing a loss function using the method described in this embodiment ensures that the results of tasks such as image inpainting and video super-resolution align with subjective human perception. In this embodiment, when evaluating distorted images, multiple distorted samples and reference samples are extracted from both the distorted and reference images. Each distorted sample is then compared with its corresponding reference sample to obtain a score. The predicted score of the distorted image is then calculated based on the scores of each distorted sample. Because samples of different positions and sizes within the distorted image are used for image evaluation, the evaluation results better reflect the overall degree of distortion in the image.

[0047] Figure 2 This is a flowchart illustrating an image quality evaluation method provided in Embodiment 2 of this application, as shown below. Figure 2 As shown, the method includes:

[0048] S201, the terminal device extracts multiple distorted samples from the distorted image and multiple reference samples from the reference image corresponding to the distorted image, with each distorted sample corresponding to one of the multiple reference samples.

[0049] The method in this embodiment can be applied to tasks that guide the improvement of image quality or video restoration, such as image denoising, image deblurring, and video restoration; it can also be used to evaluate the degree of image distortion during image compression, image processing, and image transmission.

[0050] Specifically, the aforementioned distorted image refers to an image that requires quality evaluation, and the distorted image can correspond to a clear reference image. In this embodiment, the evaluation of the distorted image is based on the reference image; that is, the reference image is used as a benchmark to evaluate the degree of distortion of the distorted image relative to the reference image.

[0051] Specifically, distorted samples are extracted from different regions of the distorted image, and the resolution of these distorted samples is greater than or equal to a preset minimum resolution. Following the same method as extracting distorted samples from the distorted image, multiple reference samples are extracted from the reference image, so that the extracted reference samples and distorted samples can correspond one-to-one.

[0052] Specifically, 32 distorted samples and 32 reference samples can be extracted to form 32 sample pairs, which can be used to evaluate the quality of distorted images.

[0053] S202, the terminal device extracts a first feature from each distorted sample and a second feature from each reference sample.

[0054] Specifically, a feature extractor composed of a convolutional neural network can be used to extract features from distorted samples and reference samples. Each distorted sample is input into the feature extractor to obtain high-dimensional features for each distorted sample; each reference sample is input into the feature extractor to obtain high-dimensional features for each reference sample.

[0055] Taking a sample with 32 sample pairs and a resolution of 76 as an example, a distorted sample with a data shape of [32, 76, 76, 3] can be input into the feature extractor. Here, 32 represents 32 sample pairs, the two 76s indicate the sample resolution is 76*76, and 3 indicates the input image is an RGB image patch with 3 dimensions. This distorted sample passes through the last convolutional layer of the feature extractor to obtain a feature map with a shape of [32, 2, 2, 256]. After adaptive spatial global average pooling, the original feature map with a shape of [32, 1, 1, 256] is obtained. Furthermore, maintaining the feature dimension of the last convolutional layer, the mean feature is calculated, resulting in a mean feature map with the same shape of [32, 2, 2, 256]. Subtracting the mean feature map from the feature map of the last convolutional layer and squaring the result yields the variance feature map. The mean and variance feature maps are adaptively processed by global average pooling. The result is then concatenated with the previous global average pooling result along the fourth dimension to obtain a feature with the shape [32, 1, 1, 256*3]. This feature is then compressed by removing the two 1-dimensional dimensions, resulting in a feature shape of [32, 756]. This high-dimensional feature is the first feature of the distorted sample. The second feature of the reference sample is obtained using the same method.

[0056] S203, the terminal device calculates the distance feature between each distorted sample and the corresponding reference sample based on the first feature and the second feature.

[0057] Specifically, the quality of a distorted sample is determined by comparing it with a corresponding reference sample. When comparing the differences between the distorted sample and the reference sample, their features can be compared. Therefore, the distance feature between the distorted sample and the reference sample can be calculated, and this distance feature represents the difference between the distorted sample and the reference sample.

[0058] Specifically, the Euclidean distance between the first feature of the distorted sample and the second feature of its corresponding reference sample can be used as the distance feature between the distorted sample and the reference sample.

[0059] S204, The terminal device determines the sample score of the distorted sample corresponding to the distance feature.

[0060] Specifically, since both the first feature and the second feature are high-dimensional features, the distance feature calculated based on the first feature and the second feature is also a high-dimensional feature. The high-dimensional feature can include multiple variables, so it can be fitted to obtain a one-dimensional vector, which can be used as the sample score of the distorted sample.

[0061] Specifically, distance features can be input into a regression network composed of fully connected layers, and the regression network can then output a one-dimensional vector as the sample score of the distorted samples. For example, inputting features of [32, 756] into the regression network will output a one-dimensional vector of shape [32, 1].

[0062] Since multiple distorted samples are extracted from the distorted image, multiple sample scores can be obtained.

[0063] S205, the terminal device determines the prediction score of the distorted image based on the sample score of each distorted sample.

[0064] Specifically, multiple distorted samples may exhibit varying degrees of distortion due to their different locations and scales, resulting in different sample scores. A higher sample score indicates a greater difference between the distorted sample and the reference sample, meaning the distortion is more pronounced. Based on prior knowledge of the human visual system, the human eye is more sensitive to areas of significant image distortion. Therefore, when evaluating distorted images, samples with higher scores can be amplified. For example, the sum of squares of the scores for each distorted sample, along with the sum of all individual distorted sample scores, can be calculated. Then, the ratio of this ratio to the sum of all individual distorted sample scores can be used as the predicted score for the distorted image. Squaring the score widens the gap between large and small sample scores, effectively amplifying the larger sample scores. This means that samples with greater distortion account for a larger proportion of the predicted score, aligning with the human eye's greater sensitivity to areas of significant image distortion.

[0065] In this embodiment, multiple different distortion samples are used to evaluate the image quality. When calculating the prediction score of the distorted image, the distorted samples with greater distortion are amplified, so that the image quality evaluation results are more in line with the subjective perception of the human eye.

[0066] Figure 3 This is a flowchart illustrating an image quality evaluation method provided in Embodiment 3 of this application, as shown below. Figure 3 As shown, the method includes:

[0067] S301, the terminal device extracts multiple distortion samples from multiple locations in the distortion image, and the resolution of each distortion sample is greater than or equal to the preset resolution.

[0068] The image quality assessment method in this embodiment can be based on a pre-trained image quality assessment system. The specific training process of this image quality assessment system can be as follows:

[0069] The training set can be comprised of 20 reference images and their corresponding 2400 distorted images from the TID2013 dataset, while the test set consists of the remaining 5 reference images and their corresponding 600 distorted images. Data normalization is achieved by dividing the pixels of each distorted image and reference image pair by 255. The normalized image pairs are then randomly cropped into 32 sub-images at various resolutions to form multiple sample sets. For example, randomly cropping a sample pair at resolutions of 68, 76, and 84 yields three sets of data with shapes of [32, 68, 68, 3], [32, 76, 76, 3], and [32, 84, 84, 3]. The labels for these three sets are equal to the subjective scores of the distorted images in the original sample pairs. These subjective scores can be the average subjective opinion score of the degree of distortion given by an expert panel. Furthermore, to verify the generalization ability of this scheme, training and testing were performed on the Kadid-10k dataset.

[0070] Figure 4 This is a schematic diagram of the model structure used in the image quality evaluation system provided in Embodiment 3 of this application. The system used in this application... Figure 4 The model shown implements feature extraction and regression of predicted scores. In this scheme, the convolutional layers of the model do not perform edge padding on the features to prevent the model from introducing zero noise into the features. Furthermore, to adapt to image pairs with different resolutions, the model structure performs global average pooling of the spatial dimensions of the image after the last convolutional layer, and calculates the variance and mean features of the obtained features. Then, referring to the features of the image, the mean and variance features are subtracted from the features corresponding to the distorted image to obtain the distance features between the two images. Finally, the distance features are regressed to the predicted score of the distorted sample image through a regression network.

[0071] The structure diagram of the image quality assessment system trained using the above steps can be shown below. Figure 5 As shown. Figure 6 These are the test results of the image quality assessment system on the test set. In the test results, the predicted score of the distorted image is basically linearly correlated with the subjective score given by the expert group. That is, the higher the subjective score of the distorted image, the higher its measured predicted score, proving that the results obtained by using the image quality assessment system are relatively accurate.

[0072] In this embodiment, the Mean Absolute Error (MAE) function can also be used as the loss function, and the Adam optimizer can be used to optimize the regression network. The total loss function can be loss = |pg| + β × L2, where p is the predicted score obtained by the image quality assessment system, g is the average subjective opinion score of the distortion degree of the distorted image given by the expert group, β is a hyperparameter that can be freely set to constrain the generalization of the system, and L2 is the regularization loss of the regression network.

[0073] Specifically, the image quality assessment system provided in this embodiment can be implemented based on TensorFlow. During the training process, the label of each sampled image corresponds to the score of the entire distorted image. In the random sampling process for test samples, the number of samples sampled from each reference distorted image is no longer 32 but 512, and the resolution can be unified to the minimum resolution of 68. Since a smaller resolution contains less information and makes image quality assessment more difficult, the lowest resolution was used during the training of the image quality assessment system. Because this image quality assessment system can evaluate images at the lowest resolution, its results will be more accurate compared to higher resolution images.

[0074] In this embodiment, the distorted image and the corresponding reference image are input into the trained image quality evaluation system, and then the predicted score of the distorted image is output. This predicted score is used to represent the difference between the distorted image and the reference image.

[0075] Therefore, when evaluating images, this quality assessment system also needs to normalize the images and extract samples first.

[0076] Specifically, before evaluating the quality of a distorted image, both the distorted image and its corresponding reference image can be normalized. For example, pixels between 0 and 225 in both the distorted and reference images can be divided by 255, thus constraining all pixels to the range of 0-1 and achieving normalization.

[0077] The higher the image resolution, the more information it contains, and the better it can distinguish whether the image is distorted. The preset resolution mentioned above refers to the minimum resolution set. When the resolution of the distorted sample is lower than this minimum resolution, the distorted sample contains too little information, which may affect the quality evaluation results.

[0078] In this embodiment, when extracting distorted samples, images from different locations in the distorted image are selected, and the resolution of each distorted sample has multiple scales, thereby ensuring that the evaluation results of image quality are more generalizable.

[0079] S302, the terminal device extracts a reference sample from the reference image corresponding to the distorted image, which corresponds to the position and size of each distorted sample.

[0080] Specifically, reference samples are extracted from the reference image using the same method as in step S201. Each reference sample corresponds to a distortion sample, the position of which in the distortion image is the same as the position of the reference sample in the reference image, and the distortion sample and the reference sample have the same resolution. Therefore, the distortion samples and reference samples can be compared one-to-one.

[0081] S303, the terminal device extracts a first feature from each distorted sample and a second feature from each reference sample.

[0082] Specifically, a feature extractor composed of convolutional neural networks can be used to extract features from each distorted sample and the reference sample respectively.

[0083] Specifically, the distorted samples are input into the feature extractor, which performs convolution calculations on the distorted samples to obtain their feature maps. Then, max pooling and global average pooling are applied to these feature maps to obtain the first feature of the distorted sample. Max pooling extracts pixels that are relatively large compared to their surrounding pixels, thus highlighting the more significant differences in the image. Since the resolutions of the distorted samples are different, global average pooling is applied to the feature maps to constrain all distorted samples to the same resolution.

[0084] Taking a sample with 32 sample pairs and a resolution of 76 as an example, a distorted sample with a data shape of [32, 76, 76, 3] can be input into the feature extractor. Here, 32 represents 32 sample pairs, the two 76s indicate the sample resolution is 76*76, and 3 indicates the input image is an RGB image patch with 3 dimensions. This distorted sample passes through the last convolutional layer of the feature extractor to obtain a feature map with a shape of [32, 2, 2, 256]. After adaptive spatial global average pooling, the original feature map with a shape of [32, 1, 1, 256] is obtained. Furthermore, maintaining the feature dimension of the last convolutional layer, the mean feature is calculated, resulting in a mean feature map with the same shape of [32, 2, 2, 256]. Subtracting the mean feature map from the feature map of the last convolutional layer and squaring the result yields the variance feature map. The mean and variance feature maps are adaptively processed by global average pooling. The result is then concatenated with the previous feature map using global average pooling, resulting in a feature of shape [32, 1, 1, 256*3] along the fourth dimension. This feature is then compressed by removing the two 1-dimensional dimensions, resulting in a feature shape of [32, 756]. This high-dimensional feature is the first feature of the distorted sample. The second feature of the reference sample is obtained using the same method.

[0085] S304, the terminal device calculates the distance feature between each distorted sample and the corresponding reference sample based on the first feature and the second feature.

[0086] Specifically, for a distorted sample and its corresponding reference sample, the Euclidean distance between the corresponding first feature and second feature can be used as the distance feature between them.

[0087] S305, The terminal device determines the sample score of the distorted sample corresponding to the distance feature.

[0088] Specifically, the distance features can be input into a regression network for processing, and the output can be a one-dimensional feature of the distorted sample, which is the sample score of the distorted sample corresponding to the distance feature.

[0089] S306, the terminal device calculates the sum of squares of multiple sample scores, and calculates the sum of multiple sample scores, including the sample score of each distorted sample.

[0090] Squaring makes large values ​​larger and small values ​​smaller. Therefore, summing the squares of the predicted scores makes the larger predicted scores even larger, thus giving them a greater weight. Since a larger predicted score indicates a greater degree of distortion, this effectively highlights distorted samples with higher levels of distortion. This aligns with the human eye's greater sensitivity to areas of significant image distortion, making the results more consistent with subjective human perception.

[0091] Specifically, the sum of squares of multiple sample scores is calculated, as is the sum of sample scores of multiple distorted samples.

[0092] S307, The terminal device calculates the ratio between the sum of squares and the sum of scores from multiple samples, and the ratio is the predicted score of the distorted image.

[0093] Specifically, the ratio of the sum of squares of the sample scores of the distorted samples to the sum of the sample scores of multiple distorted samples can be used as the prediction score of the distorted image. This prediction score is used to represent the magnitude of the difference between the distorted image and the reference image. That is, the greater the degree of distortion of the distorted image, the higher the prediction score.

[0094] In this embodiment, the prediction score is used to represent the degree of distortion of the distorted image relative to the reference image. Other methods can also be used to calculate the prediction score; for example, the sum of the squares of multiple sample scores can be directly used as the prediction score. Different calculation methods may yield different prediction score values, but for a given set, all are positively correlated with the degree of distortion of the distorted image—that is, the more pronounced the distortion of the distorted image, the higher its prediction score.

[0095] In this embodiment, when training the image quality assessment system, the training data is spatially multi-scale by sampling the data samples multiple times at different resolutions, which avoids the overfitting problem of deep networks in the image quality assessment system. In addition, the image quality assessment system can use images of multiple scales during training, so it can adapt to the input of images of different scales and has better generalization.

[0096] Figure 7 This is a schematic diagram of the structure of an image quality evaluation device provided in Embodiment 4 of this application, as shown below. Figure 7 As shown, the device includes:

[0097] The sample extraction module 71 is used to extract multiple distorted samples from the distorted image and multiple reference samples from the reference image corresponding to the distorted image, wherein the multiple distorted samples correspond one-to-one with the multiple reference samples;

[0098] The sample score determination module 72 is used to determine the sample score of each distorted sample based on multiple distorted samples and multiple reference samples;

[0099] The prediction score determination module 73 is used to determine the prediction score of the distorted image based on the sample score of each distorted sample.

[0100] The above sample score determination module 72 includes:

[0101] The feature extraction submodule is used to extract a first feature from each distorted sample and a second feature from each reference sample.

[0102] The distance feature calculation submodule is used to calculate the distance feature between each distorted sample and the corresponding reference sample based on the first feature and the second feature, respectively.

[0103] The distortion sample score determination submodule is used to determine the sample score of the distortion samples corresponding to the distance feature.

[0104] The above-mentioned device also includes:

[0105] The preprocessing module is used to divide the pixels of the distorted image and the reference image by a preset value to normalize the distorted image and the reference image.

[0106] The above sample extraction module 71 includes:

[0107] The distortion sample extraction submodule is used to extract multiple distortion samples from multiple locations in the distortion image, and the resolution of each distortion sample is greater than or equal to the preset resolution.

[0108] The reference sample extraction submodule is used to extract reference samples from the reference image corresponding to the distorted image, which correspond to the position and size of each distorted sample.

[0109] The above feature extraction submodule includes:

[0110] The first convolutional computation unit is used to perform convolutional computation on each distorted sample to obtain the feature map of each distorted sample;

[0111] The first pooling processing unit is used to sequentially perform max pooling and global average pooling on the feature map of each distorted sample to obtain the first feature of each distorted sample.

[0112] The above feature extraction submodule also includes:

[0113] The second convolution calculation unit is used to perform convolution calculation on each reference sample to obtain the feature map of each reference sample;

[0114] The second convolutional computation unit is used to sequentially perform max pooling and global average pooling on the feature map of each reference sample to obtain the second feature of each reference sample. The aforementioned distance feature determination submodule includes:

[0115] The distance feature calculation unit is used to calculate the Euclidean distance between each distorted sample and the corresponding reference sample based on the first feature and the second feature, respectively. The Euclidean distance is the distance feature between the distorted sample and the corresponding reference sample.

[0116] The above sample score determination module 72 includes:

[0117] The one-dimensional feature calculation submodule is used to input distance features into the regression network for processing and output one-dimensional features of the distorted samples. The one-dimensional features are used as the sample scores of the distorted samples.

[0118] The aforementioned predicted score calculation module 73 includes:

[0119] The first calculation submodule is used to calculate the sum of squares of multiple sample scores, and to calculate the sum of multiple sample scores, including the sample score of each distorted sample;

[0120] The second calculation submodule is used to calculate the ratio between the sum of squares and the sum of scores from multiple samples, where the ratio is the predicted score of the distorted image.

[0121] The above-mentioned device also includes:

[0122] The image inpainting module is used to input the image to be inpainted into multiple image inpainting models to be evaluated.

[0123] The distortion image acquisition module is used to acquire the distortion image output by each image restoration model;

[0124] The evaluation module is used to determine the image restoration quality of each of the image restoration models to be evaluated based on the prediction scores of each of the distorted images.

[0125] Figure 8 This is a schematic diagram of the structure of a terminal device provided in Embodiment 5 of this application. Figure 8 As shown, the terminal device 8 of this embodiment includes: at least one processor 80 ( Figure 8 (Only one is shown) a processor, a memory 81, and a computer program 82 stored in the memory 81 and executable on at least one processor 80, wherein the processor 80 executes the computer program 82 to implement the steps in any of the above method embodiments.

[0126] Terminal device 8 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This terminal device may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will understand that... Figure 8This is merely an example of terminal device 8 and does not constitute a limitation on terminal device 8. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0127] The processor 80 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0128] In some embodiments, memory 81 may be an internal storage unit of the terminal device 8, such as a hard disk or memory of the terminal device 8. In other embodiments, memory 81 may be an external storage device of the terminal device 8, such as a plug-in hard disk, SmartMediaCard (SMC), SecureDigital (SD) card, or FlashCard equipped on the terminal device 8. Furthermore, memory 81 may include both internal and external storage units of the terminal device 8. Memory 81 is used to store operating systems, applications, bootloaders, data, and other programs, such as program code for computer programs. Memory 81 can also be used to temporarily store data that has been output or will be output.

[0129] It should be noted that the information interaction and execution process between the above systems / modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0130] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0131] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0132] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0133] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0134] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0135] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0136] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of systems or units may be electrical, mechanical, or other forms.

[0137] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0138] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An image quality assessment method, characterized in that, include: Multiple distorted samples are extracted from the distorted image, and multiple reference samples are extracted from the reference image corresponding to the distorted image, wherein the multiple distorted samples correspond one-to-one with the multiple reference samples; the distorted samples are located in different positions in the distorted image and have different scales. A sample score is determined for each distorted sample based on the plurality of distorted samples and the plurality of reference samples; the higher the sample score, the greater the degree of distortion of the distorted sample. Determining the prediction score of the distorted image based on the sample score of each distorted sample includes: increasing the proportion of distorted samples with larger sample scores in the prediction score; the prediction score is used to represent the degree of distortion of the distorted image relative to the reference image. The step of determining the sample score of each distorted sample based on the plurality of distorted samples and the plurality of reference samples includes: Extract a first feature from each distorted sample and extract a second feature from each reference sample, wherein both the first and second features are high-dimensional features; Based on the first feature and the second feature, the distance feature between each distorted sample and the corresponding reference sample is calculated respectively; The distance feature is input into a regression network for processing, and the one-dimensional feature of the distorted sample is output. The one-dimensional feature is the sample score of the distorted sample corresponding to the distance feature.

2. The method as described in claim 1, characterized in that, Prior to extracting multiple distorted samples from the distorted image, the method further includes: The pixels of the distorted image and the reference image are divided by a preset value to normalize the distorted image and the reference image.

3. The method as described in claim 2, characterized in that, The step of extracting multiple distorted samples from a distorted image and extracting multiple reference samples from a reference image corresponding to the distorted image includes: Multiple distortion samples are extracted from multiple locations in the distorted image, and the resolution of each distortion sample is greater than or equal to a preset resolution. Extract reference samples corresponding to the position and size of each distorted sample from the reference image corresponding to the distorted image.

4. The method according to any one of claims 1-3, characterized in that, The step of extracting the first feature from each distorted sample includes: Convolution is performed on each distorted sample to obtain the feature map of each distorted sample; Max pooling and global average pooling are performed sequentially on the feature map of each distorted sample to obtain the first feature of each distorted sample.

5. The method as described in claim 4, characterized in that, The extraction of the second feature from each reference sample includes: Convolution is performed on each reference sample to obtain the feature map of each reference sample; Max pooling and global average pooling are performed sequentially on the feature map of each reference sample to obtain the second feature of each reference sample.

6. The method as described in claim 5, characterized in that, The step of calculating the distance feature between each distorted sample and the corresponding reference sample based on the first feature and the second feature includes: Based on the first feature and the second feature, the Euclidean distance between each distorted sample and the corresponding reference sample is calculated, whereby the Euclidean distance is the distance feature between the distorted sample and the corresponding reference sample.

7. The method as described in claim 1, characterized in that, The step of calculating the predicted score of the distorted image based on the sample score of each distorted sample includes: Calculate the sum of squares of multiple sample scores, and calculate the sum of the multiple sample scores, wherein the multiple sample scores include the sample score of each distorted sample; Calculate the ratio between the sum of squares and the sum of the scores of the plurality of samples, where the ratio is the predicted score of the distorted image.

8. The method according to any one of claims 1-3 or 5-7, characterized in that, Before extracting multiple distorted samples from a distorted image, the method further includes: The image to be repaired is input into multiple image repair models to be evaluated; Obtain the distorted image output by the image restoration model for each image to be evaluated; Accordingly, after determining the predicted score of the distorted image based on the sample score of each distorted sample, the method further includes: The image restoration quality of each of the image restoration models to be evaluated is determined based on the predicted scores of each of the distorted images.

9. An image quality evaluation device, characterized in that, include: The sample extraction module is used to extract multiple distorted samples from a distorted image and multiple reference samples from a reference image corresponding to the distorted image, wherein the multiple distorted samples correspond one-to-one with the multiple reference samples; the distorted samples are located in different positions in the distorted image and have different scales. The sample score determination module is used to determine the sample score of each distorted sample based on the plurality of distorted samples and the plurality of reference samples; the larger the sample score, the greater the degree of distortion of the distorted sample. A prediction score determination module is used to determine the prediction score of the distorted image based on the sample score of each distorted sample, including: increasing the proportion of distorted samples with larger sample scores in the prediction score; the prediction score is used to represent the degree of distortion of the distorted image relative to the reference image; The sample score determination module includes: The feature extraction submodule is used to extract a first feature from each distorted sample and a second feature from each reference sample, wherein both the first feature and the second feature are high-dimensional features. The distance feature calculation submodule is used to calculate the distance feature between each distorted sample and the corresponding reference sample based on the first feature and the second feature, respectively. The one-dimensional feature calculation submodule is used to input distance features into the regression network for processing and output one-dimensional features of the distorted samples, wherein the one-dimensional feature is the sample score of the distorted sample corresponding to the distance feature.

10. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 8.

Images