Image quality evaluation method and device and storage medium

Abstract

The invention provides an image quality evaluation method and device and a storage medium, and the method comprises the steps: obtaining a to-be-evaluated target image, carrying out the fuzzy processing of the target image, and generating a blurred image corresponding to the target image; extracting an original image feature of the target image and a blurred image feature of the blurred image, and generating a fusion feature of the original image feature and the blurred image feature; and generating quality evaluation information of the target image based on the fusion features. According to the technical scheme provided by the invention, the accuracy of the image quality evaluation information can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image quality evaluation method, device and storage medium. Background Art

[0002] Image super-resolution algorithms can reconstruct super-resolution (SR) images based on original low-resolution (LR) images. Currently, image super-resolution algorithms can be applied in many application scenarios. For example, in the application scenario of video transmission, in order to save data traffic, the server storing video resources can send low-resolution videos to the user's terminal device. After the user's terminal device receives the low-resolution video, the image super-resolution algorithm can be used to reconstruct the low-resolution video into a super-resolution video. This ensures the user's viewing experience while saving the traffic required to download data.

[0003] In order to measure the quality of the reconstructed super-resolution image, a reliable image quality evaluation method is needed. However, some current image quality evaluation methods usually assume that the quality of SR images is lower than that of native high-resolution (HR) images when determining the quality of SR images. In fact, the quality of SR images is not necessarily lower than that of HR images, which results in the low accuracy of some existing image quality evaluation methods.

[0004] In view of this, a more accurate image quality evaluation method is currently needed. Summary of the invention

[0005] In view of this, one or more embodiments of the present disclosure provide an image quality assessment method, device, and storage medium, which can improve the accuracy of image quality assessment.

[0006] On the one hand, the present disclosure provides an image quality assessment method, the method comprising: obtaining an image training set, the image training set at least comprising a plurality of image samples sorted according to image quality; for a first image sample and a second image sample with different image qualities in the image training set, blurring the first image sample and the second image sample respectively, and generating first quality assessment information of the first image sample and second quality assessment information of the second image sample respectively according to the blurring result through a preset twin model; identifying a quality sorting label between the first image sample and the second image sample, and generating sorting error information according to the quality sorting label, the first quality assessment information and the second quality assessment information, and correcting the preset twin model using the sorting error information; wherein any prediction branch in the preset twin model that has completed the correction is used as an image quality assessment model; obtaining a target image to be evaluated, blurring the target image to generate a blurred image corresponding to the target image; extracting original image features of the target image and blurred image features of the blurred image respectively; generating fusion features of the original image features and the blurred image features using the image quality assessment model, and generating quality assessment information of the target image based on the fusion features.

[0007] On the other hand, the present disclosure further provides an image quality evaluation device, the device comprising: a training set acquisition unit, used to acquire an image training set, the image training set at least comprising a plurality of image samples sorted according to image quality; a model processing unit, used to blur a first image sample and a second image sample with different image qualities in the image training set, respectively, and generate first quality evaluation information of the first image sample and second quality evaluation information of the second image sample according to the blurring result through a preset twin model; a model correction unit, used to identify a quality ranking label between the first image sample and the second image sample, and to correct the quality of the first image sample and the second image sample according to the quality ranking label. , the first quality evaluation information and the second quality evaluation information generate sorting error information, and use the sorting error information to correct the preset twin model; wherein, any prediction branch in the preset twin model that has completed the correction is used as an image quality evaluation model; an image acquisition unit, used to acquire a target image to be evaluated, and blur the target image to generate a blurred image corresponding to the target image; a feature extraction unit, used to respectively extract the original image features of the target image and the blurred image features of the blurred image; an information generation unit, used to use the image quality evaluation model to generate a fusion feature of the original image features and the blurred image features, and generate quality evaluation information of the target image based on the fusion feature.

[0008] On the other hand, the present disclosure further provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the above-mentioned image quality assessment method is implemented.

[0009] On the other hand, the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the above-mentioned image quality evaluation method is implemented.

[0010] The technical solution provided by one or more embodiments of the present disclosure is that the image samples in the image training set can be sorted according to the image quality. When generating the quality evaluation information of the image samples, the blurring processing results of the image samples can be used as reference information. The blurring processing results weaken the high-frequency texture information in the image samples, making the preset twin model more sensitive to the high-frequency texture information. The preset twin model can output the quality evaluation information of different image samples respectively. By comparing the quality evaluation information output by the preset twin model with the quality ranking label of the image sample, the ranking error information can be determined. The ranking error information can correct the preset twin model so that the quality evaluation information output by the corrected preset twin model matches the quality ranking label.

[0011] The technical solution provided by one or more embodiments of the present disclosure can improve the sensitivity of the preset twin model to image quality by training the preset twin model with image samples sorted by image quality and introducing fuzzy processing results as reference information, thereby improving the accuracy of image quality evaluation information generated by the preset twin model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The features and advantages of the embodiments of the present disclosure may be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0013] Figure 1 A schematic diagram showing steps of a training method for an image quality assessment model in one embodiment of the present disclosure is shown;

[0014] Figure 2 A schematic diagram of the structure of a preset twin model in one embodiment of the present disclosure is shown;

[0015] Figure 3 A schematic diagram of a feature extraction module in one embodiment of the present disclosure is shown;

[0016] Figure 4 A schematic diagram of a feature cross-fusion module in one embodiment of the present disclosure is shown;

[0017] Figure 5 A schematic diagram showing the steps of an image quality evaluation method in one embodiment of the present disclosure is shown;

[0018] Figure 6 A schematic diagram of functional modules of an image quality assessment device in one embodiment of the present disclosure is shown;

[0019] Figure 7 A schematic structural diagram of an electronic device in one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0021] Image super-resolution algorithms are used in many practical scenarios. For example, on a video playback platform, in order to save user traffic, the original high-resolution video can be downsampled to a low-resolution video, and then the low-resolution video can be sent to the user's terminal device. At the user's terminal device, the image super-resolution algorithm can be used to reconstruct the low-resolution video into a super-resolution video.

[0022] In practical applications, in order to measure the quality of an image processed by a super-resolution algorithm and the original high-resolution image, or to compare the image processing effects of different super-resolution algorithms, quantitative quality evaluation information can be generated through an image quality evaluation method.

[0023] Specifically, in some application scenarios, it is necessary to ensure the user's viewing experience while saving the user's data download traffic. In this case, the quality of the super-resolution image reconstructed by the super-resolution algorithm at the user's terminal device should not be lower than that of the original high-resolution image. Through the image quality evaluation method, the super-resolution image and the original high-resolution image can be processed separately to generate their respective quality evaluation information. By comparing the quality evaluation information, it can be determined whether the current super-resolution algorithm can guarantee the user's viewing experience.

[0024] In addition, in other application scenarios, when different super-resolution algorithms are used to process images, the image quality evaluation method can be used to evaluate the quality of images processed by each super-resolution algorithm, so as to determine which super-resolution algorithm has a better effect.

[0025] In the related art, image quality assessment methods can be roughly divided into full reference (FR) assessment methods, reduced reference (RR) assessment methods and no reference (NR) assessment methods. These image quality assessment methods can usually be implemented based on deep learning algorithms and training a large number of image samples in an image data set.

[0026] Among them, the full reference evaluation method requires that the super-resolution image and the original high-resolution image be input into the neural network model, and the high-resolution image is used as a reference to generate the quality evaluation information of the super-resolution image. During training, this method assumes that the quality of the original high-resolution image is better than that of the super-resolution image, but in actual applications, the quality of the super-resolution image may be better than that of the high-resolution image, resulting in inaccurate quality evaluation information generated in the end.

[0027] The semi-reference evaluation method usually inputs low-resolution images and super-resolution images into the neural network model together, and uses the low-resolution image as a reference to generate quality evaluation information for the super-resolution image. However, in this scenario, the size of the low-resolution image cannot be fixed. For example, there may be different transcoded versions of images such as 270P or 360P, and the neural network model cannot flexibly adapt the size of the low-resolution image, resulting in low efficiency in model training.

[0028] The no-reference evaluation method usually only needs to input the super-resolution image into the neural network model, and by comparing the quality evaluation information output by the model with the quality evaluation information annotated by the super-resolution image, the neural network model can be corrected so that the quality evaluation information output by the model approaches the quality evaluation information annotated for the super-resolution image at the beginning. However, due to the lack of other types of images as reference information, this method makes it impossible for the model obtained by actual training to accurately predict the quality evaluation information.

[0029] In view of this, an embodiment of the present disclosure provides an image quality assessment method, which can construct an image quality assessment model with high prediction accuracy and wide adaptability based on fuzzy processing technology and twin network technology.

[0030] Specifically, see Figure 1 In one embodiment, the training method of the image quality assessment model may include the following steps.

[0031] S1: Acquire an image training set, wherein the image training set at least includes a plurality of image samples sorted according to image quality.

[0032] In this embodiment, an image training set for training an image quality assessment model may be constructed, and the image training set may include multiple image samples sorted according to image quality. Specifically, the image samples in the image training set may be sorted in order from best to worst or from worst to best image quality.

[0033] In practical applications, the presentation form of image samples in an image training set and the process of determining the quality ranking between image samples can be achieved in a variety of ways.

[0034] In an application scenario, the image samples in the image training set can be presented in the form of sample pairs. For example, the image training set may include several groups of image sample pairs, and each group of image sample pairs may include two image samples of different qualities. The quality of the image samples in each group of image sample pairs can be identified by professionals. In this way, the two image samples in each group of image sample pairs can be sorted according to the identification results of professionals. Of course, there are many other ways to determine the quality of image samples in actual applications, and the present disclosure does not limit this, as long as the image samples in the image sample pairs can be sorted according to image quality.

[0035] In another application scenario, the image samples in the image training set can be obtained by downsampling a series of high-resolution images, and the image quality can also gradually decrease as the downsampling multiple increases. Taking one of the 1080P high-resolution images as an example, the high-resolution image can be downsampled by 2 times, 3 times, and 4 times respectively, so as to obtain 540P, 360P, and 270P downsampled images. As the downsampling multiple increases, the image quality will gradually decrease. In this way, together with the original 1080P high-resolution image, four image samples with quality sorting relationships can be obtained. By performing the above-mentioned downsampling operation on a large number of high-resolution images, multiple groups of image sample sequences can be obtained, and each group of image sample sequences can include the original high-resolution image and the derivative image obtained by downsampling based on the high-resolution image, and each image in each group of image sample sequences can be sorted according to image quality.

[0036] Of course, with the advancement of technology and the change of application scenarios, the composition of image samples in the image training set and the method of determining the quality ranking relationship can also be more diversified, and the present disclosure does not limit this.

[0037] S3: For the first image sample and the second image sample with different image qualities in the image training set, blur processing is performed on the first image sample and the second image sample respectively, and first quality evaluation information of the first image sample and second quality evaluation information of the second image sample are generated respectively according to the blur processing results through a preset twin model.

[0038] In this embodiment, if the quality of an original image is low (blurry), then after the original image is blurred and the blurred image is obtained, the similarity between the blurred image and the original image is usually high. If the quality of an original image is high (clearer), then after the original image is blurred and the blurred image is obtained, the similarity between the blurred image and the original image is usually low. The reason is that the factors that can intuitively improve the image quality are usually reflected in the texture features of the image, and the texture features are generally distributed in the high frequency. After the image is blurred, the high-frequency information of the image will be filtered out to a certain extent. Then for an image that is not of high quality, its high-frequency information is not very rich, and after blurring, the content of the image will not be greatly changed, so the similarity between the blurred image and the original image is high. However, for an image that is of high quality, its high-frequency information is relatively rich. After blurring, since part of the high-frequency information is filtered out, the content of the image will be changed more obviously, so the similarity between the blurred image and the original image is low.

[0039] Based on the above content, in this embodiment, the blurred image can be used as reference information to assist in determining whether the quality of an image sample itself is high. Specifically, the technical solution of this embodiment is described by taking the first image sample and the second image sample of different quality in the image training set as an example. The first image sample and the second image sample can be two image samples in the image sample pair in the above embodiment, or two different image samples in the image sample sequence.

[0040] In this embodiment, after selecting the first image sample and the second image sample with different image qualities from the image training set, the first image sample and the second image sample can be blurred respectively, so as to obtain the first blurred sample and the second blurred sample. Among them, the blurring method can be various. For example, the first image sample and the second image sample can be blurred by Gaussian blurring, salt and pepper blurring, motion blurring, etc. The specific blurring method can be flexibly selected according to different actual application scenarios.

[0041] In this embodiment, after obtaining the result of the blur processing, the first quality evaluation information of the first image sample and the second quality evaluation information of the second image sample can be generated by preset twin models.

[0042] For details, please refer to Figure 2 , the preset twin model may include a first prediction branch and a second prediction branch, wherein the first prediction branch may process the first image sample and the first blurred sample, thereby predicting the first quality evaluation information of the first image sample; the second prediction branch may process the second image sample and the second blurred sample, thereby predicting the second quality evaluation information of the second image sample. The first prediction branch and the second prediction branch may adopt exactly the same network architecture, and the model parameters in the network architecture may also be updated synchronously, so the first prediction branch and the second prediction branch may be regarded as the same two branches.

[0043] like Figure 2 As shown, taking the first prediction branch as an example, in the first prediction branch, a feature extraction module, a feature cross fusion module and an evaluation regression module may be included. Among them, the feature extraction module can extract the first original image feature of the first image sample, and extract the first blurred image feature of the first blurred sample. In actual application scenarios, the feature extraction module can also be implemented in various ways. For example, the feature extraction module can be implemented by networks such as VGG, ResNet, Inception, and DenseNet.

[0044] In one embodiment, in order to improve the accuracy of feature extraction, a Swin-Transformer structure can be used to implement the feature extraction module. Figure 3 , the Swin-Transformer structure can include multiple stages, and feature maps of different sizes are constructed through multiple stages. In practical applications, the input image data can be partitioned through the patch partitioning module (Patch Partition), and then in the first stage (stage1), the input block data can be processed by the linear embedding layer (Linear Embedding), and in the subsequent multiple stages, the block data can be downsampled by the patch fusion layer (Patch Merging). In each stage, the Swin-Transformer Block can be stacked repeatedly. Among them, the specific structure of the Swin-Transformer Block can adopt the W-MSA structure or the SW-MSA structure. Generally speaking, these two structures are used in pairs, that is, a W-MSA structure is used first, and then a SW-MSA structure is used, so the number of stacked Swin-Transformer Blocks in each stage is generally an even number.

[0045] In this embodiment, the feature extraction module is implemented by using the Swin-Transformer structure, which can fuse multi-stage image features, thereby improving the ability of features to express image data.

[0046] The feature cross fusion module can be implemented using the Cross-Attention Feature Fusion (CAFF) model. Figure 4 , the CAFF model may include multiple multi-head self-attention units, and the input end of the multi-head self-attention unit may include a query (Query, Q) input end, a key (Key, K) input end, and a value (Value, V) input end. Among them, the first original image feature extracted by the feature extraction module can be firstly processed by the first multi-head self-attention unit for self-attention, so as to generate the key feature and value feature corresponding to the first original image feature. Subsequently, the first fuzzy image feature extracted by the feature extraction module can be used as a query feature, and input into the second multi-head self-attention unit together with the key feature and value feature generated above, and cross-attention processing is performed. When performing cross-attention processing, the first fuzzy image feature can be used as a query condition and matched with the first original image feature for similarity. If the similarity obtained by matching is high, it indicates that the first image sample is similar to the first fuzzy sample, and further indicates that the image quality of the first graphic sample is low. If the similarity obtained by matching is low, it indicates that the first image sample is not very similar to the first fuzzy sample, and further indicates that the image quality of the first graphic sample is high. In this way, after the result output by the second multi-head self-attention unit is processed by the Multi-Layer Perceptron (MLP), the first fusion feature of the first image sample and the first blurred sample can be obtained. Subsequently, the evaluation regression module can perform regression calculation on the first fusion feature to obtain the first quality evaluation information corresponding to the first image sample.

[0047] In practical applications, the first quality evaluation information may be a numerical value indicating image quality, where a higher numerical value indicates better image quality. Of course, the first quality evaluation information may also have other forms, such as characters such as "excellent", "good", "medium", and "poor" that represent the meaning of image quality. The form of expression of the first quality evaluation information may be flexibly selected according to different requirements for image quality evaluation accuracy in practical application scenarios.

[0048] Similarly, the structure and specific processing process of the second prediction branch are similar to those of the first prediction branch. Through the second prediction branch of the preset twin model, the second fusion features of the second image sample and the second fuzzy sample can be generated, and the second quality evaluation information of the second image sample can be generated based on the second fusion features. Among them, the second original image features of the second image sample and the second fuzzy image features of the second fuzzy sample can be extracted respectively, and the second original image features can be self-attention processed to generate key features and value features corresponding to the second original image features. Subsequently, the second fuzzy image features can be used as query features, cross-attention processed with the key features and value features, and the processed features can be used as second fusion features. For the specific implementation process, please refer to the description of the first prediction branch, which will not be repeated here.

[0049] S5: Identify the quality ranking label between the first image sample and the second image sample, generate ranking error information according to the quality ranking label, the first quality evaluation information and the second quality evaluation information, and use the ranking error information to correct the preset twin model.

[0050] In this embodiment, the first quality evaluation information and the second quality evaluation information can be output respectively by the two prediction branches of the preset twin model, and the quality of the first image sample and the second image sample can be obtained by comparing the first quality evaluation information and the second quality evaluation information. However, the quality determined based on the quality evaluation information output by the preset twin model may be different from the actual quality of the first image sample and the second image sample. For example, in the image training set, the quality of the first image sample is better than the quality of the second image sample, but the first quality evaluation information and the second quality evaluation information output by the preset twin model may indicate that the quality of the first image sample is worse than the quality of the second image sample. In view of this, the result output by the preset twin model can be compared with the actual standard result to generate sorting error information, and the preset twin model can be corrected by the sorting error model, so that the judgment result output by the corrected preset twin model can be consistent with the actual standard result.

[0051] Specifically, according to the quality ranking in the image training set, a quality ranking label between the first image sample and the second image sample can be identified, and the quality ranking label can indicate which image sample has better quality between the first image sample and the second image sample. Subsequently, the ranking error information can be generated according to the quality ranking label and the first quality evaluation information and the second quality evaluation information output by the preset twin model.

[0052] In one embodiment, the quality ranking label may be represented as a specific numerical value. For example, if the image quality of the first image sample is higher than the image quality of the second image sample, a quality ranking label representing a first numerical value may be obtained; if the image quality of the first image sample is lower than the image quality of the second image sample, a quality ranking label representing a second numerical value may be obtained; wherein the first numerical value and the second numerical value are different numerical values. For example, the first numerical value may be 1, and the second numerical value may be -1.

[0053] In this embodiment, when generating the sorting error information, the difference between the first evaluation score represented by the first quality evaluation information and the second evaluation score represented by the second quality evaluation information can be calculated. For example, the first evaluation score can be represented as ps1, and the second evaluation score can be represented as ps2, then the difference can be represented as ps1-ps2.

[0054] Then, a candidate error parameter can be generated according to the difference and the numerical value represented by the quality ranking label. Specifically, the candidate error parameter can be expressed as -yr(ps1-ps2). Finally, the candidate error parameter and the preset reference coefficient can be compared, and the ranking error information can be generated based on the comparison result. Specifically, the candidate error parameter or the preset reference coefficient can be used as the numerical value represented by the ranking error information by taking the maximum value. For example, the ranking error information loss_rank can be expressed as:

[0055] loss_rank=max(0,-yr(ps1-ps2))

[0056] Among them, 0 can be used as the above-mentioned preset reference coefficient. It can be seen that in this way, when the sorting result predicted by the preset twin model is consistent with the actual sorting result, then -yr(ps1-ps2) is a negative value, and the sorting error information is 0, indicating that there is no need to correct the preset twin model. If the sorting result predicted by the preset twin model is inconsistent with the actual sorting result, then -yr(ps1-ps2) is a positive value, and the sorting error information is -yr(ps1-ps2), and the preset twin model can be corrected through this sorting error information.

[0057] In one embodiment, in order to more finely distinguish the quality differences between image samples, a margin coefficient may be introduced when determining the ranking error information. The margin coefficient may be a constant, such as 0.1. The above ranking error information loss_rank may be optimized as follows:

[0058] loss_rank=max(0,-yr(ps1-ps2)+margin)

[0059] In this way, when ps1 and ps2 are close, even if the ranking result predicted by the preset twin model is consistent with the actual ranking result, after adding the edge coefficient, -yr(ps1-ps2)+margin may still be a positive value. In this case, non-zero ranking error information will still be generated, so the preset twin model still needs to be corrected. The purpose of this treatment is to enhance the preset twin model's ability to distinguish subtle differences in different image samples, thereby improving the precision and accuracy of image quality evaluation information.

[0060] In one embodiment, considering that the first prediction branch and the second prediction branch in the preset twin model are the same branch, when the preset twin model is corrected using the sorting error information, the first prediction branch or the second prediction branch can be corrected. After one of the prediction branches is corrected, the first prediction branch and the second prediction branch can synchronously update the corrected model parameters to maintain the consistency of the two prediction branches.

[0061] In one embodiment, the image training set may further include more types of image samples. For example, the image training set may include a plurality of third image samples annotated with quality evaluation coefficients. The quality evaluation coefficient may be a value that can characterize the quality of the third image sample after the third image sample is evaluated by professionals. The higher the quality evaluation coefficient, the higher the quality of the third image sample.

[0062] For this type of image sample, since it does not involve the quality sorting between image samples, any prediction branch in the preset twin model can be used to process the third image sample and output the prediction evaluation coefficient of the third image sample. Specifically, the third image sample can be blurred first to generate a blurred image corresponding to the third image sample. Then, the original image features of the third image sample and the blurred image features of the blurred image can be extracted respectively through any prediction branch in the preset twin model, and a fusion feature of the original image features and the blurred image features can be generated. Then, the prediction evaluation coefficient of the fusion feature can be generated through the prediction branch. The specific implementation method of generating the fusion feature and generating the prediction evaluation coefficient is similar to the process of generating the first fusion feature and the second fusion feature, and generating the first quality evaluation information and the second quality evaluation information in the aforementioned steps, and will not be repeated here.

[0063] In this embodiment, after the prediction evaluation coefficient is generated, coefficient error information can be generated according to the prediction evaluation coefficient and the quality evaluation coefficient of the third image sample, wherein the coefficient error information can be represented by a cross entropy loss coefficient or a mean square error coefficient, for example.

[0064] In one embodiment, after obtaining the coefficient error information, the prediction branch can be corrected based on the coefficient error information. In addition, in order to improve the training accuracy of the preset twin model, the coefficient error information and the sorting error information can be summarized to generate comprehensive error information, and the preset twin model can be corrected using the comprehensive error information. Specifically, the coefficient error information and the sorting error information can be weighted and summed to obtain the comprehensive error information. The comprehensive error information can correct a prediction branch in the preset twin model. After the prediction branch is corrected, the corrected model parameters can be synchronously updated to another prediction branch to ensure the consistency of the two prediction branches.

[0065] It can be seen from the description of the above embodiments that in some specific application scenarios, the image samples in the image training set may include the following types:

[0066] Type 1: Image samples annotated with quality evaluation coefficients;

[0067] Type 2: two image samples in an image sample pair sorted by image quality;

[0068] Type 3: A sequence of image samples formed by downsampling a high-resolution image.

[0069] Of course, actual applications may include more types of image samples, which are not listed here.

[0070] For the above-mentioned type 1 image samples, any prediction branch in the preset twin model can be used for processing, and the preset twin model can be corrected based on the coefficient error information; for the above-mentioned type 2 and type 3 image samples, two image samples with different image qualities can be selected from the image sample pair or image sample sequence, and the two image samples can be processed respectively by the two prediction branches of the preset twin model, and the preset twin model can be corrected based on the sorting error information.

[0071] In practical applications, in type 3, as the downsampling multiple increases, the size of image samples continues to shrink. In this way, by training the preset twin model with image samples of different sizes, the preset twin model can predict the quality of images of different sizes, thereby improving the adaptability of the preset twin model.

[0072] In addition, when training the preset twin model with image samples that have a quality ranking relationship, it is not assumed that the quality of high-resolution images is necessarily better than that of super-resolution images. Instead, professionals give quality scores to different image samples from a unified evaluation dimension and rank the image samples based on the quality scores. This allows the preset twin model to output image quality evaluation information more accurately.

[0073] Furthermore, by blurring the original image samples and generating fusion features between the original image samples and the blurred image samples, the reference information is made richer, allowing the preset twin model to focus on learning texture features that can characterize image quality, thereby improving the prediction accuracy of the preset twin model.

[0074] In one embodiment, after the preset twin model is trained, since the two preset branches are the same branches, any one of the prediction branches can be selected as the image quality evaluation model. Figure 5 An embodiment of the present disclosure also provides an image quality evaluation method, which may include the following steps.

[0075] S2: Acquire a target image to be evaluated, and perform blur processing on the target image to generate a blurred image corresponding to the target image.

[0076] S4: extracting original image features of the target image and blurred image features of the blurred image.

[0077] S6: Using the image quality assessment model, generate fusion features of the original image features and the blurred image features, and generate quality assessment information of the target image based on the fusion features.

[0078] The target image can be a video frame in a video processed by a super-resolution algorithm, or a static image processed by a super-resolution algorithm. After the quality assessment model blurs the target image and generates corresponding fusion features, it can output quality assessment information of the target image based on the fusion features.

[0079] In a specific application example, the quality of the image processed by the super-resolution algorithm can be evaluated to determine whether the currently selected super-resolution algorithm can be applied to the actual video transmission process. Specifically, in the testing phase, for the video after super-resolution processing, the video frame in the video can be used as the original image, and the original image can be scaled to an image of a preset size according to the actual rendering resolution of the video frame. Considering that in the feature extraction module in the training phase, the data size processed in the Swin-Transformer structure is usually 224*224, after obtaining the image of the preset size, the target image to be evaluated of a specified size can be cropped from the image of the preset size, and the specified size can be 224*224.

[0080] In practical applications, when cropping a target image from an image of a preset size, cropping may be performed from a central area of ​​the image of the preset size, or based on a region of interest (ROI) of human eyes, which is not limited in the present disclosure.

[0081] After the target image is obtained by cropping, the target image may be processed according to the process described in steps S2 to S6 to obtain quality evaluation information of the target image.

[0082] The technical solution provided by one or more embodiments of the present disclosure is that the image samples in the image training set can be sorted according to the image quality. When generating the quality evaluation information of the image samples, the blurring processing results of the image samples can be used as reference information. The blurring processing results weaken the high-frequency texture information in the image samples, making the preset twin model more sensitive to the high-frequency texture information. The preset twin model can output the quality evaluation information of different image samples respectively. By comparing the quality evaluation information output by the preset twin model with the quality ranking label of the image sample, the ranking error information can be determined. The ranking error information can correct the preset twin model so that the quality evaluation information output by the corrected preset twin model matches the quality ranking label.

[0083] The technical solution provided by one or more embodiments of the present disclosure can improve the sensitivity of the preset twin model to image quality by training the preset twin model with image samples sorted by image quality and introducing fuzzy processing results as reference information, thereby improving the accuracy of image quality evaluation information generated by the preset twin model.

[0084] See also Figure 6 In one embodiment of the present disclosure, there is also provided an image quality evaluation device, the device comprising:

[0085] A training set acquisition unit 110 is used to acquire an image training set, wherein the image training set at least includes a plurality of image samples sorted according to image quality;

[0086] The model processing unit 210 is used to perform blur processing on the first image sample and the second image sample with different image qualities in the image training set, respectively, and generate first quality evaluation information of the first image sample and second quality evaluation information of the second image sample according to the blur processing result through a preset twin model;

[0087] A model correction unit 310 is used to identify the quality ranking label between the first image sample and the second image sample, generate ranking error information according to the quality ranking label, the first quality evaluation information and the second quality evaluation information, and use the ranking error information to correct the preset twin model; wherein any prediction branch in the preset twin model that has completed the correction is used as an image quality evaluation model;

[0088] The image acquisition unit 410 is used to acquire a target image to be evaluated, and perform blur processing on the target image to generate a blurred image corresponding to the target image;

[0089] A feature extraction unit 510, configured to extract original image features of the target image and blurred image features of the blurred image respectively;

[0090] The information generating unit 610 is used to generate a fusion feature of the original image feature and the blurred image feature by using the image quality assessment model, and to generate quality assessment information of the target image based on the fusion feature.

[0091] Among them, the specific processing logic of each functional module can be found in the description of the aforementioned method implementation method, which will not be repeated here.

[0092] Each unit described in the above embodiments may be implemented by a computer chip or a product having a certain function. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0093] For the convenience of description, the above devices are described in terms of functions and are described separately in various units. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0094] See also Figure 7 An embodiment of the present disclosure further provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the above-mentioned image quality evaluation method is implemented.

[0095] The present disclosure also provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the above-mentioned image quality evaluation method is implemented.

[0096] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0097] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiment.

[0098] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0099] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0100] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, equipment, and storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0101] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

[0102] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for evaluating image quality, characterized in that: The method comprises: Acquire an image training set, wherein the image training set at least includes a plurality of image samples sorted according to image quality; For a first image sample and a second image sample with different image qualities in the image training set, blurring the first image sample and the second image sample respectively, and generating first quality evaluation information of the first image sample and second quality evaluation information of the second image sample respectively according to the blurring result by using a preset twin model; Identify the quality ranking label between the first image sample and the second image sample, generate ranking error information according to the quality ranking label, the first quality evaluation information and the second quality evaluation information, and use the ranking error information to correct the preset twin model; wherein any prediction branch in the preset twin model that has completed the correction is used as an image quality evaluation model; Acquire a target image to be evaluated, and perform blur processing on the target image to generate a blurred image corresponding to the target image; Extracting original image features of the target image and blurred image features of the blurred image respectively; The image quality assessment model is used to generate fusion features of the original image features and the blurred image features, and quality assessment information of the target image is generated based on the fusion features.

2. The method according to claim 1, characterized in that Generating the fusion feature of the original image feature and the blurred image feature includes: Performing self-attention processing on the original image features to generate key features and value features corresponding to the original image features; The fuzzy image feature is used as a query feature, cross-attention processing is performed with the key feature and the value feature, and the processed feature is used as the fusion feature.

3. The method according to claim 1, characterized in that Obtaining the target image to be evaluated includes: Acquire an original image, and scale the original image to an image of a preset size; A target image to be evaluated of a specified size is cropped from the image of the preset size.

4. The method according to claim 1, characterized in that: Performing blur processing on the first image sample and the second image sample respectively includes: Performing a blur operation on the first image sample to obtain a first blurred sample, and performing a blur operation on the second image sample to obtain a second blurred sample; Correspondingly, the first quality evaluation information of the first image sample and the second quality evaluation information of the second image sample are generated respectively according to the result of the fuzzy processing by the preset twin model: Generate a first fusion feature of the first image sample and the first blurred sample through a first prediction branch of the preset twin model, and generate first quality evaluation information of the first image sample based on the first fusion feature; A second fusion feature of the second image sample and the second blurred sample is generated through the second prediction branch of the preset twin model, and second quality evaluation information of the second image sample is generated based on the second fusion feature.

5. The method according to claim 4, characterized in that Correcting the preset twin model using the sorting error information includes: The first prediction branch or the second prediction branch in the preset twin model is corrected using the sorting error information, wherein the first prediction branch and the second prediction branch synchronously update the corrected model parameters.

6. The method according to claim 4, characterized in that Generating a first fusion feature of the first image sample and the first blurred sample includes: respectively extracting a first original image feature of the first image sample and a first blurred image feature of the first blurred sample; Performing self-attention processing on the first original image feature to generate a key feature and a value feature corresponding to the first original image feature; The first fuzzy image feature is used as a query feature, cross-attention processing is performed with the key feature and the value feature, and the processed feature is used as the first fusion feature.

7. The method according to claim 1, characterized in that Identifying a quality ranking tag between the first image sample and the second image sample includes: If the image quality of the first image sample is higher than the image quality of the second image sample, obtaining a quality ranking label representing a first value; If the image quality of the first image sample is lower than the image quality of the second image sample, obtaining a quality ranking label representing a second value; The first value and the second value are different values.

8. The method according to claim 1 or 7, characterized in that: Generating sorting error information according to the quality sorting label, the first quality evaluation information, and the second quality evaluation information includes: Calculating a difference between a first evaluation score represented by the first quality evaluation information and a second evaluation score represented by the second quality evaluation information; Generate a candidate error parameter according to the difference and the numerical value represented by the quality ranking label; The candidate error parameters are compared with preset reference coefficients, and sorting error information is generated based on the comparison result.

9. The method according to claim 1, characterized in that: The image training set also includes a third image sample labeled with a quality evaluation coefficient; the method further includes: Performing blur processing on the third image sample to generate a blurred image corresponding to the third image sample; Through any prediction branch of the preset twin model, respectively extract the original image features of the third image sample and the blurred image features of the blurred image, and generate fusion features of the original image features and the blurred image features; Generate a prediction evaluation coefficient of the fusion feature, and generate coefficient error information according to the prediction evaluation coefficient and the quality evaluation coefficient; The coefficient error information and the sorting error information are aggregated to generate comprehensive error information, and the preset twin model is corrected using the comprehensive error information.

10. An image quality evaluation device, characterized in that: The device comprises: A training set acquisition unit, used to acquire an image training set, wherein the image training set at least includes a plurality of image samples sorted according to image quality; a model processing unit, configured to perform blur processing on a first image sample and a second image sample with different image qualities in the image training set, respectively, and generate first quality evaluation information of the first image sample and second quality evaluation information of the second image sample respectively according to a result of the blur processing by using a preset twin model; A model correction unit, configured to identify a quality ranking label between the first image sample and the second image sample, generate ranking error information according to the quality ranking label, the first quality evaluation information, and the second quality evaluation information, and use the ranking error information to correct the preset twin model; wherein any prediction branch in the preset twin model that has completed the correction is used as an image quality evaluation model; An image acquisition unit, used for acquiring a target image to be evaluated, and performing blur processing on the target image to generate a blurred image corresponding to the target image; A feature extraction unit, used to extract original image features of the target image and blurred image features of the blurred image respectively; The information generating unit is used to generate a fusion feature of the original image feature and the blurred image feature by using the image quality evaluation model, and to generate quality evaluation information of the target image based on the fusion feature.

11. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images