Image quality evaluation method, device and storage medium

Abstract

The disclosure provides an image quality evaluation method, device and storage medium, wherein the image quality evaluation method comprises: acquiring a target image to be evaluated, and performing blur processing on the target image to generate a blurred image corresponding to the target image; extracting original image features of the target image and blurred image features of the blurred image respectively, and generating fusion features of the original image features and the blurred image features; and generating quality evaluation information of the target image based on the fusion features. The technical scheme provided by the disclosure can improve the accuracy of image quality evaluation information.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to an image quality evaluation method, device, and storage medium. Background Technology

[0002] Image super-resolution algorithms can reconstruct super-resolution (SR) images from original low-resolution (LR) images. Currently, image super-resolution algorithms can be applied in many scenarios. For example, in video transmission, to save data bandwidth, servers storing video resources can send low-resolution videos to user devices. Upon receiving the low-resolution video, the user's device can use an image super-resolution algorithm to reconstruct it into a super-resolution video. This ensures a good viewing experience while saving the bandwidth required for downloading data.

[0003] To measure the quality of the reconstructed super-resolution image, a reliable image quality assessment method is needed. However, some current image quality assessment methods typically assume that the quality of the SR image is lower than that of the original high-resolution (HR) image when determining its quality. In reality, the quality of the SR image is not necessarily lower than that of the HR image, resulting in insufficient accuracy for some existing image quality assessment methods.

[0004] Therefore, a more accurate image quality assessment method is needed. Summary of the Invention

[0005] In view of this, one or more embodiments of this disclosure provide an image quality evaluation method, apparatus and storage medium that can improve the accuracy of image quality evaluation.

[0006] This disclosure provides an image quality assessment method, comprising: acquiring an image training set, the image training set including at least a plurality of image samples sorted according to image quality; performing blurring processing on a first image sample and a second image sample with different image quality in the image training set, and generating first quality assessment information of the first image sample and second quality assessment information of the second image sample respectively based on the result of the blurring processing using a preset twin model; identifying quality ranking labels between the first image sample and the second image sample, and generating ranking error information based on the quality ranking labels, the first quality assessment information and the second quality assessment information, and correcting the preset twin model using the ranking error information; wherein any prediction branch in the corrected preset twin model is used as an image quality assessment model; acquiring a target image to be evaluated, and performing blurring processing on the target image to generate a blurred image corresponding to the target image; extracting the original image features of the target image and the blurred image features of the blurred image respectively; using the image quality assessment model to generate a fusion feature of the original image features and the blurred image features, and generating quality assessment information of the target image based on the fusion feature.

[0007] This disclosure also provides an image quality assessment device, the device comprising: a training set acquisition unit, configured to acquire an image training set, the image training set including at least a plurality of image samples sorted according to image quality; a model processing unit, configured to perform blurring processing on a first image sample and a second image sample with different image quality in the image training set, respectively, and generate first quality assessment information of the first image sample and second quality assessment information of the second image sample based on the result of the blurring processing using a preset twin model; and a model correction unit, configured to identify quality ranking labels between the first image sample and the second image sample, and correct the quality ranking labels according to the quality ranking labels. The system comprises: generating sorting error information from the first quality evaluation information and the second quality evaluation information; and using the sorting error information to correct the preset twin model; wherein any prediction branch in the corrected preset twin model serves as the image quality evaluation model; an image acquisition unit is used to acquire the target image to be evaluated and to blur the target image to generate a blurred image corresponding to the target image; a feature extraction unit is used to extract the original image features of the target image and the blurred image features of the blurred image, respectively; and an information generation unit is used to generate a fusion feature of the original image features and the blurred image features using the image quality evaluation model, and to generate the quality evaluation information of the target image based on the fusion feature.

[0008] This disclosure also provides an electronic device including a memory and a processor, the memory being used to store a computer program, which, when executed by the processor, implements the above-described image quality evaluation method.

[0009] This disclosure also provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the above-described image quality evaluation method.

[0010] This disclosure provides a technical solution through one or more embodiments. Image samples in an image training set can be sorted according to image quality. When generating quality evaluation information for image samples, the blurring result of the image samples can be used as reference information. The blurring result weakens high-frequency texture information in the image samples, making the preset twin model more sensitive to high-frequency texture information. The preset twin model can output quality evaluation information for different image samples. By comparing the quality evaluation information output by the preset twin model with the quality ranking labels of the image samples, ranking error information can be determined. This ranking error information can be used to correct the preset twin model, so that the quality evaluation information output by the corrected preset twin model matches the quality ranking labels.

[0011] The technical solution provided by one or more embodiments of this disclosure trains a preset twin model by sorting image samples according to image quality, and introduces the fuzzing result as reference information, which can improve the sensitivity of the preset twin model to image quality, thereby improving the accuracy of the image quality evaluation information generated by the preset twin model. Attached Figure Description

[0012] The features and advantages of the embodiments of this disclosure will be more clearly understood by referring to the accompanying drawings, which are illustrative and should not be construed as limiting the scope of this disclosure in any way. In the drawings:

[0013] Figure 1 A schematic diagram illustrating the training method steps of an image quality assessment model according to one embodiment of this disclosure is shown;

[0014] Figure 2 A schematic diagram of the structure of a preset twin model in one embodiment of this disclosure is shown;

[0015] Figure 3 A schematic diagram of a feature extraction module in one embodiment of this disclosure is shown;

[0016] Figure 4 A schematic diagram of a feature cross-fusion module in one embodiment of this disclosure is shown;

[0017] Figure 5 A schematic diagram illustrating the steps of an image quality evaluation method according to one embodiment of this disclosure is shown.

[0018] Figure 6 A schematic diagram of the functional modules of an image quality evaluation device in one embodiment of this disclosure is shown;

[0019] Figure 7 A schematic diagram of the structure of an electronic device according to one embodiment of the present disclosure is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0021] Image super-resolution algorithms have applications in various practical scenarios. For example, on video playback platforms, to save user data, the original high-resolution video can be downsampled to a low-resolution video and then sent to the user's terminal device. At the user's terminal device, an image super-resolution algorithm can be used to reconstruct the high-resolution video from the low-resolution video.

[0022] In practical applications, in order to measure the quality difference between an image processed by a super-resolution algorithm and the original high-resolution image, or to compare the image processing effects of different super-resolution algorithms, quantitative quality assessment information can be generated through image quality assessment methods.

[0023] Specifically, in certain application scenarios, it's necessary to ensure a good viewing experience while saving users' data download bandwidth. In such cases, the super-resolution image reconstructed using a super-resolution algorithm on the user's terminal device should be of at least the same quality as the original high-resolution image. Image quality assessment methods can be used to process the super-resolution image and the original high-resolution image separately, generating their respective quality assessment information. By comparing these quality assessments, it can be determined whether the current super-resolution algorithm can guarantee the user's viewing experience.

[0024] Furthermore, in other application scenarios, when different super-resolution algorithms are used to process images, image quality evaluation methods can be used to evaluate the quality of the images processed by each super-resolution algorithm, thereby determining which super-resolution algorithm performs better.

[0025] In related technologies, image quality assessment methods can be broadly categorized into full-reference (FR) assessment methods, reduced-reference (RR) assessment methods, and no-reference (NR) assessment methods. These image quality assessment methods are typically based on deep learning algorithms and are implemented by training on a large number of image samples in an image dataset.

[0026] The full-reference evaluation method requires inputting both the super-resolution image and the original high-resolution image into a neural network model, using the high-resolution image as a reference to generate quality assessment information for the super-resolution image. This method assumes during training that the original high-resolution image is of higher quality than the super-resolution image. However, in real-world applications, the quality of the super-resolution image may be superior to the high-resolution image, leading to inaccurate quality assessment information in the final product.

[0027] Semi-reference evaluation methods typically input both low-resolution and super-resolution images into a neural network model, using the low-resolution image as a reference to generate quality assessment information for the super-resolution image. However, in this scenario, the size of the low-resolution image cannot be fixed; for example, there may be different transcoded versions such as 270P or 360P. The neural network model cannot flexibly adapt to the size of the low-resolution image, resulting in low training efficiency.

[0028] Referenceless evaluation methods typically only require inputting the super-resolution image into a neural network model. By comparing the model's output quality assessment information with the quality assessment information annotated on the super-resolution image, the neural network model can be corrected so that its output quality assessment information approximates the initial quality assessment information annotated on the super-resolution image. However, because this method lacks other types of images as reference information, the actual trained model cannot accurately predict the quality assessment information.

[0029] In view of this, one embodiment of the present disclosure provides an image quality assessment method, which can construct an image quality assessment model with high prediction accuracy and wide adaptability based on fuzzy processing technology and Siamese network technology.

[0030] Specifically, please refer to Figure 1 In one implementation, the training method for the image quality assessment model may include the following steps.

[0031] S1: Obtain an image training set, which includes at least a number of image samples sorted according to image quality.

[0032] In this embodiment, an image training set can be constructed for training the image quality evaluation model. This image training set may include multiple image samples sorted according to image quality. Specifically, the image samples in the image training set may be sorted in order of image quality from best to worst or from worst to best.

[0033] In practical applications, the presentation format of image samples in the image training set, as well as the process of determining the quality ranking among image samples, can be achieved in a variety of ways.

[0034] In one application scenario, image samples in an image training set can be represented in the form of sample pairs. For example, the image training set may include several sets of image sample pairs, each set containing two image samples of different qualities. The quality of the image samples in each set can be assessed by professionals. Thus, the two image samples in each set can be ranked according to the professional assessment results. Of course, there are many other methods to determine the quality of image samples in practical applications, and this disclosure does not limit this, as long as the image samples in the set can be ranked according to image quality.

[0035] In another application scenario, image samples in an image training set can be obtained by downsampling a series of high-resolution images. The image quality gradually decreases as the downsampling factor increases. Taking a 1080P high-resolution image as an example, it can be downsampled by 2x, 3x, and 4x to obtain downsampled images of 540P, 360P, and 270P respectively. As the downsampling factor increases, the image quality gradually decreases. In this way, together with the original 1080P high-resolution image, four image samples with a quality ranking relationship can be obtained. By performing the above downsampling operation on a large number of high-resolution images, multiple image sample sequences can be obtained. Each image sample sequence can contain the original high-resolution image and the derived images obtained by downsampling the high-resolution image, and the images in each image sample sequence can be sorted according to image quality.

[0036] Of course, with the advancement of technology and the change of application scenarios, the composition of image samples in the image training set and the determination of quality ranking relationships can be more diversified, and this disclosure does not limit this.

[0037] S3: For the first image sample and the second image sample with different image quality in the image training set, blurring is performed on the first image sample and the second image sample respectively, and a first quality evaluation information of the first image sample and a second quality evaluation information of the second image sample are generated according to the result of the blurring by a preset twin model.

[0038] In this embodiment, if an original image is of low quality (blurry), then after blurring the original image, the similarity between the blurred image and the original image is usually high. Conversely, if an original image is of high quality (sharper), then after blurring the original image, the similarity between the blurred image and the original image is usually low. This is because factors that can intuitively improve image quality are usually reflected in the image's texture features, which are generally distributed in high frequencies. Blurring an image filters out high-frequency information to some extent. For images that are originally of low quality, their high-frequency information is not very rich, and blurring will not significantly change the image content, thus the similarity between the blurred image and the original image is high. However, for images that are originally of high quality, their high-frequency information is relatively rich. After blurring, because some high-frequency information is filtered out, the image content is significantly changed, thus the similarity between the blurred image and the original image is low.

[0039] Based on the foregoing, in this embodiment, the blurred image can be used as reference information to help determine whether the quality of an image sample itself is high. Specifically, the technical solution of this embodiment is illustrated using a first image sample and a second image sample of different quality in the image training set as examples. The first image sample and the second image sample can be two image samples from the image sample pair in the aforementioned embodiments, or they can be two different image samples from an image sample sequence.

[0040] In this embodiment, after selecting first and second image samples with different image qualities from the image training set, the first and second image samples can be blurred respectively to obtain first blurred samples and second blurred samples. Various blurring methods can be used. For example, Gaussian blurring, salt-and-pepper blurring, and motion blurring can be employed to blur the first and second image samples. The specific blurring method used can be flexibly selected according to the actual application scenario.

[0041] In this embodiment, after obtaining the result of the fuzzing process, a first quality evaluation information for the first image sample and a second quality evaluation information for the second image sample can be generated by using a preset twin model.

[0042] For details, please refer to Figure 2 The pre-defined twin model can include a first prediction branch and a second prediction branch. The first prediction branch processes the first image sample and the first blurred sample to predict the first quality assessment information of the first image sample; the second prediction branch processes the second image sample and the second blurred sample to predict the second quality assessment information of the second image sample. The first and second prediction branches can adopt the exact same network architecture, and the model parameters in the network architecture are updated synchronously. Therefore, the first and second prediction branches can be regarded as two identical branches.

[0043] like Figure 2 As shown, taking the first prediction branch as an example, the first prediction branch can include a feature extraction module, a feature cross-fusion module, and an evaluation regression module. The feature extraction module can extract the first original image features of the first image sample and the first blurred image features of the first blurred sample. In practical applications, the feature extraction module can be implemented in various ways. For example, the feature extraction module can be implemented using networks such as VGG, ResNet, Inception, and DenseNet.

[0044] In one implementation, to improve the accuracy of feature extraction, a Swin-Transformer architecture can be used to implement the feature extraction module. Please refer to [link / reference]. Figure 3 The Swin-Transformer architecture can include multiple stages to construct feature maps of different sizes. In practical applications, the input image data can be divided into blocks by a patch partitioning module. In the first stage (stage 1), the input block data can be processed by a linear embedding layer. In subsequent stages, the block data can be downsampled by a patch merging layer. Swin-Transformer blocks can be repeatedly stacked in each stage. The specific structure of the Swin-Transformer block can adopt either a W-MSA structure or an SW-MSA structure. Typically, these two structures are used in pairs, i.e., a W-MSA structure is used first, followed by an SW-MSA structure. Therefore, the number of Swin-Transformer blocks stacked in each stage is generally even.

[0045] In this embodiment, the feature extraction module is implemented using a Swin-Transformer structure, which can fuse image features from multiple stages, thereby improving the expressive power of features for image data.

[0046] The feature cross-fusion module can be implemented using a Cross-Attention Feature Fusion (CAFF) model. Please refer to [link / reference]. Figure 4 The CAFF model can include multiple multi-head self-attention (MSA) units. The inputs of each MSA unit can include a query (Q), a key (K), and a value (V). The first original image features extracted by the feature extraction module are first processed by the first MSA unit to generate corresponding key and value features. Subsequently, the first blurred image features extracted by the feature extraction module can be used as query features and input together with the generated key and value features into the second MSA unit for cross-attention processing. During cross-attention processing, the first blurred image features can be used as query conditions and matched with the first original image features for similarity. A high similarity score indicates that the first image sample is similar to the first blurred sample, suggesting lower image quality. Conversely, a low similarity score indicates that the first image sample is not very similar to the first blurred sample, suggesting higher image quality. Thus, the output of the second multi-head self-attention unit, after being processed by a multi-layer perceptron (MLP), yields the first fusion feature of the first image sample and the first blurred sample. Subsequently, the evaluation regression module performs regression calculations on the first fusion feature to obtain the first quality evaluation information corresponding to the first image sample.

[0047] In practical applications, the first quality assessment information can be a numerical value indicating image quality; the higher the value, the better the image quality. Of course, the first quality assessment information can also take other forms, such as characters representing image quality like "Excellent," "Good," "Medium," and "Poor." The form of the first quality assessment information can be flexibly chosen based on the different accuracy requirements of image quality assessment in the actual application scenario.

[0048] Similarly, the structure and specific processing of the second prediction branch are similar to those of the first prediction branch. Through the second prediction branch of the preset Siamese model, the second fusion features of the second image sample and the second blurred sample can be generated, and the second quality evaluation information of the second image sample can be generated based on the second fusion features. Specifically, the second original image features of the second image sample and the second blurred image features of the second blurred sample can be extracted respectively. Self-attention processing is performed on the second original image features to generate key features and value features corresponding to the second original image features. Subsequently, the second blurred image features can be used as query features, and cross-attention processing is performed with the key features and value features. The resulting features are then used as the second fusion features. For a detailed implementation process, please refer to the description of the first prediction branch; it will not be repeated here.

[0049] S5: Identify the quality ranking label between the first image sample and the second image sample, generate ranking error information based on the quality ranking label, the first quality evaluation information and the second quality evaluation information, and use the ranking error information to correct the preset twin model.

[0050] In this embodiment, the two prediction branches of the preset twin model can output first quality evaluation information and second quality evaluation information, respectively. By comparing the first and second quality evaluation information, the quality of the first and second image samples can be determined. However, the quality evaluation information determined based on the preset twin model may differ from the actual quality of the first and second image samples. For example, in the image training set, the quality of the first image sample may be better than that of the second image sample, but the first and second quality evaluation information output by the preset twin model may indicate that the quality of the first image sample is worse than that of the second image sample. Therefore, the result output by the preset twin model can be compared with the actual standard result to generate ranking error information. The preset twin model can then be corrected using a ranking error model, so that the judgment result output by the corrected preset twin model is consistent with the actual standard result.

[0051] Specifically, based on the quality ranking in the image training set, a quality ranking label can be identified between the first and second image samples. This quality ranking label indicates which image sample is of higher quality. Subsequently, ranking error information can be generated based on this quality ranking label, as well as the first and second quality evaluation information output by the pre-defined Siamese model.

[0052] In one implementation, the quality ranking label can be represented by a specific numerical value. For example, if the image quality of the first image sample is higher than that of the second image sample, a quality ranking label representing a first numerical value can be obtained; if the image quality of the first image sample is lower than that of the second image sample, a quality ranking label representing a second numerical value can be obtained; wherein the first numerical value and the second numerical value are different values. For example, the first numerical value can be 1, and the second numerical value can be -1.

[0053] In this embodiment, when generating sorting error information, the difference between the first evaluation score represented by the first quality evaluation information and the second evaluation score represented by the second quality evaluation information can be calculated. For example, the first evaluation score can be represented as ps1, and the second evaluation score can be represented as ps2, then the difference can be represented as ps1-ps2.

[0054] Then, candidate error parameters can be generated based on the difference and the numerical values ​​represented by the quality ranking label. Specifically, the candidate error parameter can be represented as -yr(ps1-ps2). Finally, the candidate error parameter and a preset reference coefficient can be compared, and ranking error information can be generated based on the comparison result. Specifically, the maximum value can be taken, and either the candidate error parameter or the preset reference coefficient can be used as the numerical value representing the ranking error information. For example, the ranking error information loss_rank can be represented as:

[0055] loss_rank=max(0,-yr(ps1-ps2))

[0056] Here, 0 can be used as the aforementioned preset reference coefficient. It can be seen that, in this way, when the ranking result predicted by the preset twin model matches the actual ranking result, then -yr(ps1-ps2) is negative, the ranking error information is 0, indicating that no correction is needed for the preset twin model. However, if the ranking result predicted by the preset twin model does not match the actual ranking result, then -yr(ps1-ps2) is positive, the ranking error information is -yr(ps1-ps2), and this ranking error information can be used to correct the preset twin model.

[0057] In one implementation, to more precisely distinguish the quality differences between image samples, a margin coefficient can be introduced when determining the ranking error information. This margin coefficient can be a constant, for example, 0.1. The aforementioned ranking error information loss_rank can be optimized as follows:

[0058] loss_rank=max(0,-yr(ps1-ps2)+margin)

[0059] Thus, when ps1 and ps2 are relatively close, even if the ranking result predicted by the pre-defined twin model matches the actual ranking result, -yr(ps1-ps2)+margin may still be a positive value after adding the margin coefficient. In this case, non-zero ranking error information will still be generated, so the pre-defined twin model still needs to be corrected. The purpose of this processing is to enhance the pre-defined twin model's ability to distinguish subtle differences in different image samples, thereby improving the precision and accuracy of image quality assessment information.

[0060] In one implementation, considering that the first prediction branch and the second prediction branch in the preset twin model are the same branch, when using the sorting error information to correct the preset twin model, either the first prediction branch or the second prediction branch can be corrected. After one of the prediction branches is corrected, the first prediction branch and the second prediction branch can update the corrected model parameters synchronously to maintain the consistency of the two prediction branches.

[0061] In one implementation, the image training set may also include more types of image samples. For example, the image training set may include multiple third image samples labeled with quality evaluation coefficients. These quality evaluation coefficients can be numerical values ​​assigned to the third image samples by professionals after evaluation, representing their quality. A higher quality evaluation coefficient indicates a higher quality third image sample.

[0062] For this type of image sample, since it does not involve quality ranking among image samples, any prediction branch in the preset Siamese model can be used to process the third image sample and output its prediction evaluation coefficient. Specifically, the third image sample can first be blurred to generate a blurred image corresponding to the third image sample. Then, any prediction branch in the preset Siamese model can be used to extract the original image features of the third image sample and the blurred image features of the blurred image, respectively, and generate a fusion feature of the original image features and the blurred image features. Then, the prediction evaluation coefficient of the fusion feature can be generated through this prediction branch. The specific implementation of generating the fusion feature and generating the prediction evaluation coefficient is similar to the process of generating the first fusion feature and the second fusion feature, as well as generating the first quality evaluation information and the second quality evaluation information in the aforementioned steps, and will not be repeated here.

[0063] In this embodiment, after generating the predicted evaluation coefficients, coefficient error information can be generated based on these predicted evaluation coefficients and the quality evaluation coefficients of the third image sample. The coefficient error information can be represented, for example, by the cross-entropy loss coefficient or the mean squared error coefficient.

[0064] In one implementation, after obtaining the coefficient error information, the prediction branch can be corrected based on this information. Furthermore, to improve the training accuracy of the preset Siamese model, the coefficient error information and ranking error information can be summarized to generate comprehensive error information, which can then be used to correct the preset Siamese model. Specifically, the coefficient error information and ranking error information can be weighted and summed to obtain the comprehensive error information. This comprehensive error information can correct one prediction branch of the preset Siamese model. After the correction of this prediction branch, the corrected model parameters can be synchronously updated to the other prediction branch to ensure consistency between the two prediction branches.

[0065] As can be seen from the descriptions of the various implementation methods above, in some specific application scenarios, the image samples in the image training set can include the following types:

[0066] Type 1: Image samples labeled with quality evaluation coefficients;

[0067] Type 2: Two image samples in an image sample pair, sorted by image quality;

[0068] Type 3: Image sample sequence formed by downsampling a high-resolution image.

[0069] Of course, in practical applications, there can be many more types of image samples, which will not be listed here.

[0070] For image samples of type 1, any prediction branch in the preset twin model can be used for processing, and the preset twin model can be corrected based on the coefficient error information. For image samples of types 2 and 3, two image samples with different image qualities can be selected from the image sample pair or the image sample sequence, and the two image samples can be processed by the two prediction branches of the preset twin model respectively, and the preset twin model can be corrected based on the sorting error information.

[0071] In practical applications, as the downsampling factor increases in Type 3, the size of the image samples also decreases continuously. By training the preset twin model with image samples of different sizes, the preset twin model can be used to predict the quality of images of different sizes, thereby improving the adaptability of the preset twin model.

[0072] Furthermore, when training the pre-defined Siamese model using image samples with quality ranking relationships, it does not assume that high-resolution images are necessarily of higher quality than super-resolution images. Instead, it uses a unified evaluation dimension, with professionals assigning quality scores to different image samples, and then ranking the image samples based on these scores. This allows the pre-defined Siamese model to output image quality evaluation information more accurately.

[0073] Furthermore, by blurring the original image samples and generating fusion features between the original and blurred image samples, the reference information is enriched, allowing the pre-defined twin model to focus on learning texture features that can characterize image quality, thereby improving the prediction accuracy of the pre-defined twin model.

[0074] In one implementation, after training a preset Siamese model, since the two preset branches are identical, either prediction branch can be selected as the image quality assessment model. See also... Figure 5 One embodiment of this disclosure also provides an image quality evaluation method, which may include the following steps.

[0075] S2: Obtain the target image to be evaluated, and perform blurring processing on the target image to generate a blurred image corresponding to the target image.

[0076] S4: Extract the original image features of the target image and the blurred image features of the blurred image.

[0077] S6: Using the image quality assessment model, generate fused features of the original image features and the blurred image features, and generate quality assessment information of the target image based on the fused features.

[0078] The target image can be a video frame processed by a super-resolution algorithm, or a still image processed by a super-resolution algorithm. The quality assessment model blurs the target image and generates corresponding fusion features, then outputs quality assessment information based on these fusion features.

[0079] In a specific application example, the quality of the image processed by the super-resolution algorithm can be evaluated to determine whether the currently selected super-resolution algorithm can be applied to the actual video transmission process. Specifically, during the testing phase, for the super-resolution processed video, video frames can be used as the original images, and the original images can be scaled to a preset size according to the actual rendering resolution of the video frames. Considering that the data size processed in the Swin-Transformer structure in the feature extraction module during the training phase is usually 224*224, after obtaining the image of the preset size, a target image of a specified size to be evaluated can be cropped from the image of the preset size, and this specified size can be 224*224.

[0080] In practical applications, when cropping a target image from an image of a preset size, cropping can be performed from the central region of the image of the preset size, or it can be performed based on the region of interest (ROI). This disclosure does not limit the cropping in this way.

[0081] After cropping to obtain the target image, the target image can be processed according to the process described in steps S2 to S6 to obtain the quality evaluation information of the target image.

[0082] This disclosure provides a technical solution through one or more embodiments. Image samples in an image training set can be sorted according to image quality. When generating quality evaluation information for image samples, the blurring result of the image samples can be used as reference information. The blurring result weakens high-frequency texture information in the image samples, making the preset twin model more sensitive to high-frequency texture information. The preset twin model can output quality evaluation information for different image samples. By comparing the quality evaluation information output by the preset twin model with the quality ranking labels of the image samples, ranking error information can be determined. This ranking error information can be used to correct the preset twin model, so that the quality evaluation information output by the corrected preset twin model matches the quality ranking labels.

[0083] The technical solution provided by one or more embodiments of this disclosure trains a preset twin model by sorting image samples according to image quality, and introduces the fuzzing result as reference information, which can improve the sensitivity of the preset twin model to image quality, thereby improving the accuracy of the image quality evaluation information generated by the preset twin model.

[0084] Please see Figure 6 One embodiment of this disclosure also provides an image quality evaluation apparatus, the apparatus comprising:

[0085] The training set acquisition unit 110 is used to acquire an image training set, wherein the image training set includes at least a plurality of image samples sorted according to image quality;

[0086] The model processing unit 210 is used to perform blurring processing on the first image sample and the second image sample with different image quality in the image training set, respectively, and generate first quality evaluation information of the first image sample and second quality evaluation information of the second image sample according to the result of the blurring processing through a preset twin model.

[0087] The model correction unit 310 is used to identify the quality ranking label between the first image sample and the second image sample, generate ranking error information based on the quality ranking label, the first quality evaluation information and the second quality evaluation information, and use the ranking error information to correct the preset twin model; wherein, any prediction branch in the preset twin model after correction is used as the image quality evaluation model.

[0088] The image acquisition unit 410 is used to acquire the target image to be evaluated and to perform blur processing on the target image to generate a blurred image corresponding to the target image;

[0089] The feature extraction unit 510 is used to extract the original image features of the target image and the blurred image features of the blurred image, respectively.

[0090] The information generation unit 610 is used to generate fused features of the original image features and the blurred image features using the image quality evaluation model, and to generate quality evaluation information of the target image based on the fused features.

[0091] The specific processing logic of each functional module can be found in the description of the aforementioned method implementation method, and will not be repeated here.

[0092] The various units described in the above embodiments can be implemented by a computer chip or by a product with a certain function. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0093] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0094] Please see Figure 7 One embodiment of this disclosure also provides an electronic device, which includes a memory and a processor. The memory is used to store a computer program, which, when executed by the processor, implements the image quality evaluation method described above.

[0095] This disclosure also provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the above-described image quality evaluation method.

[0096] The processor can be a central processing unit (CPU). It can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.

[0097] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods in the above-described embodiments.

[0098] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0099] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0100] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, embodiments of apparatus, devices, and storage media are basically similar to method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0101] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0102] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. An image quality assessment method, characterized in that, The method includes: Obtain an image training set, wherein the image training set includes at least a plurality of image samples sorted according to image quality; For the first image sample and the second image sample with different image quality in the image training set, blurring is performed on the first image sample and the second image sample respectively, and a first quality evaluation information of the first image sample and a second quality evaluation information of the second image sample are generated according to the result of the blurring by a preset Siamese model. The quality ranking label between the first image sample and the second image sample is identified, and ranking error information is generated based on the quality ranking label, the first quality evaluation information and the second quality evaluation information. The ranking error information is then used to correct the preset twin model. Any prediction branch in the corrected preset twin model is used as the image quality evaluation model. The target image to be evaluated is acquired, and the target image is blurred to generate a blurred image corresponding to the target image. Extract the original image features of the target image and the blurred image features of the blurred image, respectively; Using the image quality assessment model, a fusion feature of the original image features and the blurred image features is generated, and quality assessment information of the target image is generated based on the fusion feature.

2. The method according to claim 1, characterized in that, The fusion features for generating the original image features and the blurred image features include: The original image features are subjected to self-attention processing to generate key features and value features corresponding to the original image features; The blurred image features are used as query features, and cross-attention processing is performed with the key features and the value features. The resulting features are then used as the fused features.

3. The method according to claim 1, characterized in that, Obtaining the target image to be evaluated includes: Acquire the original image and scale it to a preset size; The target image to be evaluated is cropped from the image of the preset size to a specified size.

4. The method according to claim 1, characterized in that, Blurring the first image sample and the second image sample respectively includes: The first image sample is blurred to obtain a first blurred sample, and the second image sample is blurred to obtain a second blurred sample; Accordingly, based on the fuzzing results, a first quality assessment information for the first image sample and a second quality assessment information for the second image sample are generated using a pre-defined twin model: The first fusion feature of the first image sample and the first blurred sample is generated through the first prediction branch of the preset twin model, and the first quality evaluation information of the first image sample is generated based on the first fusion feature; The second fusion feature of the second image sample and the second blurred sample is generated through the second prediction branch of the preset twin model, and the second quality evaluation information of the second image sample is generated based on the second fusion feature.

5. The method according to claim 4, characterized in that, Correcting the preset twin model using the sorting error information includes: The sorting error information is used to correct the first prediction branch or the second prediction branch in the preset twin model, wherein the first prediction branch and the second prediction branch are updated synchronously with the corrected model parameters.

6. The method according to claim 4, characterized in that, The first fusion feature for generating the first image sample and the first blurred sample includes: Extract the first original image features of the first image sample and the first blurred image features of the first blurred sample, respectively; Self-attention processing is performed on the first original image features to generate key features and value features corresponding to the first original image features; The first blurred image feature is used as the query feature, and cross-attention processing is performed with the key feature and the value feature. The resulting feature is then used as the first fused feature.

7. The method according to claim 1, characterized in that, Identifying the quality ranking labels between the first image sample and the second image sample includes: If the image quality of the first image sample is higher than that of the second image sample, obtain the quality ranking label representing the first value; If the image quality of the first image sample is lower than that of the second image sample, obtain a quality ranking label representing the second value; The first value and the second value are different values.

8. The method according to claim 1 or 7, characterized in that, Generating sorting error information based on the quality sorting label, the first quality evaluation information, and the second quality evaluation information includes: Calculate the difference between the first evaluation score represented by the first quality evaluation information and the second evaluation score represented by the second quality evaluation information; Based on the difference and the numerical value represented by the quality ranking label, candidate error parameters are generated; The candidate error parameters are compared with the preset reference coefficients, and sorting error information is generated based on the comparison results.

9. The method according to claim 1, characterized in that, The image training set also includes third image samples labeled with quality evaluation coefficients; the method further includes: The third image sample is blurred to generate a blurred image corresponding to the third image sample; By using any prediction branch of the preset twin model, the original image features of the third image sample and the blurred image features of the blurred image are extracted respectively, and a fusion feature of the original image features and the blurred image features is generated. Generate the predicted evaluation coefficients of the fusion features, and generate coefficient error information based on the predicted evaluation coefficients and the quality evaluation coefficients; The coefficient error information and the sorting error information are summarized to generate comprehensive error information, and the comprehensive error information is used to correct the preset twin model.

10. An image quality evaluation device, characterized in that, The device includes: A training set acquisition unit is used to acquire an image training set, wherein the image training set includes at least a plurality of image samples sorted according to image quality; The model processing unit is used to perform blurring processing on the first image sample and the second image sample with different image quality in the image training set, respectively, and generate first quality evaluation information of the first image sample and second quality evaluation information of the second image sample according to the result of the blurring processing through a preset twin model. The model correction unit is used to identify the quality ranking label between the first image sample and the second image sample, generate ranking error information based on the quality ranking label, the first quality evaluation information and the second quality evaluation information, and use the ranking error information to correct the preset twin model; wherein, any prediction branch in the preset twin model after correction is used as the image quality evaluation model. An image acquisition unit is used to acquire a target image to be evaluated and to blur the target image to generate a blurred image corresponding to the target image. The feature extraction unit is used to extract the original image features of the target image and the blurred image features of the blurred image, respectively. The information generation unit is used to generate fused features of the original image features and the blurred image features using the image quality evaluation model, and to generate quality evaluation information of the target image based on the fused features.

11. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 9.

Images