Image quality evaluation method, electronic device, storage medium and program product

Abstract

The invention discloses an image quality evaluation method, electronic equipment, a storage medium and a program product, belongs to the technical field of image processing, and is used for improving the efficiency of image quality evaluation. The method comprises the following steps: carrying out distortion transformation on a target image until a distorted image obtained by carrying out distortion transformation on the target image meets a preset condition; and inputting the target image and the distorted image into an image quality evaluation network to obtain quality evaluation scores and distortion types of the target image and the distorted image.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an image quality evaluation method, an electronic device, a storage medium and a program product. BACKGROUND

[0002] In some areas, the complexity of traffic is gradually increasing, and the traffic monitoring system is becoming more and more perfect. The monitoring video has become an important basis for the relevant departments to mediate traffic disputes, determine illegal behavior and plan urban roads. Tens of thousands of traffic monitoring cameras record and store massive video data every day. The relevant departments hope to establish a reasonable video image quality evaluation system under the premise of controllable cost according to key indicators such as picture quality, so as to ensure the usability of video quality and avoid related losses and investment waste caused by video image failure.

[0003] At present, the mainstream video image quality evaluation method basically adopts traditional image processing technology, and mainly processes image data in the back end to evaluate its quality. First, the traditional image processing technology is relatively mature, and can be directly applied to different scenes by making slight modifications on the basis of the existing application program interface, and the cost is very low. It has a faster speed in the scene with low accuracy requirement. Second, the non-embedded module used in the back-end system can meet the space requirements of more modules due to its sufficient memory, and can run more algorithms. Moreover, the software and hardware are easier to expand, and have good compatibility. The appropriate hardware facilities can be replaced according to the algorithm requirements.

[0004] The traditional image processing technology is a processing technology based on digital images. The main method for identifying video image quality problems is to use a lot of labeled data for supervised training. This process requires frequent manual screening and manual labeling of video images on a large scale, which wastes a lot of time and manpower. There are still defects in precision and real-time performance, and the degree of automation is not high, and it is not intelligent enough, resulting in low efficiency of video image quality evaluation. SUMMARY

[0005] The embodiments of the present application provide an image quality evaluation method, an electronic device, a storage medium and a program product, which can solve the problem of low efficiency of video image quality evaluation caused by defects in precision and real-time performance, low degree of automation and insufficient intelligence.

[0006] In a first aspect, the embodiments of the present application provide an image quality evaluation method, which comprises: performing a distortion transformation on a target image until a distorted image obtained by performing the distortion transformation on the target image meets a preset condition; and inputting the target image and the distorted image into an image quality evaluation network to obtain a quality evaluation score and a distortion type of the target image and the distorted image.

[0007] In a second aspect, an embodiment of the present application provides an image quality evaluation device, which comprises: a transformation module configured to perform a distortion transformation on a target image until a distorted image obtained by performing the distortion transformation on the target image meets a preset condition; and an evaluation module configured to input the target image and the distorted image into an image quality evaluation network to obtain a quality evaluation score and a distortion type of the target image and the distorted image.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which comprises a processor, a memory, and a program or instructions stored in the memory and executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the method according to the first aspect.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, which stores a program or instructions, and the program or instructions, when executed by a processor, implement the steps of the method according to the first aspect.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product, which comprises a computer program stored in a non-transitory computer readable storage medium, and the computer program comprises program instructions, and the program instructions, when executed by a computer, cause the computer to perform the steps of the method according to the first aspect.

[0011] In the embodiments of the present application, the target image is subjected to a distortion transformation until a distorted image obtained by performing the distortion transformation on the target image meets a preset condition, the target image and the distorted image are input into an image quality evaluation network to obtain a quality evaluation score and a distortion type of the target image and the distorted image, the quality of the generated distorted image is controlled through a feedback control mechanism, the distorted image is efficiently and high-quality generated, the image quality data does not need to be manually labeled, the quality evaluation score and the distortion type of the target image are automatically output by the image quality evaluation network, and the image quality evaluation efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a flowchart of an image evaluation method provided by an embodiment of the present application; Figure 2 is a flowchart of generating a distorted image provided by an embodiment of the present application; Figure 3 is a feedback mechanism diagram provided by an embodiment of the present application; Figure 4 is a structure diagram of an image quality evaluation network provided by an embodiment of the present application; Figure 5is a Conformer structure schematic diagram provided by an embodiment of the present application. Figure 6 is a structure schematic diagram of an image evaluation device provided by an embodiment of the present application. Figure 7 is a structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0014] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a category and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in an "or" relationship.

[0015] Figure 1 An embodiment of the present application provides an image quality evaluation method, which can be executed by an electronic device, which can include a server and / or a terminal device. In other words, the method can be executed by software or hardware installed in the electronic device, and the method includes the following steps: Step S102: warp transformation is performed on the target image until the distortion image obtained by warp transformation on the target image meets a preset condition.

[0016] The image quality evaluation method provided by the embodiments of the present application can be applied to monitoring videos in a traffic monitoring scene. The subsequent embodiments take monitoring videos in a traffic monitoring scene as an example for description. The image quality average method provided by the embodiments of the present application can also be applied to video images in other scenes, and the application scene of the image quality average method of the embodiments of the present application is not specifically limited herein.

[0017] The embodiment of the application is directed to the traffic monitoring video image quality evaluation problem in the actual scene, realizes the image quality evaluation and distortion transformation type judgment without manual annotation, greatly reduces the difficulty of manual screening data, and improves the efficiency. In order to obtain a large number of images close to real traffic, the Feedback Mechanism for Image Quality Assessment (FMIQA) algorithm based on objective quality evaluation is designed, a large number of actual traffic monitoring image data are constructed, and images with higher quality difference are generated under the condition of meeting the data requirements of the experiment.

[0018] The FMIQA algorithm designed in the embodiment of the application constructs the self-supervised data set used in the embodiment of the application. By adding a feedback control mechanism and combining the SSIM method, the quality of the generated image data is controlled, the data set is efficiently and high-quality generated, and the types of image data are enriched. The key point mainly lies in the ingenious combination of the feedback mechanism, which is equivalent to image augmentation plus automatic control. FMIQA can adaptively adjust the distortion related parameters to control the quality of the generated distorted images and ensure the quality of the generated distorted images. Data is crucial to the performance of AI algorithms, and through quality control, the data set can have high quality.

[0019] In order to construct the actual data set, based on the public image quality evaluation data set TID2013, the FMIQA algorithm is designed to generate distorted images with different quality conditions, different climates, and different distortion degrees in the actual traffic monitoring scene. Due to the characteristics of the generation method, the generated distorted images contain two labels of distortion type and distortion degree, and these data will be used for self-supervised training of the model.

[0020] In the embodiment of the application, a custom number of video frames can be extracted from, for example, traffic monitoring videos, and images without signals, completely black and white, and blank images are screened out. The remaining images are the target images. In the embodiment of the application, different climate and season images can be generated from the target images using data augmentation methods, such as fogging, raining, and snowing the target images, to enrich the data types and enable the model to adapt to more diverse road traffic environments. At the same time, the quality of the generated images is controlled using the FMIQA method, and the augmented data and the original data are merged. Then, the FMIQA algorithm is used to perform distortion transformation on the target images.

[0021] In one implementation manner, the distortion transformation includes at least one of the following: brightness transformation, color cast transformation, blur transformation, occlusion transformation, black and white transformation, screen transformation, and noise transformation.

[0022] In the embodiments of the present application, seven types of abnormalities are provided, i.e., brightness abnormality, color cast abnormality, blur abnormality, occlusion abnormality, black and white abnormality, screen abnormality and noise abnormality. Each type of abnormality is divided into five distortion levels. Therefore, the embodiments of the present application provide seven distortion transformations, i.e., brightness transformation, color cast transformation, blur transformation, occlusion transformation, black and white transformation, screen transformation and noise transformation. At least one of the seven distortion transformations can be selected to perform distortion transformation on the target image. In the embodiments of the present application, some images that cannot be completely generated in five distortion levels can be deleted. The embodiments of the present application take the noise abnormality (noise transformation) algorithm as an example to show the process of some algorithms.

[0023] For noise transformation, various methods such as additive Gaussian noise, comfort noise, lossy compression, spatially dependent noise and high-frequency noise can be used to enrich the types of blur distortion, so that the obtained distorted image can better adapt to complex actual environments. In actual application, one method can be selected each time to perform blur transformation.

[0024] Figure 2 The process diagram for generating a distorted image with comfort noise is shown. Comfort noise is obtained by converting the target image into YCbCr form, processing Y, Cb and Cr components respectively, and obtaining the blurred distorted image. Taking the Y channel as an example: the Y channel is compressed by adaptive discrete cosine transform (ADCT) and then decompressed to obtain Y. By setting appropriate coefficients, Y is replaced by nY1-Y. Cb and Cr channels are processed in the same way to obtain Cbn and Crn. Then, the three channels are combined to convert into an RGB channel image, and a distorted image with comfort noise abnormality is obtained.

[0025] For other blur distortion generation methods, such as additive Gaussian noise, the target image is added with additive Gaussian noise and then ADCT encoded and decompressed, and the noise standard deviation is used to control the distortion level to obtain a distorted image. For example, spatially dependent noise, by changing the RGB three-channel color components of the target image and performing motion blur, the blur degree and transformation value are used to control the distortion degree, and a distorted image is obtained.

[0026] For other distortion types, appropriate distortion algorithms can also be designed, combined with FMIQA, and various types and different degrees of distortion transformation are used to finally obtain distorted images of different distortion types and different distortion degrees, which are used to form a data set for subsequent self-supervised learning of the model.

[0027] In this embodiment of the application, the distorted image obtained after distorting the target image may not meet the actual requirements. Therefore, this embodiment of the application cleverly incorporates a feedback mechanism into FMIQA, that is, it is necessary to determine whether the obtained distorted image meets the preset conditions. If the obtained distorted image does not meet the preset conditions, it is necessary to circulate the distorting transformation of the target image until the obtained distorted image meets the preset conditions.

[0028] In one implementation, the step of distorting the target image until the distorted image obtained by distorting the target image meets a preset condition includes: distorting the target image using preset control parameters; if the similarity value between the obtained distorted image and the target image does not meet the preset condition, adjusting the control parameters, and distorting the distorted image according to the adjusted control parameters until the similarity value between the distorted image and the target image meets the preset condition, wherein the preset condition includes that the similarity value between the distorted image and the target image is within a preset similarity range.

[0029] Figure 3 A schematic diagram of the feedback mechanism provided in an embodiment of this application is shown, such as... Figure 3 As shown, the FMIQA algorithm takes the target image as input. Different distortion transformation methods are designed based on different distortion types, and the degree of distortion transformation is controlled by the control parameter K. First, the target image is distorted to obtain a distorted image. Then, the objective quality assessment method SSIM is used as an indicator to measure the degree of distortion between the distorted image and the target image. The similarity (SSIM) value between the two is calculated. If the SSIM value is too large, exceeding the set threshold Tr, the similarity value between the distorted image and the target image is not within the preset similarity range, indicating that the generated distorted image is not significantly different from the target image, which is not conducive to the model's differentiation of different levels of distortion. Therefore, the generated image needs to be deleted, and the control parameter K is adjusted, and the distortion transformation is performed again. If the SSIM value is too small, less than the set threshold T... l If the similarity value between the distorted image and the target image is not within the preset similarity range, it indicates that the generated distorted image is too distorted. Therefore, this generated image needs to be deleted, the control parameter K needs to be readjusted, and the distortion transformation needs to be performed again, and this process is repeated. If the SSIM value is within the preset similarity range (T... l Within Tr), the generated distorted image is considered to meet the preset conditions.

[0030] In practical applications, for the same type of distortion, up to 5 different quality conditions of the distorted image can be generated, and the maximum cycle can be set to 10 times. If the SSIM value between the generated distorted image and the original target image is still not within the preset similarity range after the cycle ends, an effective distorted image cannot be obtained, and then it is considered that the target image is not suitable for application in the data set generation method, and therefore the target image can be deleted. Through this feedback mechanism, combined with the objective quality evaluation method, the quality of the generated distorted image is controlled, so that the image quality between the distorted image and the target image has a certain hierarchical difference, but it does not deviate completely from the target image.

[0031] wherein the similarity SSIM is often used to represent the similarity between the distorted image and the target image, and the specific calculation method is:

[0032] x denotes the target image, y denotes the distorted image, μ x denotes the mean value of the target image, μ y denotes the mean value of the distorted image, σ² x denotes the variance of the target image, σ² y denotes the variance of the distorted image, σ xy denotes the covariance of the target image and the distorted image, c 1 and c 2 are constants, used to avoid the denominator being 0. From the above formula, it can be seen that SSIM mainly compares the distorted image and the target image from the aspects of structure, brightness and contrast. Compared with MSE and PSNR, SSIM can better reflect the quality evaluation of the human eye on the image, and the larger the value is, the more similar the two are.

[0033] The FMIQA provided by the embodiments of the present application ingeniously combines the feedback mechanism, controls the image augmentation and automatic control, which is equivalent to controlling the quality of the generated data set, and generates the data set efficiently and with high quality, and enriches the types of data. Through the designed FMIQA method, a large amount of effective data set for self-supervised learning can be generated, and time cost can also be saved. This greatly expands the number of data sets, obtains a large amount of high-quality data, and is used for subsequent model training.

[0034] Step S104: inputting the target image and the distorted image into the image quality evaluation network to obtain the quality evaluation score and the distortion type of the target image and the distorted image.

[0035] In order to meet the needs of actual traffic monitoring scenes, reduce the dependence on labeled images, and assist related personnel in judging abnormal image types, an embodiment of the present application proposes an IQA method based on full reference. Without additional labeled data quality scores, the above target image and generated distorted image are directly used as a data set, and the quality score and transformation type can be learned. In the embodiment of the present application, after obtaining the target image and the distorted image, the target image and the distorted image can be input into the image quality evaluation network, and the quality evaluation score and the distortion type of the target image and the distorted image are output through the image quality evaluation network, as shown in Figure 4 The network quality evaluation network adopts the full reference IQA method, and needs the target image when applied. The network quality evaluation network is composed of a pair of twin networks and two branch networks, and the number of iterations of the transformation is replaced by the quality score. The weak label is constructed in combination with the distortion transformation type. After the target image and the distorted image pair are extracted by the twin network, they are respectively input into the quality evaluation branch and the distortion transformation classification branch, so as to realize the image quality evaluation and the abnormal type classification functions.

[0036] The image quality evaluation method provided by the embodiment of the present application can generate a distorted image by twisting and transforming a target image until the distorted image obtained by twisting and transforming the target image meets a preset condition; input the target image and the distorted image into an image quality evaluation network to obtain a quality evaluation score and a distortion type of the target image and the distorted image. The feedback mechanism is ingeniously combined, the image augmentation + automatic control is realized, the quality of the generated data is controlled, the data set is efficiently and high-quality generated, and the types of data are enriched. The FMIQA method designed by the present application can generate a large number of effective data sets for self-supervised learning, and can also save time cost. This greatly expands the number of data sets, obtains a large number of high-quality data, and is used for subsequent model training. The image quality evaluation efficiency is improved by automatically outputting the quality evaluation score and the distortion type of the target image through the image quality evaluation network. The embodiment of the present application does not need manual labeling of image data, eliminates the error factors caused by personal subjective misjudgment factors, improves the accuracy of the algorithm, can better overcome the shortcomings of traditional digital image processing technology, and better conforms to the trend of intelligent transportation.

[0037] In an implementation manner, the image quality evaluation network comprises a quality evaluation branch network, and the inputting the target image and the distorted image into the image quality evaluation network to obtain the quality evaluation score and the distortion type of the target image and the distorted image comprises: inputting the target image and the distorted image into the image quality evaluation network to extract a first feature of the target image and a second feature of the distorted image; calculating a feature difference between the first feature and the second feature; inputting the feature difference into the quality evaluation branch network to output the quality evaluation score of the target image and the distorted image.

[0038] In the embodiments of the present application, the target image X ref and the distorted image X dist after the distortion transformation can be input into a twin network as a pile of image pairs, and after feature extraction, a first feature Z ref of the target image and a second feature Z dist of the distorted image are obtained, and an L1 distance between the first feature Z ref and the second feature Z dist is calculated, that is, a feature difference |Z ref -Z dist | between the first feature and the second feature, and the generated feature difference is taken as an input of a quality evaluation branch network, and the quality evaluation score of the target image and the distorted image is output through the quality evaluation branch network.

[0039] The twin network adopts Inception-ResNet-v2 pre-trained from ImageNet to extract the features of the target image and the distorted image. And part of the intermediate layer feature maps of Inception-Resnet-V2 are connected, and the target image and the distorted image respectively obtain Z ref and Z dist after passing through the twin network. In order to obtain the difference information between the target image and the distorted image, |Z ref -Z dist | is used for calculation, and is taken as an input of the subsequent two branches.

[0040] In an implementation, the inputting the feature difference into the quality evaluation branch network and outputting a quality evaluation score of the target image and the distorted image include: converting the feature difference by a decoder-encoder module of the quality evaluation branch network to obtain a difference image; inputting the difference image into a multiple binary classifier to determine a reference quality score of the difference image; determining a prediction probability of each binary classifier of the multiple binary classifier according to the reference quality score and a score threshold of each binary classifier; and determining an average value of the prediction probability of each binary classifier as the quality evaluation score.

[0041] The feature difference |Z ref -Z dist may be first input into a decoder-encoder module of the quality evaluation branch network, and the feature difference is converted by the decoder-encoder module to obtain a difference image. Specifically, the decoder-encoder uses a single ConformerBlock. The Conformer is based on the Transformer network, adds a CNN structure, combines the local information association of the CNN and the global information interaction of the Transformer, and realizes efficient use of local and global features. The overall structure of the ConformerEncoder is shown in Figure 5 , and the main part is the ConformerBlock. Compared with the TransformerEncoder, an additional feedforward network and a convolution module are added, and the whole is composed of a first feedforward network, a multi-head attention module, a convolution module and a second feedforward network. The input x i When passing through the ConformerBlock, the calculation is as follows:

[0042] After the feature difference |Z ref -Z dist is input into the decoder-encoder module to obtain the difference image, a multiple binary classifier (MBC) is used to obtain a similarity index of the input image pair. The similarity index obtained by this branch is corresponding to the label value, so as to learn the quality score of the image. The loss function selects the BCELoss.

[0043] Specifically, the difference images are input into a multiple binary classifier (MBC), which predicts the quality score. Instead of a traditional regressor, MBC uses N binary classifiers to learn the quality score of the generated image. Specifically, the classifiers are trained based on whether the reference quality score of the generated difference image is greater than the score threshold of each binary classifier in the multiple binary classifier.

[0044] After inputting the difference image into a multi-binary classifier to determine the reference quality score of the difference image, the method further includes: determining a score threshold for each binary classifier based on the number of each binary classifier in the multi-binary classifier.

[0045] For the n One classifier, This represents a defined score threshold, and =( n -1) / , n =1, 2, ..., ,by p n Indicates the first n The prediction probabilities of each binary classifier, and the final quality score predicted by this branch is the prediction probabilities of each binary classifier. p ( x ref , x dist )=[ p 1, p 2, ..., p N The average value of ]. The MBC module will input pairs ( x ref , x dist Reference quality score Based on whether it is greater than the score threshold of each binary classifier Division, transformation A binary classification task was performed, and a new form was obtained. l ( x ref , x dist )=[ l 1, l 2, ..., l N ], for example, setting =5, and reference quality score =0.4, then the obtained quality fraction will be converted to y= [1, 1, 0, 0, 0]. In this way, the traditional quality score prediction task is converted into a classification task, meeting the above generation data form. The loss function selects BCELoss:

[0046] wherein P ( x ref , x dist ) represents the prediction vector of the branch, l ( x ref , x dist ) represents the binary vector after label conversion, in the embodiment of the application, = 5.

[0047] In an implementation mode, the image quality evaluation network includes an abnormal type classification branch network, and the inputting of the target image and the distorted image into the image quality evaluation network to obtain the quality evaluation score and the distortion type of the target image and the distorted image includes: inputting the target image and the distorted image into the image quality evaluation network to extract a first feature of the target image and a second feature of the distorted image; calculating a feature difference between the first feature and the second feature; and extracting and classifying the feature difference through a convolutional neural network and a multi-layer perceptron of the abnormal type classification branch network to obtain the distortion type.

[0048] In the embodiment of the application, the image quality evaluation network further includes an abnormal type classification branch network, and the inputting of the target image and the distorted image into the image quality evaluation network extracts a first feature Z ref of the target image and a second feature Z dist of the distorted image. For the abnormal type classification branch network, the feature difference |Z ref -Z dist | is further extracted and classified through a convolutional neural network (CNN) to then pass through a multi-layer perceptron (MLP) module to extract and classify the special vehicle, so as to learn the distortion type of the target image and the distorted image, and the loss function selects BCELoss.

[0049] For the abnormal type classification branch network, the feature difference |Z ref -Z dist | is extracted and classified through a 5-layer CNN and an MLP to extract and classify the classification features of the input, x ref , x distthe distortion type (such as color cast, blur, occlusion, and black and white) classification task of the target image. The loss adopts a BCELoss:

[0050] In the embodiments of the present application, the loss functions of the quality evaluation branch network and the abnormal type classification branch network are weighted and summed to obtain the final loss function of the quality evaluation network: = +

[0051] The data set used in the present application is the target image and the distortion image obtained by performing a distortion transformation on the target image. According to the characteristics of the generation algorithm, the generated distortion image contains two pieces of information: the distortion type and the distortion degree. The higher the distortion degree, the more distorted the image, and the lower the quality score of the distortion image, and the two are in a negative correlation relationship, so the (1-distortion level / total distortion level number) is used as the quality score of the coarse label for the quality evaluation branch. In addition, the distortion type is used as the label for the abnormal type classification branch.

[0052] The specific implementation process of the embodiments of the present application converts the regression problem into a classification problem, thereby effectively utilizing the previously generated data without the need for additional manually labeled image quality data.

[0053] The entire network of the embodiments of the present application does not need to manually label the quality score of the data, and can realize the functions of quality evaluation and abnormal type classification of the target image and the distortion image. The quality evaluation network first extracts the first feature of the target image and the second feature of the distortion image using a twin network, calculates the feature difference between the two, obtains the feature similarity between the two, and then passes through the quality evaluation branch network and the abnormal classification branch network, respectively, on the basis of fully utilizing the target image and the distortion image obtained by performing a distortion transformation on the target image, uses the distortion level as a coarse label to realize the evaluation of the image quality, and at the same time uses the distortion type to realize the judgment of the image abnormal category. The design of the network as a whole draws on the idea of the full-reference image quality evaluation method, but overcomes the difficulty of the full-reference method for massive labeled data, greatly relieving the pressure of data labeling.

[0054] The IQA algorithm provided in the embodiments of the present application takes the number of iterations and the type of distortion transformation as corresponding labels, without the need for manual labeling of images, thereby saving a large amount of labor cost and greatly improving efficiency. The algorithm can realize quality evaluation of a single image and judgment of the distortion type of the single image under the condition that there is a target image. The algorithm combines a twin network and takes a Conformer as an Encoder-Decoder module to further extract image macro and detail features and generate a quality evaluation branch. In order to intuitively display the distortion transformation type, the network further adds a classification branch. The quality evaluation network can finally output the quality score of the target object and the distorted image and the distortion transformation type. Not only can the quality of the picture be evaluated, but also the type of picture abnormality can be obtained, so that the output is more rich.

[0055] The image quality evaluation method provided in the embodiments of the present application combines a twin network and takes a Conformer as an Encoder-Decoder module to further extract image macro and detail features and generate a quality evaluation branch. In order to intuitively display the distortion transformation type, the network further adds a classification branch. The network can finally generate the quality score of the input pair and the distortion transformation type. Not only can the quality of the picture be evaluated, but also the type of picture abnormality can be obtained, so that the output is more rich. The method meets the requirement that no manual labeling is needed and no label data with quality scores is needed, realizes quality evaluation of an image and judgment of the distortion transformation type, greatly reduces the difficulty of manual data screening, and improves efficiency.

[0056] It should be noted that the image quality evaluation method provided in the embodiments of the present application can be executed by an image quality evaluation device or a control module in the image quality evaluation device for executing the image quality evaluation method. In the embodiments of the present application, the image quality evaluation device is taken as an example to illustrate the image quality evaluation device provided in the embodiments of the present application.

[0057] Figure 6 FIG. 1 is a structural schematic diagram of an image quality evaluation device according to the embodiments of the present application. As shown in FIG. 1, the image quality evaluation device 600 includes a transformation module 610 and an evaluation module 620. Figure 6

[0058] The transformation module 610 is configured to perform a distortion transformation on a target image until a distorted image obtained by performing the distortion transformation on the target image meets a preset condition. The evaluation module 620 is configured to input the target image and the distorted image into an image quality evaluation network to obtain a quality evaluation score and a distortion type of the target image and the distorted image.

[0059] ​In an implementation manner, the transformation module 610 is configured to perform a distortion transformation on the target image according to preset control parameters; in a case where a similarity value between the obtained distorted image and the target image does not meet a preset condition, the control parameters are adjusted, and the distorted image is transformed according to the adjusted control parameters until the similarity value between the distorted image and the target image meets the preset condition, and the preset condition includes that the similarity value between the distorted image and the target image is within a preset similarity range.

[0060] In an implementation manner, the image quality evaluation network includes a quality evaluation branch network, and the evaluation module 620 is configured to input the target image and the distorted image into the image quality evaluation network, extract a first feature of the target image and a second feature of the distorted image, calculate a feature difference between the first feature and the second feature, input the feature difference into the quality evaluation branch network, and output a quality evaluation score of the target image and the distorted image.

[0061] In an implementation manner, the evaluation module 620 is configured to convert the feature difference by using a decoder-encoder module of the quality evaluation branch network to obtain a difference image, input the difference image into a multi-binary classifier to determine a reference quality score of the difference image, determine a prediction probability of each score threshold of each binary classifier of the multi-binary classifier according to the reference quality score and the score threshold of each binary classifier, and determine an average value of the prediction probability of each binary classifier as the quality evaluation score.

[0062] In an implementation manner, the evaluation module 620 is further configured to determine the score threshold of each binary classifier according to a number of each binary classifier in the multi-binary classifier.

[0063] In an implementation manner, the image quality evaluation network includes an abnormal type classification branch network, and the evaluation module 620 is configured to input the target image and the distorted image into the image quality evaluation network, extract a first feature of the target image and a second feature of the distorted image, calculate a feature difference between the first feature and the second feature, and extract and classify the feature difference by using a convolutional neural network and a multi-layer perceptron of the abnormal type classification branch network to obtain the distortion type.

[0064] In an implementation manner, the distortion transformation includes at least one of the following: a brightness transformation, a color cast transformation, a blur transformation, an occlusion transformation, a black and white transformation, a screen distortion transformation, and a noise transformation.

[0065] The image quality evaluation device in the embodiments of the present application can be a device, or a component, an integrated circuit, or a chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and the like, and the non-mobile electronic device can be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the like, which are not limited in the embodiments of the present application.

[0066] The image quality evaluation device in the embodiments of the present application can be a device with an operating system. The operating system can be an Android operating system, an ios operating system, or other possible operating systems, which are not limited in the embodiments of the present application.

[0067] The image quality evaluation device provided in the embodiments of the present application can implement Figures 1 to 5 The method embodiments implement various processes, which are not repeated here to avoid repetition.

[0068] As shown in Figure 7 The embodiments of the present application further provide an electronic device 700, which includes a processor 701 and a memory 702, and the memory 702 stores programs or instructions executable on the processor 701, and the programs or instructions are executed by the processor 701 to implement: performing a distortion transformation on a target image until a distorted image obtained by performing the distortion transformation on the target image meets a preset condition; and inputting the target image and the distorted image into an image quality evaluation network to obtain a quality evaluation score and a distortion type of the target image and the distorted image.

[0069] In an implementation manner, the target image is distorted by a preset control parameter; in a case where a similarity value of the distorted image and the target image does not meet the preset condition, the control parameter is adjusted, and the distorted image is distorted according to the adjusted control parameter until the similarity value of the distorted image and the target image meets the preset condition, and the preset condition includes that the similarity value of the distorted image and the target image is within a preset similarity range.

[0070] In an implementation manner, the image quality evaluation network comprises a quality evaluation branch network, the target image and the distorted image are input into the image quality evaluation network, first features of the target image and second features of the distorted image are extracted, a feature difference between the first features and the second features is calculated, the feature difference is input into the quality evaluation branch network, and a quality evaluation score of the target image and the distorted image is output.

[0071] In an implementation manner, the feature difference is converted by a decoder-encoder module of the quality evaluation branch network to obtain a difference image, the difference image is input into a multi-binary classifier to determine a reference quality score of the difference image, according to the reference quality score and a score threshold of each binary classifier of the multi-binary classifier, a prediction probability of each binary classifier whose score threshold is less than or equal to the reference quality score is determined, and an average value of the prediction probabilities of each binary classifier is determined as the quality evaluation score.

[0072] In an implementation manner, after the difference image is input into the multi-binary classifier to determine the reference quality score of the difference image, according to a number of each binary classifier in the multi-binary classifier, a score threshold of each binary classifier is determined.

[0073] In an implementation manner, the image quality evaluation network comprises an abnormal type classification branch network, the target image and the distorted image are input into the image quality evaluation network, first features of the target image and second features of the distorted image are extracted, a feature difference between the first features and the second features is calculated, and the feature difference is extracted and classified by a convolutional neural network and a multi-layer perceptron of the abnormal type classification branch network to obtain the distortion type.

[0074] In an implementation manner, the distortion transformation comprises at least one of the following: a brightness transformation, a color cast transformation, a blur transformation, an occlusion transformation, a black and white transformation, a screen distortion transformation, and a noise transformation.

[0075] The specific implementation steps can refer to the steps of the image quality evaluation method embodiments, and the same technical effects can be achieved. To avoid repetition, the details are not described here.

[0076] It should be noted that the electronic device in the embodiments of the present application includes a server, a terminal, or other devices other than a terminal.

[0077] The above electronic device structure does not constitute a limitation on the electronic device, which can include more or fewer components than those shown, or combine some components, or have different arrangements of components, for example, the input unit can include a Graphics Processing Unit (GPU) and a microphone, and the display unit can be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes at least one of a touch panel and other input devices. The touch panel is also referred to as a touch screen. Other input devices can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, and the like, which are not described here.

[0078] The memory can be used to store software programs and various data. The memory can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), etc. In addition, the memory can include a volatile memory or a non-volatile memory, or the memory can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synchlink DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM).

[0079] The processor can include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes a wireless communication signal, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor.

[0080] The embodiment of the present application further provides a readable storage medium, wherein the readable storage medium stores a program or instructions, and the program or instructions are executed by a processor to implement each process of the above-mentioned image quality evaluation method embodiment and achieve the same technical effects. To avoid repetition, details are not described herein.

[0081] The processor is the processor in the electronic device in the above-mentioned embodiment. The readable storage medium includes a computer readable storage medium, such as a ROM, a RAM, a magnetic disc, or an optical disc.

[0082] The embodiment of the present application further provides a computer program product, wherein the computer program product includes a computer program stored on a non-transitory computer readable storage medium, and the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes each process of the above-mentioned image quality evaluation method embodiment and achieves the same technical effects. To avoid repetition, details are not described herein.

[0083] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusions, so that processes, methods, articles, or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or include elements inherent to such processes, methods, articles, or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article, or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to the order of performing functions as shown or discussed, and can also include performing functions in a substantially simultaneous manner or in a reverse order, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted, or combined. In addition, the features described with reference to certain examples can be combined in other examples.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned example methods can be realized by means of software and a necessary general hardware platform, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a computer software product in essence or in the form of a part that contributes to the prior art, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a plurality of instructions for causing a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0085] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative and not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. An image quality assessment method, characterized in that, include: The target image is distorted until the distorted image obtained by distorting the target image meets the preset conditions; The target image and the distorted image are input into an image quality assessment network to obtain the quality assessment scores and distortion types of the target image and the distorted image.

2. The method according to claim 1, characterized in that, The process of distorting the target image until the distorted image obtained by the distortion transformation meets preset conditions includes: The target image is distorted using preset control parameters; If the similarity value between the obtained distorted image and the target image does not meet the preset condition, the control parameters are adjusted, and the distorted image is distorted according to the adjusted control parameters until the similarity value between the distorted image and the target image meets the preset condition. The preset condition includes that the similarity value between the distorted image and the target image is within a preset similarity range.

3. The method according to claim 1, characterized in that, The image quality assessment network includes a quality assessment branch network. The process of inputting the target image and the distorted image into the image quality assessment network to obtain the quality assessment scores and distortion types of the target image and the distorted image includes: The target image and the distorted image are input into an image quality assessment network to extract the first feature of the target image and the second feature of the distorted image. Calculate the feature difference between the first feature and the second feature; The feature difference is input into the quality evaluation branch network, and the quality evaluation scores of the target image and the distorted image are output.

4. The method according to claim 3, characterized in that, The step of inputting the feature difference into the quality evaluation branch network and outputting the quality evaluation scores of the target image and the distorted image includes: The feature difference is transformed by the decoder-encoder module of the quality evaluation branch network to obtain a difference image; The difference image is input into a multi-binary classifier to determine a reference quality score for the difference image; Based on the reference quality score and the score threshold of each binary classifier of the multi-binary classifier, determine the predicted probability that the score threshold of each binary classifier is less than or equal to the reference quality score; The average predicted probability of each of the binary classifiers is determined as the quality evaluation score.

5. The method according to claim 4, characterized in that, After inputting the difference image into a multi-binary classifier to determine the reference quality score of the difference image, the method further includes: The score threshold for each binary classifier is determined based on the number of each binary classifier in the multiple binary classifier.

6. The method according to claim 1, characterized in that, The image quality assessment network includes an anomaly type classification branch network. The process of inputting the target image and the distorted image into the image quality assessment network to obtain the quality assessment scores and distortion types of the target image and the distorted image includes: The target image and the distorted image are input into an image quality assessment network to extract the first feature of the target image and the second feature of the distorted image. Calculate the feature difference between the first feature and the second feature; The distortion type is obtained by extracting and classifying the feature differences through the convolutional neural network and multilayer perceptron of the anomaly type classification branch network.

7. The method according to claim 1, characterized in that, The distortion transformation includes at least one of the following: brightness transformation, color cast transformation, blur transformation, occlusion transformation, black and white transformation, screen distortion transformation, and noise transformation.

8. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the image quality evaluation method as described in any one of claims 1-7.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the image quality evaluation method as described in any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the steps of the image quality evaluation method as described in any one of claims 1-7.

Images