Model training method and device, image quality evaluation method and device, equipment and medium

Abstract

The embodiment of the invention provides a model training method, an image quality evaluation method and device, equipment and a medium. Wherein the first network model comprises a first sub-model, a second sub-model and a third sub-model; the model training method comprises the following steps: inputting a plurality of first images into a first sub-model to obtain first visual information of the first images; inputting the first image into the second sub-model to obtain first weight information corresponding to the first visual information; inputting the first visual information and the first weight information into a third sub-model to obtain first quality information of the first image; determining perception quality information of the human vision system on the first image; training a second sub-model according to the difference between the first quality information and the perceived quality information; and iteratively executing the above steps until the difference satisfies a first condition, and obtaining a trained first network model.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and specifically to a model training method, an image quality evaluation method and apparatus, equipment, and medium. Background Art

[0002] After the image is processed by algorithms such as super-resolution and denoising, it needs to be evaluated for quality before it can be put into normal use. Otherwise, if the processed result is too different from the original image, or the expected effect is not achieved, the algorithm needs to be adjusted and reprocessed. However, there are currently two main image quality evaluation methods commonly used in the industry: subjective evaluation (manual) and objective evaluation (algorithm). Although subjective evaluation can reflect human visual perception, there are problems such as time consumption, high cost, and poor repeatability. Although objective evaluation methods can give results quickly and consistently, they may not be fully aligned with human eye perception, have limited applicability, lack context perception, may require parameter adjustment, and have problems such as complex calculations and limited generalization capabilities. Summary of the invention

[0003] The present application provides a model training method, an image quality evaluation method and apparatus, a device, and a medium, which are useful for obtaining image quality information that is consistent with the perceived quality information of the human visual system.

[0004] In a first aspect, an embodiment of the present application provides a model training method, wherein a first network model includes: a first sub-model, a second sub-model and a third sub-model; the method includes: inputting multiple first images into the first sub-model to obtain first visual information of the first image; inputting the first image into the second sub-model to obtain first weight information corresponding to the first visual information; inputting the first visual information and the first weight information into the third sub-model to obtain first quality information of the first image; determining the perceived quality information of the first image by the human visual system; training the second sub-model according to the difference between the first quality information and the perceived quality information; iteratively performing the above steps until the difference satisfies the first condition to obtain a trained first network model.

[0005] It can be understood that in the model training method provided in the embodiment of the present application, in the process of training the first network model, the second sub-model of the first network model is trained through the perceptual quality information of the first image and the first quality information of the first image by the human visual system; wherein the first quality information is obtained by the third sub-model based on the first visual information and the first weight information. In this way, it is beneficial to adjust the first weight information of the first visual information through the perceptual quality information of the first image by the human visual system, so as to obtain the first quality information consistent with the perceptual quality information of the human visual system, and further to obtain the quality information of the image consistent with the perceptual quality information of the human visual system based on the trained first network model.

[0006] In some embodiments, the first sub-model includes multiple objective evaluation algorithms, and the step of inputting multiple first images into the first sub-model to obtain first visual information of the first images includes: inputting the first images into multiple objective evaluation algorithms to respectively obtain objective evaluation values ​​of the first images; wherein the objective evaluation values ​​are used to characterize the visual information of the first images from different dimensions; and normalizing the objective evaluation values ​​to obtain the first visual information.

[0007] It can be understood that in the model training method provided in the embodiment of the present application, the objective evaluation value of the first image is obtained based on multiple different objective evaluation algorithms; wherein the objective evaluation value is used to characterize the visual information of the first image from different dimensions; the objective evaluation value is normalized to obtain the first visual information. In this way, since each objective evaluation value used to characterize the visual information of the first image has certain advantages, it also has certain limitations. Therefore, obtaining the first visual information based on the objective evaluation value of the visual information of the first image characterizing it from different dimensions is beneficial to improving the accuracy of the first visual information, thereby improving the performance of the first network model trained based on the first visual information, and further obtaining the quality information of the image consistent with the perceptual quality information of the human visual system based on the trained first network model.

[0008] In some embodiments, the training of the second sub-model based on the difference between the first quality information and the perceptual quality information includes: when the perceptual quality information is obtained based on a score for the quality of the first image, normalizing the first quality information based on the perceptual quality information to obtain second quality information; and training the second sub-model based on the difference between the second quality information and the perceptual quality information.

[0009] It can be understood that in the model training method provided in the embodiment of the present application, when the perceptual quality information is obtained based on the scoring of the quality of the first image, the first quality information is normalized according to the perceptual quality information to obtain the second quality information; and the second sub-model is trained according to the difference between the second quality information and the perceptual quality information. In this way, the quality information of the image and the perceptual quality information are in the same dimension, which is beneficial to determine the difference between the quality information of the image and the perceptual quality information, and further beneficial to the perceptual quality information of the first image through the human visual system, and the first weight information of the first visual information is adjusted to obtain the first quality information consistent with the perceptual quality information of the human visual system, and is beneficial to obtain the quality information of the image consistent with the perceptual quality information of the human visual system based on the trained first network model.

[0010] In some embodiments, the training of the second sub-model based on the difference between the first quality information and the perceptual quality information includes: when the perceptual quality information is based on image quality, and a plurality of the first images are sorted to obtain a first sorting result of the first images, sorting the plurality of the first quality information to obtain a second sorting result of the first images; and training the second sub-model based on the difference between the first sorting result and the second sorting result.

[0011] It can be understood that in the model training method provided in the embodiment of the present application, when the perceptual quality information is based on the image quality, and multiple first images are sorted to obtain the first sorting result of the first image, the multiple first quality information are sorted to obtain the second sorting result of the first image; and the second sub-model is trained according to the difference between the first sorting result and the second sorting result. In this way, the image quality information and the perceptual quality information are in the same dimension, which is beneficial to determine the difference between the image quality information and the perceptual quality information, and further beneficial to the perceptual quality information of the first image through the human visual system, and the first weight information of the first visual information is adjusted to obtain the first quality information consistent with the perceptual quality information of the human visual system, and beneficial to obtain the image quality information consistent with the perceptual quality information of the human visual system based on the trained first network model.

[0012] In second aspect, an embodiment of the present application provides an image quality assessment method, the method comprising: inputting a second image into a trained first network model to obtain third quality information of the second image; wherein the trained first network model is obtained by training the first network model based on the difference between the first quality information of the first image and the perceptual quality information of the first image, and the trained first network model is used to determine the quality of the second image.

[0013] It can be understood that in the image quality evaluation method provided in the embodiment of the present application, the first network model is trained using the difference between the first quality information based on the first image and the perceptual quality information of the first image, and the quality of the second image is determined by the trained first network model. In this way, since the trained first network model is obtained by training the first network model based on the difference between the first quality information and the perceptual quality information of the first image. Therefore, based on the trained first network model, it is beneficial to obtain image quality information that is consistent with the perceptual quality information of the human visual system.

[0014] In some embodiments, the trained first network model includes: a first sub-model, a trained second sub-model and a third sub-model; the inputting the second image into the trained first network model to obtain the third quality information of the second image includes: inputting the second image into the first sub-model to obtain the second visual information of the second image; inputting the second image into the trained second sub-model to obtain the second weight matrix corresponding to the second visual information; inputting the second visual information and the second weight matrix into the third sub-model to obtain the third quality information of the second image.

[0015] It can be understood that in the image quality evaluation method provided in the embodiment of the present application, the second image is input into the first sub-model to obtain the second visual information of the second image; the second image is input into the trained second sub-model to obtain the second weight matrix corresponding to the second visual information; the second visual information and the second weight matrix are input into the third sub-model to obtain the third quality information of the second image. In this way, since the second sub-model is obtained by adjusting the weight information based on the perceptual quality information of the first image. Therefore, based on the trained second sub-model, it is beneficial to obtain more accurate weight information of visual information, thereby obtaining image quality information consistent with the perceptual quality information of the human visual system.

[0016] In a third aspect, an embodiment of the present application provides a training device, comprising: a first input module, configured to input multiple first images into a first sub-model to obtain first visual information of the first image; a second input module, configured to input the first image into a second sub-model to obtain first weight information corresponding to the first visual information; a third input module, configured to input the first visual information and the first weight information into a third sub-model to obtain first quality information of the first image; a determination module, configured to determine the perceptual quality information of the first image by a human visual system; a training module, configured to train the second sub-model according to the difference between the first quality information and the perceptual quality information; an iteration module, configured to iteratively execute the above steps until the difference satisfies a first condition to obtain a trained first network model; wherein the first network model comprises: the first sub-model, the second sub-model and the third sub-model.

[0017] In a fourth aspect, an embodiment of the present application provides a quality evaluation device, comprising: a fourth input module, configured to input a second image into a trained first network model to obtain third quality information of the second image; wherein the trained first network model is obtained by training the first network model based on the difference between the first quality information of the first image and the perceptual quality information of the first image, and the trained first network model is used to determine the quality of the second image.

[0018] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the method described in the first aspect is implemented, or when the processor executes the program, the method described in the second aspect is implemented.

[0019] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the method described in the first aspect when the computer program is executed by a processor, or implements the method described in the second aspect when the computer program is executed by a processor.

[0020] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the method described in the first aspect, or implements the method described in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of an implementation flow of a model training method provided in an embodiment of the present application;

[0022] Figure 2A schematic diagram of an implementation flow of an image quality evaluation method provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of an image quality evaluation system provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of a processing flow of a subjective evaluation module provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of a processing flow of an objective evaluation module provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of a processing flow of a gate signal matrix optimization module provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of a training device provided in an embodiment of the present application;

[0028] Figure 8 A schematic diagram of an image quality evaluation device provided in an embodiment of the present application;

[0029] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0031] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0032] In the following description, reference is made to “some embodiments\other embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments\other embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0035] (1) Subjective evaluation of image quality

[0036] The subjective evaluation of image quality depends on human observation and scoring. Common methods include Mean Opinion Score (MOS) and Differential Mean Opinion Score (DMOS).

[0037] Among them, MOS is used to evaluate the subjective quality of images. Observers rate images based on factors such as image details, clarity, distortion, and noise, and then calculate the average score of all observers as the subjective quality score of the image. Based on the subjective evaluation of MOS, a limited number of 1 to 5 or 1 to 7 is usually used for rating, with 1 representing the worst or complete disagreement and 5 (or 7) representing the best or complete agreement.

[0038] DMOS is used to measure the quality difference between distorted and undistorted images. Observers rate the undistorted and distorted images respectively, and then calculate the difference between the two. Generally, the smaller the DMOS value, the smaller the quality difference between the distorted and undistorted images, and the better the image quality; conversely, the larger the DMOS value, the greater the quality difference between the distorted and undistorted images, and the worse the image quality.

[0039] (2) Objective evaluation of image quality

[0040] Objective evaluation of image quality refers to the use of mathematical models and algorithms to automatically evaluate image quality without human intervention. Common methods include: Full-Reference (FR) indicator, Reduced-Reference (RR) indicator, and No-Reference (NR) indicator.

[0041] Among them, the FR index refers to selecting an ideal image as a reference image, comparing the difference between the image to be evaluated and the reference image, analyzing the degree of distortion of the image to be evaluated, and thus obtaining a quality assessment of the image to be evaluated. Algorithms belonging to the FR index include: Peak Signal-to-Noise Ratio (PSNR), Mean Squared Error (MSE), Structural Similarity Index (SSIM), etc.

[0042] The RR index is used when the complete original image cannot be obtained, but some information is available. Algorithms belonging to the RR index include: Reduced-Reference Structural Similarity Index (RR-SSIM).

[0043] The NR index evaluates the quality only through the processed image itself, without the need for the original image. The algorithms belonging to the NR index include: Natural Image Quality Evaluator (NIQE); Blind / Referenceless Image Spatial Quality Evaluator (BRISQUE), etc.

[0044] In the related art, image quality evaluation methods are mainly divided into two categories: subjective evaluation and objective evaluation. Although subjective evaluation can reflect human visual perception, it has problems such as time consumption, high cost, and poor repeatability. Although objective evaluation can give results quickly and consistently, it may not be fully aligned with human eye perception, has limited applicability, lacks context perception, may require parameter adjustment, and has problems such as complex calculations and limited generalization capabilities.

[0045] In view of this, an embodiment of the present application provides a model training method, wherein the first network model includes: a first sub-model, a second sub-model, and a third sub-model; Figure 1 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes the following steps 101 to 106:

[0046] Step 101, inputting a plurality of first images into a first sub-model to obtain first visual information of the first images;

[0047] Step 102, inputting the first image into the second sub-model to obtain first weight information corresponding to the first visual information;

[0048] Step 103, inputting the first visual information and the first weight information into the third sub-model to obtain first quality information of the first image;

[0049] Step 104, determining the perception quality information of the first image by the human visual system;

[0050] Step 105, training the second sub-model according to the difference between the first quality information and the perceived quality information;

[0051] Step 106, iteratively execute the above steps until the difference satisfies the first condition, thereby obtaining a trained first network model.

[0052] It can be understood that in the model training method provided in the embodiment of the present application, in the process of training the first network model, the second sub-model of the first network model is trained through the perceptual quality information of the first image and the first quality information of the first image by the human visual system; wherein the first quality information is obtained by the third sub-model based on the first visual information and the first weight information. In this way, it is beneficial to adjust the first weight information of the first visual information through the perceptual quality information of the first image by the human visual system, so as to obtain the first quality information consistent with the perceptual quality information of the human visual system, and further to obtain the quality information of the image consistent with the perceptual quality information of the human visual system based on the trained first network model.

[0053] The following describes further optional implementations and related terms of each of the above steps.

[0054] In step 101, a plurality of first images are input into a first sub-model to obtain first visual information of the first images.

[0055] It should be understood that in the embodiments of the present application, the plurality of first images are not limited, and the first images are used to train the first network model. In some embodiments, the first images include: images processed by algorithms such as super-resolution, denoising, compression, and enhancement.

[0056] In the embodiment of the present application, the first sub-model is not limited, and the first sub-model is used to determine the visual information of the first image. In some embodiments, the first sub-model includes multiple objective evaluation algorithms, and the inputting of multiple first images into the first sub-model to obtain the first visual information of the first image includes: inputting the first image into multiple objective evaluation algorithms to obtain objective evaluation values ​​of the first image respectively; wherein the objective evaluation values ​​are used to characterize the visual information of the first image from different dimensions; and the objective evaluation values ​​are normalized to obtain the first visual information.

[0057] It can be understood that in the model training method provided in the embodiment of the present application, the objective evaluation value of the first image is obtained based on multiple different objective evaluation algorithms; wherein the objective evaluation value is used to characterize the visual information of the first image from different dimensions; the objective evaluation value is normalized to obtain the first visual information. In this way, since each objective evaluation value used to characterize the visual information of the first image has certain advantages, it also has certain limitations. Therefore, obtaining the first visual information based on the objective evaluation value of the visual information of the first image characterizing it from different dimensions is beneficial to improving the accuracy of the first visual information, thereby improving the performance of the first network model trained based on the first visual information, and further obtaining the quality information of the image consistent with the perceptual quality information of the human visual system based on the trained first network model.

[0058] It should be understood that in the embodiments of the present application, the objective evaluation algorithm is not limited, and the objective evaluation algorithm can be preset, or determined based on the first image, or determined based on the application scenario. In some embodiments, the objective evaluation algorithm includes at least one of the following: an objective evaluation algorithm belonging to a full reference indicator, an objective evaluation algorithm belonging to a partial reference indicator, and an objective evaluation algorithm belonging to a no-reference indicator; wherein the algorithms belonging to the full reference indicator include: PSNR, MSE, SSIM, Mean Absolute Error (MAE), Signal to Noise Ratio (SNR), Information Fidelity Criterion (IFC), Visual Information Fidelity (VIF), multi-scale SSIM, information content weighted SSIM, feature similarity based on scale transformation, feature similarity, etc.; algorithms belonging to partial reference indicators include: RR-SSIM, etc.; algorithms belonging to no-reference indicators include: NIQE, BRISQUE, etc.

[0059] It should be noted that, since it is impossible for the embodiments of the present application to enumerate all objective evaluation algorithms, even if some objective evaluation algorithms are not among the objective evaluation algorithms listed above, they should also be within the protection scope of the present application.

[0060] It should be understood that in the embodiments of the present application, the visual information of the image is not limited. In some embodiments, the visual information of the image includes: visual information such as brightness, contrast, clarity, color fidelity, noise, and structural information of the image.

[0061] In step 102, the first image is input into the second sub-model to obtain first weight information corresponding to the first visual information.

[0062] It should be understood that in the embodiment of the present application, the second sub-model is not limited, and the second sub-model is used to determine the weight information corresponding to the visual information. And the second sub-model is used to output the first weight information of the same dimension as the first visual information. In some embodiments, the first weight information includes: the weights corresponding to the objective evaluation values.

[0063] In an embodiment of the present application, the second sub-model is a neural network model. In some embodiments, the second sub-model is a convolutional neural network; in other embodiments, the second sub-model is a residual network; in still other embodiments, the second sub-model is a deep convolutional neural network.

[0064] In some embodiments, the second sub-model includes multiple types of neural network models. Exemplarily, in some embodiments, the second sub-model includes at least two of the following neural network models: recurrent convolutional neural network, generative adversarial convolutional neural network, long short-term memory network, feedforward neural network, multi-layer perceptron, convolutional neural network including attention mechanism, deep belief network, etc.

[0065] In step 103, the first visual information and the first weight information are input into the third sub-model to obtain first quality information of the first image.

[0066] It should be understood that in the embodiments of the present application, the third sub-model is not limited. In some embodiments, the third sub-model is used to determine the product of the first visual information and the first weight information, and to use the product of the first visual information and the first weight information as the first quality information of the first image.

[0067] In step 104, the perceived quality information of the first image by the human visual system is determined.

[0068] It should be understood that in the embodiments of the present application, the perceptual quality information is not limited. The perceptual quality information is used to characterize the perceptual quality of the first image by the human visual system. In some embodiments, the perceptual quality information may be the mean opinion score of the first image, the perceptual quality information may also be the difference mean opinion score of the first image, or the perceptual quality information may also be the quality ranking result of the first image.

[0069] In step 105, the second sub-model is trained according to the difference between the first quality information and the perceptual quality information.

[0070] In some embodiments, the training of the second sub-model based on the difference between the first quality information and the perceptual quality information includes: when the perceptual quality information is obtained based on a score for the quality of the first image, normalizing the first quality information based on the perceptual quality information to obtain second quality information; and training the second sub-model based on the difference between the second quality information and the perceptual quality information.

[0071] It can be understood that in the model training method provided in the embodiment of the present application, when the perceptual quality information is obtained based on the scoring of the quality of the first image, the first quality information is normalized according to the perceptual quality information to obtain the second quality information; and the second sub-model is trained according to the difference between the second quality information and the perceptual quality information. In this way, the quality information of the image and the perceptual quality information are in the same dimension, which is beneficial to determine the difference between the quality information of the image and the perceptual quality information, and further beneficial to the perceptual quality information of the first image through the human visual system, and the first weight information of the first visual information is adjusted to obtain the first quality information consistent with the perceptual quality information of the human visual system, and is beneficial to obtain the quality information of the image consistent with the perceptual quality information of the human visual system based on the trained first network model.

[0072] It should be understood that in the embodiments of the present application, the scoring criteria for scoring the quality of the first image are not limited, and the type of the numerical value of the score is not limited. In some embodiments, a scoring range of 1-5 points is used, where 1 point indicates extremely poor image quality and 5 points indicates excellent image quality. In other embodiments, a scoring range of 0-100 points is used, where 0 points indicates the worst image quality and 100 points indicates the best image quality.

[0073] In some embodiments, the training of the second sub-model based on the difference between the first quality information and the perceptual quality information includes: when the perceptual quality information is based on image quality, and a plurality of the first images are sorted to obtain a first sorting result of the first images, sorting the plurality of the first quality information to obtain a second sorting result of the first images; and training the second sub-model based on the difference between the first sorting result and the second sorting result.

[0074] It can be understood that in the model training method provided in the embodiment of the present application, when the perceptual quality information is based on the image quality, and multiple first images are sorted to obtain the first sorting result of the first image, the multiple first quality information are sorted to obtain the second sorting result of the first image; and the second sub-model is trained according to the difference between the first sorting result and the second sorting result. In this way, the image quality information and the perceptual quality information are in the same dimension, which is beneficial to determine the difference between the image quality information and the perceptual quality information, and further beneficial to the perceptual quality information of the first image through the human visual system, and the first weight information of the first visual information is adjusted to obtain the first quality information consistent with the perceptual quality information of the human visual system, and beneficial to obtain the image quality information consistent with the perceptual quality information of the human visual system based on the trained first network model.

[0075] It should be understood that in the embodiments of the present application, the sorting order is not limited, and the sorting order corresponding to the first sorting result and the sorting order corresponding to the second sorting result can be consistent. In some embodiments, the sorting order can be to sort the image quality from best to worst, or the sorting order can be to sort the image quality from worst to best.

[0076] Step 106, iteratively execute the above steps until the difference satisfies the first condition, thereby obtaining a trained first network model.

[0077] In some embodiments, when the difference is less than or equal to the difference threshold, the training of the first network model is stopped; or, when the second sub-model converges, the training of the first network model is stopped; or, when the number of training times is equal to a preset number of training times, the training of the first network model is stopped.

[0078] It should be understood that in the embodiment of the present application, the difference threshold is not limited. The difference threshold can be preset or set according to actual conditions.

[0079] An embodiment of the present application provides an image quality assessment method, the method comprising: inputting a second image into a trained first network model to obtain third quality information of the second image; wherein the trained first network model is obtained by training a first network model based on the difference between the first quality information of the first image and the perceptual quality information of the first image, and the trained first network model is used to determine the quality of the second image.

[0080] It can be understood that in the image quality evaluation method provided in the embodiment of the present application, the first network model is trained using the difference between the first quality information based on the first image and the perceptual quality information of the first image, and the quality of the second image is determined by the trained first network model. In this way, since the trained first network model is obtained by training the first network model based on the difference between the first quality information and the perceptual quality information of the first image. Therefore, based on the trained first network model, it is beneficial to obtain image quality information that is consistent with the perceptual quality information of the human visual system.

[0081] In some embodiments, the trained first network model includes: a first sub-model, a trained second sub-model, and a third sub-model; Figure 2 A schematic diagram of an implementation flow of an image quality evaluation method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes the following steps 201 to 203:

[0082] Step 201, inputting the second image into the first sub-model to obtain second visual information of the second image;

[0083] Step 202, inputting the second image into the trained second sub-model to obtain a second weight matrix corresponding to the second visual information;

[0084] Step 203: input the second visual information and the second weight matrix into the third sub-model to obtain third quality information of the second image.

[0085] It can be understood that in the image quality evaluation method provided in the embodiment of the present application, the second image is input into the first sub-model to obtain the second visual information of the second image; the second image is input into the trained second sub-model to obtain the second weight matrix corresponding to the second visual information; the second visual information and the second weight matrix are input into the third sub-model to obtain the third quality information of the second image. In this way, since the second sub-model is obtained by adjusting the weight information based on the perceptual quality information of the first image. Therefore, based on the trained second sub-model, it is beneficial to obtain more accurate weight information of visual information, thereby obtaining image quality information consistent with the perceptual quality information of the human visual system.

[0086] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0087] The embodiment of the present application provides a novel image quality evaluation system that integrates the perceptual alignment of subjective evaluation and the rapid consistency of objective evaluation. The system controls the proportion of objective evaluation results (i.e., an example of visual information) through a gated signal matrix (i.e., an example of a weight matrix), and continuously feeds back the optimization matrix parameters (i.e., the parameters of the second sub-model) through subjective evaluation (i.e., an example of perceptual quality information), thereby training a highly adaptable gated signal matrix for different scenarios, and realizing the fusion of subjective and objective evaluations.

[0088] Figure 3 A schematic diagram of an image quality evaluation system provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the system 30 includes: a subjective evaluation module 301 , an objective evaluation module 302 , and a gated signal matrix optimization module 303 .

[0089] Figure 4 A processing flow diagram of a subjective evaluation module provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the subjective evaluation module 301 is used to perform the following steps:

[0090] Step 401, collecting human eye evaluation data;

[0091] Step 402, calculating the mean opinion score;

[0092] Step 403, calculating the difference mean opinion score.

[0093] It should be understood that in the embodiment of the present application, the execution order of step 402 and step 403 is not limited.

[0094] Figure 5 A processing flow diagram of an objective evaluation module provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the objective evaluation module 302 is used to perform the following steps:

[0095] Step 501, calculating an objective evaluation value using all reference indicators;

[0096] Step 502, calculating an objective evaluation value using some reference indicators;

[0097] Step 503, calculating the objective evaluation value using the reference-free index;

[0098] Step 504: Generate an objective evaluation matrix according to the objective evaluation values.

[0099] It should be understood that in the embodiment of the present application, the execution order of steps 501 to 503 is not limited.

[0100] Figure 6A schematic diagram of a processing flow of a gate signal matrix optimization module provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the gated signal matrix optimization module 303 is used to perform the following steps:

[0101] Step 601, obtaining an objective evaluation matrix;

[0102] Step 602, multiplying the objective evaluation matrix and the gated signal matrix to obtain a comprehensive evaluation value;

[0103] Step 603, comparing the comprehensive evaluation value with the subjective evaluation result;

[0104] Step 604, using an optimization algorithm to adjust matrix parameters;

[0105] Step 605: iterative training to optimize the gate signal matrix.

[0106] In some embodiments, the optimization of the gating signal matrix optimization module includes: taking the objective evaluation matrix as input; multiplying the objective evaluation matrix with the gating signal matrix to obtain a comprehensive evaluation value; comparing the comprehensive evaluation value with the subjective evaluation result and calculating the difference; using an optimization algorithm (such as a gradient descent method) to adjust the parameters of the gating signal matrix to reduce the difference between the comprehensive evaluation value and the subjective evaluation result, so that the comprehensive evaluation value is closer to the subjective evaluation result; through iterative training, the gating signal matrix is ​​continuously optimized so that the comprehensive evaluation value is closer to the subjective evaluation result, and different matrix parameters are generated for different scenarios.

[0107] It can be understood that the beneficial effects that can be obtained by optimizing the gated signal matrix optimization module in the above manner include: (1) Improving evaluation accuracy: By integrating subjective evaluation and objective evaluation, it can more accurately reflect human visual perception and improve the accuracy of image quality evaluation. (2) Reducing evaluation costs: This method has a high degree of automation and reduces the time and cost of manual participation. (3) Enhancing applicability: Through the optimization of the gated signal matrix, the method can adjust the evaluation strategy for different scenarios, thereby improving the generalization ability and applicability of the system. (4) Improving evaluation efficiency: The system can quickly and consistently give evaluation results, saving a lot of computing time. Different gated signal matrices are trained for different image processing scenarios to enhance the generalization ability and applicability of the system.

[0108] In some embodiments, system initialization includes: initializing the subjective evaluation module 301: setting the evaluation criteria and scoring method; initializing the objective evaluation module 302: selecting appropriate objective evaluation indicators and algorithms, and setting the method for generating an objective evaluation matrix; initializing the gated signal matrix optimization module 303: setting the initial matrix parameters and optimization algorithm.

[0109] In some embodiments, the collection and processing of subjective evaluations includes: selecting a certain number of participants to give subjective scores by observing the processed images; counting the participants' scoring results, calculating the mean opinion score (MOS) and the difference mean opinion score (DMOS); and inputting the subjective scoring results into the system as benchmark data for optimizing the gating signal matrix.

[0110] In some embodiments, the calculation and processing of objective evaluation includes: evaluating the processed image using full reference indicators, partial reference indicators and no reference indicators, and generating an objective evaluation matrix including PSNR, SSIM, NIQE, BRISQUE, etc.

[0111] In the actual application of image quality evaluation, the image quality evaluation system can quickly give evaluation results and guide the optimization and adjustment of image processing algorithms. The image quality evaluation system has strong scalability. For different needs and scenarios, according to the subjective evaluation results, the weights corresponding to the objective evaluation values ​​obtained based on different objective evaluation algorithms in the objective evaluation matrix are adjusted to meet different image quality evaluation needs.

[0112] Through the above specific implementation methods, this application realizes the effective integration of subjective evaluation and objective evaluation, and improves the accuracy, efficiency and applicability of image quality evaluation. The system has high practical value and broad application prospects, and can play an important role in image processing, video surveillance, medical imaging and other fields.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present application, which should be included in the scope of protection of the present application.

[0114] It should be noted that although the steps of the method in the present application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution, etc.; or, the steps in different embodiments may be combined into a new technical solution. Based on the foregoing embodiments, an embodiment of the present application provides a device, which includes the modules included and the units included in the modules, which can be implemented by a processor; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor may be an AI acceleration engine (such as NPU, etc.), a graphics processor (GPU), a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0115] Figure 7 A schematic diagram of a training device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the training device 70 includes: a first input module 701, a second input module 702, a third input module 703, a determination module 704, a training module 705 and an iteration module 706; wherein,

[0116] A first input module 701 is configured to input a plurality of first images into a first sub-model to obtain first visual information of the first images;

[0117] A second input module 702 is configured to input the first image into a second sub-model to obtain first weight information corresponding to the first visual information;

[0118] A third input module 703 is configured to input the first visual information and the first weight information into a third sub-model to obtain first quality information of the first image;

[0119] A determination module 704 is configured to determine the perception quality information of the first image by the human visual system;

[0120] A training module 705, configured to train the second sub-model according to the difference between the first quality information and the perceptual quality information;

[0121] The iteration module 706 is configured to iteratively execute the above steps until the difference satisfies the first condition, thereby obtaining a trained first network model; wherein the first network model includes: the first sub-model, the second sub-model and the third sub-model.

[0122] In some embodiments, the first sub-model includes multiple objective evaluation algorithms, and the first input module 701 is configured to input the first image into the multiple objective evaluation algorithms to obtain objective evaluation values ​​of the first image respectively; wherein the objective evaluation values ​​are used to characterize the visual information of the first image from different dimensions; and the objective evaluation values ​​are normalized to obtain the first visual information.

[0123] In some embodiments, the training module 705 is configured to normalize the first quality information according to the perceptual quality information to obtain second quality information when the perceptual quality information is obtained based on a score for the quality of the first image; and train the second sub-model according to the difference between the second quality information and the perceptual quality information.

[0124] In some embodiments, the training module 705 is configured to, when the perceptual quality information is based on image quality and a first sorting result of the first images is obtained by sorting a plurality of the first quality information, sort the plurality of the first quality information to obtain a second sorting result of the first images; and train the second sub-model according to the difference between the first sorting result and the second sorting result.

[0125] Figure 8 A schematic diagram of an image quality evaluation device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the image quality evaluation device 80 includes: a fourth input module 801; wherein,

[0126] The fourth input module 801 is configured to input the second image into the trained first network model to obtain third quality information of the second image; wherein the trained first network model is obtained by training the first network model based on the difference between the first quality information of the first image and the perceptual quality information of the first image, and the trained first network model is used to determine the quality of the second image.

[0127] In some embodiments, the trained first network model includes: a first sub-model, a trained second sub-model and a third sub-model; the fourth input module 801 is configured to: input the second image into the first sub-model to obtain second visual information of the second image; input the second image into the trained second sub-model to obtain a second weight matrix corresponding to the second visual information; input the second visual information and the second weight matrix into the third sub-model to obtain third quality information of the second image.

[0128] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0129] It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.

[0130] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0131] An embodiment of the present application provides an electronic device, Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the electronic device 90 includes a memory 901 and a processor 902, wherein the memory 901 stores a computer program that can be run on the processor 902, and the processor 902 implements the steps in the method provided in the above embodiment when executing the program.

[0132] It should be noted that the memory 901 is configured to store instructions and applications executable by the processor 902, and can also cache data to be processed or processed by the processor 902 and various modules in the electronic device 90, which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0133] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the above embodiment are implemented.

[0134] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the method provided in the above method embodiment.

[0135] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0136] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similarities can be referenced to each other. For the sake of brevity, this article will not repeat them.

[0137] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0138] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0139] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0140] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0141] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0142] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0143] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0144] The methods disclosed in several method embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments. The features disclosed in several product embodiments provided in this application can be combined arbitrarily without conflict to obtain new product embodiments.

[0145] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0146] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: The first network model includes: a first sub-model, a second sub-model and a third sub-model; the method includes: Inputting a plurality of first images into the first sub-model to obtain first visual information of the first images; Inputting the first image into the second sub-model to obtain first weight information corresponding to the first visual information; Inputting the first visual information and the first weight information into the third sub-model to obtain first quality information of the first image; Determining the perceived quality information of the first image by a human visual system; training the second sub-model according to a difference between the first quality information and the perceptual quality information; The above steps are iteratively performed until the difference satisfies the first condition, thereby obtaining a trained first network model.

2. The model training method according to claim 1, characterized in that: The first sub-model includes a plurality of objective evaluation algorithms, and the step of inputting a plurality of first images into the first sub-model to obtain first visual information of the first images includes: Inputting the first image into a plurality of objective evaluation algorithms to obtain objective evaluation values ​​of the first image respectively; wherein the objective evaluation values ​​are used to characterize visual information of the first image from different dimensions; The objective evaluation value is normalized to obtain the first visual information.

3. The model training method according to claim 1 or 2, characterized in that: The training of the second sub-model according to the difference between the first quality information and the perceptual quality information includes: In a case where the perceptual quality information is obtained based on scoring the quality of the first image, normalizing the first quality information according to the perceptual quality information to obtain second quality information; The second sub-model is trained according to the difference between the second quality information and the perceptual quality information.

4. The model training method according to claim 1 or 2, characterized in that: The training of the second sub-model according to the difference between the first quality information and the perceptual quality information includes: In a case where the perceptual quality information is a first sorting result of the first images obtained by sorting the plurality of first quality information based on image quality, sorting the plurality of first quality information to obtain a second sorting result of the first images; The second sub-model is trained according to the difference between the first sorting result and the second sorting result.

5. A method for evaluating image quality, characterized in that: The method comprises: The second image is input into the trained first network model to obtain third quality information of the second image; wherein the trained first network model is obtained by training the first network model based on the difference between the first quality information of the first image and the perceptual quality information of the first image, and the trained first network model is used to determine the quality of the second image.

6. The image quality evaluation method according to claim 5, characterized in that: The trained first network model includes: a first sub-model, a trained second sub-model and a third sub-model; and inputting the second image into the trained first network model to obtain the third quality information of the second image includes: Inputting the second image into the first sub-model to obtain second visual information of the second image; Inputting the second image into the trained second sub-model to obtain a second weight matrix corresponding to the second visual information; The second visual information and the second weight matrix are input into the third sub-model to obtain third quality information of the second image.

7. A training device, characterized in that: The device comprises: A first input module, configured to input a plurality of first images into a first sub-model to obtain first visual information of the first images; a second input module, configured to input the first image into a second sub-model to obtain first weight information corresponding to the first visual information; a third input module, configured to input the first visual information and the first weight information into a third sub-model to obtain first quality information of the first image; A determination module, configured to determine the perception quality information of the first image by a human visual system; a training module, configured to train the second sub-model according to a difference between the first quality information and the perceptual quality information; The iteration module is configured to iteratively execute the above steps until the difference satisfies the first condition, thereby obtaining a trained first network model; wherein the first network model includes: the first sub-model, the second sub-model and the third sub-model.

8. A quality evaluation device, characterized in that: The device comprises: The fourth input module is configured to input the second image into the trained first network model to obtain third quality information of the second image; wherein the trained first network model is obtained by training the first network model based on the difference between the first quality information of the first image and the perceptual quality information of the first image, and the trained first network model is used to determine the quality of the second image.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented, or when the processor executes the program, the method according to claim 5 or 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented; or when the computer program is executed by a processor, the method according to claim 5 or 6 is implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method described in any one of claims 1 to 4 is implemented, or the method described in claim 5 or 6 is implemented.

Images