Image quality assessment method, device and equipment

Abstract

The embodiment of the present application discloses an image quality assessment method, apparatus, and device, the method comprising inputting a fourth reference image and a fourth image to be assessed into an image quality assessment model to obtain an assessment prediction result of the fourth image to be assessed; the image quality assessment model is trained based on a first image difference encoding result between a first reference image and a first image to be assessed, a second image difference encoding result between a second reference image and a second image to be assessed, an assessment prediction result of the second image to be assessed, and an assessment calibration result of the second image to be assessed; the first reference image has the same image content as the first image to be assessed; and the second reference image has the same or different image content as the second image to be assessed. In this way, when the image quality assessment model is applied to perform image quality assessment, a relatively accurate image quality assessment result can also be obtained when a high-definition image that is unrelated to the content of the image to be assessed is used as a reference.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to an image quality assessment method, device, and equipment. Background Art

[0002] In the field of image processing, image quality assessment has a wide range of applications. Reliable image quality assessment methods are helpful in filtering out low-quality image data and improving image review efficiency.

[0003] There are various image quality assessment methods, such as full-reference quality assessment and no-reference quality assessment. Full-reference quality assessment uses a high-definition image with the same content and size as the image to be assessed as a reference, while no-reference quality assessment uses manually calibrated scores as a reference.

[0004] The image quality assessment results obtained by the currently used full-reference quality assessment methods and no-reference quality assessment methods are both inaccurate. Therefore, how to train a better image quality assessment model is an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present application provide an image quality assessment method, apparatus, and device, which can improve the accuracy of image quality assessment.

[0006] To solve the above problems, the technical solutions provided in the embodiments of the present application are as follows:

[0007] In a first aspect, a method for image quality assessment is provided, the method comprising:

[0008] Acquiring a fourth reference image and a fourth image to be evaluated, wherein the fourth reference image and the fourth image to be evaluated have the same or different image contents;

[0009] Inputting the fourth reference image and the fourth image to be evaluated into an image quality assessment model to obtain an evaluation prediction result of the fourth image to be evaluated;

[0010] The image quality assessment model is trained based on a first image difference coding result between a first reference image and a first image to be evaluated, a second image difference coding result between a second reference image and a second image to be evaluated, an assessment prediction result of the second image to be evaluated, and an assessment calibration result of the second image to be evaluated; wherein, the first reference image has the same image content as the first image to be evaluated, and the image quality of the first reference image is not lower than that of the first image to be evaluated; the second reference image has the same or different image content as the second image to be evaluated, and the image quality of the second reference image is not lower than that of the second image to be evaluated.

[0011] According to a second aspect, an image quality assessment device is provided, the device comprising:

[0012] an acquiring unit, configured to acquire a fourth reference image and a fourth image to be evaluated, wherein the image contents of the fourth reference image and the fourth image to be evaluated are the same or different;

[0013] an input unit, configured to input the fourth reference image and the fourth image to be evaluated into an image quality assessment model to obtain an assessment prediction result of the fourth image to be evaluated;

[0014] The image quality assessment model is trained based on a first image difference coding result between a first reference image and a first image to be evaluated, a second image difference coding result between a second reference image and a second image to be evaluated, an assessment prediction result of the second image to be evaluated, and an assessment calibration result of the second image to be evaluated; wherein, the first reference image has the same image content as the first image to be evaluated, and the image quality of the first reference image is not lower than that of the first image to be evaluated; the second reference image has the same or different image content as the second image to be evaluated, and the image quality of the second reference image is not lower than that of the second image to be evaluated.

[0015] According to a third aspect, an electronic device is provided, including:

[0016] one or more processors;

[0017] a storage device having one or more programs stored thereon,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned image quality assessment method.

[0019] According to a fourth aspect, a computer-readable medium is provided, on which a computer program is stored, wherein when the program is executed by a processor, the image quality assessment method as described above is implemented.

[0020] It can be seen that the embodiments of the present application have the following beneficial effects:

[0021] The embodiments of the present application provide an image quality assessment method, apparatus, and device, wherein a fourth reference image and the fourth image to be assessed are input into an image quality assessment model to obtain an assessment prediction result of the fourth image to be assessed, wherein the image content of the fourth reference image is the same as or different from that of the fourth image to be assessed. The image quality assessment model is trained based on a first image difference encoding result between a first reference image and the first image to be assessed, a second image difference encoding result between a second reference image and the second image to be assessed, an assessment prediction result of the second image to be assessed, and an assessment calibration result of the second image to be assessed. The first reference image has the same image content as the first image to be assessed, and the second reference image has the same image content as or different from that of the second image to be assessed.

[0022] Among them, the first image difference coding result mainly represents the relevant features caused by different image qualities. When it is used to train the image quality assessment model, when the image quality assessment model inputs images with different contents, the second image difference coding result it generates is also more inclined to represent the relevant features caused by different image qualities, rather than the relevant features caused by different image contents.

[0023] Therefore, when applying a trained image quality assessment model to image quality assessment, using a high-definition reference image unrelated to the image being assessed can also yield relatively accurate image quality assessment results. Compared to no-reference quality assessment methods, the inclusion of a second reference image during training improves the accuracy of the model's image quality assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1a A flowchart of an image quality assessment method provided in an embodiment of the present application;

[0025] Figure 1b A schematic diagram of an exemplary framework for a training method for an image quality assessment model provided in an embodiment of the present application;

[0026] Figure 2 A flowchart of a method for training an image quality assessment model provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of the structure of a full-reference teacher model provided in an embodiment of the present application;

[0028] Figure 4 A flowchart of a full-reference teacher model pre-training provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of the structure of a Resnet network provided in an embodiment of the present application;

[0030] Figure 6 A schematic diagram of the structure of a multilayer perceptron provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of the structure of an image quality assessment model provided in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of the structure of a full-reference teacher model and an image quality assessment model provided in an embodiment of the present application;

[0033] Figure 9 A schematic structural diagram of an image quality assessment device provided in an embodiment of the present application;

[0034] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0036] In order to facilitate understanding and explanation of the technical solutions provided by the embodiments of the present application, the background technology of the present application will be described below.

[0037] Image quality assessment has a wide range of applications in image processing. Reliable and accurate image quality assessment methods play an important role in guiding downstream tasks such as image compression and reconstruction. Furthermore, in real-world scenarios, such as user-generated content, reliable image quality assessment methods can help filter out low-quality image data and improve image review efficiency.

[0038] There are many methods for image quality assessment, such as full-reference quality assessment methods, semi-reference quality assessment methods, and no-reference quality assessment methods. Among them, when performing image quality assessment, the full-reference quality assessment method uses a high-definition image with the same content size as the image to be assessed as a reference. The lack of a high-definition image with the same content size as the image to be assessed will limit the application of this method, resulting in low image quality assessment accuracy. In addition, the no-reference quality assessment method only uses manually calibrated scores as a reference, which will reduce the accuracy of the image quality assessment results of this method in real-world application scenarios. Therefore, improving the accuracy of image quality assessment results is an urgent problem to be solved.

[0039] Based on this, the embodiments of the present application provide a method, apparatus and device for image quality assessment. Figure 1a As shown, Figure 1aA flowchart of an image quality assessment method provided in an embodiment of the present application is shown in FIG. Figure 1a As shown, the method may include S101-S102:

[0040] S101: Acquire a fourth reference image and a fourth image to be evaluated, where the image contents of the fourth reference image and the fourth image to be evaluated are the same or different.

[0041] S102: Inputting a fourth reference image and a fourth image to be evaluated into an image quality assessment model to obtain an evaluation prediction result of the fourth image to be evaluated; the image quality assessment model is trained based on a first image difference coding result between the first reference image and the first image to be evaluated, a second image difference coding result between the second reference image and the second image to be evaluated, an evaluation prediction result of the second image to be evaluated, and an evaluation calibration result of the second image to be evaluated; wherein the first reference image and the first image to be evaluated have the same image content, and the image quality of the first reference image is not lower than that of the first image to be evaluated; and the second reference image and the second image to be evaluated have the same or different image content, and the image quality of the second reference image is not lower than that of the second image to be evaluated.

[0042] In the embodiment of the present application, the first image difference coding result primarily represents features related to differences in image quality. When this first image difference coding result is used to train an image quality assessment model, when the image quality assessment model is fed images of different content, the second image difference coding result it generates is more likely to represent features related to differences in image quality, rather than features related to differences in image content. Consequently, when the trained image quality assessment model is used to assess image quality, even when using a high-definition image unrelated to the content of the image being assessed as a reference, a relatively accurate image quality assessment result can be obtained.

[0043] For ease of understanding, the present application embodiment first provides an illustrative example of a training method for an image quality assessment model.

[0044] The following combination Figure 1b See the example scenario shown. Figure 1b As shown in the figure, this figure is a framework diagram of the training method of the image quality assessment model provided in an embodiment of the present application.

[0045] In practical applications, a first reference image and a first image to be evaluated are first input into a full-reference teacher model to obtain a first image difference encoding result between the first reference image and the first image to be evaluated. The first reference image and the first image to be evaluated have identical image content, and the image quality of the first reference image is no less than that of the first image to be evaluated. The full-reference teacher model is pre-trained. It is understood that the full-reference teacher model employs a full-reference image quality assessment method.

[0046] Similarly, a second reference image and a second image to be evaluated are input into the image quality assessment model to obtain a second image difference coding result between the second reference image and the second image to be evaluated and an evaluation prediction result of the second image to be evaluated; the image content of the second reference image and the second image to be evaluated is the same or different, and the image quality of the second reference image is not lower than that of the second image to be evaluated.

[0047] Then, a first loss value is calculated based on the evaluation prediction result of the second image to be evaluated and the evaluation calibration result of the second image to be evaluated. A second loss value is calculated based on the difference encoding result of the first image and the difference encoding result of the second image. The image quality assessment model is trained using the first and second loss values. Inputting the first reference image and the first image to be evaluated into the full-reference teacher model and subsequent steps are repeated until a first preset condition is met. Thus, after training, a trained image quality assessment model can be obtained.

[0048] Those skilled in the art will understand that Figure 1b The framework diagram shown is only an example in which the embodiments of the present application can be implemented. The scope of application of the embodiments of the present application is not limited by any aspect of the framework.

[0049] To facilitate understanding of the present application, the training method of the image quality assessment model provided in the embodiment of the present application is described below with reference to the accompanying drawings.

[0050] See also Figure 2 As shown in FIG, this figure is a flow chart of a training method for an image quality assessment model provided in an embodiment of the present application, as shown in FIG. Figure 2 As shown, the method may include S201-S205:

[0051] S201: Input a first reference image and a first image to be evaluated into a full-reference teacher model to obtain a first image difference encoding result between the first reference image and the first image to be evaluated; the first reference image and the first image to be evaluated have the same image content, and the image quality of the first reference image is not lower than that of the first image to be evaluated.

[0052] In this step, the full-reference teacher model is pre-trained. In an embodiment of the present application, the pre-trained full-reference teacher model is used to assist in the training of the image quality assessment model. Specifically, the first image to be evaluated and the first reference image are first input into the pre-trained full-reference teacher model to obtain a first image difference encoding result between the first reference image and the first image to be evaluated. The first image difference encoding result is used to represent the difference in image quality between the first reference image and the first image to be evaluated. The first image difference encoding result is used in the subsequent training process of the image quality assessment model.

[0053] The first reference image and the first image to be evaluated have the same image content. It is understood that the full-reference teacher model utilizes a full-reference quality assessment method. Based on this, in one or more embodiments, the first reference image and the first image to be evaluated have the same image content and image size. That is, the first reference image is a full reference image that is pixel-aligned with the first image to be evaluated. Pixel alignment can be understood as having the same image size and image content.

[0054] Furthermore, the image quality of the first reference image is no less than that of the first image to be evaluated. To evaluate the first image to be evaluated, a high-quality image, i.e., a high-definition image, whose image quality is greater than or equal to that of the first image to be evaluated is typically required. Thus, the first reference image can be used as a reference to obtain an image quality evaluation result for the first image to be evaluated.

[0055] Since the first reference image and the first image to be evaluated have the same image content, the first image difference encoding result generated by the full-reference teacher model mainly represents the relevant features of the differences caused by different image qualities.

[0056] In one possible implementation, the embodiment of the present application provides a specific implementation method of a pre-trained full-reference teacher model, please see S301-S306 below for details.

[0057] In one possible implementation, an embodiment of the present application provides a specific implementation method for inputting a first reference image and a first image to be evaluated into a full reference teacher model to obtain a first image difference encoding result between the first reference image and the first image to be evaluated, see A1-A2 below.

[0058] S202: Inputting a second reference image and a second image to be evaluated into an image quality assessment model to obtain a second image difference coding result between the second reference image and the second image to be evaluated and an evaluation prediction result of the second image to be evaluated; the image content of the second reference image and the second image to be evaluated is the same or different, and the image quality of the second reference image is not lower than that of the second image to be evaluated.

[0059] After the full reference teacher model is obtained through pre-training, the image quality assessment model is first obtained. In an embodiment of the present application, since the image quality assessment model needs to be trained with the help of the full reference teacher model. Therefore, in a possible implementation method, after the full reference teacher model is obtained through pre-training, the model parameters of the trained full reference teacher model are fixed, and a model with the same structure as the full reference teacher model is reselected as the image quality assessment model. That is, the structures of the image quality assessment model and the full reference teacher model are the same. Moreover, before training the image quality assessment model, the initial model parameters of the image quality assessment model are the same as the model parameters of the pre-trained full reference teacher model. As the image quality assessment model is trained, the model parameters of the image quality assessment model are updated.

[0060] The second reference image and the second image to be evaluated are input into the image quality assessment model to obtain a second image difference encoding result between the second reference image and the second image to be evaluated, as well as an evaluation prediction result for the second image to be evaluated. The second image difference encoding result is used to represent the difference between the second reference image and the second image to be evaluated. The second image difference encoding result is used in the subsequent training process of the image quality assessment model. The evaluation prediction result of the second image to be evaluated is the predicted evaluation result of the second image to be evaluated output by the image quality assessment model. The second image to be evaluated is also used in the subsequent training process of the image quality assessment model.

[0061] In one or more embodiments, when the structure and initial model parameters of the image quality assessment model and the full-reference teacher model are the same as those of the full-reference teacher model, in order to train the image quality assessment model with the help of the full-reference teacher model, the first image to be assessed and the second image to be assessed in the embodiments of the present application can be the same image to be assessed. That is, after obtaining the pre-trained full-reference teacher model and determining the image quality assessment model, the first image to be assessed and the second image to be assessed are respectively input into the full-reference teacher model and the image quality assessment model. At the same time, the first reference image and the second reference image are respectively input into the full-reference teacher model and the image quality assessment model.

[0062] Furthermore, the image quality of the second reference image is no less than that of the second image to be evaluated. To evaluate the second image to be evaluated, a high-quality image, i.e., a high-definition image, whose image quality is greater than or equal to that of the second image to be evaluated is typically required. Thus, the second reference image can be used as a reference to obtain an image quality evaluation result for the second image to be evaluated.

[0063] In an embodiment of the present application, the second reference image has the same or different image content as the second image to be evaluated. It is understood that in order to enable a relatively accurate image quality assessment of the second image to be evaluated even in the absence of a high-quality image that is pixel-aligned with the second image to be evaluated, the image quality assessment model in the present application uses a second reference image with the same or different image content as the second image to be evaluated during training. Thus, when the trained image quality assessment model is applied, even if the high-quality reference image has different image content from the image to be evaluated, a relatively accurate assessment result for the image to be evaluated can be obtained based on the high-quality reference image and the image quality assessment model.

[0064] In one possible implementation, an embodiment of the present application provides a specific implementation method for inputting a second reference image and a second image to be evaluated into an image quality assessment model to obtain a second image difference encoding result between the second reference image and the second image to be evaluated and an evaluation prediction result of the second image to be evaluated, please see B1-B4 below.

[0065] It should be noted that the embodiment of the present application does not limit the execution order of S201 and S202. As an optional example, S202 may be executed first, and then S201. Alternatively, S201 and S202 may be executed simultaneously.

[0066] S203: Calculate a first loss value according to the evaluation prediction result of the second image to be evaluated and the evaluation calibration result of the second image to be evaluated.

[0067] After obtaining the evaluation prediction result of the second image to be evaluated, an evaluation calibration result of the second image to be evaluated is also obtained. In one or more embodiments, the evaluation calibration result of the second image to be evaluated can be a score or a grade. In one possible implementation, the evaluation calibration result of the second image to be evaluated can be obtained by manually marking the differences between the second image to be evaluated and the second reference image. For example, the evaluation calibration result of the second image to be evaluated is manually calibrated to 100 points or the first grade. It is understandable that the evaluation calibration result of the second image to be evaluated is essentially the score or grade of the second reference image.

[0068] After obtaining the evaluation prediction result and the evaluation calibration result of the second image to be evaluated, a first loss value is calculated based on the evaluation prediction result and the evaluation calibration result of the second image to be evaluated. The first loss value is used to train the image quality assessment model.

[0069] S204: Calculate a second loss value according to the first image difference encoding result and the second image difference encoding result.

[0070] After obtaining the first image difference coding result and the second image difference coding result, a second loss value is calculated based on the first image difference coding result and the second image difference coding result. When the image quality assessment model and the full reference teacher model have the same structure and initial model parameters as the full reference teacher model, and the first image to be evaluated and the second image to be evaluated are the same image to be evaluated, since the full reference teacher model is pre-trained, the first image difference coding result can be used as the expected value of the second image difference coding result, and the second image difference coding result is the result that needs to be adjusted to be closer to the first image difference coding result.

[0071] S205: Using the first loss value and the second loss value, train the image quality assessment model, and repeatedly input the first reference image and the first image to be evaluated into the full reference teacher model and subsequent steps until the first preset condition is met.

[0072] The image quality assessment model is trained using the first loss value and the second loss value, and a determination is made as to whether a first preset condition is met. If the first preset condition is not met, steps S201-S205 are repeated until the first preset condition is met. When the first preset condition is met, the image quality assessment model training is determined to be complete, and a trained image quality assessment model is obtained.

[0073] It can be understood that since the high-quality reference image input by the image quality assessment model in this application can be an image that is irrelevant to the content of the image to be evaluated and the training process of the image quality assessment model is assisted by a pre-trained full-reference teacher model, the image quality assessment model can be regarded as a content-independent reference student model.

[0074] As an optional example, the full-reference teacher model and the image quality assessment model in this application have exactly the same structure. The full-reference teacher model is pre-trained as the teacher network, and the full-reference teacher model using the content-independent reference image is used as the student network. The teacher network is used to guide the student network. While ensuring the stable performance of the content-independent reference model, more high-definition-low-quality difference prior information is introduced into the student network, which can improve the robustness and accuracy of the image quality assessment model. The trained image quality assessment model focuses more on image quality and reduces the interference caused by changes in image content.

[0075] Based on the contents of S201-S205, it can be seen that the embodiment of the present application provides a method for training an image quality assessment model. First, a first reference image and a first image to be assessed are input into a full-reference teacher model to obtain a first image difference encoding result between the first reference image and the first image to be assessed. Then, a second reference image and a second image to be assessed are input into the image quality assessment model to obtain a second image difference encoding result between the second reference image and the second image to be assessed, as well as an assessment prediction result of the second image to be assessed. Furthermore, a first loss value is calculated based on the assessment prediction result of the second image to be assessed and the assessment calibration result of the second image to be assessed. A second loss value is calculated based on the first image difference encoding result and the second image difference encoding result. The image quality assessment model is trained using the first and second loss values, and the input of the first reference image and the first image to be assessed into the full-reference teacher model and subsequent steps are repeated until a first preset condition is met. The image quality of the first reference image is not lower than that of the first image to be assessed, and the image quality of the second reference image is not lower than that of the second image to be assessed. The first reference image and the first image to be assessed have the same image content, and the second reference image and the second image to be assessed have the same or different image content.

[0076] Since the first image difference coding result generated by the full-reference teacher model mainly represents the relevant features caused by different image quality, it is used to train the image quality assessment model so that when the image quality assessment model inputs images with different contents, the second image difference coding result it generates is also more inclined to represent the relevant features caused by different image quality, rather than the relevant features caused by different image content.

[0077] Compared to full-reference quality assessment methods, the second reference image input during image quality assessment model training can have content that differs from the second image to be assessed. Therefore, when applying the trained image quality assessment model to image quality assessment, using a high-definition image with no content associated with the image to be assessed as a reference can also yield relatively accurate image quality assessment results. Compared to no-reference quality assessment methods, adding a second reference image as a reference image during image quality assessment training can improve the model's image quality assessment accuracy.

[0078] See also Figure 3 , Figure 3 A schematic diagram of the structure of a full reference teacher model provided in an embodiment of the present application. In a possible implementation, Figure 3 As shown, the full-reference teacher model includes: a first feature extractor, a first difference encoder, a first autoencoder, and a first regressor. Based on this, the embodiment of the present application provides a specific implementation method of pre-training the full-reference teacher model. Figure 4 , Figure 4 This is a flowchart of the full reference teacher model pre-training provided in this application implementation. Figure 4 As shown, this specific implementation includes S301-S306:

[0079] S301: Inputting a third reference image and a third image to be evaluated into a first feature extractor, obtaining image features of the third reference image and image features of the third image to be evaluated, and calculating third differential features between the image features of the third reference image and the image features of the third image to be evaluated; the third reference image and the third image to be evaluated have the same image content, and the image quality of the third reference image is not lower than that of the third image to be evaluated.

[0080] In this embodiment of the present application, the full-reference teacher model includes a first feature extractor. The first feature extractor is used to extract features of the input image. In this step, the first feature extractor is a trained feature extractor. When a third reference image is input into the first feature extractor, feature extraction is performed on the third reference image to obtain image features of the third reference image. When a third image to be evaluated is input into the first feature extractor, feature extraction is performed on the third image to be evaluated to obtain image features of the third image to be evaluated.

[0081] As an optional example, the first feature extractor is a Resnet network, the first feature extractor includes multiple residual module groups, each residual module group includes multiple residual modules, and the outputs of each residual module group together constitute the output of the first feature extractor. Figure 5 , Figure 5 A schematic diagram of the structure of a ResNet network provided in an embodiment of the present application. Figure 5 The Resnet network shown is a structural diagram of the Resnet50 network, which includes four residual module groups, namely residual module group 1, residual module group 2, residual module group 3 and residual module group 4. Each residual module group includes multiple residual modules. For example Figure 5 Residual module group 1 in the example includes three residual modules, namely Resblock1-1 to Resblock1-3. Residual module group 2 includes four residual modules, namely Resblock2-1 to Resblock2-4. Residual module group 3 includes six residual modules, namely Resblock3-1 to Resblock3-6. Residual module group 4 includes three residual modules, namely Resblock4-1 to Resblock4-3.

[0082] The input of the Resnet50 network is an image, which is the third reference image and the third image to be evaluated in this step. The output of each residual module group ( Figure 5(output 1-output 4) are the extracted features. The outputs of each residual module group together constitute the output of the first feature extractor. In one or more embodiments, the output of each residual module group is then passed through an average pooling layer and a convolution layer to output the final features, and the dimensions of the features output by the convolution layer are the same. Furthermore, the four groups of features that have passed through the convolution layer are combined into the features of the final output, that is, the output of the first feature extractor, which combines the outputs of different residual module groups and is a multi-scale image feature. Multi-scale image features are obtained by fusing image features of different scales, which can represent images more accurately and more comprehensively.

[0083] As an example, Figure 5 As shown in the figure, the feature dimensions of the Resnet50 input image are represented as [1, m, 3, 224, 224]. The first parameter, 1, represents the first batch of images. The second parameter, m, represents the number of image blocks, which are obtained by cropping the input image. When the input image is not cropped, m = 1. The third parameter, 3, indicates that there are three Resnet layers. The fourth and fifth parameters indicate a feature scale of 224*224. Output 1 is [1, m, 3, 224, 224], output 2 is [1, m, 512, 28, 28], output 3 is [1, m, 1024, 14, 14], and output 4 is [1, m, 2048, 7, 7]. Output 1 passes through the 1_256_64 convolution layer, resulting in an output of [1, m, 64, 7, 7]. Output 2 passes through a convolutional layer with a size of 1_512_64, resulting in an output of [1,m,64,7,7]. Output 3 passes through a convolutional layer with a size of 1_1024_64, resulting in an output of [1,m,64,7,7]. Output 4 passes through a convolutional layer with a size of 1_2048_64, resulting in an output of [1,m,64,7,7]. The final multi-scale image feature output is [1,m,64*4,7,7], where 4 represents four scales.

[0084] In one possible implementation, the number of first feature extractors is one or two. When there is one first feature extractor, the third reference image and the third image to be evaluated are both input into the same first feature extractor to extract corresponding features. When there are two first feature extractors, the third reference image and the third image to be evaluated are each input into their respective corresponding first feature extractors to extract corresponding features.

[0085] After acquiring the image features of the third reference image and the image features of the third image to be evaluated, a third differential feature between the image features of the third reference image and the image features of the third image to be evaluated is calculated, where the third differential feature is used to represent the difference between the image features of the third reference image and the image features of the third image to be evaluated.

[0086] In addition, the third reference image in this step has the same image content as the third image to be evaluated, and the image quality of the third reference image is no less than that of the third image to be evaluated. The third reference image and the third image to be evaluated are used to train the full-reference teacher model. For details regarding the third reference image and the third image to be evaluated, refer to the first reference image and the first image to be evaluated in the above-mentioned embodiment.

[0087] S302: Input the third differential feature into the first difference encoder to obtain a third image difference encoding result.

[0088] The third image difference encoding result is also used to represent the difference between the image features of the third reference image and the image features of the third image to be evaluated.

[0089] S303: Inputting the image features of the third image to be evaluated into the first autoencoder to obtain an autoencoding result of the third image to be evaluated.

[0090] The image features of the third image to be evaluated are input into the first autoencoder to obtain an autoencoding result of the third image to be evaluated. It can be understood that the autoencoding result of the third image to be evaluated is another feature representation of the image features of the third image to be evaluated.

[0091] As an optional example, the first difference encoder and the first autoencoder each include multiple multilayer perceptrons (MLPs). Using multiple layers of MLPs can improve the accuracy of the first difference encoder and the first autoencoder while increasing the inference speed of the first difference encoder and the first autoencoder. For example, the first difference encoder includes 18 MLPs and the first autoencoder includes 9 MLPs. Figure 6 , Figure 6 This is a schematic diagram of the structure of a multilayer perceptron provided in an embodiment of the present application. Figure 6 As shown, when the multilayer perceptron constitutes the first difference encoder, the input feature is the third differential feature. When the multilayer perceptron constitutes the first autoencoder, the input feature is the image feature of the third image to be evaluated.

[0092] In specific implementations, the input features are matrix-transformed to obtain the matrix-transformed features. For example, if the input features are [1, m, c, h, w], the matrix-transformed features are [1, h*w, m*c]. Here, c is the product of the number of residual module layers and the scale. For example, if the number of layers is 64 and the scale is 4, then c is 64*4. h and w are the two feature scales. The matrix-transformed features are input into the linear layer and then into multiple MLPs (e.g., n MLPs). Finally, average pooling is performed to output mixed features. Within an MLP, cross-block mixing and cross-channel mixing operations are performed. The cross-block mixing operation involves normalizing the input MLP features, performing a matrix rank transformation, and then performing a matrix rank transformation again through two linear layers to obtain quadratically transformed features. The quadratically transformed features are combined with the input MLP features to obtain the quadratically input features, i.e., the features after the cross-block mixing operation. Furthermore, the cross-channel mixing operation is to normalize the secondary input features, enter the two linear layers, output the features after the linear layers, and then merge the features with the features obtained by the cross-image block mixing operation to obtain the features after the cross-channel mixing operation.

[0093] S304: After merging features of the autoencoding result of the third image to be evaluated and the third image difference encoding result, the results are input into the first regressor to obtain an evaluation prediction result of the third image to be evaluated.

[0094] After obtaining the autoencoding result of the third image to be evaluated and the third image difference encoding result, feature merging is performed, and the merged features are input into the first regressor to output the evaluation prediction result of the third image to be evaluated.

[0095] S305: Calculate a third loss value according to the evaluation prediction result of the third image to be evaluated and the evaluation calibration result of the third image to be evaluated.

[0096] The evaluation calibration result of the third image to be evaluated is an expected value of the evaluation prediction result of the third image to be evaluated. As an optional example, the evaluation calibration result of the third image to be evaluated is a score or a grade. When the evaluation calibration result of the third image to be evaluated is a score, the evaluation prediction result of the third image to be evaluated is also a score.

[0097] A third loss value is calculated based on the evaluation prediction result of the third image to be evaluated and the evaluation calibration result of the third image to be evaluated. The third loss value is used to train the full-reference teacher model.

[0098] S306: Using the third loss value, train the first difference encoder, the first autoencoder, and the first regressor, and repeatedly input the third reference image and the third image to be evaluated into the first feature extractor and subsequent steps until a second preset condition is met.

[0099] Specifically, the first difference encoder, the first autoencoder, and the first regressor in the full-reference teacher model are trained using the third loss value. After training is completed once, it is determined whether the second preset condition is met. If the second preset condition is not met, S301-S306 are repeated until the second preset condition is met.

[0100] Based on the contents of S301-S306, it can be seen that the first difference encoder, the first autoencoder, and the first regressor in the full-reference teacher model can be trained using the third reference image and the third image to be evaluated to obtain a trained full-reference teacher model. The full-reference teacher model uses the full-reference image quality assessment method to evaluate the image quality.

[0101] In this embodiment of the present application, to better utilize the local-global information of the third reference image and the third image to be evaluated, random image block cropping is performed on the third reference image to obtain at least one third reference image block. Furthermore, random image block cropping is performed on the third image to be evaluated to obtain at least one third image block to be evaluated. Based on this, processing the third reference image effectively processes the at least one third reference image block. Processing the third image to be evaluated effectively processes the at least one third image block to be evaluated.

[0102] Based on the above content, in a possible implementation, an embodiment of the present application provides a specific implementation of inputting the third reference image and the third image to be evaluated into the first feature extractor in S301, including: inputting at least one third reference image block and at least one third image block to be evaluated into the first feature extractor.

[0103] Based on this, as an optional example, Figure 5 and Figure 6 The input image shown in is an image patch.

[0104] like Figure 3 The structure of the full-reference teacher model shown in FIG. The full-reference teacher model includes: a first feature extractor and a first difference encoder. Based on this, in one possible implementation, the embodiment of the present application provides a specific implementation of inputting a first reference image and a first image to be evaluated into the full-reference teacher model in S201 to obtain a first image difference encoding result between the first reference image and the first image to be evaluated, including:

[0105] A1: Input a first reference image and a first image to be evaluated into a first feature extractor, obtain image features of the first reference image and image features of the first image to be evaluated, and calculate a first differential feature between the image features of the first reference image and the image features of the first image to be evaluated.

[0106] When the full-reference teacher model includes a first feature extractor, when a first reference image is input into the first feature extractor, feature extraction is performed on the first reference image to obtain image features of the first reference image. When a first image to be evaluated is input into the first feature extractor, feature extraction is performed on the first image to be evaluated to obtain image features of the first image to be evaluated.

[0107] As an optional example, the number of first feature extractors is one or two. When there is one first feature extractor, the first reference image and the first image to be evaluated are both input into the same first feature extractor to extract corresponding features. When there are two first feature extractors, the first reference image and the first image to be evaluated are each input into their respective corresponding first feature extractors to extract corresponding features.

[0108] A2: Inputting the first differential feature into a first difference encoder to obtain a first image difference encoding result between the first reference image and the first image to be evaluated.

[0109] Furthermore, the first differential feature is input into a first difference encoder to obtain a first image difference encoding result.

[0110] In this embodiment of the present application, to better utilize the local-global information of the first reference image and the first image to be evaluated, random image block cropping is performed on the first reference image to obtain at least one first reference image block. Random image block cropping is also performed on the first image to be evaluated to obtain at least one first image block to be evaluated. Subsequent processing can then be performed based on the image blocks.

[0111] Based on this, in a possible implementation method, an embodiment of the present application provides a specific implementation method of inputting the first reference image and the first image to be evaluated into the full reference teacher model in S201, including: inputting at least one first reference image block and at least one first image block to be evaluated into the full reference teacher model.

[0112] Additionally, when the full-reference teacher model includes a first feature extractor, inputting at least one first reference image block and at least one first image block to be evaluated into the full-reference teacher model comprises:

[0113] At least one first reference image block and at least one first image block to be evaluated are input into a first feature extractor.

[0114] See also Figure 7 , Figure 7 This is a structural diagram of an image quality assessment model provided in an embodiment of the present application. Figure 7 As shown, in one or more embodiments, the image quality assessment model includes: a second feature extractor, a second difference encoder, a second autoencoder, and a second regressor.

[0115] based on Figure 7 In one possible implementation of the image quality assessment model shown, the embodiment of the present application provides a specific implementation method of inputting the second reference image and the second image to be evaluated into the image quality assessment model in S202 to obtain a second image difference encoding result between the second reference image and the second image to be evaluated and an evaluation prediction result of the second image to be evaluated, including:

[0116] B1: Input the second reference image and the second image to be evaluated into a second feature extractor, obtain image features of the second reference image and image features of the second image to be evaluated, and calculate second differential features between the image features of the second reference image and the image features of the second image to be evaluated.

[0117] In this embodiment of the present application, the image quality assessment model includes a second feature extractor. The second feature extractor is used to extract features of the input image. In this step, the second feature extractor is a trained feature extractor. When a second reference image is input into the second feature extractor, feature extraction is performed on the second reference image to obtain image features of the second reference image. When a second image to be assessed is input into the second feature extractor, feature extraction is performed on the second image to be assessed to obtain image features of the second image to be assessed.

[0118] As an optional example, the second feature extractor is a Resnet network, and the second feature extractor includes multiple residual module groups, each residual module group includes multiple residual modules, and the outputs of each residual module group together constitute the output of the second feature extractor. As an optional example, the second feature extractor is the same as the first feature extractor. Specifically, please refer to the technical details of the first feature extractor above, which will not be repeated here.

[0119] In one possible implementation, the number of second feature extractors is one or two. When there is one second feature extractor, the second reference image and the second image to be evaluated are both input into the same second feature extractor to extract corresponding features. When there are two second feature extractors, the second reference image and the second image to be evaluated are each input into their respective corresponding second feature extractors to extract corresponding features.

[0120] After acquiring the image features of the second reference image and the image features of the second image to be evaluated, a second differential feature between the image features of the second reference image and the image features of the third image to be evaluated is calculated, where the second differential feature is used to represent the difference between the image features of the second reference image and the image features of the second image to be evaluated.

[0121] B2: Input the second differential feature into the second difference encoder to obtain a second image difference encoding result.

[0122] The second image difference encoding result is also used to represent the difference between the image features of the second reference image and the image features of the second image to be evaluated.

[0123] B3: Input the image features of the second image to be evaluated into the second autoencoder to obtain an autoencoding result of the second image to be evaluated.

[0124] The image features of the second image to be evaluated are input into the second autoencoder to obtain an autoencoding result of the second image to be evaluated. It can be understood that the autoencoding result of the second image to be evaluated is another feature representation of the image features of the second image to be evaluated.

[0125] As an optional example, the second difference encoder and the second autoencoder each include multiple MLPs. Using multiple layers of MLPs can improve the accuracy of the first difference encoder and the first autoencoder while also increasing their inference speed. For example, the second difference encoder includes 18 MLPs, and the second autoencoder includes 9 MLPs. For technical details about the MLPs, refer to S303 and are not further elaborated here.

[0126] B4: After merging the features of the autoencoding result of the second image to be evaluated and the second image difference encoding result, the results are input into the second regressor to obtain the evaluation prediction result of the second image to be evaluated.

[0127] After obtaining the autoencoding result of the second image to be evaluated and the second image difference encoding result, feature merging is performed, and the merged features are input into the second regressor to output the evaluation prediction result of the second image to be evaluated.

[0128] Based on the contents of B1-B4, it can be seen that when the image quality assessment model includes a second feature extractor, a second difference encoder, a second autoencoder and a second regressor, a specific implementation of S202 is provided based on the second feature extractor, the second difference encoder, the second autoencoder and the second regressor.

[0129] based on Figure 7 In one possible implementation of the structure of the image quality assessment model shown, the embodiment of the present application provides a specific implementation method of using the first loss value and the second loss value in S205 to train the image quality assessment model, including:

[0130] The second difference encoder, the second autoencoder, and the second regressor are trained using the first loss value and the second loss value.

[0131] It is understandable that the second feature extractor in the image quality assessment model is pre-selected and trained. Therefore, training the image quality assessment model means only training the second difference encoder, the second autoencoder, and the second regressor in the image quality assessment model.

[0132] In this embodiment of the present application, to better utilize the local-global information of the first reference image and the first image to be evaluated, random image block cropping is performed on the second reference image to obtain at least one second reference image block. Random image block cropping is also performed on the second image to be evaluated to obtain at least one second image block to be evaluated. Subsequent processing can then be performed based on these image blocks.

[0133] Based on this, in one possible implementation, the embodiment of the present application provides a specific implementation of inputting the second reference image and the second image to be evaluated into the image quality assessment model in S202, including:

[0134] At least one second reference image block and at least one second image block to be evaluated are input into the image quality assessment model.

[0135] In addition, when the image quality assessment model includes a second feature extractor, inputting at least one second reference image block and at least one second image block to be assessed into the image quality assessment model comprises:

[0136] At least one second reference image block and at least one second image block to be evaluated are input into a second feature extractor.

[0137] See also Figure 8 , Figure 8 A structural diagram of a full reference teacher model and an image quality assessment model provided in the embodiment of the present application. In order to better understand the image quality assessment method provided in the embodiment of the present application, combined with Figure 8 , further describing the image quality assessment method provided in the embodiments of the present application.

[0138] like Figure 8 As shown, Figure 8 The invention comprises a full-reference teacher model and an image quality assessment model. The full-reference teacher model comprises a first feature extractor, a first difference encoder, a first autoencoder, and a first regressor. The image quality assessment model comprises a second feature extractor, a second difference encoder, a second autoencoder, and a second regressor.

[0139] In the embodiment of the present application, the full-reference teacher model is pre-trained. After the full-reference teacher model training is completed, the full-reference teacher model is trained again.

[0140] When training the full-reference teacher model, the third reference image is randomly cropped to obtain M third reference image blocks. The third image to be evaluated is also randomly cropped to obtain M third image blocks to be evaluated. The third reference image and the third image to be evaluated have the same image content, and the image quality of the third reference image is no less than that of the third image to be evaluated. Furthermore, the M third reference image blocks and the M third image blocks to be evaluated are each input into a first feature extractor to obtain image features of the third reference image and image features of the third image to be evaluated. The feature dimensions of the image features of the third reference image are represented by [1, m, 64*4, 7, 7], and the feature dimensions of the image features of the third image to be evaluated are represented by [1, m, 64*4, 7, 7]. Both first feature extractors are pre-trained and share weights.

[0141] Furthermore, the image features of the third reference image and the image features of the third image to be evaluated are both matrix-transformed to obtain corresponding image features of the third reference image after the matrix transformation (feature dimensions are [1, 49, m*256]) and image features of the third image to be evaluated after the matrix transformation (feature dimensions are [1, 49, m*256]). A third differential feature is calculated between the image features of the third reference image after the matrix transformation and the image features of the third image to be evaluated after the matrix transformation.

[0142] The third differential feature is input into the first difference encoder to obtain the third image difference encoding result (feature dimension is [1,256]). The image feature of the third image to be evaluated after matrix transformation is input into the first autoencoder to obtain the autoencoding result of the third image to be evaluated (feature dimension is [1,256]). The first difference encoder and the first autoencoder are called a dual-stream cross-image block hybrid encoder. After merging the features of the autoencoding result of the third image to be evaluated and the third image difference encoding result, the merged features (feature dimension is [1,512]) are input into the first regressor to obtain the evaluation prediction result of the third image to be evaluated. Among them, the first regressor is composed of two fully connected layers. As Figure 8 As shown in FIG, the first difference encoder and the first autoencoder are both composed of multiple layers of MLP. The first difference encoder is composed of 18 MLPs, and the first autoencoder is composed of 9 MLPs.

[0143] A third loss value is calculated based on the evaluation prediction result of the third image to be evaluated and the evaluation calibration result of the third image to be evaluated. The third loss value is used to train the first difference encoder, the first autoencoder, and the first regressor until a second preset condition is met, thereby obtaining a pre-trained full-reference teacher model.

[0144] Furthermore, the image quality assessment model is trained using the pre-trained full-reference teacher model. The pre-trained full-reference teacher model includes a first feature extractor and a first difference encoder. The number of the first feature extractors is one or more. Figure 8 The number of first feature extractors shown is two.

[0145] Specifically, the first reference image is randomly cropped to obtain M first reference image blocks. The first image to be evaluated is randomly cropped to obtain M first image blocks to be evaluated. The first reference image and the first image to be evaluated have the same image content, and the image quality of the first reference image is no less than that of the first image to be evaluated. Furthermore, the M first reference image blocks and the M first image blocks to be evaluated are respectively input into a first feature extractor to obtain image features of the first reference image and image features of the first image to be evaluated. The feature dimensions of the image features of the first reference image are represented by [1, m, 64*4, 7, 7], and the feature dimensions of the image features of the first image to be evaluated are represented by [1, m, 64*4, 7, 7]. Furthermore, the image features of the first reference image and the image features of the first image to be evaluated are both matrix transformed to obtain the corresponding image features of the first reference image after matrix transformation (feature dimensions of [1, 49, m*256]) and the image features of the first image to be evaluated after matrix transformation (feature dimensions of [1, 49, m*256]). A first differential feature is calculated between the image features of the first reference image after the matrix transformation and the image features of the first image to be evaluated after the matrix transformation. The first differential feature is input into a first difference encoder to obtain a first image difference encoding result between the first reference image and the first image to be evaluated.

[0146] When determining the image quality assessment model, a model with the same structure as the full-reference teacher model is selected as the image quality assessment model to be trained. The image quality assessment model includes a second feature extractor, a second difference encoder, a second autoencoder, and a second regressor. If the second feature extractor is pre-trained, the second difference encoder, second autoencoder, and second regressor in the image quality assessment model are trained during training.

[0147] Specifically, the second reference image is randomly cropped with image blocks to obtain M second reference image blocks. The second image to be evaluated is randomly cropped with image blocks to obtain M second image blocks to be evaluated. The image content of the second reference image is the same as or different from that of the second image to be evaluated, and the image quality of the second reference image is no less than that of the second image to be evaluated. It should be noted that the second image to be evaluated is the same as the first image to be evaluated. Then, the M second reference image blocks and the M second image blocks to be evaluated are respectively input into a second feature extractor to obtain image features of the second reference image and image features of the second image to be evaluated. The feature dimensions of the image features of the second reference image are represented by [1, m, 64*4, 7, 7], and the feature dimensions of the image features of the second image to be evaluated are represented by [1, m, 64*4, 7, 7]. Both second feature extractors are pre-trained and share weights.

[0148] Furthermore, the image features of the second reference image and the image features of the second image to be evaluated are both subjected to matrix transformation to obtain corresponding image features of the second reference image after matrix transformation (feature dimensions are [1, 49, m*256]) and image features of the third image to be evaluated after matrix transformation (feature dimensions are [1, 49, m*256]). Second differential features are calculated between the image features of the second reference image after matrix transformation and the image features of the second image to be evaluated after matrix transformation.

[0149] The second differential feature is input into the second difference encoder to obtain the second image difference encoding result (feature dimension is [1,256]). The image feature of the second image to be evaluated is input into the second autoencoder to obtain the autoencoding result of the second image to be evaluated (feature dimension is [1,256]). The second difference encoder and the second autoencoder are also called dual-stream cross-image block hybrid encoders. After merging the features of the autoencoding result of the second image to be evaluated and the second image difference encoding result, the merged features (feature dimension is [1,512]) are input into the second regressor to obtain the evaluation prediction result of the second image to be evaluated. Among them, the first regressor consists of two fully connected layers. As Figure 8 As shown in FIG, the second difference encoder and the second autoencoder are both composed of multiple layers of MLP. The second difference encoder is composed of 18 MLPs, and the second autoencoder is composed of 9 MLPs.

[0150] A first loss value is calculated based on the evaluation prediction result of the second image to be evaluated and the evaluation calibration result of the second image to be evaluated. A second loss value is calculated based on the difference encoding result of the first image and the difference encoding result of the second image. The image quality assessment model is trained using the first and second loss values ​​until a first preset condition is met, thereby obtaining a pre-trained image quality assessment model.

[0151] The structures of the full-reference teacher model and the image quality assessment model in this application are exactly the same. The full-reference teacher model is pre-trained and used as the teacher network. The full-reference teacher model using the content-independent reference image is used as the student network. The teacher network is used to guide the student network. While ensuring the stable performance of the content-independent reference model, more high-definition-low-quality difference prior information is introduced into the student network, which can improve the robustness and accuracy of the image quality assessment model. The trained image quality assessment model focuses more on image quality and reduces the interference caused by changes in image content. Therefore, compared with the full-reference quality assessment method, in the method provided in the embodiment of the present application, the second reference image input when training the image quality assessment model can have an image content different from that of the second image to be evaluated. When applying the trained image quality assessment model to perform image quality assessment, using a high-definition image that is irrelevant to the content of the image to be evaluated as a reference can also obtain a more accurate image quality assessment result. Compared with the no-reference quality assessment method, adding a second reference image as a reference image for image quality assessment when training the image quality assessment model can improve the image quality assessment accuracy of the model.

[0152] Based on the training method for the image quality assessment model provided in the above embodiment, the high-quality reference images used during training can have the same or different content as the image to be assessed. Therefore, even when there is no high-quality reference image that is pixel-aligned with the image to be assessed, using a high-quality image with different content as the reference image can still yield relatively accurate image quality assessment results. Thus, in Figure 1, the fourth reference image can have different image content than the fourth image to be assessed.

[0153] By inputting the fourth reference image and the fourth image to be evaluated into the trained image quality assessment model, an evaluation prediction result of the fourth image to be evaluated can be obtained.

[0154] As an optional example, the image quality assessment model includes: a second feature extractor, a second difference encoder, a second autoencoder, and a second regressor.

[0155] Based on this, in one possible implementation, the embodiment of the present application provides a specific implementation method of inputting the fourth reference image and the fourth image to be evaluated into the image quality assessment model in S102 to obtain the evaluation prediction result of the fourth image to be evaluated, including:

[0156] C1: Inputting the fourth reference image and the fourth image to be evaluated into the second feature extractor, obtaining image features of the fourth reference image and image features of the fourth image to be evaluated, and calculating fourth differential features between the image features of the fourth reference image and the image features of the fourth image to be evaluated.

[0157] C2: Input the fourth differential feature into the second difference encoder to obtain a fourth difference encoding result.

[0158] C3: Input the image features of the fourth image to be evaluated into the second autoencoder to obtain an autoencoding result of the fourth image to be evaluated.

[0159] C4: After merging the features of the autoencoding result and the fourth difference encoding result of the fourth image to be evaluated, the results are input into the second regressor to obtain the evaluation prediction result of the fourth image to be evaluated.

[0160] It is understandable that the technical details of C1-C4 can be referred to B1-B4 and will not be repeated here.

[0161] Based on the contents of S101-S102, compared to the full-reference quality assessment method, the second reference image input when training the image quality assessment model can have different image content than the second image to be assessed. Therefore, when applying the trained image quality assessment model to image quality assessment, using a high-definition image with no content related to the image to be assessed as a reference can also produce relatively accurate image quality assessment results. Compared to the no-reference quality assessment method, adding a second reference image as a reference image for image quality assessment when training the image quality assessment model can improve the model's image quality assessment accuracy.

[0162] Based on the image quality assessment method provided in the above method embodiment, the embodiment of the present application further provides an image quality assessment device, which will be described below with reference to the accompanying drawings.

[0163] See also Figure 9 As shown, Figure 9 This is a structural diagram of an image quality assessment device provided in an embodiment of the present application. Figure 9 As shown, the image quality assessment device includes:

[0164] An acquiring unit 1001 is configured to acquire a fourth reference image and a fourth image to be evaluated, wherein the fourth reference image and the fourth image to be evaluated have the same or different image contents;

[0165] Input unit 1002 is used to input the fourth reference image and the fourth image to be evaluated into an image quality assessment model to obtain an assessment prediction result of the fourth image to be evaluated; the image quality assessment model is trained based on a first image difference encoding result between a first reference image and the first image to be evaluated, a second image difference encoding result between a second reference image and the second image to be evaluated, an assessment prediction result of the second image to be evaluated, and an assessment calibration result of the second image to be evaluated; wherein the first reference image has the same image content as the first image to be evaluated, and the image quality of the first reference image is not lower than that of the first image to be evaluated; the second reference image has the same or different image content as the second image to be evaluated, and the image quality of the second reference image is not lower than that of the second image to be evaluated.

[0166] In a possible implementation, the image quality assessment apparatus further includes: a training unit, which is used to train the image quality assessment model and includes:

[0167] a first input unit, configured to input a first reference image and a first image to be evaluated into a full-reference teacher model to obtain a first image difference encoding result between the first reference image and the first image to be evaluated; the first reference image and the first image to be evaluated have the same image content, and the image quality of the first reference image is not lower than that of the first image to be evaluated;

[0168] a second input unit, configured to input a second reference image and a second image to be evaluated into the image quality assessment model, to obtain a second image difference coding result between the second reference image and the second image to be evaluated, and an assessment prediction result of the second image to be evaluated; the image content of the second reference image and the second image to be evaluated being the same or different, and the image quality of the second reference image being no less than that of the second image to be evaluated;

[0169] a first calculation unit, configured to calculate a first loss value according to an evaluation prediction result of the second image to be evaluated and an evaluation calibration result of the second image to be evaluated;

[0170] a second calculation unit, configured to calculate a second loss value according to the first image difference encoding result and the second image difference encoding result;

[0171] The first training unit is used to train the image quality assessment model using the first loss value and the second loss value, and repeatedly perform the steps of inputting the first reference image and the first image to be evaluated into the full reference teacher model and subsequent steps until a first preset condition is met.

[0172] In one possible implementation, the full-reference teacher model includes: a first feature extractor and a first difference encoder;

[0173] The first input unit includes:

[0174] a first input subunit, configured to input a first reference image and a first image to be evaluated into the first feature extractor, obtain image features of the first reference image and image features of the first image to be evaluated, and calculate a first differential feature between the image features of the first reference image and the image features of the first image to be evaluated;

[0175] The second input subunit is configured to input the first differential feature into the first difference encoder to obtain a first image difference encoding result between the first reference image and the first image to be evaluated.

[0176] In one possible implementation, the image quality assessment model includes: a second feature extractor, a second difference encoder, a second autoencoder, and a second regressor;

[0177] The second input unit includes:

[0178] a third input subunit, configured to input a second reference image and a second image to be evaluated into the second feature extractor, obtain image features of the second reference image and image features of the second image to be evaluated, and calculate second differential features between the image features of the second reference image and the image features of the second image to be evaluated;

[0179] a fourth input subunit, configured to input the second differential feature into the second difference encoder to obtain a second image difference encoding result;

[0180] a fifth input subunit, configured to input the image features of the second image to be evaluated into the second autoencoder to obtain an autoencoding result of the second image to be evaluated;

[0181] a sixth input subunit, configured to perform feature merging on the autoencoding result of the second image to be evaluated and the second image difference encoding result, and input the merged features into the second regressor to obtain an evaluation prediction result of the second image to be evaluated;

[0182] The first training unit includes:

[0183] The first training subunit is used to train the second difference encoder, the second autoencoder, and the second regressor using the first loss value and the second loss value.

[0184] In one possible implementation, the full-reference teacher model includes: a first feature extractor, a first difference encoder, a first autoencoder, and a first regressor; the training unit further includes:

[0185] a third input unit, configured to input a third reference image and a third image to be evaluated into the first feature extractor, obtain image features of the third reference image and image features of the third image to be evaluated, and calculate third differential features between the image features of the third reference image and the image features of the third image to be evaluated; the third reference image and the third image to be evaluated have the same image content, and the image quality of the third reference image is not lower than that of the third image to be evaluated;

[0186] a fourth input unit, configured to input the third differential feature into the first difference encoder to obtain a third image difference encoding result;

[0187] a fifth input unit, configured to input the image features of the third image to be evaluated into the first autoencoder to obtain an autoencoding result of the third image to be evaluated;

[0188] a sixth input unit, configured to combine the features of the autoencoding result of the third image to be evaluated and the third image difference encoding result, and input the combined features into the first regressor to obtain an evaluation prediction result of the third image to be evaluated;

[0189] a calculation unit, configured to calculate a third loss value according to an evaluation prediction result of the third image to be evaluated and an evaluation calibration result of the third image to be evaluated;

[0190] The second training unit is used to use the third loss value to train the first difference encoder, the first autoencoder and the first regressor, and repeatedly perform the steps of inputting the third reference image and the third image to be evaluated into the first feature extractor and subsequent steps until a second preset condition is met.

[0191] In a possible implementation, the training unit further includes:

[0192] A first cropping unit, configured to perform random image block cropping on the first reference image to obtain at least one first reference image block;

[0193] a second cropping unit, configured to perform random image block cropping on the first image to be evaluated to obtain at least one first image block to be evaluated;

[0194] a third cropping unit, configured to perform random image block cropping on the second reference image to obtain at least one second reference image block;

[0195] a fourth cropping unit, configured to perform random image block cropping on the second image to be evaluated to obtain at least one second image block to be evaluated;

[0196] The first input unit includes:

[0197] a seventh input subunit, configured to input the at least one first reference image block and the at least one first image block to be evaluated into a full-reference teacher model;

[0198] The second input unit includes:

[0199] The eighth input subunit is configured to input the at least one second reference image block and the at least one second image block to be evaluated into the image quality assessment model.

[0200] In a possible implementation, the training unit further includes:

[0201] a fifth cropping unit, configured to perform random image block cropping on the third reference image to obtain at least one third reference image block;

[0202] a sixth cropping unit, configured to perform random image block cropping on the third image to be evaluated to obtain at least one third image block to be evaluated;

[0203] The third input unit includes:

[0204] The ninth input subunit is configured to input the at least one third reference image block and the at least one third image block to be evaluated into the first feature extractor.

[0205] In a possible implementation, the first feature extractor is a Resnet network, the first feature extractor includes multiple residual module groups, each of the residual module groups includes multiple residual modules, and the outputs of the respective residual module groups together constitute the output of the first feature extractor.

[0206] In a possible implementation, the second feature extractor is a Resnet network, the second feature extractor includes multiple residual module groups, each of the residual module groups includes multiple residual modules, and the outputs of each of the residual module groups together constitute the output of the second feature extractor.

[0207] In one possible implementation, the second difference encoder and the second autoencoder both include multiple multilayer perceptrons.

[0208] In one possible implementation, the first difference encoder and the first autoencoder both include multiple multilayer perceptrons.

[0209] In one possible implementation, the image quality assessment model includes: a second feature extractor, a second difference encoder, a second autoencoder, and a second regressor;

[0210] The input unit 1002 includes:

[0211] a first input subunit, configured to input a fourth reference image and a fourth image to be evaluated into the second feature extractor, obtain image features of the fourth reference image and image features of the fourth image to be evaluated, and calculate fourth differential features between the image features of the fourth reference image and the image features of the fourth image to be evaluated;

[0212] a second input subunit, configured to input the fourth differential feature into the second difference encoder to obtain a fourth difference encoding result;

[0213] a third input subunit, configured to input the image features of the fourth image to be evaluated into the second autoencoder to obtain an autoencoding result of the fourth image to be evaluated;

[0214] The fourth input subunit is configured to perform feature merging on the autoencoding result of the fourth image to be evaluated and the fourth difference encoding result, and input the merged features into the second regressor to obtain an evaluation prediction result of the fourth image to be evaluated.

[0215] Based on the image quality assessment method provided in the above method embodiment, the present application also provides an electronic device, including: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the image quality assessment method described in any of the above embodiments.

[0216] Reference below Figure 10 , which shows a schematic structural diagram of an electronic device 1200 suitable for implementing an embodiment of the present application. The terminal device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (portable Android devices), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs (televisions) and desktop computers. Figure 10 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0217] like Figure 10 As shown, the electronic device 1200 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage device 1206 into a random access memory (RAM) 1203. Various programs and data required for the operation of the electronic device 1200 are also stored in the RAM 1203. The processing device 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0218] Typically, the following devices may be connected to the I / O interface 1205: an input device 1206 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1207 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1206 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1209. The communication device 1209 may allow the electronic device 1200 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 10 The electronic device 1200 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0219] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1209, or installed from the storage device 1206, or installed from the ROM 1202. When the computer program is executed by the processing device 1201, the above-mentioned functions defined in the method of the embodiment of the present application are performed.

[0220] The electronic device provided in the embodiment of the present application and the image quality assessment method provided in the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0221] Based on the image quality assessment method provided in the above method embodiment, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the image quality assessment method as described in any of the above embodiments is implemented.

[0222] It should be noted that the computer-readable medium mentioned above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0223] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0224] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0225] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the image quality assessment model training method or image quality assessment method.

[0226] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0227] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0228] The units involved in the embodiments described in this application may be implemented by software or hardware. In some cases, the name of a unit / module does not constitute a limitation of the unit itself. For example, a voice data acquisition module may also be described as a "data acquisition module."

[0229] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0230] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0231] According to one or more embodiments of the present application, [Example 1] provides an image quality assessment method, the method comprising:

[0232] Acquiring a fourth reference image and a fourth image to be evaluated, wherein the fourth reference image and the fourth image to be evaluated have the same or different image contents;

[0233] Inputting the fourth reference image and the fourth image to be evaluated into an image quality assessment model to obtain an evaluation prediction result of the fourth image to be evaluated;

[0234] The image quality assessment model is trained based on a first image difference coding result between a first reference image and a first image to be evaluated, a second image difference coding result between a second reference image and a second image to be evaluated, an assessment prediction result of the second image to be evaluated, and an assessment calibration result of the second image to be evaluated; wherein, the first reference image has the same image content as the first image to be evaluated, and the image quality of the first reference image is not lower than that of the first image to be evaluated; the second reference image has the same or different image content as the second image to be evaluated, and the image quality of the second reference image is not lower than that of the second image to be evaluated.

[0235] According to one or more embodiments of the present application, [Example 2] provides an image quality assessment method, wherein the training process of the image quality assessment model includes:

[0236] Inputting a first reference image and a first image to be evaluated into a full-reference teacher model to obtain a first image difference encoding result between the first reference image and the first image to be evaluated; the first reference image and the first image to be evaluated have the same image content, and the image quality of the first reference image is not lower than that of the first image to be evaluated;

[0237] Inputting a second reference image and a second image to be evaluated into an image quality assessment model, obtaining a second image difference coding result between the second reference image and the second image to be evaluated, and an evaluation prediction result of the second image to be evaluated; the image content of the second reference image and the second image to be evaluated being identical or different, and the image quality of the second reference image being no less than that of the second image to be evaluated;

[0238] Calculating a first loss value according to the evaluation prediction result of the second image to be evaluated and the evaluation calibration result of the second image to be evaluated;

[0239] Calculating a second loss value according to the first image difference encoding result and the second image difference encoding result;

[0240] The image quality assessment model is trained using the first loss value and the second loss value, and the step of inputting the first reference image and the first image to be evaluated into the full-reference teacher model and subsequent steps are repeatedly performed until a first preset condition is met.

[0241] According to one or more embodiments of the present application, [Example 3] provides an image quality assessment method, wherein the full-reference teacher model includes: a first feature extractor and a first difference encoder;

[0242] The step of inputting the first reference image and the first image to be evaluated into a full reference teacher model to obtain a first image difference encoding result between the first reference image and the first image to be evaluated includes:

[0243] Inputting a first reference image and a first image to be evaluated into the first feature extractor, obtaining image features of the first reference image and image features of the first image to be evaluated, and calculating a first differential feature between the image features of the first reference image and the image features of the first image to be evaluated;

[0244] The first differential feature is input into the first difference encoder to obtain a first image difference encoding result between the first reference image and the first image to be evaluated.

[0245] According to one or more embodiments of the present application, [Example 4] provides an image quality assessment method, wherein the image quality assessment model includes: a second feature extractor, a second difference encoder, a second autoencoder, and a second regressor;

[0246] Inputting the second reference image and the second image to be evaluated into the image quality assessment model to obtain a second image difference encoding result between the second reference image and the second image to be evaluated and an evaluation prediction result of the second image to be evaluated includes:

[0247] Inputting a second reference image and a second image to be evaluated into the second feature extractor, obtaining image features of the second reference image and image features of the second image to be evaluated, and calculating second differential features between the image features of the second reference image and the image features of the second image to be evaluated;

[0248] Inputting the second differential feature into the second difference encoder to obtain a second image difference encoding result;

[0249] Inputting the image features of the second image to be evaluated into the second autoencoder to obtain an autoencoding result of the second image to be evaluated;

[0250] After merging features of the autoencoding result of the second image to be evaluated and the second image difference encoding result, the results are input into the second regressor to obtain an evaluation prediction result of the second image to be evaluated;

[0251] The using the first loss value and the second loss value to train the image quality assessment model includes:

[0252] The second difference encoder, the second autoencoder, and the second regressor are trained using the first loss value and the second loss value.

[0253] According to one or more embodiments of the present application, [Example 5] provides an image quality assessment method, wherein the full-reference teacher model includes: a first feature extractor, a first difference encoder, a first autoencoder, and a first regressor; the training process of the full-reference teacher model includes:

[0254] inputting a third reference image and a third image to be evaluated into the first feature extractor, obtaining image features of the third reference image and image features of the third image to be evaluated, and calculating third differential features between the image features of the third reference image and the image features of the third image to be evaluated; the third reference image and the third image to be evaluated have the same image content, and the image quality of the third reference image is not lower than that of the third image to be evaluated;

[0255] Inputting the third differential feature into the first difference encoder to obtain a third image difference encoding result;

[0256] Inputting the image features of the third image to be evaluated into the first autoencoder to obtain an autoencoding result of the third image to be evaluated;

[0257] After merging features of the autoencoding result of the third image to be evaluated and the third image difference encoding result, the results are input into the first regressor to obtain an evaluation prediction result of the third image to be evaluated;

[0258] Calculating a third loss value according to the evaluation prediction result of the third image to be evaluated and the evaluation calibration result of the third image to be evaluated;

[0259] The first difference encoder, the first autoencoder, and the first regressor are trained using the third loss value, and the step of inputting the third reference image and the third image to be evaluated into the first feature extractor and subsequent steps are repeated until a second preset condition is met.

[0260] According to one or more embodiments of the present application, [Example 6] provides an image quality assessment method, wherein the training process of the image quality assessment model further includes:

[0261] Performing random image block cropping on the first reference image to obtain at least one first reference image block;

[0262] Performing random image block cropping on the first image to be evaluated to obtain at least one first image block to be evaluated;

[0263] Performing random image block cropping on the second reference image to obtain at least one second reference image block;

[0264] Performing random image block cropping on the second image to be evaluated to obtain at least one second image block to be evaluated;

[0265] The step of inputting the first reference image and the first image to be evaluated into the full-reference teacher model comprises:

[0266] inputting the at least one first reference image block and the at least one first image block to be evaluated into a full-reference teacher model;

[0267] The step of inputting the second reference image and the second image to be evaluated into the image quality assessment model comprises:

[0268] The at least one second reference image block and the at least one second image block to be evaluated are input into an image quality assessment model.

[0269] According to one or more embodiments of the present application, [Example 7] provides an image quality assessment method, wherein the training process of the full-reference teacher model further includes:

[0270] Performing random image block cropping on the third reference image to obtain at least one third reference image block;

[0271] Performing random image block cropping on the third image to be evaluated to obtain at least one third image block to be evaluated;

[0272] Inputting the third reference image and the third image to be evaluated into the first feature extractor includes:

[0273] The at least one third reference image block and the at least one third image block to be evaluated are input into the first feature extractor.

[0274] According to one or more embodiments of the present application, [Example Eight] provides an image quality assessment method, wherein the first feature extractor is a Resnet network, the first feature extractor includes multiple residual module groups, each of the residual module groups includes multiple residual modules, and the outputs of each of the residual module groups together constitute the output of the first feature extractor.

[0275] According to one or more embodiments of the present application, [Example Nine] provides an image quality assessment method, wherein the second feature extractor is a Resnet network, the second feature extractor includes multiple residual module groups, each of the residual module groups includes multiple residual modules, and the outputs of each of the residual module groups together constitute the output of the second feature extractor.

[0276] According to one or more embodiments of the present application, [Example 10] provides an image quality assessment method, wherein the second difference encoder and the second autoencoder both include multiple multilayer perceptrons.

[0277] According to one or more embodiments of the present application, [Example 11] provides an image quality assessment method, wherein the first difference encoder and the first autoencoder both include multiple multi-layer perceptrons.

[0278] According to one or more embodiments of the present application, [Example 12] provides an image quality assessment method, wherein the image quality assessment model includes: a second feature extractor, a second difference encoder, a second autoencoder, and a second regressor;

[0279] Inputting the fourth reference image and the fourth image to be evaluated into an image quality assessment model to obtain an assessment prediction result of the fourth image to be evaluated includes:

[0280] inputting a fourth reference image and a fourth image to be evaluated into the second feature extractor, obtaining image features of the fourth reference image and image features of the fourth image to be evaluated, and calculating fourth differential features between the image features of the fourth reference image and the image features of the fourth image to be evaluated;

[0281] inputting the fourth differential feature into the second difference encoder to obtain a fourth difference encoding result;

[0282] Inputting the image features of the fourth image to be evaluated into the second autoencoder to obtain an autoencoding result of the fourth image to be evaluated;

[0283] After feature merging, the autoencoding result of the fourth image to be evaluated and the fourth difference encoding result are input into the second regressor to obtain an evaluation prediction result of the fourth image to be evaluated.

[0284] According to one or more embodiments of the present application, [Example 13] provides an image quality assessment device, the device comprising:

[0285] an acquiring unit, configured to acquire a fourth reference image and a fourth image to be evaluated, wherein the image contents of the fourth reference image and the fourth image to be evaluated are the same or different;

[0286] an input unit, configured to input the fourth reference image and the fourth image to be evaluated into an image quality assessment model to obtain an assessment prediction result of the fourth image to be evaluated;

[0287] The image quality assessment model is trained based on a first image difference coding result between a first reference image and a first image to be evaluated, a second image difference coding result between a second reference image and a second image to be evaluated, an assessment prediction result of the second image to be evaluated, and an assessment calibration result of the second image to be evaluated; wherein, the first reference image has the same image content as the first image to be evaluated, and the image quality of the first reference image is not lower than that of the first image to be evaluated; the second reference image has the same or different image content as the second image to be evaluated, and the image quality of the second reference image is not lower than that of the second image to be evaluated.

[0288] According to one or more embodiments of the present application, [Example 14] provides an image quality assessment device, the image quality assessment device further comprising: a training unit, comprising: a first input unit, configured to input a first reference image and a first image to be assessed into a full-reference teacher model, and obtain a first image difference encoding result between the first reference image and the first image to be assessed; the first reference image and the first image to be assessed have the same image content, and the image quality of the first reference image is not lower than that of the first image to be assessed;

[0289] a second input unit, configured to input a second reference image and a second image to be evaluated into the image quality assessment model, to obtain a second image difference coding result between the second reference image and the second image to be evaluated, and an assessment prediction result of the second image to be evaluated; the image content of the second reference image and the second image to be evaluated being the same or different, and the image quality of the second reference image being no less than that of the second image to be evaluated;

[0290] a first calculation unit, configured to calculate a first loss value according to an evaluation prediction result of the second image to be evaluated and an evaluation calibration result of the second image to be evaluated;

[0291] a second calculation unit, configured to calculate a second loss value according to the first image difference encoding result and the second image difference encoding result;

[0292] The first training unit is used to train the image quality assessment model using the first loss value and the second loss value, and repeatedly perform the steps of inputting the first reference image and the first image to be evaluated into the full reference teacher model and subsequent steps until a first preset condition is met.

[0293] According to one or more embodiments of the present application, [Example 15] provides an image quality assessment device, wherein the full-reference teacher model includes: a first feature extractor and a first difference encoder;

[0294] The first input unit includes:

[0295] a first input subunit, configured to input a first reference image and a first image to be evaluated into the first feature extractor, obtain image features of the first reference image and image features of the first image to be evaluated, and calculate a first differential feature between the image features of the first reference image and the image features of the first image to be evaluated;

[0296] The second input subunit is configured to input the first differential feature into the first difference encoder to obtain a first image difference encoding result between the first reference image and the first image to be evaluated.

[0297] According to one or more embodiments of the present application, [Example 16] provides an image quality assessment device, wherein the image quality assessment model includes: a second feature extractor, a second difference encoder, a second autoencoder, and a second regressor;

[0298] The second input unit includes:

[0299] a third input subunit, configured to input a second reference image and a second image to be evaluated into the second feature extractor, obtain image features of the second reference image and image features of the second image to be evaluated, and calculate second differential features between the image features of the second reference image and the image features of the second image to be evaluated;

[0300] a fourth input subunit, configured to input the second differential feature into the second difference encoder to obtain a second image difference encoding result;

[0301] a fifth input subunit, configured to input the image features of the second image to be evaluated into the second autoencoder to obtain an autoencoding result of the second image to be evaluated;

[0302] a sixth input subunit, configured to perform feature merging on the autoencoding result of the second image to be evaluated and the second image difference encoding result, and input the merged features into the second regressor to obtain an evaluation prediction result of the second image to be evaluated;

[0303] The first training unit includes:

[0304] The first training subunit is used to train the second difference encoder, the second autoencoder, and the second regressor using the first loss value and the second loss value.

[0305] According to one or more embodiments of the present application, [Example 17] provides an image quality assessment device, wherein the full-reference teacher model includes: a first feature extractor, a first difference encoder, a first autoencoder, and a first regressor; the training unit further includes:

[0306] a third input unit, configured to input a third reference image and a third image to be evaluated into the first feature extractor, obtain image features of the third reference image and image features of the third image to be evaluated, and calculate third differential features between the image features of the third reference image and the image features of the third image to be evaluated; the third reference image and the third image to be evaluated have the same image content, and the image quality of the third reference image is not lower than that of the third image to be evaluated;

[0307] a fourth input unit, configured to input the third differential feature into the first difference encoder to obtain a third image difference encoding result;

[0308] a fifth input unit, configured to input the image features of the third image to be evaluated into the first autoencoder to obtain an autoencoding result of the third image to be evaluated;

[0309] a sixth input unit, configured to combine the features of the autoencoding result of the third image to be evaluated and the third image difference encoding result, and input the combined features into the first regressor to obtain an evaluation prediction result of the third image to be evaluated;

[0310] a calculation unit, configured to calculate a third loss value according to an evaluation prediction result of the third image to be evaluated and an evaluation calibration result of the third image to be evaluated;

[0311] The second training unit is used to use the third loss value to train the first difference encoder, the first autoencoder and the first regressor, and repeatedly perform the steps of inputting the third reference image and the third image to be evaluated into the first feature extractor and subsequent steps until a second preset condition is met.

[0312] According to one or more embodiments of the present application, [Example 18] provides an image quality assessment device, wherein the training unit further includes:

[0313] A first cropping unit, configured to perform random image block cropping on the first reference image to obtain at least one first reference image block;

[0314] a second cropping unit, configured to perform random image block cropping on the first image to be evaluated to obtain at least one first image block to be evaluated;

[0315] a third cropping unit, configured to perform random image block cropping on the second reference image to obtain at least one second reference image block;

[0316] a fourth cropping unit, configured to perform random image block cropping on the second image to be evaluated to obtain at least one second image block to be evaluated;

[0317] The first input unit includes:

[0318] a seventh input subunit, configured to input the at least one first reference image block and the at least one first image block to be evaluated into a full-reference teacher model;

[0319] The second input unit includes:

[0320] The eighth input subunit is configured to input the at least one second reference image block and the at least one second image block to be evaluated into the image quality assessment model.

[0321] According to one or more embodiments of the present application, [Example 19] provides an image quality assessment device, wherein the training unit further includes:

[0322] a fifth cropping unit, configured to perform random image block cropping on the third reference image to obtain at least one third reference image block;

[0323] a sixth cropping unit, configured to perform random image block cropping on the third image to be evaluated to obtain at least one third image block to be evaluated;

[0324] The third input unit includes:

[0325] The ninth input subunit is configured to input the at least one third reference image block and the at least one third image block to be evaluated into the first feature extractor.

[0326] According to one or more embodiments of the present application, [Example 20] provides an image quality assessment device, wherein the first feature extractor is a Resnet network, the first feature extractor includes multiple residual module groups, each of the residual module groups includes multiple residual modules, and the outputs of each of the residual module groups together constitute the output of the first feature extractor.

[0327] According to one or more embodiments of the present application, [Example 21] provides an image quality assessment device, wherein the second feature extractor is a Resnet network, the second feature extractor includes multiple residual module groups, each of the residual module groups includes multiple residual modules, and the outputs of each of the residual module groups together constitute the output of the second feature extractor.

[0328] According to one or more embodiments of the present application, [Example 22] provides an image quality assessment device, wherein the second difference encoder and the second autoencoder both include multiple multi-layer perceptrons.

[0329] According to one or more embodiments of the present application, [Example 23] provides an image quality assessment device, wherein the first difference encoder and the first autoencoder both include multiple multi-layer perceptrons.

[0330] According to one or more embodiments of the present application, [Example 24] provides an image quality assessment device, wherein the image quality assessment model includes: a second feature extractor, a second difference encoder, a second autoencoder, and a second regressor;

[0331] The input unit includes:

[0332] a first input subunit, configured to input a fourth reference image and a fourth image to be evaluated into the second feature extractor, obtain image features of the fourth reference image and image features of the fourth image to be evaluated, and calculate fourth differential features between the image features of the fourth reference image and the image features of the fourth image to be evaluated;

[0333] a second input subunit, configured to input the fourth differential feature into the second difference encoder to obtain a fourth difference encoding result;

[0334] a third input subunit, configured to input the image features of the fourth image to be evaluated into the second autoencoder to obtain an autoencoding result of the fourth image to be evaluated;

[0335] The fourth input subunit is configured to perform feature merging on the autoencoding result of the fourth image to be evaluated and the fourth difference encoding result, and input the merged features into the second regressor to obtain an evaluation prediction result of the fourth image to be evaluated.

[0336] According to one or more embodiments of the present application, [Example 25] provides an electronic device, including:

[0337] one or more processors;

[0338] a storage device having one or more programs stored thereon,

[0339] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-described image quality assessment methods.

[0340] According to one or more embodiments of the present application, [Example 26] provides a computer-readable medium on which a computer program is stored, wherein when the program is executed by a processor, an image quality assessment method as described above is implemented.

[0341] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0342] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0343] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0344] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0345] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for image quality assessment, characterized in that: The method comprises: Acquiring a fourth reference image and a fourth image to be evaluated, wherein the fourth reference image and the fourth image to be evaluated have the same or different image contents; Inputting the fourth reference image and the fourth image to be evaluated into an image quality assessment model to obtain an evaluation prediction result of the fourth image to be evaluated; The image quality assessment model is trained based on a first image difference coding result between a first reference image and a first image to be assessed, a second image difference coding result between a second reference image and a second image to be assessed, an assessment prediction result of the second image to be assessed, and an assessment calibration result of the second image to be assessed; wherein the first reference image has the same image content as the first image to be assessed, and the image quality of the first reference image is not lower than that of the first image to be assessed; the second reference image has the same or different image content as the second image to be assessed, and the image quality of the second reference image is not lower than that of the second image to be assessed; The training process of the image quality assessment model includes: Inputting a first reference image and a first image to be evaluated into a full-reference teacher model to obtain a first image difference encoding result between the first reference image and the first image to be evaluated; Inputting a second reference image and a second image to be evaluated into an image quality assessment model, obtaining a second image difference coding result between the second reference image and the second image to be evaluated and an evaluation prediction result of the second image to be evaluated; Calculating a first loss value according to the evaluation prediction result of the second image to be evaluated and the evaluation calibration result of the second image to be evaluated; Calculating a second loss value according to the first image difference encoding result and the second image difference encoding result; The image quality assessment model is trained using the first loss value and the second loss value, and the step of inputting the first reference image and the first image to be evaluated into the full-reference teacher model and subsequent steps are repeatedly performed until a first preset condition is met.

2. The method according to claim 1, characterized in that The full-reference teacher model includes: a first feature extractor and a first difference encoder; The step of inputting the first reference image and the first image to be evaluated into a full reference teacher model to obtain a first image difference encoding result between the first reference image and the first image to be evaluated includes: Inputting a first reference image and a first image to be evaluated into the first feature extractor, obtaining image features of the first reference image and image features of the first image to be evaluated, and calculating a first differential feature between the image features of the first reference image and the image features of the first image to be evaluated; The first differential feature is input into the first difference encoder to obtain a first image difference encoding result between the first reference image and the first image to be evaluated.

3. The method according to claim 1, characterized in that The image quality assessment model includes: a second feature extractor, a second difference encoder, a second autoencoder and a second regressor; Inputting the second reference image and the second image to be evaluated into the image quality assessment model to obtain a second image difference encoding result between the second reference image and the second image to be evaluated and an evaluation prediction result of the second image to be evaluated includes: Inputting a second reference image and a second image to be evaluated into the second feature extractor, obtaining image features of the second reference image and image features of the second image to be evaluated, and calculating second differential features between the image features of the second reference image and the image features of the second image to be evaluated; Inputting the second differential feature into the second difference encoder to obtain a second image difference encoding result; Inputting the image features of the second image to be evaluated into the second autoencoder to obtain an autoencoding result of the second image to be evaluated; After merging features of the autoencoding result of the second image to be evaluated and the second image difference encoding result, the results are input into the second regressor to obtain an evaluation prediction result of the second image to be evaluated; The using the first loss value and the second loss value to train the image quality assessment model includes: The second difference encoder, the second autoencoder, and the second regressor are trained using the first loss value and the second loss value.

4. The method according to claim 1 or 2, characterized in that The full-reference teacher model includes: a first feature extractor, a first difference encoder, a first autoencoder, and a first regressor; the training process of the full-reference teacher model includes: inputting a third reference image and a third image to be evaluated into the first feature extractor, obtaining image features of the third reference image and image features of the third image to be evaluated, and calculating third differential features between the image features of the third reference image and the image features of the third image to be evaluated; the third reference image and the third image to be evaluated have the same image content, and the image quality of the third reference image is not lower than that of the third image to be evaluated; Inputting the third differential feature into the first difference encoder to obtain a third image difference encoding result; Inputting the image features of the third image to be evaluated into the first autoencoder to obtain an autoencoding result of the third image to be evaluated; After merging features of the autoencoding result of the third image to be evaluated and the third image difference encoding result, the results are input into the first regressor to obtain an evaluation prediction result of the third image to be evaluated; Calculating a third loss value according to the evaluation prediction result of the third image to be evaluated and the evaluation calibration result of the third image to be evaluated; The first difference encoder, the first autoencoder, and the first regressor are trained using the third loss value, and the step of inputting the third reference image and the third image to be evaluated into the first feature extractor and subsequent steps are repeated until a second preset condition is met.

5. The method according to any one of claims 1 to 3, characterized in that The training process of the image quality assessment model further includes: Performing random image block cropping on the first reference image to obtain at least one first reference image block; Performing random image block cropping on the first image to be evaluated to obtain at least one first image block to be evaluated; Performing random image block cropping on the second reference image to obtain at least one second reference image block; Performing random image block cropping on the second image to be evaluated to obtain at least one second image block to be evaluated; The step of inputting the first reference image and the first image to be evaluated into the full-reference teacher model comprises: inputting the at least one first reference image block and the at least one first image block to be evaluated into a full-reference teacher model; The step of inputting the second reference image and the second image to be evaluated into the image quality assessment model comprises: The at least one second reference image block and the at least one second image block to be evaluated are input into an image quality assessment model.

6. The method according to claim 4, characterized in that The training process of the full-reference teacher model also includes: Performing random image block cropping on the third reference image to obtain at least one third reference image block; Performing random image block cropping on the third image to be evaluated to obtain at least one third image block to be evaluated; Inputting the third reference image and the third image to be evaluated into the first feature extractor includes: The at least one third reference image block and the at least one third image block to be evaluated are input into the first feature extractor.

7. The method according to claim 2, characterized in that The first feature extractor is a Resnet network, and the first feature extractor includes multiple residual module groups, each of the residual module groups includes multiple residual modules, and the outputs of each of the residual module groups together constitute the output of the first feature extractor.

8. The method according to claim 4, characterized in that The first feature extractor is a Resnet network, and the first feature extractor includes multiple residual module groups, each of the residual module groups includes multiple residual modules, and the outputs of each of the residual module groups together constitute the output of the first feature extractor.

9. The method according to claim 3, characterized in that The second feature extractor is a Resnet network, and the second feature extractor includes multiple residual module groups, each of the residual module groups includes multiple residual modules, and the outputs of each residual module group together constitute the output of the second feature extractor.

10. The method according to claim 3, characterized in that The second difference encoder and the second autoencoder each include a plurality of multilayer perceptrons.

11. The method according to claim 4, characterized in that The first difference encoder and the first autoencoder each include a plurality of multilayer perceptrons.

12. The method according to claim 1, characterized in that The image quality assessment model includes: a second feature extractor, a second difference encoder, a second autoencoder and a second regressor; Inputting the fourth reference image and the fourth image to be evaluated into an image quality assessment model to obtain an assessment prediction result of the fourth image to be evaluated includes: inputting a fourth reference image and a fourth image to be evaluated into the second feature extractor, obtaining image features of the fourth reference image and image features of the fourth image to be evaluated, and calculating fourth differential features between the image features of the fourth reference image and the image features of the fourth image to be evaluated; inputting the fourth differential feature into the second difference encoder to obtain a fourth difference encoding result; Inputting the image features of the fourth image to be evaluated into the second autoencoder to obtain an autoencoding result of the fourth image to be evaluated; After feature merging, the autoencoding result of the fourth image to be evaluated and the fourth difference encoding result are input into the second regressor to obtain an evaluation prediction result of the fourth image to be evaluated.

13. An image quality assessment device, characterized in that: The device comprises: an acquiring unit, configured to acquire a fourth reference image and a fourth image to be evaluated, wherein the image contents of the fourth reference image and the fourth image to be evaluated are the same or different; an input unit, configured to input the fourth reference image and the fourth image to be evaluated into an image quality assessment model to obtain an assessment prediction result of the fourth image to be evaluated; The image quality assessment model is trained based on a first image difference coding result between a first reference image and a first image to be assessed, a second image difference coding result between a second reference image and a second image to be assessed, an assessment prediction result of the second image to be assessed, and an assessment calibration result of the second image to be assessed; wherein the first reference image has the same image content as the first image to be assessed, and the image quality of the first reference image is not lower than that of the first image to be assessed; the second reference image has the same or different image content as the second image to be assessed, and the image quality of the second reference image is not lower than that of the second image to be assessed; The training process of the image quality assessment model includes: Inputting a first reference image and a first image to be evaluated into a full-reference teacher model to obtain a first image difference encoding result between the first reference image and the first image to be evaluated; Inputting a second reference image and a second image to be evaluated into an image quality assessment model, obtaining a second image difference coding result between the second reference image and the second image to be evaluated and an evaluation prediction result of the second image to be evaluated; Calculating a first loss value according to the evaluation prediction result of the second image to be evaluated and the evaluation calibration result of the second image to be evaluated; Calculating a second loss value according to the first image difference encoding result and the second image difference encoding result; The image quality assessment model is trained using the first loss value and the second loss value, and the step of inputting the first reference image and the first image to be evaluated into the full-reference teacher model and subsequent steps are repeatedly performed until a first preset condition is met.

14. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the image quality assessment method according to any one of claims 1 to 12.

15. A computer-readable medium, characterized in that A computer program is stored thereon, wherein when the program is executed by a processor, the image quality assessment method according to any one of claims 1 to 12 is implemented.

Images