Model training method, image quality evaluation method, device, and storage medium

Abstract

The present disclosure relates to a model training method, an image quality evaluation method, a device and a storage medium. The method comprises: obtaining a plurality of preset images; the object categories corresponding to the plurality of preset images are different; performing image damage processing on the plurality of preset images to obtain a plurality of sample images; each sample image corresponds to one kind of damage information, and the damage information represents the image quality of the sample image; from the sample images corresponding to the same preset image, sample images with the same damage information are selected to construct a positive sample pair; based on the sample images corresponding to any two preset images and the sample images with different damage information corresponding to the same preset image, a negative sample pair is constructed; based on the first preset model, the positive sample pair and the negative sample pair are compared and learned to train an initial feature extraction sub-model; and based on the initial feature extraction sub-model, an image quality evaluation model is determined. The present disclosure improves the effectiveness of the image quality evaluation model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model training method, an image quality evaluation method, an apparatus, and a storage medium. Background Technology

[0002] In recent years, thanks to the development of deep learning theory, NR-IQA research has made significant progress. As a typical data-driven algorithm, the performance of NR-IQA algorithms based on deep neural networks (DNNs) is inextricably linked to the scale of the training data; the larger the training data, the better the algorithm's performance tends to be. Compared to common image recognition, detection, and segmentation tasks, the data annotation process for IQA tasks is extremely cumbersome and costly, which limits the size of labeled datasets. To date, the largest real-world image quality assessment dataset in the IQA field, FLIVE, contains fewer than 40,000 images. In contrast, the classic image classification dataset ImageNet1k contains over 1.28 million images. Therefore, existing IQA algorithms are limited by the size of labeled IQA datasets, resulting in lower effectiveness of image quality assessment models obtained through supervised training. Summary of the Invention

[0003] This disclosure provides a model training method, an image quality assessment method, an apparatus, and a storage medium to at least address the problem of low effectiveness of image quality assessment models in related technologies. The technical solution of this disclosure is as follows:

[0004] According to a first aspect of the present disclosure, a method for training an image quality assessment model is provided, comprising:

[0005] Acquire multiple preset images; each of the multiple preset images corresponds to a different object category;

[0006] Image impairment processing is performed on the multiple preset images to obtain multiple sample images; each sample image corresponds to a type of impairment information, and the impairment information characterizes the image quality of the sample image.

[0007] From the sample images corresponding to the same preset image, sample images with the same damage information are selected to construct positive sample pairs;

[0008] Based on the sample images corresponding to any two preset images, and the sample images with different damage information in the sample images corresponding to the same preset image, construct negative sample pairs;

[0009] Based on the first preset model, the positive sample pairs and the negative sample pairs are compared and trained to obtain the initial feature extraction sub-model;

[0010] Based on the initial feature extraction sub-model, an image quality evaluation model is determined.

[0011] In one exemplary embodiment, the first preset model includes an object feature extraction network, an image damage feature extraction network, and a feature fusion network. The step of performing comparative learning training on the positive sample pairs and the negative sample pairs based on the first preset model to obtain an initial feature extraction sub-model includes:

[0012] The sample image pairs are input into the object feature extraction network for object feature extraction processing to obtain sample object features; the sample image pairs include the positive sample pairs and the negative sample pairs.

[0013] The sample image is input to the image damage feature extraction network for image damage feature extraction processing to obtain sample damage features;

[0014] Input the sample object features and sample damage features corresponding to any sample image in the sample image pair into the feature fusion network to obtain the sample image features corresponding to any sample image.

[0015] Based on the sample image features corresponding to each of the two sample images in the sample image pair, the object feature extraction network, the image damage feature extraction network, and the feature fusion network are trained to obtain the initial feature extraction sub-model.

[0016] In an exemplary embodiment, the step of inputting the sample object features and sample damage features corresponding to any sample image in the sample image pair into the feature fusion network to obtain the sample image features corresponding to any sample image includes:

[0017] The sample object features and sample damage features corresponding to any sample image in the sample image pair are input into the feature fusion network;

[0018] Based on the feature fusion network, a first weight of the sample object feature corresponding to any sample image and a second weight of the sample damage feature corresponding to any sample image are determined.

[0019] The first sample image features are determined based on the product of the sample object features corresponding to any sample image and the first weight.

[0020] The second sample image features are determined based on the product of the sample damage features corresponding to any sample image and the second weight;

[0021] Based on the first sample image features and the second sample image features, determine the sample image features corresponding to any one of the sample images.

[0022] In an exemplary embodiment, the step of training the object feature extraction network, the image damage feature extraction network, and the feature fusion network based on the sample image features corresponding to each of the two sample images in the sample image pair to obtain the initial feature extraction sub-model includes:

[0023] Based on the sample image features corresponding to each of the two sample images in the sample image pair, adjust the parameters of the object feature extraction network, the image damage feature extraction network and the feature fusion network until the first training termination condition is met.

[0024] The initial feature extraction sub-model is determined based on the object feature extraction network, image damage feature extraction network, and feature fusion network when the first training termination condition is met.

[0025] In an exemplary implementation, adjusting the parameters of the object feature extraction network, the image damage feature extraction network, and the feature fusion network based on the sample image features corresponding to each of the two sample images in the sample image pair until the first training termination condition is met includes:

[0026] Based on the sample image features corresponding to each of the two positive sample images in the positive sample pair, the first similarity between the two positive sample images in the positive sample pair is determined;

[0027] Based on the sample image features corresponding to each of the two negative sample images in the negative sample pair, a second similarity between the two negative sample images in the negative sample pair is determined;

[0028] Based on the difference between the first similarity and the second similarity, loss information is determined;

[0029] The parameters of the object feature extraction network, the image damage feature extraction network, and the feature fusion network are adjusted according to the loss information until the first training termination condition is met.

[0030] In one exemplary implementation, determining the image quality assessment model based on the initial feature extraction sub-model includes:

[0031] Obtain training images labeled with training impairment information;

[0032] The training image features of the training image are extracted based on the initial feature extraction sub-model;

[0033] The damage information recognition result corresponding to the features of the training image is determined based on the second preset model; the damage information recognition result characterizes the image quality of the training image.

[0034] Based on the difference between the damage information identification result and the training damage information label, the initial feature extraction sub-model and the second preset model are trained to obtain the image quality evaluation model.

[0035] In one exemplary embodiment, the step of training the initial feature extraction sub-model and the second preset model based on the difference between the damage information recognition result and the trained damage information label to obtain the image quality evaluation model includes:

[0036] Based on the difference between the damage information identification result and the training damage information label, train the initial feature extraction sub-model and the second preset model until the second training termination condition is met;

[0037] The image quality evaluation model is determined based on the image feature extraction sub-model and the damage information recognition sub-model; the image feature extraction sub-model is the initial feature extraction sub-model that meets the second training termination condition, and the damage information recognition sub-model is the second preset model that meets the second training termination condition.

[0038] In one exemplary implementation, the step of constructing negative sample pairs based on sample images corresponding to any two preset images, and sample images with different damage information in the same preset image, includes:

[0039] Construct the first negative sample pair based on the sample images corresponding to any two preset images;

[0040] From the sample images corresponding to the same preset image, sample images with different damage information are selected to construct a second negative sample pair;

[0041] The negative sample pair is constructed based on the first negative sample pair and the second negative sample pair.

[0042] According to a second aspect of the present disclosure, an image quality evaluation method is provided, the method comprising:

[0043] Acquire the image to be recognized;

[0044] The image to be identified is input into an image quality assessment model for image quality assessment processing to obtain target damage information of the image to be identified; the target damage information characterizes the image quality of the image to be identified.

[0045] The image quality evaluation model is trained based on the method described above.

[0046] According to a third aspect of the present disclosure, a training apparatus for an image quality assessment model is provided, comprising:

[0047] The preset image acquisition module is configured to acquire multiple preset images; each of the multiple preset images corresponds to a different object category;

[0048] The sample image determination module is configured to perform image damage processing on the plurality of preset images to obtain a plurality of sample images; each sample image corresponds to a damage information, and the damage information characterizes the image quality of the sample image.

[0049] The positive sample construction module is configured to select sample images with the same damage information from sample images corresponding to the same preset image and construct positive sample pairs;

[0050] The negative sample construction module is configured to construct negative sample pairs based on sample images corresponding to any two preset images, and sample images with different damage information in the sample images corresponding to the same preset image.

[0051] The contrast training module is configured to perform contrast learning training on the positive sample pairs and the negative sample pairs based on a first preset model to obtain an initial feature extraction sub-model.

[0052] The model determination module is configured to execute an image quality evaluation model based on the initial feature extraction sub-model.

[0053] In one exemplary embodiment, the first preset model includes an object feature extraction network, an image damage feature extraction network, and a feature fusion network, and the contrast training module includes:

[0054] The sample object feature determination submodule is configured to perform object feature extraction processing on the input sample image pairs to the object feature extraction network to obtain sample object features; the sample image pairs include the positive sample pairs and the negative sample pairs.

[0055] The sample damage feature determination submodule is configured to perform image damage feature extraction processing on the input image damage feature extraction network of the sample image to obtain sample damage features;

[0056] The feature fusion submodule is configured to input the sample object features and sample damage features corresponding to any sample image in the sample image pair into the feature fusion network to obtain the sample image features corresponding to any sample image.

[0057] The training submodule is configured to train the object feature extraction network, the image damage feature extraction network, and the feature fusion network based on the sample image features corresponding to the two sample images in the sample image pair, to obtain the initial feature extraction submodel.

[0058] In one exemplary implementation, the feature fusion submodule includes:

[0059] The feature input unit is configured to input the sample object features and sample damage features corresponding to any sample image in the sample image pair into the feature fusion network;

[0060] The weight determination unit is configured to perform a process based on the feature fusion network to determine a first weight of the sample object feature corresponding to any sample image and a second weight of the sample damage feature corresponding to any sample image.

[0061] The first feature determination unit is configured to perform a product of the sample object features corresponding to any sample image and the first weight to determine the features of the first sample image;

[0062] The second feature determination unit is configured to perform a second sample image feature determination based on the product of the sample damage feature corresponding to any sample image and the second weight.

[0063] The sample image feature determination unit is configured to perform the task of determining the sample image features corresponding to any one of the sample images based on the first sample image features and the second sample image features.

[0064] In one exemplary implementation, the training submodule includes:

[0065] The first training unit is configured to perform adjustments to the parameters of the object feature extraction network, the image damage feature extraction network, and the feature fusion network based on the sample image features corresponding to the two sample images in the sample image pair, until the first training termination condition is met.

[0066] The model determination unit is configured to execute an object feature extraction network, an image damage feature extraction network, and a feature fusion network based on the first training termination condition to determine the initial feature extraction sub-model.

[0067] In one exemplary implementation, the first training unit includes:

[0068] The first similarity determination subunit is configured to perform the task of determining the first similarity between the two positive sample images in the positive sample pair based on the sample image features corresponding to the two positive sample images in the positive sample pair.

[0069] The second similarity determination subunit is configured to perform the following: determine the second similarity between the two negative sample images in the negative sample pair based on the sample image features corresponding to the two negative sample images in the negative sample pair.

[0070] The loss information determination subunit is configured to determine loss information based on the difference between the first similarity and the second similarity;

[0071] The parameter adjustment subunit is configured to adjust the parameters of the object feature extraction network, the image damage feature extraction network, and the feature fusion network according to the loss information until the first training termination condition is met.

[0072] In one exemplary embodiment, the model determination module includes:

[0073] The training image acquisition unit is configured to acquire training images labeled with training impairment information.

[0074] The training feature extraction unit is configured to extract training image features from the training image based on the initial feature extraction sub-model;

[0075] The identification result determination unit is configured to perform a damage information identification result based on a second preset model to determine the damage information identification result corresponding to the features of the training image; the damage information identification result characterizes the image quality of the training image.

[0076] The second training unit is configured to train the initial feature extraction sub-model and the second preset model based on the difference between the damage information recognition result and the training damage information label, so as to obtain the image quality evaluation model.

[0077] In one exemplary embodiment, the second training unit includes:

[0078] The training subunit is configured to train the initial feature extraction submodel and the second preset model based on the difference between the damage information identification result and the training damage information label, until the second training termination condition is met.

[0079] The model determination subunit is configured to execute an image feature extraction submodel and a damage information recognition submodel to determine the image quality evaluation model; the image feature extraction submodel is an initial feature extraction submodel that meets the second training termination condition, and the damage information recognition submodel is a second preset model that meets the second training termination condition.

[0080] In one exemplary implementation, the negative sample construction module includes:

[0081] The first construction submodule is configured to construct a first negative sample pair based on the sample images corresponding to any two preset images.

[0082] The second construction submodule is configured to perform the task of selecting sample images with different damage information from the sample images corresponding to the same preset image, and constructing a second negative sample pair.

[0083] The negative sample pair construction submodule is configured to perform the construction of the negative sample pair based on the first negative sample pair and the second negative sample pair.

[0084] According to a fourth aspect of the present disclosure, an image quality evaluation apparatus is provided, the apparatus comprising:

[0085] The image acquisition module is configured to acquire the image to be recognized.

[0086] The target damage information determination module is configured to perform image quality evaluation processing by inputting the image to be identified into an image quality evaluation model to obtain target damage information of the image to be identified; the target damage information characterizes the image quality of the image to be identified.

[0087] The image quality evaluation model is trained based on the method described above.

[0088] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising:

[0089] processor;

[0090] Memory used to store the processor's executable instructions;

[0091] The processor is configured to execute the instructions to implement the training method or image quality assessment method of the image quality assessment model as described above.

[0092] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when the instructions in the computer-readable storage medium are executed by an electronic device processor, the electronic device is enabled to perform the training method or image quality evaluation method of the image quality evaluation model as described above.

[0093] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the training method or image quality evaluation method of the image quality evaluation model as described above.

[0094] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects:

[0095] This disclosure obtains multiple preset images; each preset image corresponds to a different object category; image damage processing is performed on the multiple preset images to obtain multiple sample images; each sample image corresponds to a type of damage information, which characterizes the image quality of the sample image; from the sample images corresponding to the same preset image, sample images with the same damage information are selected to construct positive sample pairs; based on the sample images corresponding to any two preset images, and sample images with different damage information in the sample images corresponding to the same preset image, negative sample pairs are constructed; based on a first preset model, the positive sample pairs and negative sample pairs are trained through comparative learning to obtain an initial feature extraction sub-model; based on the initial feature extraction sub-model, an image quality evaluation model is determined. This disclosure obtains multiple sample images by performing image damage processing on preset images; then constructs positive and negative sample pairs based on the preset images corresponding to the sample images and damage information; thus, an initial feature extraction sub-model can be obtained through unsupervised comparative learning training using positive and negative sample pairs; this avoids dependence on a large amount of labeled data during model training, reducing model training complexity; and comparative learning training can improve the effectiveness of the initial feature extraction sub-model, thereby improving the effectiveness of the image quality evaluation model.

[0096] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0097] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0098] Figure 1 This is an application environment diagram illustrating a training method for an image quality assessment model according to an exemplary embodiment.

[0099] Figure 2 This is a flowchart illustrating a training method for an image quality assessment model according to an exemplary embodiment.

[0100] Figure 3 This is a schematic diagram illustrating a process for constructing positive and negative sample pairs according to an exemplary embodiment.

[0101] Figure 4 This is a flowchart illustrating a method for obtaining an initial feature extraction sub-model by comparative learning and training the positive sample pairs and the negative sample pairs based on a first preset model, according to an exemplary embodiment.

[0102] Figure 5This is a flowchart illustrating a method, according to an exemplary embodiment, for inputting sample object features and sample damage features corresponding to any sample image in the above sample image pair into the feature fusion network to obtain sample image features corresponding to any sample image.

[0103] Figure 6 This is a flowchart illustrating a method for training the object feature extraction network, the image damage feature extraction network, and the feature fusion network based on the sample image features corresponding to the two sample images in the sample image pair, according to an exemplary embodiment, to obtain the initial feature extraction sub-model.

[0104] Figure 7 This is a flowchart illustrating a method, according to an exemplary embodiment, for adjusting the parameters of the object feature extraction network, the image damage feature extraction network, and the feature fusion network based on the sample image features corresponding to the two sample images in the above sample image pair, until a first training termination condition is met.

[0105] Figure 8 This is a flowchart illustrating a method for determining an image quality evaluation model based on the aforementioned initial feature extraction sub-model, according to an exemplary embodiment.

[0106] Figure 9 This is a flowchart illustrating a method for training an initial feature extraction sub-model and a second preset model based on the difference between the damage information recognition result and the training damage information label, according to an exemplary embodiment, to obtain the image quality evaluation model.

[0107] Figure 10 This is a flowchart illustrating an image quality evaluation method according to an exemplary embodiment.

[0108] Figure 11 This is a block diagram of a training apparatus for an image quality evaluation model according to an exemplary embodiment.

[0109] Figure 12 This is a block diagram of an image quality evaluation device according to an exemplary embodiment.

[0110] Figure 13 This is a block diagram illustrating an electronic device for image quality evaluation according to an exemplary embodiment. Detailed Implementation

[0111] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0112] First, some of the nouns or terms that appear in the description of the embodiments in this specification are explained as follows:

[0113] IQA (Image Quality Assessment): Image quality assessment

[0114] QoE (Quality of Experience): The user's subjective experience.

[0115] DNN (Deep Neural Network): Deep Neural Network

[0116] NR-IQA (No Reference Video Quality Assessment): No-reference image quality assessment

[0117] RR-IQA (Reduced Reference Video Quality Assessment): Semi-reference quality assessment

[0118] FR-IQA (Full Reference Video Quality Assessment): A quality assessment with reference images.

[0119] SRCC (Spearman's Rank Correlation Coefficient): Spearman's rank correlation coefficient

[0120] PLCC (Pearson's Linear Correlation Coefficient): Pearson's linear correlation coefficient

[0121] SOTA (State-of-the-art): The best quality-aware pretrained task (QPT): Pretraining that takes subjective quality into account.

[0122] Mean Opinion Score (MOS): Subjective quality average score

[0123] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0124] To avoid relying on a large amount of labeled data during model training and to improve the effectiveness of image quality assessment models, this disclosure provides a model training method, an image quality assessment method, an apparatus, and a storage medium.

[0125] Please see Figure 1 The diagram illustrates an application environment for a training method of an image quality assessment model according to an exemplary embodiment. The application environment may include a server 01 and a client 02.

[0126] Specifically, in the embodiments of this specification, server 01 may include a standalone server, a distributed server, or a server cluster composed of multiple servers. It may also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 01 may include a network communication unit, a processor, and a memory, etc. Specifically, server 01 can be used to acquire multiple preset images; each preset image corresponds to a different object category; image damage processing is performed on the multiple preset images to obtain multiple sample images; each sample image corresponds to a type of damage information, and the damage information characterizes the image quality of the sample image; from the sample images corresponding to the same preset image, sample images with the same damage information are selected to construct positive sample pairs; based on the sample images corresponding to any two preset images, and sample images with different damage information in the sample images corresponding to the same preset image, negative sample pairs are constructed; based on the first preset model, the positive sample pairs and negative sample pairs are compared and trained to obtain an initial feature extraction sub-model; based on the initial feature extraction sub-model, an image quality evaluation model is determined; and the image to be identified sent by client 02 is input into the image quality evaluation model for image quality evaluation processing to obtain the target damage information of the image to be identified, and the target damage information of the image to be identified is sent to client 02.

[0127] Specifically, in this embodiment, the client 02 may include physical devices such as smartphones, desktop computers, tablets, laptops, digital assistants, smart wearable devices, and in-vehicle terminals, or software running on the physical device, such as web pages provided to users by service providers, or applications provided to users by such service providers. Specifically, the client 02 can be used to send an image to be identified to the server 01, receive target damage information of the image to be identified sent by the server 01, and display the target damage information of the image to be identified, wherein the target damage information characterizes the image quality of the image to be identified.

[0128] Figure 2 This is a flowchart illustrating a training method for an image quality assessment model according to an exemplary embodiment, such as... Figure 2 As shown, this method can be applied to Figure 1 The server 01 shown includes the following steps S201-S2011.

[0129] In step S201, multiple preset images are acquired; the object categories corresponding to the preset images are different.

[0130] In this embodiment, each preset image includes a preset object, and the object categories of the preset objects corresponding to each preset image are different. The preset images can be images from the ImageNet-1k dataset. The ImageNet project is a large visualization database used for research on visual object recognition software. The ImageNet-1k dataset contains more than 1.28 million images with different content, belonging to 1,000 different categories, including buildings, animals, plants, landscapes, etc. Each category can include multiple object categories. For example, if the preset image is an image of a cat, then the object category is cat; if the preset image is an image of a lotus flower, then the object category is lotus flower.

[0131] In step S203, image damage processing is performed on the above-mentioned multiple preset images to obtain multiple sample images; each sample image corresponds to a damage information, and the damage information characterizes the image quality of the sample image.

[0132] In this embodiment, an image damage generator can be constructed based on common image damage methods in real-world environments, including brightness adjustment, contrast adjustment, color adjustment, noise interference, Gaussian blur, and JPEG or JPEG2000 compression. This generator can process the input source image to generate a degraded image that closely resembles reality but exhibits subjective quality differences. The image damage generator may include, but is not limited to, image brightness adjustment algorithms, contrast adjustment algorithms, color adjustment algorithms, noise interference algorithms, Gaussian blur algorithms, and JPEG or JPEG2000 compression algorithms. Multiple preset images can be processed using the image damage generator to obtain multiple sample images, all of which are degraded images.

[0133] In some embodiments, one or more damage treatments can be applied to each preset image, and each damage treatment can include different damage methods (types) and damage levels; thereby obtaining damaged images, which are used as sample images; the damaged images can also be cropped to obtain multiple sample images. Each sample image corresponds to a type of damage information, which includes damage type and damage level; the damage type characterizes the damage treatment method, and the damage level characterizes the degree of damage to the image; the damage information is used to characterize the image quality of the sample image, that is, to characterize the image quality of the sample image, and an image quality score can be determined based on the damage type and damage level.

[0134] In step S205, sample images with the same damage information are selected from the sample images corresponding to the same preset image to construct positive sample pairs.

[0135] In this embodiment, QPT categorizes the factors affecting the subjective quality of an image into two aspects: image content and image quality degradation. Changes in either element will affect the subjective quality of the image. Positive sample pairs can be constructed by selecting sample images with identical degradation information from sample images corresponding to the same preset image; alternatively, positive sample pairs can be constructed by cropping multiple sample images from the same damaged image. A positive sample pair includes two sample images with identical content (from the same preset image) and identical degradation information. In other words, sample images with completely identical content and image quality degradation constitute a positive sample pair.

[0136] In step S207, negative sample pairs are constructed based on the sample images corresponding to any two preset images and the sample images with different damage information in the sample images corresponding to the same preset image.

[0137] In this embodiment of the disclosure, the above-mentioned construction of negative sample pairs based on sample images corresponding to any two preset images and sample images with different damage information in the same preset image includes S2071-S2073.

[0138] S2071. Construct the first negative sample pair based on the sample images corresponding to any two preset images.

[0139] In this embodiment of the disclosure, sample images corresponding to any two preset images, i.e., two sample images with different content, can be constructed as a first negative sample pair.

[0140] S2072. From the sample images corresponding to the same preset image, select sample images with different damage information to construct a second negative sample pair.

[0141] In this embodiment of the present disclosure, sample images with different damage information can be selected from the sample images corresponding to the same preset image, that is, sample images with the same content but different damage information (image quality) are constructed as a second negative sample pair.

[0142] S2073. Based on the first negative sample pair and the second negative sample pair, construct the negative sample pair.

[0143] In this embodiment of the disclosure, a first negative sample pair with different content and a second negative sample pair with the same content but different damage information (image quality) can be identified as a negative sample pair.

[0144] In this embodiment of the disclosure, a first negative sample pair can be constructed based on two sample images with different content; a second negative sample pair can be constructed based on sample images with the same content but different damage information (image quality); this increases the number and diversity of negative sample pairs, thereby improving the effectiveness of the model.

[0145] In some embodiments, such as Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the process of constructing positive and negative sample pairs.

[0146] in, Figure 3 Figure a shows a preset image including y n and

[0147] Figure 3 Image b shows the damage images corresponding to the two preset images. as well as

[0148] Figure 3 Figure c shows a sample image constructed based on the damage image; where the damage image... The sample image obtained after cropping is and Damage images The sample image obtained after cropping is and Damage images The sample image obtained after cropping is and Damage images The sample image obtained after cropping is and in, To form a positive sample pair, the sample images corresponding to any two preset images, i.e., two sample images with different content, can be determined as the first negative sample pair; for example... and The first negative sample pair is formed; from the sample images corresponding to the same preset image, sample images with different damage information can be selected, that is, sample images with the same content but different damage information (image quality) are determined as the second negative sample pair; for example. and Forming the second negative sample pair and This constitutes the second negative sample pair; it should be noted that the first negative sample pair and the second negative sample pair are not limited to the sample pairs listed in this embodiment.

[0149] In step S209, the positive sample pairs and the negative sample pairs are compared and trained based on the first preset model to obtain the initial feature extraction sub-model.

[0150] In some embodiments, the first preset model includes an object feature extraction network, an image damage feature extraction network, and a feature fusion network. The first preset model may include, but is not limited to, residual networks, convolutional neural networks, etc. Residual networks (ResNet) are diverse and may include, but are not limited to, ResNet-18, ResNet-34, ResNet-50, ResNet-101, ResNet152, etc., but their basic structure is the same—composed of multiple layers of identical residual blocks (ResNet blocks) stacked repeatedly. A ResNet block is characterized by consisting of several network layers (referred to as Layers here), and the output is the sum of the network's output and input. The numbers 18, 34, 50, 101, and 152 represent the number of residual blocks in the network. The object feature extraction network, image damage feature extraction network, and feature fusion network mentioned above correspond to different residual blocks in the residual network. The residual block corresponding to the object feature extraction network is located before the residual block corresponding to the image damage feature extraction network, and the residual block corresponding to the image damage feature extraction network is located before the residual block corresponding to the feature fusion network. Different residual blocks correspond to different processing functions.

[0151] Convolutional Neural Networks (CNNs) are a class of deep feedforward neural networks that incorporate convolutional computations. Structurally, they can be divided into a feature extraction stage and a classification stage. In the feature extraction stage, multiple feature layers are typically stacked, each consisting of convolutional and pooling layers. The feature layers at the front of the network capture local image details, while later feature layers capture more abstract information. In the classification stage, a simple classifier, such as a fully connected network or a support vector machine, typically receives the output of the last feature layer and performs classification and recognition. The aforementioned feature extraction network, image damage feature extraction network, and feature fusion network are each composed of stacked feature layers, including convolutional and pooling layers. The feature layer corresponding to the object feature extraction network precedes the feature layer corresponding to the image damage feature extraction network, and the feature layer corresponding to the image damage feature extraction network precedes the feature layer corresponding to the feature fusion network.

[0152] In one example, the object feature extraction network could be an encoder network. For example... Figure 4 As shown, based on the first preset model, comparative learning training is performed on the above positive sample pairs and the above negative sample pairs to obtain the initial feature extraction sub-model, including S2091-S2097:

[0153] S2091. Input the sample image pairs into the above-mentioned object feature extraction network to perform object feature extraction processing to obtain sample object features; the sample image pairs include the above-mentioned positive sample pairs and the above-mentioned negative sample pairs.

[0154] In this embodiment of the disclosure, sample image pairs can be constructed based on the aforementioned positive sample pairs and negative sample pairs; sample object features of each sample image in the sample image pair are extracted using an object feature extraction network, and the sample object features characterize the content features of the sample image. In one example, the sample object features are the features of the objects in the sample image.

[0155] S2093. The above sample images are input into the above image damage feature extraction network to perform image damage feature extraction processing to obtain sample damage features.

[0156] In this embodiment of the disclosure, the image damage feature extraction network can be an encoder network; the image damage feature extraction network can extract sample damage features of the sample image; the sample damage features can characterize the damage information of the sample image.

[0157] S2095. Input the sample object features and sample damage features corresponding to any sample image in the above sample image pair into the above feature fusion network to obtain the sample image features corresponding to any sample image.

[0158] In some embodiments, such as Figure 5 As shown, the sample object features and sample damage features corresponding to any sample image in the above sample image pair are input into the above feature fusion network to obtain the sample image features corresponding to any of the above sample images, including S20951-S20959:

[0159] S20951. Input the sample object features and sample damage features corresponding to any sample image in the sample image pair into the above feature fusion network.

[0160] In this embodiment of the disclosure, a feature fusion network can be used to fuse the sample object features and sample damage features corresponding to any sample image in a sample image pair.

[0161] S20953. Based on the above feature fusion network, determine the first weight of the sample object feature corresponding to any of the above sample images and the second weight of the sample damage feature corresponding to any of the above sample images.

[0162] In this embodiment of the disclosure, before feature fusion, initial weights can be assigned to the sample object features and sample damage features respectively through a feature fusion network; a first weight for the sample object feature corresponding to any of the above sample images and a second weight for the sample damage feature corresponding to any of the above sample images are determined; wherein, the sum of the first weight and the second weight is 1. In the subsequent model training process, the values ​​of the first weight and the second weight can be continuously adjusted until the training termination condition is met.

[0163] S20955. Based on the product of the sample object features corresponding to any of the above sample images and the above first weight, determine the features of the first sample image.

[0164] In this embodiment of the disclosure, the first sample image feature can be obtained by multiplying the sample object feature corresponding to any sample image with the first weight mentioned above through a feature fusion network.

[0165] S20957. Based on the product of the sample damage features corresponding to any of the above sample images and the above second weight, determine the features of the second sample image.

[0166] In this embodiment of the disclosure, the second sample image features can be obtained by multiplying the sample damage features corresponding to any sample image with the aforementioned second weight using a feature fusion network.

[0167] S20959. Based on the first sample image features and the second sample image features, determine the sample image features corresponding to any of the sample images.

[0168] In this embodiment of the disclosure, the first sample image features and the second sample image features can be fused by a feature fusion network to obtain the sample image features corresponding to any of the above sample images; the sample image features can be vectors of a preset dimension.

[0169] In this embodiment of the disclosure, an initial weight can be assigned to the sample object features and sample damage features respectively by a feature fusion network, and the image features corresponding to the sample object features and sample damage features can be determined according to the weights to obtain the first sample image features and the second sample image features. Then, the first sample image features and the second sample image features are fused to obtain the sample image features corresponding to any sample image. The sample image features fuse the sample object features and the sample damage features, and represent the comprehensive features of the content and image quality of the sample image, thereby improving the representation effectiveness of the sample image features.

[0170] In some embodiments, such as Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the process of constructing positive and negative sample pairs. Figure 3 Figure d shows the sample image features extracted based on the sample images, where the sample images... Corresponding sample image features Sample images Corresponding sample image features Sample image is Corresponding sample image features Sample images Corresponding sample image features Sample images Corresponding sample image features Sample images Corresponding sample image features Sample images Corresponding sample image features Sample images Corresponding sample image features Figure 3 The middle d-image shows the sample image features corresponding to each positive and negative sample pair; the reference box (anchor) is a set or several fixed reference boxes of different scales and aspect ratios, used for image cropping; the negative sample pair includes the first negative sample pair and the second negative sample pair.

[0171] S2097. Based on the sample image features corresponding to the two sample images in the above sample image pair, train the above object feature extraction network, the above image damage feature extraction network, and the above feature fusion network to obtain the above initial feature extraction sub-model.

[0172] In this embodiment of the disclosure, a loss function can be constructed based on the sample image features corresponding to each of the two sample images in the sample image pair, and the network parameters corresponding to the object feature extraction network, the image damage feature extraction network, and the feature fusion network can be adjusted according to the loss function to obtain an initial feature extraction sub-model.

[0173] In an exemplary embodiment, the loss function (optimization objective) L is determined based on the features of the sample image. QC As shown below:

[0174]

[0175] Where K represents the total number of samples for comparison; τ is the temperature coefficient, which controls the distribution shape of the output value; β is the weighting coefficient, which controls the importance of the two optimization objectives (i.e., different content and different image quality); k is the kth sample image in the total sample images; the value of k ranges from 1 to K; i and j correspond to two image damage methods, i and j indicate that different damage methods are used to process the sample images; Sample images The corresponding sample image features. The first * takes a value of 1-K, and the second * takes a value of i or j; The value of * in the value can be either i or j.

[0176] In this embodiment of the disclosure, the sample object features of the sample image can be extracted by an object feature extraction network, the sample damage features of the sample image can be extracted by an image damage feature extraction network, and the sample object features and sample damage features of the same sample image can be fused by a feature fusion network to obtain sample image features. The obtained sample image features fuse both sample object features and sample damage features, so that the image quality can be evaluated from both the image content and image quality aspects.

[0177] In some embodiments, such as Figure 6 As shown, based on the sample image features corresponding to each of the two sample images in the above sample image pair, the above object feature extraction network, the above image damage feature extraction network, and the above feature fusion network are trained to obtain the above initial feature extraction sub-model, including S20971-S20973:

[0178] S20971. Based on the sample image features corresponding to the two sample images in the above sample image pair, adjust the parameters of the above object feature extraction network, the above image damage feature extraction network, and the above feature fusion network until the first training termination condition is met.

[0179] In some embodiments, such as Figure 7 As shown, based on the sample image features corresponding to each of the two sample images in the above sample image pair, the parameters of the above object feature extraction network, the above image damage feature extraction network, and the above feature fusion network are adjusted until the first training termination condition is met, including S209711-S209717:

[0180] S209711. Based on the sample image features corresponding to the two positive sample images in the above positive sample pair, determine the first similarity between the two positive sample images in the above positive sample pair.

[0181] In this embodiment of the disclosure, the similarity of the sample image features corresponding to the two sample images in the positive sample pair can be calculated as the first similarity between the two positive sample images in the positive sample pair; the method for calculating the feature similarity may include, but is not limited to, cosine similarity, Euclidean distance, Manhattan distance and other methods.

[0182] S209713. Based on the sample image features corresponding to the two negative sample images in the above negative sample pair, determine the second similarity between the two negative sample images in the above negative sample pair.

[0183] In this embodiment of the disclosure, the similarity of the sample image features corresponding to the two sample images in the negative sample pair can be calculated as the second similarity between the two negative sample images in the negative sample pair; the method for calculating the feature similarity may include, but is not limited to, cosine similarity, Euclidean distance, Manhattan distance and other methods.

[0184] S209715. Based on the difference between the first similarity and the second similarity, determine the loss information.

[0185] In this embodiment of the disclosure, the difference between the first similarity and the second similarity can be calculated to obtain loss information; then, the model is trained based on the loss information, and the training objective of the model is to make the value of the first similarity larger and the value of the second similarity smaller.

[0186] S209717. Adjust the parameters of the object feature extraction network, the image damage feature extraction network, and the feature fusion network according to the above loss information until the first training termination condition is met.

[0187] In this embodiment of the disclosure, during model training, the network parameters corresponding to the object feature extraction network, the image damage feature extraction network, and the feature fusion network can be continuously adjusted based on the loss information until the first training termination condition is met. The first training termination condition can be set to the loss information being greater than a preset threshold; it can also be set to reaching a preset number of iterations.

[0188] In this embodiment of the disclosure, loss information can be determined based on the first similarity corresponding to positive sample pairs and the second similarity corresponding to negative sample pairs, and comparative learning training can be performed based on the loss information, thereby achieving unsupervised training without label annotation, avoiding dependence on a large amount of labeled data during model training, and reducing the complexity of model training.

[0189] S20973. Based on the object feature extraction network, image damage feature extraction network, and feature fusion network when the first training termination condition is met, the above initial feature extraction sub-model is determined.

[0190] In this embodiment, the first training termination condition can be set to the loss information being greater than a preset threshold; training terminates when the damage information is greater than the preset threshold; and an initial feature extraction sub-model is determined based on the object feature extraction network, image damage feature extraction network, and feature fusion network at the end of training. The object feature extraction network, image damage feature extraction network, and feature fusion network at the end of training can be combined to obtain the initial feature extraction sub-model.

[0191] In this embodiment of the present disclosure, the parameters of the object feature extraction network, the image damage feature extraction network, and the feature fusion network can be adjusted according to the sample image features corresponding to the two sample images in the sample image pair until the first training termination condition is met; thereby realizing the rapid training of the object feature extraction network, the image damage feature extraction network, and the feature fusion network; and then, based on the object feature extraction network, the image damage feature extraction network, and the feature fusion network when the first training termination condition is met, the initial feature extraction sub-model can be quickly determined.

[0192] In step S2011, an image quality evaluation model is determined based on the above-mentioned initial feature extraction sub-model.

[0193] In this embodiment of the disclosure, the initial feature extraction sub-model can be used as a pre-trained model, thereby further training to obtain an image quality evaluation model.

[0194] In the embodiments disclosed herein, such as Figure 8 As shown, based on the above initial feature extraction sub-model, the image quality assessment model is determined, including S20111-S20117:

[0195] S20111. Obtain training images labeled with training impairment information.

[0196] In this embodiment of the disclosure, the training image can be an image from the IQA dataset, and the training image is labeled with training impairment information, which characterizes the image quality of the training image.

[0197] S20113. Based on the above initial feature extraction sub-model, extract the training image features of the training image.

[0198] In this embodiment of the disclosure, the training image can be processed by an initial feature extraction sub-model to obtain training image features. Specifically, the training object features in the training image can be extracted by the object feature extraction network in the initial feature extraction sub-model, and the training damage features in the training image can be extracted by the image damage feature extraction network in the initial feature extraction sub-model. Finally, the training object features and training damage features are fused by the feature fusion network in the initial feature extraction sub-model to obtain the training image features of the training image.

[0199] S20115. Determine the damage information recognition result corresponding to the features of the above-mentioned training images based on the second preset model; the damage information recognition result characterizes the image quality of the above-mentioned training images.

[0200] In this embodiment, the second preset model is used to perform damage information recognition processing on the training image features. The second preset model can predict damage information from the training image features to obtain damage information recognition results. The second preset model is a classification network, which may include, but is not limited to, Support Vector Machines (SVM), Random Forest, etc. Random Forest refers to a classifier that uses multiple trees to train and predict samples. The damage information recognition results include the damage type and damage level of the training image; these results can characterize the image quality of the training image.

[0201] S20117. Based on the difference between the above-mentioned damage information recognition results and the above-mentioned training damage information labels, train the above-mentioned initial feature extraction sub-model and the above-mentioned second preset model to obtain the above-mentioned image quality evaluation model.

[0202] In this embodiment of the disclosure, a loss function can be constructed based on the difference between the damage information recognition result and the training damage information label, and the above-mentioned initial feature extraction sub-model and the above-mentioned second preset model can be trained according to the loss function, thereby determining the combination of the initial feature extraction sub-model and the second preset model after training as the image quality evaluation model.

[0203] In this embodiment of the disclosure, the initial feature extraction sub-model can be fine-tuned by training images, thereby improving the effectiveness of the initial feature extraction sub-model. Based on the initial feature extraction sub-model at the end of training, an image quality evaluation model is constructed, thereby improving the effectiveness of the image quality evaluation model.

[0204] In the embodiments disclosed herein, such as Figure 9 As shown, based on the difference between the above-mentioned damage information recognition results and the above-mentioned training damage information labels, the above-mentioned initial feature extraction sub-model and the above-mentioned second preset model are trained to obtain the above-mentioned image quality evaluation model, including S201171-S201173:

[0205] S201171. Based on the difference between the above-mentioned damage information recognition results and the above-mentioned training damage information labels, train the above-mentioned initial feature extraction sub-model and the above-mentioned second preset model until the second training termination condition is met.

[0206] In this embodiment of the disclosure, training loss information can be determined based on the difference between the damage information identification result and the above-mentioned training damage information label, and the model parameters of the initial feature extraction sub-model and the above-mentioned second preset model can be adjusted according to the training loss information until the second training termination condition is met; the second training termination condition can be set to the training loss information being less than the target threshold, or it can be set to reach a preset number of iterations.

[0207] S201173. Based on the image feature extraction sub-model and the damage information recognition sub-model, determine the above image quality evaluation model; the above image feature extraction sub-model is the initial feature extraction sub-model that meets the second training termination condition, and the above damage information recognition sub-model is the second preset model that meets the second training termination condition.

[0208] In this embodiment of the disclosure, the initial feature extraction sub-model that meets the second training termination condition can be determined as the image feature extraction sub-model, and the second preset model that meets the second training termination condition can be determined as the damage information recognition sub-model; and the image feature extraction sub-model and the damage information recognition sub-model can be combined to obtain the image quality evaluation model.

[0209] In this embodiment of the present disclosure, during the fine-tuning training stage of the initial feature extraction sub-model, the initial feature extraction sub-model and the second preset model can be trained based on the difference between the damage information recognition result and the training damage information label. The initial feature extraction sub-model that meets the second training termination condition is determined as the image feature extraction sub-model, and the second preset model that meets the second training termination condition is determined as the damage information recognition sub-model. Thus, an image quality evaluation model can be quickly obtained based on the image feature extraction sub-model and the damage information recognition sub-model.

[0210] In an exemplary embodiment, the trained image quality assessment model can be tested for performance; Pearson linear correlation coefficient (PLCC) and Spearman rank correlation coefficient (SRCC) are used as evaluation metrics to measure the consistency and accuracy between the model's predicted score and the reviewer's subjective quality mean (MOS). Five commonly used no-reference video quality assessment datasets—BID, CLIVIE, KonIQ10k, SPAQ, and FLIVE—were selected to evaluate the generalization performance of the image quality assessment model obtained in this embodiment. Table 1 describes the performance data of the five datasets under different testing methods. The testing methods include Natural Image Quality Evaluator (NIQE), Integrated Local Natural Image Quality Evaluator (IL NIQE), Blind / Referenceless Image Spatial Quality Evaluator (BRISQUE), Pseudo-Reference Image Quality Evaluator (BMPRI), Convolutional Neural Network Image Quality Assessment (CNNIQA), Deep Image Quality Assessment (WaDIQaM-NR), Stochastic Frontier Approach (SFA), and Deep Bilinear Convolutional Neural Network (DBN). The proposed methods include CNN (DB-CNN), Hyper Image Quality Assessment (HyperIQA), UNIQUE (a subjective quality assessment model considering subjective quality uncertainty), Multi-scale Image Quality Transformer (MUSIQ), and NR-IQA (Relative Ranking and Self-consistency, TreS) based on rank learning. As shown in Table 2, the image quality assessment model (QPT-ResNet50) in this embodiment achieves state-of-the-art (SOTA) performance on all five datasets. Particularly on the smaller BID and CLIVY datasets, compared to the Residual Network (ResNet50) which uses image classification pre-trained weights, the SRCC index is improved by 0.0452 and 0.042 respectively, demonstrating significant improvement.

[0211] Table 1

[0212]

[0213] Table 2

[0214]

[0215]

[0216] In one exemplary embodiment, to further enhance the generalization and effectiveness of the image quality assessment model of this embodiment, the state-of-the-art (SOTA) models HyperIQA and TReS, which also employ ResNet50 as their backbone, were selected, and their initial network weights were replaced with parameters obtained from training the image quality assessment model. As shown in Table 3 below, with other settings remaining consistent, the image quality assessment model of this embodiment significantly outperformed the original models on the BID, CLIIVE, and KonIQ10k datasets.

[0217] Table 3

[0218]

[0219] This disclosure obtains multiple preset images; each preset image corresponds to a different object category; image damage processing is performed on the multiple preset images to obtain multiple sample images; each sample image corresponds to a type of damage information, which characterizes the image quality of the sample image; from the sample images corresponding to the same preset image, sample images with the same damage information are selected to construct positive sample pairs; based on the sample images corresponding to any two preset images, and sample images with different damage information in the sample images corresponding to the same preset image, negative sample pairs are constructed; based on a first preset model, the positive sample pairs and negative sample pairs are trained through comparative learning to obtain an initial feature extraction sub-model; based on the initial feature extraction sub-model, an image quality evaluation model is determined. This disclosure obtains multiple sample images by performing image damage processing on preset images; then constructs positive and negative sample pairs based on the preset images corresponding to the sample images and damage information; thus, an initial feature extraction sub-model can be obtained through unsupervised comparative learning training using positive and negative sample pairs; this avoids dependence on a large amount of labeled data during model training, reducing model training complexity; and comparative learning training can improve the effectiveness of the initial feature extraction sub-model, thereby improving the effectiveness of the image quality evaluation model.

[0220] This disclosure also provides an image quality assessment method, such as... Figure 10 As shown, the method includes:

[0221] In step S1001, the image to be identified is acquired;

[0222] In this embodiment of the disclosure, the image to be identified can be an image from a database or a video.

[0223] In some embodiments, obtaining the image to be identified may include:

[0224] Get preset video;

[0225] In this embodiment of the disclosure, the preset video can be a video requested by the user to be published on the business platform; before the video is published, the quality of the preset video needs to be detected, and the exposure rate of the preset video is determined based on the quality detection results.

[0226] Extract video frames from a preset video as images to be identified.

[0227] In this embodiment of the present disclosure, all video frames or a portion of video frames in a preset video can be extracted as images to be identified; then, quality identification is performed on multiple images to be identified to determine the quality of the preset video.

[0228] In step S1003, the image to be identified is input into the image quality evaluation model for image quality evaluation processing to obtain the target damage information of the image to be identified; the target damage information characterizes the image quality of the image to be identified.

[0229] In this embodiment of the disclosure, the image quality evaluation model includes an image feature extraction sub-model and a damage information recognition sub-model. The image feature extraction sub-model can extract image features from the image to be identified, and the damage information recognition sub-model can then identify the image features to obtain target damage information. The target damage information includes the target damage type and the target damage level. The target damage information can characterize the image quality of the image to be identified.

[0230] In this embodiment of the disclosure, if the image to be identified is an image in a preset video, and there are multiple images to be identified, the comprehensive damage information of the preset video can be determined based on the target damage information corresponding to each of the multiple images to be identified; for example, the average value of the target damage information corresponding to each of the multiple images to be identified can be calculated as the comprehensive damage information of the preset video; thereby, the image quality of the preset video can be determined.

[0231] In this embodiment of the disclosure, the method further includes:

[0232] Based on the image quality corresponding to the preset video, the exposure rate of the preset video on the business platform is determined.

[0233] In this embodiment of the disclosure, before the video is published, the quality of the preset video needs to be detected, and the exposure rate of the preset video is determined based on the quality detection results. A higher exposure rate can be set for videos with poor image quality, and a lower exposure rate can be set for videos with good image quality. This enables the recommendation of images with better image quality to other users in the business platform, thereby improving the user's subjective experience (QoE) of the business platform.

[0234] In the embodiments of this disclosure, the target damage information of the image to be identified can be quickly and accurately identified through the image quality evaluation model, thereby quickly and accurately obtaining the image quality of the image to be identified.

[0235] Figure 11 This is a block diagram of a training apparatus for an image quality assessment model according to an exemplary embodiment. (Refer to...) Figure 11 The device includes:

[0236] The preset image acquisition module 1110 is configured to acquire multiple preset images; each of the multiple preset images corresponds to a different object category.

[0237] The sample image determination module 1120 is configured to perform image damage processing on the above-mentioned multiple preset images to obtain multiple sample images; each sample image corresponds to a damage information, and the damage information characterizes the image quality of the sample image.

[0238] The positive sample construction module 1130 is configured to select sample images with the same damage information from sample images corresponding to the same preset image and construct positive sample pairs;

[0239] The negative sample construction module 1140 is configured to construct negative sample pairs based on sample images corresponding to any two preset images and sample images with different damage information in the sample images corresponding to the same preset image.

[0240] The contrast training module 1150 is configured to perform contrast learning training on the positive sample pairs and the negative sample pairs based on the first preset model to obtain an initial feature extraction sub-model.

[0241] The model determination module 1160 is configured to execute the determination of the image quality evaluation model based on the above-mentioned initial feature extraction sub-model.

[0242] In one exemplary embodiment, the first preset model includes an object feature extraction network, an image damage feature extraction network, and a feature fusion network, and the comparison training module includes:

[0243] The sample object feature determination submodule is configured to perform object feature extraction processing on the sample image pairs input to the above-mentioned object feature extraction network to obtain sample object features; the above-mentioned sample image pairs include the above-mentioned positive sample pairs and the above-mentioned negative sample pairs.

[0244] The sample damage feature determination submodule is configured to perform image damage feature extraction processing on the above sample image input to the above image damage feature extraction network to obtain sample damage features;

[0245] The feature fusion submodule is configured to input the sample object features and sample damage features corresponding to any sample image in the above sample image pair into the feature fusion network to obtain the sample image features corresponding to any sample image.

[0246] The training submodule is configured to train the object feature extraction network, the image damage feature extraction network, and the feature fusion network based on the sample image features corresponding to the two sample images in the above sample image pair, so as to obtain the above initial feature extraction submodel.

[0247] In one exemplary implementation, the feature fusion submodule includes:

[0248] The feature input unit is configured to input the sample object features and sample damage features corresponding to any sample image in the above sample image pair into the above feature fusion network;

[0249] The weight determination unit is configured to perform, based on the feature fusion network described above, determine the first weight of the sample object feature corresponding to any of the sample images and the second weight of the sample damage feature corresponding to any of the sample images.

[0250] The first feature determination unit is configured to perform a product of the sample object features corresponding to any of the above sample images and the above first weight to determine the features of the first sample image;

[0251] The second feature determination unit is configured to perform a product of the sample damage features corresponding to any of the above sample images and the above second weight to determine the features of the second sample image.

[0252] The sample image feature determination unit is configured to determine the sample image features corresponding to any of the sample images based on the first sample image features and the second sample image features.

[0253] In one exemplary implementation, the training submodule includes:

[0254] The first training unit is configured to perform adjustments to the parameters of the object feature extraction network, the image damage feature extraction network, and the feature fusion network based on the sample image features corresponding to the two sample images in the above sample image pair, until the first training termination condition is met.

[0255] The model determination unit is configured to execute the object feature extraction network, image damage feature extraction network, and feature fusion network based on the first training termination condition to determine the above initial feature extraction sub-model.

[0256] In one exemplary implementation, the first training unit includes:

[0257] The first similarity determination subunit is configured to perform the task of determining the first similarity between the two positive sample images in the positive sample pair based on the sample image features corresponding to the two positive sample images in the positive sample pair.

[0258] The second similarity determination subunit is configured to determine the second similarity between the two negative sample images in the negative sample pair based on the sample image features corresponding to the two negative sample images in the negative sample pair.

[0259] The loss information determination subunit is configured to determine loss information based on the difference between the first similarity and the second similarity.

[0260] The parameter adjustment subunit is configured to adjust the parameters of the object feature extraction network, the image damage feature extraction network, and the feature fusion network according to the loss information, until the first training termination condition is met.

[0261] In one exemplary embodiment, the model determination module includes:

[0262] The training image acquisition unit is configured to acquire training images labeled with training impairment information.

[0263] The training feature extraction unit is configured to extract training image features from the training image based on the initial feature extraction sub-model described above;

[0264] The recognition result determination unit is configured to perform damage information recognition results based on a second preset model to determine the damage information recognition results corresponding to the features of the training images; the damage information recognition results characterize the image quality of the training images.

[0265] The second training unit is configured to train the initial feature extraction sub-model and the second preset model based on the difference between the damage information recognition result and the training damage information label, so as to obtain the image quality evaluation model.

[0266] In one exemplary embodiment, the second training unit includes:

[0267] The training subunit is configured to train the initial feature extraction submodel and the second preset model based on the difference between the damage information recognition result and the training damage information label, until the second training termination condition is met.

[0268] The model determination sub-unit is configured to execute an image feature extraction sub-model and a damage information recognition sub-model to determine the image quality evaluation model; the image feature extraction sub-model is the initial feature extraction sub-model that meets the second training termination condition, and the damage information recognition sub-model is the second preset model that meets the second training termination condition.

[0269] In one exemplary implementation, the negative sample construction module described above includes:

[0270] The first construction submodule is configured to construct a first negative sample pair based on the sample images corresponding to any two preset images.

[0271] The second construction submodule is configured to perform the task of selecting sample images with different damage information from the sample images corresponding to the same preset image, and constructing a second negative sample pair.

[0272] The negative sample pair construction submodule is configured to construct the aforementioned negative sample pair based on the aforementioned first negative sample pair and the aforementioned second negative sample pair.

[0273] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0274] Figure 12 This is a block diagram illustrating an image quality evaluation device according to an exemplary embodiment. (Refer to...) Figure 12 The device includes:

[0275] The image acquisition module 1210 is configured to acquire the image to be recognized.

[0276] The target damage information determination module 1220 is configured to perform image quality evaluation processing by inputting the image to be identified into an image quality evaluation model to obtain target damage information of the image to be identified; the target damage information characterizes the image quality of the image to be identified.

[0277] The image quality assessment model described above was trained using the method described above.

[0278] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0279] In one exemplary embodiment, an electronic device is also provided, including a processor; a memory for storing processor-executable instructions; wherein, when the processor is configured to execute the instructions stored in the memory, it implements the training method or image quality evaluation method of the image quality evaluation model provided in any of the above embodiments.

[0280] The electronic device can be a terminal, a server, or a similar computing device. Taking a server as an example... Figure 13 This is a block diagram illustrating an electronic device according to an exemplary embodiment, such as... Figure 13 As shown, the server 1300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1310 (CPUs 1310 may include, but are not limited to, microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 1330 for storing data, and one or more storage media 1320 (e.g., one or more mass storage devices) for storing application programs 1323 or data 1322. The memory 1330 and storage media 1320 may be temporary or persistent storage. The program stored in the storage media 1320 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 1310 may be configured to communicate with the storage media 1320 and execute the series of instruction operations stored in the storage media 1320 on the server 1300. Server 1300 may also include one or more power supplies 1360, one or more wired or wireless network interfaces 1350, one or more input / output interfaces 1340, and / or one or more operating systems 1321, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0281] The input / output interface 1340 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 1300. In one example, the input / output interface 1340 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 1340 may be a radio frequency (RF) module for wireless communication with the Internet.

[0282] Those skilled in the art will understand that Figure 13 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 1300 may also include... Figure 13 The more or fewer components shown, or having the same Figure 13 The different configurations shown.

[0283] In one exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1330 including instructions, which can be executed by a processor 1310 of the device 1300 to perform the above-described method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0284] In one exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the training method or image quality evaluation method of the image quality evaluation model provided in any of the above embodiments.

[0285] This disclosure obtains multiple preset images; each preset image corresponds to a different object category; image damage processing is performed on the multiple preset images to obtain multiple sample images; each sample image corresponds to a type of damage information, which characterizes the image quality of the sample image; from the sample images corresponding to the same preset image, sample images with the same damage information are selected to construct positive sample pairs; based on the sample images corresponding to any two preset images, and sample images with different damage information in the sample images corresponding to the same preset image, negative sample pairs are constructed; based on a first preset model, the positive sample pairs and negative sample pairs are trained through comparative learning to obtain an initial feature extraction sub-model; based on the initial feature extraction sub-model, an image quality evaluation model is determined. This disclosure obtains multiple sample images by performing image damage processing on preset images; then constructs positive and negative sample pairs based on the preset images corresponding to the sample images and damage information; thus, an initial feature extraction sub-model can be obtained through unsupervised comparative learning training using positive and negative sample pairs; this avoids dependence on a large amount of labeled data during model training, reducing model training complexity; and comparative learning training can improve the effectiveness of the initial feature extraction sub-model, thereby improving the effectiveness of the image quality evaluation model.

[0286] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0287] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0288] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for an image quality assessment model, characterized in that, include: Acquire multiple preset images; each of the multiple preset images corresponds to a different object category; Image impairment processing is performed on the multiple preset images to obtain multiple sample images; Each sample image corresponds to a type of damage information, which characterizes the image quality of the sample image. From the sample images corresponding to the same preset image, sample images with the same damage information are selected to construct positive sample pairs; Based on the sample images corresponding to any two preset images, and the sample images with different damage information in the sample images corresponding to the same preset image, construct negative sample pairs; The sample image is used to extract object features from the input object feature extraction network to obtain sample object features; the sample image pair includes the positive sample pair and the negative sample pair. The sample image is used to extract image damage features from the input image damage feature extraction network to obtain sample damage features that characterize the damage information; the damage information includes damage type and damage level. Input the sample object features and sample damage features corresponding to any sample image in the sample image pair into the feature fusion network to obtain the sample image features corresponding to any sample image. Based on the sample image features corresponding to each of the two sample images in the sample image pair, the object feature extraction network, the image damage feature extraction network, and the feature fusion network are trained to obtain an initial feature extraction sub-model; Based on the initial feature extraction sub-model, an image quality evaluation model is determined.

2. The method according to claim 1, characterized in that, The step of inputting the sample object features and sample damage features corresponding to any sample image in the sample image pair into a feature fusion network to obtain the sample image features corresponding to any sample image includes: The sample object features and sample damage features corresponding to any sample image in the sample image pair are input into the feature fusion network; Based on the feature fusion network, a first weight of the sample object feature corresponding to any sample image and a second weight of the sample damage feature corresponding to any sample image are determined. The first sample image features are determined based on the product of the sample object features corresponding to any sample image and the first weight. The second sample image features are determined based on the product of the sample damage features corresponding to any sample image and the second weight; Based on the first sample image features and the second sample image features, determine the sample image features corresponding to any one of the sample images.

3. The method according to claim 1, characterized in that, The initial feature extraction sub-model is obtained by training the object feature extraction network, the image damage feature extraction network, and the feature fusion network based on the sample image features corresponding to the two sample images in the sample image pair, including: Based on the sample image features corresponding to each of the two sample images in the sample image pair, adjust the parameters of the object feature extraction network, the image damage feature extraction network and the feature fusion network until the first training termination condition is met. The initial feature extraction sub-model is determined based on the object feature extraction network, image damage feature extraction network, and feature fusion network when the first training termination condition is met.

4. The method according to claim 3, characterized in that, The step of adjusting the parameters of the object feature extraction network, the image damage feature extraction network, and the feature fusion network based on the sample image features corresponding to the two sample images in the sample image pair until the first training termination condition is met includes: Based on the sample image features corresponding to each of the two positive sample images in the positive sample pair, the first similarity between the two positive sample images in the positive sample pair is determined; Based on the sample image features corresponding to each of the two negative sample images in the negative sample pair, a second similarity between the two negative sample images in the negative sample pair is determined; Based on the difference between the first similarity and the second similarity, loss information is determined; The parameters of the object feature extraction network, the image damage feature extraction network, and the feature fusion network are adjusted according to the loss information until the first training termination condition is met.

5. The method according to claim 1, characterized in that, The step of determining the image quality assessment model based on the initial feature extraction sub-model includes: Obtain training images labeled with training impairment information; The training image features of the training image are extracted based on the initial feature extraction sub-model; The damage information recognition result corresponding to the features of the training image is determined based on the second preset model; the damage information recognition result characterizes the image quality of the training image. Based on the difference between the damage information identification result and the training damage information label, the initial feature extraction sub-model and the second preset model are trained to obtain the image quality evaluation model.

6. The method according to claim 5, characterized in that, The step of training the initial feature extraction sub-model and the second preset model based on the difference between the damage information identification result and the trained damage information label to obtain the image quality evaluation model includes: Based on the difference between the damage information identification result and the training damage information label, train the initial feature extraction sub-model and the second preset model until the second training termination condition is met; The image quality evaluation model is determined based on the image feature extraction sub-model and the damage information recognition sub-model; the image feature extraction sub-model is the initial feature extraction sub-model that meets the second training termination condition, and the damage information recognition sub-model is the second preset model that meets the second training termination condition.

7. The method according to claim 1, characterized in that, The construction of negative sample pairs based on sample images corresponding to any two preset images, and sample images with different damage information in the same preset image, includes: Construct the first negative sample pair based on the sample images corresponding to any two preset images; From the sample images corresponding to the same preset image, sample images with different damage information are selected to construct a second negative sample pair; The negative sample pair is constructed based on the first negative sample pair and the second negative sample pair.

8. An image quality assessment method, characterized in that, The method includes: Acquire the image to be recognized; The image to be identified is input into an image quality assessment model for image quality assessment processing to obtain target damage information of the image to be identified; the target damage information characterizes the image quality of the image to be identified. The image quality evaluation model is trained based on the method described in any one of claims 1-7.

9. A training device for an image quality assessment model, characterized in that, include: The preset image acquisition module is configured to acquire multiple preset images; The multiple preset images each correspond to a different object category; The sample image determination module is configured to perform image damage processing on the plurality of preset images to obtain a plurality of sample images; each sample image corresponds to a damage information, and the damage information characterizes the image quality of the sample image. The positive sample construction module is configured to select sample images with the same damage information from sample images corresponding to the same preset image and construct positive sample pairs; The negative sample construction module is configured to construct negative sample pairs based on sample images corresponding to any two preset images, and sample images with different damage information in the sample images corresponding to the same preset image. The contrast training module is configured to perform contrast learning training on the positive sample pairs and the negative sample pairs based on a first preset model to obtain an initial feature extraction sub-model. The first preset model includes an object feature extraction network, an image damage feature extraction network, and a feature fusion network. The comparison training module includes: a sample object feature determination submodule, configured to perform object feature extraction processing on the object feature extraction network by inputting a pair of sample images to obtain sample object features; the sample image pair includes the positive sample pair and the negative sample pair; a sample damage feature determination submodule, configured to perform image damage feature extraction processing on the image damage feature extraction network by inputting the pair of sample images to obtain sample damage features characterizing the damage information; the damage information includes damage type and damage level; a feature fusion submodule, configured to input the sample object features and sample damage features corresponding to any sample image in the sample image pair into the feature fusion network to obtain sample image features corresponding to any sample image; and a training submodule, configured to train the object feature extraction network, the image damage feature extraction network, and the feature fusion network based on the sample image features corresponding to each of the two sample images in the sample image pair to obtain the initial feature extraction submodel. The model determination module is configured to execute an image quality evaluation model based on the initial feature extraction sub-model.

10. An image quality evaluation device, characterized in that, The device includes: The image acquisition module is configured to acquire the image to be recognized. The target damage information determination module is configured to perform image quality evaluation processing by inputting the image to be identified into an image quality evaluation model to obtain target damage information of the image to be identified; the target damage information characterizes the image quality of the image to be identified. The image quality evaluation model is trained based on the method described in any one of claims 1-7.

11. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the model training method as described in any one of claims 1-7 or the image quality evaluation method as described in claim 8.

12. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of an electronic device, the electronic device is able to perform the model training method as described in any one of claims 1-7 or the image quality evaluation method as described in claim 8.

Images