No-reference image quality evaluation method and device based on continuous learning

Abstract

The invention discloses a non-reference image quality evaluation method and device based on continuous learning, and the method comprises the steps: obtaining a first image sample set which comprises at least one first image pair; the second image sample set is updated according to the first image sample set to obtain a third image sample set, the second image sample set is pre-stored in a preset storage unit, the second image sample set comprises at least one second image pair, and the third image sample set comprises at least one first image pair and at least one second image pair; according to the first image sample set, the third image sample set and a preset loss function, the first model is trained to obtain an image quality determination model, the second image sample set is used for training the first model and determining the image quality of the to-be-evaluated image, the evaluation precision of cross-data-set continuous learning can be improved, the model parameter quantity can be reduced, and the evaluation efficiency is improved. And the model training efficiency is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a method and apparatus for evaluating the quality of no-reference images based on continuous learning. Background Technology

[0002] Images facilitate information transmission, and image quality has a crucial impact on the effectiveness of information transmission. Image quality determination methods, also known as image quality evaluation methods, can be divided into subjective image quality evaluation methods and objective image quality evaluation methods.

[0003] Objective image quality assessment methods can be categorized into full-reference image quality assessment methods, half-reference image quality assessment methods, and no-reference image quality assessment methods based on their dependence on reference images. The goal of no-reference image quality assessment is to accurately predict image quality by analyzing the image's own features and content.

[0004] Existing no-reference image quality assessment methods lack sufficient accuracy for older datasets during continuous learning. In 2022, Elahe Arani et al. published a paper titled "LearningFast,LearningSlow:A General Continual Learning Method based on ComplementaryLearning System" on arXiv preprint arXiv:2201.12604. This method models the brain's learning and memory mechanisms, integrating the differences between new and old datasets into no-reference image quality assessment based on fast and slow learning mechanisms. However, due to the significant differences between different real-world image datasets, the model trained by this method cannot effectively accommodate different datasets, resulting in insufficient prediction accuracy. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a method and apparatus for referenceless image quality assessment based on continuous learning. The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] In a first aspect, this invention provides a no-reference image quality assessment method based on continuous learning, applied to electronic devices, the method comprising:

[0007] Obtain a first image sample set, which includes at least one first image pair;

[0008] The second image sample set is updated according to the first image sample set to obtain the third image sample set. The third image sample set is pre-stored in a preset storage unit. The second image sample set includes at least one second image pair, and the third image sample set contains at least one first image pair and at least one second image pair.

[0009] The first model is trained based on the first image sample set, the third image sample set, and the preset loss function to obtain an image quality determination model, which is used to evaluate the image quality of the image to be evaluated and obtain the evaluation result.

[0010] Optionally, the first model includes a quality assessment model, a plasticity model, and a stability model.

[0011] Based on the first image sample set, the third image sample set, and a preset loss function, the first model is trained to obtain an image quality determination model, which is used to evaluate the image quality of the image to be evaluated, and the evaluation results are obtained, including:

[0012] The quality assessment model is trained based on the first image sample set and the preset loss function to determine the parameters of the quality assessment model.

[0013] Update the parameters of the plastic model according to the first update frequency corresponding to the plastic model and based on the third image sample set;

[0014] The parameters of the stable model are updated according to the second update frequency corresponding to the stable model and based on the third image sample set.

[0015] Based on the parameters of the obtained quality evaluation model, the parameters of the plasticity model, and the parameters of the stability model, an image quality determination model is constructed to evaluate the image quality of the image to be evaluated and obtain the evaluation results.

[0016] Optionally, it includes:

[0017] The first update frequency corresponding to the plastic model is greater than the second update frequency corresponding to the preset stable model.

[0018] Optionally, an image quality determination model is constructed based on the parameters of the obtained quality assessment model, the plasticity model, and the stability model, including:

[0019] Based on the obtained parameters of the quality evaluation model and the first decay probability corresponding to the plastic model and the second decay probability corresponding to the stable model, the parameters of the plastic model and the stable model are processed by exponential moving average to obtain the updated parameters of the plastic model and the updated parameters of the stable model.

[0020] An image quality determination model is constructed based on the parameters of the obtained quality assessment model, the updated plasticity model, and the updated stability model.

[0021] Optionally, the first decay probability corresponding to the plastic model is less than or equal to the second decay frequency corresponding to the preset stable model.

[0022] Optionally, the quality assessment model is trained based on the first image sample set and a preset loss function to determine the parameters of the quality assessment model, including:

[0023] Step 1: Input the target first image pair into the quality assessment model, train it, and obtain the prediction result. The target first image pair is the t-th first image pair in the first image sample set.

[0024] Step 2: Based on the prediction results and the preset loss function, perform backpropagation to update the parameters of the quality assessment model, obtaining the updated quality assessment model. The loss function is expressed as:

[0025]

[0026] Where λ represents the weighting factor, θ represents the parameters of the multi-scale feature extraction module in the quality assessment model, and φ t The parameter x represents the quality prediction module. t ,v t Let each represent the input image for the t-th task. This represents the image pairs in the third image sample set. Represents image pairs (x t ,v t The image sample set to which ) belongs Representing image pairs The image sample set to which it belongs This can be a subset or the entirety of the third image sample set.

[0027] Step 3: Take the (t+1)th first image pair in the first image sample set as the target image pair, and repeat steps 1 and 2 until all first image pairs have been trained. Determine the parameters of the quality evaluation model based on the training results.

[0028] Secondly, the present invention also provides a referenceless image quality assessment method based on continuous learning, applied to electronic devices, the method comprising:

[0029] Obtain the image to be evaluated;

[0030] The image to be evaluated is input into the image quality determination model for processing to obtain the image quality evaluation result, wherein the image quality determination model is generated by performing any of the methods in the first aspect.

[0031] Thirdly, the present invention provides a referenceless image quality assessment device based on continuous learning, the device comprising:

[0032] An acquisition module is used to acquire a first image sample set, which includes at least one first image pair.

[0033] The processing module is used to update the second image sample set according to the first image sample set to obtain the third image sample set. The second image sample set is pre-stored in a preset storage unit. The second image sample set includes at least one second image pair. The third image sample set includes at least one first image pair and at least one second image pair.

[0034] The processing module is also used to train the first model based on the first image sample set, the third image sample set and the preset loss function to obtain an image quality determination model, so as to evaluate the image quality of the image to be evaluated and obtain the evaluation result.

[0035] Optionally, the first model includes a quality assessment model, a malleable model, and a stable model. The processing module is specifically used to train the quality assessment model based on a first image sample set and a preset loss function to determine the parameters of the quality assessment model; update the parameters of the malleable model based on a third image sample set according to a first update frequency corresponding to the malleable model; update the parameters of the stable model based on a third image sample set according to a second update frequency corresponding to the stable model; and construct an image quality determination model based on the obtained parameters of the quality assessment model, the parameters of the malleable model, and the parameters of the stable model.

[0036] Optionally, the first update frequency for the plastic model is greater than the second update frequency for the stable model.

[0037] Optionally, the processing module is specifically used to perform exponential moving average processing on the parameters of the plastic model and the parameters of the stable model according to the obtained parameters of the quality evaluation model and the first decay probability corresponding to the plastic model and the second decay probability corresponding to the stable model, respectively, to obtain the updated parameters of the plastic model and the updated parameters of the stable model; and to construct an image quality determination model based on the obtained parameters of the quality evaluation model, the updated parameters of the plastic model and the updated parameters of the stable model.

[0038] Optionally, the first decay probability corresponding to the plastic model is less than or equal to the second decay frequency corresponding to the stable model.

[0039] Optional, processing module, specifically used for

[0040] Step 1: Input the target first image pair into the quality assessment model, train it, and obtain the prediction result. The target first image pair is the t-th first image pair in the first image sample set.

[0041] Step 2: Based on the prediction results and the preset loss function, perform backpropagation to update the parameters of the quality assessment model, obtaining the updated quality assessment model. The loss function is expressed as:

[0042]

[0043] Where λ represents the weighting factor, θ represents the parameters of the multi-scale feature extraction module in the quality assessment model, and φ t The parameter x represents the quality prediction module. t ,v t Let each represent the input image for the t-th task. This represents the image pairs in the third image sample set. Represents image pairs (x t ,v t The image sample set to which ) belongs Representing image pairs The image sample set to which it belongs This can be a subset or the entirety of the third image sample set.

[0044] Step 3: Take the (t+1)th first image pair in the first image sample set as the target image pair, and repeat steps 1 and 2 until all first image pairs have been trained. Determine the parameters of the quality evaluation model based on the training results.

[0045] Fourthly, the present invention also provides a referenceless image quality assessment device based on continuous learning, the device comprising:

[0046] The acquisition module is used to acquire the image to be evaluated.

[0047] The processing module is used to input the image to be evaluated into the image quality determination model for processing, and obtain the image quality evaluation result.

[0048] The image quality determination model is generated by performing the methods provided in Example 1 or Example 2.

[0049] Fifthly, the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform as in the first aspect and the alternative methods of the first aspect, or the method provided in the second aspect.

[0050] In a sixth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method provided in the first aspect or an alternative to the first aspect, or the method provided in the second aspect.

[0051] In a seventh aspect, this disclosure provides a computer program product including a computer program / instructions that, when executed by a processor, implement as in the first aspect or an alternative to the first aspect, or the method provided in the second aspect.

[0052] The beneficial effects of this invention are:

[0053] This model effectively accommodates different datasets, achieving better continuous learning on both new and old datasets with significant differences in distortion types or content scenarios, thus improving the evaluation accuracy of cross-dataset continuous learning. Furthermore, by setting up a data cache to simulate a memory mechanism, the number of model parameters is effectively reduced, improving model training efficiency.

[0054] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0055] Figure 1 A flowchart illustrating a referenceless image quality assessment method based on continuous learning, provided in an embodiment of the present invention;

[0056] Figure 2 A flowchart illustrating another referenceless image quality assessment method based on continuous learning provided in an embodiment of the present invention;

[0057] Figure 3 A schematic diagram of the structure of a first model provided in an embodiment of the present invention;

[0058] Figure 4 A flowchart illustrating the process of determining parameters for a quality evaluation model, provided as an embodiment of the present invention;

[0059] Figure 5 A schematic diagram illustrating the training process of an image quality determination model provided in an embodiment of the present invention;

[0060] Figure 6 This is a schematic diagram of the structure of a quality evaluation model provided in an embodiment of the present invention;

[0061] Figure 7 A flowchart illustrating another referenceless image quality assessment method based on continuous learning provided in an embodiment of the present invention;

[0062] Figure 8 A schematic diagram of a referenceless image quality assessment device based on continuous learning provided in an embodiment of the present invention;

[0063] Figure 9 A schematic diagram of another referenceless image quality assessment device based on continuous learning provided in an embodiment of the present invention;

[0064] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0065] The accompanying drawings have illustrated specific embodiments of the invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0066] Images help convey information, record memories, inspire creativity, and evoke emotional resonance. However, the effectiveness of an image is often limited by its quality. A poor-quality image, such as one with blurry pixels, distorted colors, or unclear details, may fail to accurately convey information or produce the desired effect. For example, in the medical field, a low-quality medical image may lead to misdiagnosis or miss important details. Similarly, in the advertising and media industries, low-quality images may fail to capture the audience's interest. Image quality assessment methods can guide the processing and enhancement of image quality.

[0067] Image quality assessment methods can be categorized into subjective and objective image quality assessment. Objective image quality assessment involves establishing mathematical models to objectively score images. By simulating the human visual system's perception of images, an objective evaluation model is designed to reflect the subjective quality of the image, thereby evaluating the quality of the image under test. With the development of machine learning and deep learning, objective image quality assessment methods are constantly evolving and improving. By utilizing large-scale image databases and deep neural networks, more accurate and refined objective evaluation models can be constructed, further improving the efficiency and reliability of image quality assessment.

[0068] Objective image quality assessment methods can be categorized into full-reference image quality assessment methods, half-reference image quality assessment methods, and referenceless image quality assessment methods based on their dependence on reference images. Among these, referenceless image quality assessment methods, due to their elimination of the need for reference images, have become a current research hotspot and focus. In practical applications, we often face situations where the original reference image is unavailable, thus requiring a method to accurately assess image quality without a reference image. This situation is particularly common in image transmission, storage, and processing. Therefore, referenceless image quality assessment has become a key research focus in the field of image quality assessment. The goal of referenceless image quality assessment is to accurately predict image quality by analyzing the image's own features and content. This requires extracting features related to human subjective perception from the image and establishing a mapping relationship between these features and the subjective evaluation results. This method does not rely on reference images and can perform accurate assessments in the absence of reference information, possessing significant practical application value. Research on referenceless image quality assessment involves the intersection of multiple fields such as computer vision, image processing, and machine learning, requiring full consideration of factors such as image structure, content, distortion type, and the perceptual mechanisms of the human visual system. Through in-depth research and exploration, no-reference image quality assessment methods have made continuous breakthroughs and progress, providing new approaches and solutions for solving image quality assessment problems in practical applications.

[0069] Existing no-reference image quality assessment methods, after being trained on old datasets, experience significant changes in model weights when new datasets are added for training. This causes a rapid decline in model performance on the old datasets, leading to a catastrophic forgetting problem. Secondly, image quality assessment methods, when faced with the need for continuous learning, have numerous model parameters, resulting in high memory consumption and low training efficiency.

[0070] To address the aforementioned problems, this invention provides a method, apparatus, and device for evaluating the quality of referenceless images based on continuous learning. The invention will be further described in detail below with reference to specific embodiments, but the implementation of this invention is not limited thereto.

[0071] Example 1

[0072] Figure 1 This is a flowchart illustrating a referenceless image quality assessment method based on continuous learning, as provided in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0073] S101. Obtain the first image sample set.

[0074] The first image sample set includes at least one first image pair.

[0075] The image information in the first image pair is determined according to the same standard, that is, it follows a common quality perception scale. For example, the image quality score corresponding to any image in the first image pair is obtained according to the same scoring standard.

[0076] Optionally, obtaining the first image sample set includes: obtaining multiple pre-selected images and image quality labels corresponding to each pre-selected image; and determining the first image sample set from each pre-selected image dataset based on the image quality labels corresponding to each pre-selected image.

[0077] For example, multiple pre-selected images and image quality labels corresponding to each pre-selected image are obtained; when the evaluation criteria of the image quality labels corresponding to the multiple images are not the same, for any image corresponding to any evaluation criterion, two images are randomly selected to form a first image pair; when the evaluation criteria of the image quality labels corresponding to the multiple images are the same, two images are randomly selected to form a first image pair.

[0078] Optionally, the multiple pre-selected images can be combined into multiple image pairs, and these multiple image pairs can be divided into a training sample set and a validation sample set according to a preset ratio. The training sample set, i.e., the first image sample set, is used to train the image quality determination model, and the validation sample set is used to validate the trained model after training is completed.

[0079] This method ensures that any training sample in the first training sample corresponds to the same quality evaluation standard, thus guaranteeing the effectiveness of the training samples, improving training efficiency, and reducing resource waste.

[0080] S102. Update the second image sample set according to the first image sample set to obtain the third image sample set.

[0081] The second image sample set is pre-stored in a preset storage unit; the second image sample set includes at least one second image pair.

[0082] The image information in the second image pair is determined according to the same standard. For example, the image quality score corresponding to any image in the second image pair is obtained according to the same scoring standard.

[0083] Optionally, the capacity of the preset storage unit can be a fixed value or a variable value, and the size of the capacity can be determined according to the storage capacity of the corresponding electronic device.

[0084] This method ensures the normal operation of electronic devices while maintaining a sufficient number of samples, thereby improving the accuracy of model training.

[0085] The third image sample set contains at least one first image pair and at least one second image pair.

[0086] For example, the first image sample set is {A, B}, the second image sample set is {C, D}, and the third image sample set is {A, C}, where A, B, C, and D represent different image pairs.

[0087] Optionally, when the second image sample set includes multiple subsets, an equal number of image pairs can be extracted from the multiple subsets of the first and second image sample sets to form the third image sample set.

[0088] The model training sample set constructed by this method includes not only historical training samples used to train the image quality model, but also new training samples, taking into account both old and new data. By utilizing relative quality information to learn the perceptual scale of all tasks, that is, by using image pairs formed by memory images from the same old task in a preset storage unit for data playback, the update of the quality assessment model is constrained. This helps to overcome the problem that existing image quality determination methods experience a rapid decline in performance on previous datasets when new datasets are added for training after training on old datasets during continuous learning.

[0089] S103. Based on the first image sample set, the third image sample set, and the preset loss function, the first model is trained to obtain an image quality determination model, which is used to evaluate the image quality of the image to be evaluated and obtain the evaluation result.

[0090] Among them, the image quality determination model is used to determine the image quality of the image to be evaluated.

[0091] The second set of image samples was used to train the first model.

[0092] That is, the samples in the second image sample set are historical training samples, i.e. old data. It should be noted that the data in the second image sample set is not equal to all the training data corresponding to the first model.

[0093] The image quality assessment method provided by this invention obtains a first image sample set including at least one first image pair, updates a second image sample set pre-stored in a preset storage unit for training a first model based on the first image sample set, and obtains a third image sample set including at least one first image pair and at least one second image pair. Then, the first model is trained based on the first image sample set, the third image sample set, and a preset loss function to obtain an image quality determination model for determining the image quality of the image to be evaluated. This method simulates the brain's memory mechanism by setting a data cache to cache old model training sample data, effectively reducing model parameters. At the same time, by combining new and old training sample data to train the model, when the image quality determination model is trained on the old dataset and a new dataset is added for training, the degree of change in model weights is reduced, the performance degradation of the model on the previous dataset is reduced, and the catastrophic forgetting problem is avoided.

[0094] Example 2

[0095] Figure 2 This is a flowchart illustrating another referenceless image quality assessment method based on continuous learning provided in an embodiment of the present invention. Figure 3 A schematic diagram of the structure of a first model provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the first model includes a quality assessment model, a malleable model, and a stable model. For example, the quality assessment model consists of a multi-scale feature extraction module, a multi-scale feature fusion module, and a quality prediction module. The network structures of the malleable and stable models are consistent with those of the quality assessment model, but the parameters set during training are different. The malleable and stable models form a complementary learning system. Specifically, the multi-scale feature extraction module performs multiple convolutions on the image using a ResNet50 residual network. Convolution is the process of converting image information from the spatial domain to the frequency domain; deeper convolutions extract more specific information. The tensor obtained after convolution is called a feature. Multi-scale features refer to the sum of deep and shallow features after convolutions at different depths. The feature fusion module combines deep and shallow features into a single tensor. The quality regression module regresses the feature data to a scalar (i.e., a quality score) through fully connected layers.

[0096] like Figure 2 As shown, the method includes:

[0097] S201. Obtain the first image sample set.

[0098] The first image sample set includes at least one first image pair.

[0099] S202. Update the second image sample set according to the first image sample set to obtain the third image sample set.

[0100] S201 and S202 have the same or corresponding technical features as S101 and S102. For a detailed description, please refer to S101 and S102. They will not be repeated here.

[0101] S203. Based on the first image sample set and the preset loss function, train the quality assessment model to determine the parameters of the quality assessment model.

[0102] Quality assessment models are used to predict image quality.

[0103] Optionally, the quality assessment model can be iteratively trained using training samples from the first image sample set through backpropagation of the loss function.

[0104] Figure 4 This is a flowchart illustrating the process of determining parameters for a quality assessment model, as provided in an embodiment of the present invention. Figure 4 As shown, the quality assessment model is trained based on the first image sample set and a preset loss function to determine the parameters of the quality assessment model, including:

[0105] S401. Train the target first image into the input quality evaluation model to obtain the prediction result.

[0106] Here, the target first image pair is the t-th first image pair in the first image sample set. t is a positive integer greater than 0.

[0107] The t-th first image pair can be a first image pair randomly obtained from the first image sample set; or it can be the t-th first image pair obtained sequentially after sorting the first image pairs in the first image sample set according to a preset rule.

[0108] Optionally, before S401, the method may further include: randomly selecting a first image pair from the first image training set as the target first image pair.

[0109] S402. Based on the prediction results and the preset loss function, perform backpropagation to update the parameters of the quality assessment model and obtain the updated quality assessment model.

[0110] The loss function is expressed as:

[0111]

[0112] Where λ represents the weighting factor, θ represents the parameters of the multi-scale feature extraction module in the quality assessment model, and φ t The parameter x represents the quality prediction module. t ,v tLet each represent the input image for the t-th task. This represents the image pairs in the third image sample set. Represents image pairs (x t ,v t The image sample set to which ) belongs Representing image pairs The image sample set to which it belongs It is either a subset or the entire set of the third image sample set.

[0113] S403. Take the (t+1)th first image pair in the first image sample set as the target image pair, and repeat S401 and S402 in sequence until all first image pairs have been trained. Determine the parameters of the quality evaluation model based on the training results.

[0114] The loss function constructed using this method can take into account the features of both historical and new training sample data, resulting in a quality assessment model with stronger predictive power and higher prediction accuracy.

[0115] S204. Update the parameters of the plastic model according to the first update frequency corresponding to the plastic model and the third image sample set.

[0116] S205. Update the parameters of the stable model according to the second update frequency corresponding to the stable model and the third image sample set.

[0117] Continuous learning is a process of constantly evolving based on complex environments and behaviors, and building upon already learned skills to create more complex ones.

[0118] Model plasticity refers to its ability to adapt to new training tasks, that is, its ability to learn new knowledge; model stability refers to its ability to retain the skills learned on old tasks, that is, its ability to retain old knowledge.

[0119] Plasticity models are used to improve the plasticity of image quality determination models; stability models are used to improve the stability of image quality determination models.

[0120] Optionally, the first update frequency for the plastic model is greater than the second update frequency for the stable model.

[0121] This method allows for a higher number of training iterations for the plastic model compared to the stable model during training, enabling more frequent parameter updates for the plastic model. This improves the plasticity of the trained image quality determination model and enhances its learning ability for new data.

[0122] S206. Based on the parameters of the obtained quality evaluation model, the parameters of the plastic model, and the parameters of the stable model, construct an image quality determination model to evaluate the image quality of the image to be evaluated and obtain the evaluation result.

[0123] Optionally, an image quality determination model is constructed based on the parameters of the obtained quality assessment model, the parameters of the plastic model, and the parameters of the stable model. This includes: performing exponential moving average processing on the parameters of the plastic model and the parameters of the stable model based on the parameters of the obtained quality assessment model and the first decay probability corresponding to the plastic model and the second decay probability corresponding to the stable model, respectively, to obtain updated parameters of the plastic model and the updated parameters of the stable model; constructing an image quality determination model based on the parameters of the obtained quality assessment model, the updated parameters of the plastic model, and the updated parameters of the stable model, and using the image quality determination model to evaluate the image quality of the image to be evaluated, thereby obtaining an evaluation result.

[0124] Optionally, the first decay probability corresponding to the plastic model is less than or equal to the second decay frequency corresponding to the stable model.

[0125] This method allows the plastic model to simulate the brain's rapid adaptation to new information, while the stable model simulates the brain's slow acquisition of structured knowledge.

[0126] Figure 5 This is a schematic diagram of the training process of an image quality determination model provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the image quality determination model includes the quality evaluation model F(·Θ). t ), Plastic model F(·;Θ P ) and stable model F(·;Θ S ).

[0127] The following example illustrates this. Figure 5 The process shown is explained below:

[0128] For a given distorted image pair (x, v), its binary quality label is calculated as follows:

[0129]

[0130] Where q(x) and q(v) represent the MOS labels of images x and v, respectively. When learning the t-th task, the data stream... Datasets in Convert to in This represents the number of combinations of choosing m elements from n distinct elements.

[0131] First, the input image is encoded into a feature vector using a feature extraction network f(·;θ). Then, a quality regression network f(·;φ) is used. t The quality score of the t-th task image is calculated using the given formula. The quality assessment model is denoted as F(·Θ). t ={θ,φ t Then, the probability that the perceived quality of image x is higher than that of image v in the image pair is estimated by the following formula:

[0132]

[0133] Where Φ(·) is the Standard Normal Cumulative Distribution Function, and the variance of the quality prediction is fixed at 1.

[0134] For the current t-th evaluation task, fidelity loss is used to measure the statistical distance between the binary quality label and the predicted probability, i.e.:

[0135]

[0136] Direct optimization of equation (3) leads to catastrophic forgetting of old evaluation tasks by the model. Therefore, two additional EMA-weighted models are set up to maintain evaluation memory, namely the plastic model F(·Θ). P ) and stable model F(·;Θ S These two models are similar to the quality evaluation model F(·; Θ). t The structures are completely identical. By retrieving and recalling short-term evaluation memories encoded in the plastic model and long-term evaluation memories encoded in the stable model, a consistency loss is enforced on the quality evaluation model so that it does not deviate from previously learned experience when learning new tasks. Formally, a data cache is first utilized. Images from the same old task are combined into image pairs to obtain... Then, when training the t-th task, the plastic model F(v; Θ) is used. P Calculate the image pair according to equation (2) Predicted probability Similarly, using the stable model F(·;Θ) S Calculate the predicted probability And based on which model is used for the image pair Having a higher prediction probability to select the final prediction probability This is equivalent to creating a dataset with pseudo-labels to constrain the new prediction probabilities. Close to pseudo-labels This constraint is implemented using consistency loss:

[0137]

[0138] Finally, using image pairs Perform fidelity loss to replay the data, that is:

[0139]

[0140] In the actual training of the model, respectively from and Randomly select training batches and And a variant of stochastic gradient descent is used to minimize the following empirical loss:

[0141]

[0142] Where λ is the weighting factor. In each training step, after updating the quality assessment model, r is used as the weighting factor. P and r S To update the plastic and stable models probabilistically and stably, let r P >r S This allows the plastic model to be updated more frequently. Finally, the decay parameter α is used. P and α S An exponential moving average is applied to the weights of the quality assessment model to update the short-term and long-term assessment memories, i.e.:

[0143] Θ i =α i Θ i +(1-α i )Θ,i∈{P,S} (7)

[0144] Let α P ≤α S This allows the plastic model to simulate the brain's rapid adaptation to new information, while the stable model simulates the brain's slow acquisition of structured knowledge.

[0145] Figure 6 This is a schematic diagram of the structure of a quality evaluation model provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the quality assessment model includes a feature extraction network and a quality prediction module, wherein the feature extraction network consists of a multi-scale feature extraction module and a multi-scale feature fusion module.

[0146] Specifically, for the multi-scale feature extraction module, a ResNet18 network with the terminal average pooling layer and fully connected layers removed is used as the backbone network, and the corresponding network parameters are initialized using an ImageNet pre-trained model. Specifically, for the input image x, firstly, multi-scale feature maps fm1, fm2, fm3, and fm4 are obtained using the residual blocks conv2_x, conv3_x, conv4_x, and conv5_x of the ResNet18 network. Then, fm1 is passed through a 1×1 convolutional layer and a global average pooling layer with an output size of 8×8 to obtain the feature vector fv1; fm2 is passed through a 1×1 convolutional layer and a global average pooling layer with an output size of 4×4 to obtain the feature vector fv2; fm3 is passed through a 1×1 convolutional layer and a global average pooling layer with an output size of 2×2 to obtain the feature vector fv3; and fm4 is passed through a 1×1 convolutional layer and a global average pooling layer with an output size of 1×1 to obtain the feature vector fv4. Then, fv1, fv2, fv3 and fv4 are reduced to vectors of length 128 by passing them through a fully connected layer.

[0147] For the multi-scale feature fusion module, fv′1, fv′2, fv′3, and fv′4 are first concatenated to form a feature vector fv′. fv′ is then passed through an encoding block consisting of fully connected layers with a structure of 512-256 and a decoding block consisting of fully connected layers with a structure of 256-512 to obtain the feature vector v. Next, an L2 norm normalization layer projects the feature vector v onto the unit hypersphere to obtain the feature vector z. This scales the predictions of all quality regression networks to a similar range, making subsequent calculations numerically more stable. Finally, the quality regression network is used to map z to a quality score. The quality regression network consists of three fully connected layers with a structure of 512-128-32-1. Furthermore, the activation function of all the fully connected layers is the ReLU function.

[0148] The image quality assessment method provided by this invention, based on the above embodiments, further improves the evaluation accuracy of the image quality assessment model by constructing a complementary learning system using a plastic model and a stable model. This enables the image quality determination model to achieve better continuous learning on new and old image datasets with significant differences in distortion type or content scene, thereby improving the evaluation accuracy of the image quality determination model's continuous learning across image datasets.

[0149] Example 3

[0150] Figure 7 This is a flowchart illustrating another referenceless image quality assessment method based on continuous learning provided in an embodiment of the present invention. This method is applied to electronic devices, such as... Figure 7 As shown, the method includes:

[0151] S701. Obtain the image to be evaluated.

[0152] Electronic devices can acquire images to be evaluated through interactive interfaces, such as by scanning or taking pictures, or through virtual or physical storage units, such as internal storage units or cloud storage units.

[0153] The image to be evaluated refers to the image whose image quality is to be evaluated.

[0154] S702. Input the image to be evaluated into the image quality determination model for processing to obtain the image quality evaluation result.

[0155] The image quality determination model is generated by performing the methods provided in Example 1 or Example 2. The specific training process of the image quality determination model can be found in the relevant descriptions in Example 1 and Example 2, and will not be repeated here.

[0156] This method enables a more accurate evaluation of image quality, ensuring the effective transmission of information.

[0157] Example 4

[0158] This invention also provides a referenceless image quality assessment device based on continuous learning. Figure 8 A schematic diagram of a referenceless image quality assessment device based on continuous learning provided in an embodiment of the present invention is shown below. Figure 8 As shown, the device includes:

[0159] The acquisition module 81 is used to acquire a first image sample set, which includes at least one first image pair.

[0160] The processing module 82 is used to update the second image sample set according to the first image sample set to obtain the third image sample set. The second image sample set is pre-stored in a preset storage unit. The second image sample set includes at least one second image pair. The third image sample set includes at least one first image pair and at least one second image pair.

[0161] The processing module 82 is also used to train the first model based on the first image sample set, the third image sample set and the preset loss function to obtain an image quality determination model, and to evaluate the image quality of the image to be evaluated to obtain the evaluation result.

[0162] Optionally, the first model includes a quality assessment model, a malleable model, and a stable model. The processing module 82 is specifically used to train the quality assessment model according to the first image sample set and a preset loss function to determine the parameters of the quality assessment model; update the parameters of the malleable model according to the first update frequency corresponding to the malleable model and the third image sample set; update the parameters of the stable model according to the second update frequency corresponding to the stable model and the third image sample set; and construct an image quality determination model based on the obtained parameters of the quality assessment model, the malleable model, and the stable model.

[0163] Optionally, the first update frequency for the plastic model is greater than the second update frequency for the stable model.

[0164] Optionally, the processing module 82 is specifically used to perform exponential moving average processing on the parameters of the plastic model and the parameters of the stable model according to the obtained parameters of the quality evaluation model and the first decay probability corresponding to the plastic model and the second decay probability corresponding to the stable model, respectively, to obtain the updated parameters of the plastic model and the updated parameters of the stable model; and to construct an image quality determination model based on the obtained parameters of the quality evaluation model, the updated parameters of the plastic model and the updated parameters of the stable model.

[0165] Optionally, the first decay probability corresponding to the plastic model is less than or equal to the second decay frequency corresponding to the stable model.

[0166] Optionally, processing module 82 is specifically used for: Step 1: Inputting the target first image pair into the quality assessment model for training to obtain a prediction result, where the target first image pair is the t-th first image pair in the first image sample set; Step 2: Performing backpropagation based on the prediction result and a preset loss function to update the parameters of the quality assessment model, resulting in an updated quality assessment model, where the loss function is expressed as:

[0167]

[0168] Where λ represents the weighting factor, θ represents the parameters of the multi-scale feature extraction module in the quality assessment model, and φ t The parameter x represents the quality prediction module. t ,v t Let each represent the input image for the t-th task. This represents the image pairs in the third image sample set. Represents image pairs (x t ,v t The image sample set to which ) belongs Representing image pairs The image sample set to which it belongs The first image is a subset or the entire set of the third image sample set. Step 3: Take the (t+1)th first image pair in the first image sample set as the target image pair, and repeat steps 1 and 2 in sequence until all first image pairs have been trained. Determine the parameters of the quality evaluation model based on the training results.

[0169] The no-reference image quality assessment device can perform the no-reference image quality assessment method based on continuous learning provided in Embodiment 1 or Embodiment 2 above. Its content and effects can be found in the method embodiment section, and will not be repeated here.

[0170] Example 5

[0171] This invention also provides a referenceless image quality assessment device based on continuous learning. Figure 9 A schematic diagram of another referenceless image quality assessment device based on continuous learning provided in an embodiment of the present invention is shown below. Figure 9 As shown, the device includes:

[0172] The acquisition module 91 is used to acquire the image to be evaluated.

[0173] The processing module 92 is used to input the image to be evaluated into the image quality determination model for processing, and obtain the image quality evaluation result.

[0174] The image quality determination model is generated by performing the methods provided in Example 1 or Example 2.

[0175] The image quality assessment device can perform the referenceless image quality assessment method based on continuous learning provided in Embodiment 3 above. Its content and effects can be found in the method embodiment section, and will not be repeated here.

[0176] This invention also provides an electronic device. Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 10 As shown, the electronic device of this embodiment includes a processor 101 and a memory 102; the processor 101 and the memory 102 are communicatively connected. The memory 102 is used to store computer programs. The processor 101 is used to call the computer program stored in the memory 102 to implement the method in the above embodiment.

[0177] Optionally, the electronic device also includes a transceiver 103 for communicating with other devices.

[0178] The electronic device can execute the referenceless image quality assessment method based on continuous learning provided in Embodiment 1, Embodiment 2 or Embodiment 3 above. The content and effect can be found in the method embodiment section, and will not be repeated here.

[0179] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0180] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0181] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0182] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0183] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods as described in any of the above method embodiments.

[0184] When the computer-readable storage medium stores the computer-executable instructions, it can implement the above-mentioned image quality evaluation method when executed by the processor. The content and effect can be referred to the method embodiment section, and will not be repeated here.

[0185] The present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the method as described in any of the above method embodiments.

[0186] When the computer-readable storage medium stores the computer-executable instructions, it can implement the above-mentioned image quality evaluation method when executed by the processor. The content and effect can be referred to the method embodiment section, and will not be repeated here.

[0187] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0188] For the embodiments of the device / electronic device / storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0189] It should be noted that the device, electronic device and storage medium in the embodiments of the present invention are respectively the device, electronic device and storage medium for applying the above image quality evaluation. Therefore, all embodiments of the above image quality evaluation method are applicable to the device, electronic device and storage medium, and can achieve the same or similar beneficial effects.

[0190] The terminal device provided by the embodiments of the present invention can display proper nouns and / or fixed phrases for users to select, thereby reducing user input time and improving user experience.

[0191] This terminal device exists in various forms, including but not limited to:

[0192] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0193] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0194] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes: audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

[0195] (4) Other electronic devices with data interaction functions.

[0196] In this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0197] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0198] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, the disclosure, and the appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0199] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus (devices), or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects, all of which are collectively referred to herein as "modules" or "systems." Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The computer program may be stored / distributed in a suitable medium, provided with or as part of other hardware, or may take other distribution forms, such as via the Internet or other wired or wireless telecommunications systems.

[0200] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0201] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0202] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0203] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A no-reference image quality assessment method based on continuous learning, characterized in that, Applied to electronic devices, the method includes: Obtain a first image sample set, the first image sample set including at least one first image pair; The second image sample set is updated according to the first image sample set to obtain the third image sample set. The second image sample set is pre-stored in a preset storage unit. The second image sample set includes at least one second image pair. The third image sample set includes at least one first image pair and at least one second image pair. The first model is trained based on the first image sample set, the third image sample set, and a preset loss function to obtain an image quality determination model, which is used to evaluate the image quality of the image to be evaluated and obtain the evaluation result.

2. The method according to claim 1, characterized in that, The first model includes a quality assessment model, a malleable model, and a stable model. The first model is trained using the first image sample set, the third image sample set, and a preset loss function to obtain an image quality determination model, which is used to evaluate the image quality of the image to be evaluated, and to obtain the evaluation result, including: The quality assessment model is trained based on the first image sample set and a preset loss function to determine the parameters of the quality assessment model. The parameters of the plastic model are updated according to the third image sample set based on the first update frequency corresponding to the plastic model. The parameters of the stable model are updated according to the third image sample set at the second update frequency corresponding to the stable model. Based on the parameters of the obtained quality evaluation model, the parameters of the plasticity model, and the parameters of the stability model, an image quality determination model is constructed to evaluate the image quality of the image to be evaluated and obtain the evaluation result.

3. The method according to claim 2, characterized in that, include: The first update frequency corresponding to the plastic model is greater than the second update frequency corresponding to the stable model.

4. The method according to claim 2 or 3, characterized in that, The step of constructing an image quality determination model based on the parameters of the obtained quality evaluation model, the parameters of the plasticity model, and the parameters of the stability model includes: Based on the obtained parameters of the quality evaluation model and the first decay probability corresponding to the plastic model and the second decay probability corresponding to the stable model, the parameters of the plastic model and the stable model are respectively processed by exponential moving average to obtain the updated parameters of the plastic model and the updated parameters of the stable model. An image quality determination model is constructed based on the parameters of the obtained quality assessment model, the updated parameters of the plasticity model, and the updated parameters of the stability model.

5. The method according to any one of claims 1-3, characterized in that, The step of training the quality assessment model based on the first image sample set and a preset loss function to determine the parameters of the quality assessment model includes: Step 1: Input the target first image pair into the quality assessment model for training to obtain the prediction result. The target first image pair is the t-th first image pair in the first image sample set. Step 2: Based on the prediction results and the preset loss function, perform backpropagation to update the parameters of the quality assessment model, obtaining the updated quality assessment model. The loss function is expressed as: Where λ represents the weighting factor, θ represents the parameters of the multi-scale feature extraction module in the quality assessment model, and φ t The parameter x represents the quality prediction module. t ,v t Let each represent the input image for the t-th task. This represents the image pairs in the third image sample set. Represents image pairs (x t ,v t The image sample set to which ) belongs Representing image pairs The image sample set to which it belongs This can be a subset or the entirety of the third image sample set. Step 3: Take the (t+1)th first image pair in the first image sample set as the target image pair, and repeat Step 1 and Step 2 in sequence until all first image pairs have been trained. Determine the parameters of the quality evaluation model based on the training results.

6. A no-reference image quality assessment method based on continuous learning, characterized in that, Applied to electronic devices, the method includes: Obtain the image to be evaluated; The image to be evaluated is input into the image quality determination model for processing to obtain the image quality evaluation result, wherein the image quality determination model is generated by performing the method as described in any one of claims 1-5.

7. A referenceless image quality assessment device based on continuous learning, characterized in that, include: The acquisition module is used to acquire the first image sample set; The processing module is used to update the second image sample set according to the first image sample set to obtain a third image sample set. The second image sample set is pre-stored in a preset storage unit. The second image sample set includes at least one second image pair. The third image sample set includes at least one first image pair and at least one second image pair. The processing module is further configured to train the first model based on the first image sample set, the third image sample set, and a preset loss function to obtain an image quality determination model, so as to evaluate the image quality of the image to be evaluated and obtain an evaluation result.

8. A referenceless image quality assessment device based on continuous learning, characterized in that, include: The acquisition module is used to acquire the image to be evaluated; The processing module is used to input the image to be evaluated into the image quality determination model for processing to obtain the image quality evaluation result, wherein the image quality determination model is generated by performing the method as described in any one of claims 1-5.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method steps of any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method steps of any one of claims 1-6.

Images