Image quality evaluation continuous learning method based on prototype playback strategy

Abstract

The invention discloses an image quality evaluation continuous learning method based on a prototype replay strategy, and the method comprises the following steps: constructing an evaluation model based on a compression distortion prototype replay strategy module, an image quality attention contrast learning module, and a loss optimization function and quality feature regression module; storing the distorted picture compact groups of different distortion type prototypes into a memory with a limited size to obtain a training set; acquiring quality features of distorted images in the training set; optimizing the evaluation model to obtain an optimized evaluation model; and training the optimization evaluation model to obtain a final image quality evaluation model. According to the method, the problem that the overall accuracy of the model for the multi-scene image is not high due to the fact that the problem of knowledge disastrous forgetting of different scene distortion types is insufficient to solve can be effectively relieved, and therefore accurate prediction of the quality of different distortion scene images is achieved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a continuous learning method for image quality evaluation based on a prototype replay strategy. Background Art

[0002] With the rapid development of image processing systems, the types of image distortion in real-world open environments have become complex and unpredictable, and quality assessment methods may face the challenge of processing images with distortion types that have never been seen before deployment. In practical applications, distortion artifacts and evaluation criteria may change with the changes in task scenarios, which puts new requirements on cross-task quality assessment methods. The image data space can be divided into various distortion scenarios, such as synthetic, real, and visually enhanced distortion scenarios. Each distortion scene contains one or more data sets, and the existing data sets only cover the distortion types of specific distortion scenes, resulting in the existing methods generally lacking the ability to integrate distortion knowledge between different distortion types and difficult to deal with the problem of distortion displacement.

[0003] For the problem of distortion displacement between different distortion type scenes, the most direct approach is to train a separate model for each distortion type scene, and finally integrate multiple separate models into one model to achieve accurate quality assessment of different types of images in multiple distortion type scenes. Although this method is the simplest way to solve the generalization problem for multiple distortion type scenes, it allows the integrated model to obtain good prediction performance in different scenes and effectively retain performance for different distortion type scenes. However, this method requires a lot of storage and computing resources, and when new distortion type scenes appear, the model needs to be retrained and integrated, which is challenging in practice. Another solution is to use only one static model and fine-tune the old model to adapt to the new task. Its storage and computing overhead will not increase with the increase of new tasks. However, due to the lack of a designed training strategy, the model tends to forget the old distortion scene knowledge after learning the new distortion scene task. This phenomenon occurs when the model learns multiple distortion scene tasks in succession. This phenomenon is called catastrophic forgetting, that is, the model tends to change the weights required for the previous task to adapt to the new task, resulting in a sharp drop in the performance of the original (old) task. This greatly limits the deployment and application of quality assessment methods in real-world scenarios. Obviously, facing the dynamic environment in practical applications, static models are not optimal.

[0004] Some pioneering works have been proposed to address the problem of catastrophic forgetting in cross-distortion scene quality assessment tasks. A direct way to mitigate catastrophic forgetting is joint training. In 2021, Zhang Weixia et al. from Shanghai Jiaotong University proposed a joint training strategy for dataset combination to mitigate the catastrophic forgetting problem of quality assessment models, that is, combining all training data and retraining the model from scratch. Although it is robust to different datasets used in training, for new distortion scenarios that emerge in the future, it is necessary to retrain by adding new datasets suitable for new distortion scenarios, so this method is very inefficient. And due to storage limitations and privacy issues, previous data may not be accessible, so it is difficult to adopt a joint training strategy. In 2023, Zhang Weixia et al. from Shanghai Jiaotong University further proposed a reference-free method based on a shared backbone network to combat catastrophic forgetting. This method adds prediction heads for new datasets and enforces regularizers during continuous learning, so that all prediction heads evolve with new data to adapt to new tasks while resisting catastrophic forgetting. However, limited by the hard parameter sharing of the backbone network, the performance of the model drops sharply when facing distorted scene tasks with significant differences. In 2023, Wu Qingbo and others from the University of Electronic Science and Technology of China proposed a reference-free evaluation method called SILF. By using a dynamic parameter isolation strategy, specific parameter subsets of non-overlapping distortion scenes are sequentially updated, solving the problem that backbone-based hard parameter sharing cannot be applied to tasks with significant differences. This method prunes subsets of old task parameters and releases memory capacity, allowing the model to adapt to tasks with new distortion scenarios that may appear in the future. However, this will cause the model to forget some knowledge in old tasks, and as the training sequence lengthens, the robustness to distortion scenes that are far apart will gradually decrease. Summary of the invention

[0005] The present application provides a continuous learning method for image quality evaluation based on a prototype replay strategy to solve the technical problems that the prior art causes the model to forget some knowledge in old tasks, and as the training sequence lengthens, the robustness to distorted scenes that are far apart will gradually decrease.

[0006] In order to solve the above technical problems, a technical solution adopted in this application is: a continuous learning method for image quality evaluation based on a prototype replay strategy, comprising the following steps:

[0007] An evaluation model is constructed based on the compression distortion prototype replay strategy module, image quality attention contrast learning module, loss optimization function and quality feature regression module;

[0008] Based on the attention average drift algorithm in the compression distortion prototype replay strategy module, the distorted images of different distortion type prototypes are compactly stored in a memory of limited size to obtain the training set;

[0009] Based on the image quality attention contrast learning module, the quality features of the distorted images in the training set are obtained;

[0010] Based on the loss function of attention focus contrast and the quality characteristics of the distorted image, the evaluation model is optimized to obtain the optimized evaluation model;

[0011] Based on the quality feature regression module and the distorted images in the training set, the optimization evaluation model is trained to obtain the final image quality evaluation model.

[0012] Further, the method of obtaining the training set includes:

[0013] Based on the distorted image set, different distortion type prototypes in the distorted image set are obtained;

[0014] Fill the representative distorted image samples in the distorted image set into the corresponding distorted type prototype, obtain a compact group of distorted images away from other types and store them in the memory, and obtain the first training set;

[0015] The sample selection ratio is set, and the distorted images in the distorted image set are randomly selected and stored in the memory to obtain the second training set;

[0016] The first training set and the second training set together constitute the training set.

[0017] Furthermore, a method for constructing an image quality attention contrast learning module includes:

[0018] Based on formula (1), the attention head in the attention comparison mechanism is obtained; wherein formula (1) is:

[0019]

[0020] Among them, σ(·) represents the softmax activation function, and A h is represented as a dot product attention matrix; d represents the dimension of the key, H is the number of attention heads in the attention mechanism; Q, K, B, and V represent query, learnable foreign key, attention bias, and value embedding, respectively; and Q h , K h , B h and V h are the linear projections relative to Q, K, B and V respectively; Ψ h is the standard token interaction pattern in the standard Transformer.

[0021] Further, the method of constructing the loss function includes:

[0022] Based on formula (2), the loss function is obtained; wherein formula (2) is:

[0023]

[0024] Among them, L CLA is the loss function, x ki is stored in the memory buffer B Mk The i-th image in ; t represents the current task; t-1 represents the previous task; h θ (x ki ) is the cosine similarity between attention focus vectors; is the evaluation model parameter of the current task; B M represents the distorted image samples in memory.

[0025] Furthermore, the method for obtaining the final image quality evaluation model includes:

[0026] Attach a multi-layer perceptron prediction head during multiple training runs to obtain quality estimates for the current task;

[0027] Based on the quality estimate of the current task, obtain the loss optimization function;

[0028] Based on the loss optimization coefficient and the loss function, the total loss function is obtained;

[0029] Based on the total loss function, the final image quality evaluation model is obtained.

[0030] Further, the method for obtaining the loss optimization function includes:

[0031] Based on formula (3), the difference between the actual subjective score value of the image and the quality estimation value is calculated; where formula (3) is:

[0032]

[0033] in, The i-th image in a batch B in the current task t is predicted by the prediction head The calculated prediction score, y ti is the true subjective quality score corresponding to the image; φ, χ t are the parameters of the multilayer perceptron;

[0034] Based on formula (4), the memory metric loss function is obtained; wherein formula (4) is:

[0035]

[0036] Based on formula (5), the loss optimization function is obtained; wherein formula (5) is:

[0037]

[0038] Among them, λ is a hyperparameter weight that measures the importance of the loss function.

[0039] Further, the method of obtaining the total loss function includes:

[0040] Based on formula (6), the total loss function is obtained; wherein formula (6) is:

[0041]

[0042] Among them, L(.) is the total loss function, L CLA (.) is the loss function, L score (.) is the loss optimization function.

[0043] The beneficial effect of the present application is: the present application deeply explores the distortion displacement generation mechanism between different distortion scenes, and combines the self-attention mechanism and data-driven strategy of the Transformer architecture to construct a multi-scene image quality evaluation model, which can effectively alleviate the problem of catastrophic forgetting of knowledge of different scene distortion types caused by insufficient solution to the method, resulting in the low overall accuracy of the model for multi-scene images, thereby achieving accurate prediction of the image quality of different distortion scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flowchart of an embodiment of a method for continuous learning of image quality assessment based on a prototype replay strategy of the present application;

[0045] Figure 2 yes Figure 1 A schematic flow chart of an embodiment of step S2 in FIG.

[0046] Figure 3 It is an overall architecture diagram of an embodiment of a method for continuous learning of image quality assessment based on a prototype replay strategy of the present application;

[0047] Figure 4 yes Figure 1 A schematic flow chart of an embodiment of step S5 in FIG.

[0048] Figure 5 It is a schematic diagram of prediction results for distorted images of different scenes according to an embodiment of the continuous learning method for image quality assessment based on the prototype replay strategy of the present application. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with specific embodiments.

[0050] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0051] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of a method for continuous learning of image quality assessment based on a prototype replay strategy of the present application. The method comprises:

[0052] Step S1. Construct an evaluation model based on the compression distortion prototype replay strategy module, the image quality attention contrast learning module, the loss optimization function and the quality feature regression module.

[0053] Step S2. Based on the attention average drift algorithm in the compression distortion prototype replay strategy module, the distorted images of different distortion type prototypes are compactly grouped and stored in a memory of limited size to obtain a training set.

[0054] For details, see Figure 2 , Figure 2 yes Figure 1 Schematic diagram of the process of step S2 in an embodiment. Step S2 includes:

[0055] Step S21. Based on the distorted image set, different distortion type prototypes in the distorted image set are obtained.

[0056] Specifically, when the evaluation model learns a set of distorted images D defined in task t t When the data samples are clustered into k distortion categories, for each distortion category k∈D t , will construct a distortion type d i =F(x i ; θ)’s sample representation.

[0057] Step S22: Fill the representative distorted image samples in the distorted image set into the corresponding distorted type prototype, obtain a compact group of distorted images away from other types and store them in the memory, and obtain the first training set.

[0058] Specifically, the distortion category k is combined with the concentrated distortion prototype DP k Connect them and apply the attention average drift algorithm to DP k Move to the high-density area of ​​the representation space. During the model training process, DP k All samples belonging to the distortion category k in are updated, and the new DP k Perform normalization operation, and the calculation process is as follows:

[0059]

[0060] Among them, λ controls the step size of the average attention drift, B t is the training set D for task t during model training t The batch size set in . The attention average drift algorithm used in this section is different from the original average drift algorithm. The main difference is that the weight η k Each distorted sample d in the representation space i Prototype DP with distortion type k The learnable dot product attention between them determines the weight η k The calculation process is given by the following equation:

[0061]

[0062] in, It is a learnable dot product attention. The detailed calculation process of dot product attention is given by the following equation:

[0063]

[0064] Through the above calculation method, the training set D of the current task t can be calculated t The compression distortion type prototype DP corresponding to different distortion type scenarios k and the samples represented by each distortion type prototype.

[0065] Step S23: Set the sample selection ratio, randomly select distorted images from the distorted image set and store them in the memory to obtain a second training set.

[0066] Specifically, due to the limited capacity of memory M, the number of data samples that have been learned and can be used for replay is limited. In order to balance the amount of newly arrived data and the amount of data used for replay, the model selects data samples of each type of compression distortion prototype Create a new dataset The sample selection ratio β is used to control the number of samples in each category. The calculation process of the sample selection ratio β is as follows:

[0067]

[0068] where L is the total number of distortion types seen by the model so far, is the number of data samples of the kth class. When entering the next task t+1, the model will store multiple distorted samples randomly selected from each prototype calculated in the previous task in the memory according to the memory budget to form the second training set.

[0069] Step S24: The first training set and the second training set together constitute a training set.

[0070] Specifically, the first training set stored in the memory and the second training set in the new task together form a training set and are used together to train the evaluation model.

[0071] Step S3. Based on the image quality attention contrast learning module, the quality features of the distorted images in the training set are obtained.

[0072] Specifically, in order to make the Transformer architecture suitable for continuous learning tasks, an inter-task attention mechanism is adopted, which can implicitly absorb the information of previous tasks and slow down the drift of important attention between previous tasks and current tasks. Specifically, the introduced external attention mechanism modifies the calculation method of the attention map in the typical visual Transformer to be calculated by the affinity between the self-query and the learnable external key K and the attention bias B, which implicitly injects the previous task information into the attention mechanism. Considering that there are a total of H attention heads in the model, the representation of each attention head is as follows:

[0073]

[0074] where σ(·) represents the softmax activation function, and A h is represented as a dot product attention matrix. d represents the dimension of the key, and H is the number of attention heads in the attention mechanism. Q, K, B, and V represent query, learnable foreign key, attention bias, and value embedding, respectively, while Q h , K h , B h and V h are the linear projections relative to Q, K, B and V respectively. h is the standard token interaction mode in the standard Transformer, which performs attention calculations evenly and intensively between all image-standard token pairs.

[0075] However, in continuous learning on image quality assessment tasks, it is difficult to use the standard token interaction mode in a typical visual Transformer. This is because during continuous learning, the weights of the attention in the standard token interaction mode are adjusted according to the new task, while the weights of the old task are forgotten, resulting in catastrophic forgetting. In order to meet the requirements of continuous learning, a learnable attention bias B is added to the attention map to avoid catastrophic interference by injecting new representations and accumulate new and previous knowledge in a balanced way to improve the overall performance. Therefore, the formula for each attention head can be rewritten as:

[0076]

[0077] Among them, B his the linear projection relative to B. The quality characteristics of the distorted images in the training set can be obtained by calculating the above formula.

[0078] Step S4. Based on the loss function of attention focus contrast and the quality characteristics of the distorted image, the evaluation model is optimized to obtain the optimized evaluation model.

[0079] For details, see Figure 3 Each input batch of the evaluation model in the training phase consists of two parts of data samples: the first part of the data samples comes from the part of the representative distortion samples of the old task distortion type prototype stored in the limited memory buffer M in step S1 The second part of the data samples corresponds to the current task T t The corresponding dataset D t The distorted samples Bt of the current batch size sampled in . and has the same architecture, but with parameters θ t and So for the vector z of the focus of attention new and z old The generation process is based on the new task network parameters θ t and the corresponding old task network parameters θ k The specific calculation process is given as follows:

[0080]

[0081] Among them, x ki ∈B Mk , and k = {1, ...t-1}, which is used to represent the partial representative distorted samples in the old tasks stored in the memory buffer M.

[0082] At the same time, in order to learn the characteristics of different distortion scenes and obtain complementary knowledge between them, and better tap the potential of the self-attention mechanism in the Transformer architecture, this scheme proposes to use the cosine similarity comparison of two attention focus vectors to measure the consistency of the perceptual area of ​​the same distorted sample. By comparing the cosine similarity of the two attention focus vectors, the model can consolidate the knowledge of the old task in the process of learning the new task, and at the same time tap the commonality between the representative distorted samples of the new distorted scene and the distorted scene of the old task. Specifically, for the attention focus vector z new and z old The cosine similarity calculation process between is given by the following formula:

[0083]

[0084] in, It is a very small positive number used to prevent calculation errors that occur when the denominator is calculated to be 0 during the calculation process.

[0085] Based on formula (2), the final loss function based on attention focus comparison is as follows:

[0086]

[0087] Among them, x ki is B stored in memory buffer M Mk The loss function based on attention focus contrast can effectively supervise the model by comparing the similarity between the model's attention focus in new task learning and the attention focus of previous old tasks, effectively consolidating the distorted scene knowledge in previous old tasks, which helps to alleviate the catastrophic forgetting of old tasks.

[0088] Step S5. Based on the quality feature regression module and the distorted image samples in the training set, an optimization evaluation model is trained to obtain a final image quality evaluation model.

[0089] For details, see Figure 4 , Figure 4 for Figure 1 Schematic diagram of the process of step S5 of an embodiment. Step S5 includes:

[0090] Step S51: attach a multi-layer perceptron prediction head during multiple training processes to obtain a quality estimate of the current task.

[0091] Specifically, in order to adapt it to continuous learning, during the training process, as the task sequence proceeds, the quality feature regression module adds a t Parameterized Multilayer Perceptron (MLP) prediction head To regress the quality prediction value of the first t tasks.

[0092] Step S52: Based on the quality estimation value of the current task, obtain the loss optimization function.

[0093] Specifically, for the complete training process of the proposed method, the complete parameter set of the MLP prediction head in the model is {φ, χ 1 , χ 2 ,x T For the first t tasks, the mean absolute error loss optimization function is used to calculate the difference between the actual subjective score value of the image and the model prediction value, which can be expressed as

[0094]

[0095] in, The i-th image in a batch B in the current task t is predicted by the prediction head The calculated prediction score, y ti is the true subjective quality score corresponding to the image.

[0096] However, if we simply use the mean absolute error loss optimization function (as shown in formula (3)) to guide the optimization of the model, the model will only focus on learning the distortion type scene knowledge of the current task, resulting in catastrophic forgetting of the old task knowledge, which does not meet the requirements of continuous learning of different distortion type scene knowledge. The loss function in the training process should also be changed accordingly to adapt to the replay strategy, and further use the memory metric loss to guide the model to consolidate the memory of the old task knowledge. The specific calculation is given by the following formula:

[0097]

[0098] The training process of the proposed method is as follows: First, from the training dataset D corresponding to the current task t t Randomly select a small batch of B t , which is then compared with the The data sets are merged into B = B t ∪B M As the training sample of the model in the current task t. Finally, it is input into the model for training optimization, and a variant of stochastic gradient descent is used to minimize the following loss optimization function:

[0099] Based on formula (5), the loss optimization function is obtained; wherein formula (5) is:

[0100]

[0101] Among them, λ is a hyperparameter weight that measures the importance of the loss function, and this scheme sets it to 0.6.

[0102] Step S53: Based on the loss optimization function and the loss function, obtain the total loss function.

[0103] Specifically, the total loss is given by:

[0104]

[0105] Among them, L(.) is the total loss function, L CLA (.) is the loss function, L score (.) is the loss optimization function.

[0106] Step S54: Based on the total loss function, obtain a final image quality evaluation model.

[0107] See also Figure 5 , Figure 5The visualization diagram of the prediction results of the image quality evaluation continuous learning method based on the prototype replay strategy proposed in this application for distorted images of different scenes is given. From the visualization results, it can be seen that the method proposed in this application can effectively evaluate the quality information of distorted images of different distorted scenes, and its prediction score is similar to the subjective score, with an error of less than 2%.

[0108] This application proposes a continuous learning method for image quality assessment based on a prototype replay strategy, which deeply analyzes the potential of the self-attention mechanism in the Transformer architecture, designs a visual quality feature perception module suitable for continuous learning and a loss optimization function based on attention focus comparison for the image quality assessment task of multiple distortion scenes, and proposes a quality feature regression module suitable for continuous learning in multiple distortion scenes in combination with data-driven. And in order to further reduce the method's demand for memory budget, a replay strategy for compressed distortion prototypes is also proposed. By selecting only a few representative samples from the concentrated distortion prototypes of each distortion type during replay, the model's demand for memory budget is effectively reduced, and the model is made more robust to different distortion scenes. Experiments show that the performance of the continuous learning method for image quality assessment based on the prototype replay strategy proposed in this application on public datasets is significantly superior to existing advanced image quality assessment methods.

[0109] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A continuous learning method for image quality assessment based on prototype replay strategy, characterized in that: The following steps are involved: An evaluation model is constructed based on the compression distortion prototype replay strategy module, image quality attention contrast learning module, loss optimization function and quality feature regression module; Based on the attention average drift algorithm in the compression distortion prototype replay strategy module, the distorted images of different distortion type prototypes are compactly stored in a memory of limited size to obtain the training set; Based on the image quality attention contrast learning module, obtaining quality features of the distorted images in the training set; Based on the loss function of the attention focus contrast and the quality characteristics of the distorted image, optimizing the evaluation model to obtain an optimized evaluation model; Based on the quality feature regression module and the distorted image samples in the training set, the optimization evaluation model is trained to obtain a final image quality evaluation model.

2. The method according to claim 1, characterized in that Methods for obtaining training sets include: Based on the distorted image set, obtaining different distortion type prototypes in the distorted image set; Filling representative distorted image samples in the distorted image set into the corresponding distorted type prototype, obtaining a compact group of distorted images away from other types and storing them in the memory, to obtain a first training set; Setting a sample selection ratio, randomly selecting distorted images from the distorted image set and storing them in the memory, to obtain a second training set; The first training set and the second training set together constitute the training set.

3. The method according to claim 1, characterized in that The method for constructing the image quality attention contrast learning module comprises: Based on formula (1), the attention head in the attention comparison mechanism is obtained; wherein the formula (1) is: Among them, σ(·) represents the softmax activation function, and A h is represented as a dot product attention matrix; d represents the dimension of the key, H is the number of attention heads in the attention mechanism; Q, K, B, and V represent query, learnable foreign key, attention bias, and value embedding, respectively; and Q h , K h , B h and V h are the linear projections relative to Q, K, B and V respectively; Ψ h is the standard token interaction pattern in the standard Transformer.

4. The method according to claim 1, characterized in that: The method for constructing the loss function comprises: Based on formula (2), the loss function is obtained; wherein, formula (2) is: Among them, L CLA is the loss function, x ki is stored in the memory buffer B Mk The i-th image in ; t represents the current task; t-1 represents the previous task; h θ (x ki ) is the cosine similarity between attention focus vectors; is the evaluation model parameter of the current task; B M represents the distorted image samples in memory.

5. The method according to claim 4, characterized in that The method for obtaining the final image quality evaluation model includes: Attach a multi-layer perceptron prediction head during multiple training runs to obtain quality estimates for the current task; Based on the quality estimation value of the current task, obtaining a loss optimization function; Based on the loss optimization function and the loss function, obtaining a total loss function; Based on the total loss function, the final image quality assessment model is obtained.

6. The method according to claim 5, characterized in that Methods for obtaining loss optimization functions include: Based on formula (3), the difference between the actual subjective score value of the image and the quality estimation value is calculated; wherein formula (3) is: in, The i-th image in a batch B in the current task t is predicted by the prediction head The calculated prediction score, y ti is the true subjective quality score corresponding to the image; φ, χ t are the parameters of the multilayer perceptron; Based on formula (4), the memory metric loss function is obtained; wherein the formula (4) is: Based on formula (5), the loss optimization function is obtained; wherein the formula (5) is: Among them, λ is a hyperparameter weight that measures the importance of the loss function.

7. The method according to claim 6, characterized in that Methods for obtaining the total loss function include: Based on formula (6), the total loss function is obtained; wherein the formula (6) is: Among them, L(.) is the total loss function, L CLA (.) is the loss function, L score (.) is the loss optimization function.

Images