Image restoration model generation method and image restoration method based on continuous learning

Abstract

This invention provides a method for generating image restoration models based on continuous learning and an image restoration method, relating to the field of image processing technology. The method includes: acquiring a baseline model and a training set corresponding to a target degradation type; the training set includes a first training set and a second training set, wherein the first training set contains fewer training data sets than the second training set; each training data set includes a sample image to be restored and a corresponding restored image; training the baseline model corresponding to the target degradation type based on the first training set to obtain a target model corresponding to the target degradation type; determining key convolutional kernels based on the target model and the baseline model; and training the baseline model based on the second training set, updating the parameters of the key convolutional kernels to obtain the image restoration model corresponding to the target degradation type. This invention can improve the generation efficiency of image restoration models for various image restoration tasks.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for generating an image restoration model based on continuous learning and an image restoration method. Background Technology

[0002] Image restoration refers to the process of recovering a clear image from a degraded image. It is the foundation of advanced vision tasks (such as image recognition and object detection). With the continuous development of deep learning theory, deep convolutional networks have been applied to various computer vision tasks, including a wide range of image restoration tasks.

[0003] However, existing image restoration methods tend to handle different degradation factors separately, and design and train a completely new specific network model for each task (each degradation task corresponds to different degradation factors, such as noise, blur, rain streaks, etc.), which is inefficient. Summary of the Invention

[0004] This invention provides a method for generating an image restoration model based on continuous learning and an image restoration method, which solves the problem of low efficiency in the prior art, which requires designing and training a specific network model for each degradation task.

[0005] This invention provides a method for generating image restoration models based on continuous learning, comprising:

[0006] Obtain the baseline model and training set corresponding to the target degradation type. The training set includes a first training set and a second training set. The number of training data sets in the first training set is less than the number of training data sets in the second training set. Each training data set includes a sample image to be restored and a restored image corresponding to the sample image to be restored.

[0007] The baseline model corresponding to the target degradation type is trained based on the first training set to obtain the target model corresponding to the target degradation type;

[0008] Key convolutional kernels are determined based on the target model and the baseline model corresponding to the target degradation type;

[0009] The baseline model is trained based on the second training set, and the parameters of the key convolutional kernels are updated to obtain the image restoration model corresponding to the target degradation type.

[0010] According to the present invention, an image restoration model generation method based on continuous learning is provided, wherein determining key convolutional kernels based on the target model corresponding to the target degradation type and the baseline model includes:

[0011] The key convolutional kernel is determined based on the ensemble gradient of the convolutional kernel;

[0012] The ensemble gradient of the convolutional kernel is obtained based on the loss functions of the target model and the baseline model corresponding to the target degradation type.

[0013] According to the image restoration model generation method based on continuous learning provided by the present invention, the determination of the key convolutional kernel based on the ensemble gradient of the convolutional kernel includes:

[0014] The ensemble gradients of each convolutional level are obtained, and multiple target convolutional levels are determined based on the ensemble gradients of the convolutional levels. Each convolutional level includes at least one convolutional kernel, and the ensemble gradient of the convolutional level is determined based on the ensemble gradients of all convolutional kernels included in the convolutional level.

[0015] The key convolutional kernel is determined based on the target convolutional level.

[0016] According to the image restoration model generation method based on continuous learning provided by the present invention, the integrated gradient of the convolution kernel is determined based on a first formula or a second formula;

[0017] The first formula is:

[0018]

[0019] The second formula is:

[0020]

[0021] Among them, IG i (θ, x) represents the ensemble gradient of the i-th convolutional kernel in the model. α∈[0,1], γ(0)=θ, θ represents the parameters of the baseline model, and θ represents the parameters of the target model. It is a loss function used to measure the distance between the output of the model with parameter θ for the input sample image to be restored x and the restored image corresponding to x, where N is a constant.

[0022] According to the image restoration model generation method based on continuous learning provided by the present invention, the step of determining the key convolutional kernel based on the target convolutional level includes:

[0023] The target convolutional kernel is determined based on the ensemble gradient corresponding to each convolutional kernel in the target convolutional level;

[0024] Based on the cross-similarity of candidate convolutional kernels, the cross-similarity of candidate convolutional kernels reflects the similarity between candidate convolutional kernels and the similarity between candidate convolutional kernels and target convolutional kernels;

[0025] A supplementary set is determined based on the cross-similarity, and supplementary convolutional kernels are determined in the supplementary set, wherein the cross-similarity of the convolutional kernels in the supplementary set is lower than the cross-similarity of the other candidate convolutional kernels;

[0026] The target convolutional kernel and the supplementary convolutional kernel are used as the key convolutional kernel.

[0027] According to the image restoration model generation method based on continuous learning provided by the present invention, after obtaining the image restoration model corresponding to the target degradation type, the method further includes:

[0028] The image restoration model corresponding to the target degradation type is stored as the baseline model for the next degradation type.

[0029] The present invention also provides an image restoration method, comprising:

[0030] Obtain the image to be restored, input the image to be restored into the image restoration model corresponding to the image to be restored, and obtain the restored image output by the image restoration model;

[0031] The image restoration model is generated based on the continuous learning-based image restoration model generation method described in any of the preceding claims.

[0032] The present invention also provides an image restoration model generation device based on continuous learning, comprising:

[0033] The data acquisition module is used to acquire the baseline model and training set corresponding to the target degradation type. The training set includes a first training set and a second training set. The number of training data sets in the first training set is less than the number of training data sets in the second training set. Each training data set includes a sample image to be restored and a restored image corresponding to the sample image to be restored.

[0034] The pre-training module is used to train the baseline model corresponding to the target degradation type based on the first training set to obtain the target model corresponding to the target degradation type.

[0035] A key convolutional kernel localization module is used to determine key convolutional kernels based on the target model and the baseline model corresponding to the target degradation type.

[0036] The training module is used to train the baseline model based on the second training set, update the parameters of the key convolutional kernels, and obtain the image restoration model corresponding to the target degradation type.

[0037] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image restoration model generation method based on continuous learning as described above.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image restoration model generation method based on continuous learning as described above.

[0039] The image restoration model generation method and image restoration method based on continuous learning provided by this invention first train a baseline model using a small portion of training data for the target degradation type to obtain a target model. Then, the key convolutional kernels that play a crucial role in processing the target degradation type are identified using the target model and the baseline model. The model is then regressed to the baseline model, and a large amount of training data for the target degradation type is used to train the model based on the baseline model. At this time, only the parameters of the key convolutional kernels are updated, while the parameters of convolutional kernels that are not very useful for the restoration task of the target degradation type are not updated but frozen. In this way, for each degradation type, it is not necessary to design and train a completely new model. It is only necessary to learn some convolutional kernel parameters based on the baseline model, which improves the generation efficiency of the image restoration model. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating the image restoration model generation method based on continuous learning provided by the present invention;

[0042] Figure 2 This is a schematic diagram of the model learning process of the image restoration model generation method based on continuous learning provided by the present invention;

[0043] Figure 3 This is a schematic diagram illustrating the effect of the image restoration model generation method based on continuous learning provided by the present invention. Figure 1 ;

[0044] Figure 4 This is a schematic diagram illustrating the effect of the image restoration model generation method based on continuous learning provided by the present invention. Figure 2 ;

[0045] Figure 5This is a schematic diagram of the structure of the image restoration model generation device based on continuous learning provided by the present invention;

[0046] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0048] The following is combined Figures 1-4 This invention describes the image restoration model generation method based on continuous learning provided by the present invention.

[0049] like Figure 1 As shown, the image restoration model generation method based on continuous learning provided by the present invention includes the following steps:

[0050] S110. Obtain the baseline model and training set corresponding to the target degradation type. The training set includes a first training set and a second training set. The number of training data sets in the first training set is less than the number of training data sets in the second training set. Each training data set includes a sample image to be restored and the restored image corresponding to the sample image to be restored.

[0051] S120. Train the baseline model corresponding to the target degradation type based on the first training set to obtain the target model corresponding to the target degradation type;

[0052] S130. Determine key convolutional kernels based on the target model and baseline model corresponding to the target degradation type;

[0053] S140. Train the baseline model based on the second training set, update the parameters of the key convolutional kernels, and obtain the image restoration model corresponding to the target degradation type.

[0054] The baseline model corresponding to the target degradation type can be the image restoration model corresponding to the previous degradation type. That is, in the method provided by this invention, after training an image restoration model corresponding to one degradation type, it is used as the baseline model for the next degradation type. Specifically, after obtaining the image restoration model corresponding to the target degradation type, the method further includes storing the image restoration model corresponding to the target degradation type as the baseline model for the next degradation type. In existing technologies, a specific network model is designed and trained separately for each degradation type restoration task. This fails to maintain the restoration capability of old degradation types while learning new degradation types, leading to a catastrophic forgetting problem during continuous learning. For example, when a model learned for image deraining is fine-tuned on an image denoising dataset, it quickly forgets its deraining capabilities. However, in the embodiments of the method provided by this invention, such as... Figure 2 As shown, the next degradation type restoration task t+1 is learned based on the image restoration model corresponding to the previous degradation type restoration task t. Each time the image restoration model corresponding to the degradation type is learned, the key convolutional kernels closely related to the degradation type being learned are located, and only the parameters of these key convolutional kernels are updated, while the parameters of the remaining convolutional kernels are fixed. In this way, the restoration capability of the old degradation type can be maintained while learning the new degradation type, which can realize the effective message of the continuous image restoration task and avoid disaster forgetting.

[0055] The method provided by this invention, in order to achieve key shutdowns and localizations that are closely related to the task processing performance of the target degradation type, firstly pre-trains a baseline model on a dataset of the target degradation type, and then uses the pre-trained baseline model... The integrated gradient values ​​of the parameters of each convolutional kernel in the trained target model F are calculated. The key convolutional kernels are determined based on the integrated gradient values.

[0056] Specifically, key convolutional kernels are determined based on the target model and baseline model corresponding to the target degradation type, including:

[0057] Determining key convolutional kernels based on ensemble gradients of convolutional kernels;

[0058] The ensemble gradient of the convolution kernel is obtained based on the loss functions of the target model and the baseline model corresponding to the target degradation type.

[0059] The ensemble gradient of a convolutional kernel reflects the degree of change in its parameters during model training, and thus reflects the importance of the kernel in handling the target degradation type task. A higher ensemble gradient indicates greater importance of the kernel in recovering the target degradation type. To find convolutional kernels that play a crucial role in recovering specific degradation types, the method provided in this invention trains a baseline model that only learns to reconstruct the input image itself and cannot recover any degradation. Based on the baseline model, a small amount of pre-training is performed on a recovery task for a specific type of degradation to obtain the target model F. The ensemble gradient values ​​of all convolutional kernels are then removed from an intermediate model between these two models. The parameters of the intermediate model are determined by... The calculation yields the result where the parameter α∈[0,1], satisfying the condition... γ(0) = θ. Thus, for a given input x, the degree of change in the convolution kernel can be calculated using path integrals, thereby assessing its importance.

[0060]

[0061] in It is a loss function used to measure the distance between the output of a network model with parameters θ and the ground truth value for an input image x. The gradient of the i-th parameter in the model can be calculated using the first formula, which is:

[0062]

[0063] In practice, the integral of the first formula can be approximated by sampling discrete points uniformly along the path:

[0064]

[0065] That is, the ensemble gradient of the i-th convolutional kernel in the model can be calculated using the second formula, which is:

[0066]

[0067] Where N is the total number of steps in the approximate calculation of the ensemble gradient, and an empirical value is generally set to 100. In this way, the ensemble gradient can represent the contribution of the convolutional kernel to the model's recovery of a certain type of degradation.

[0068] In one possible implementation, the ensemble gradient of each convolutional kernel can be calculated separately, and the top few convolutional kernels with larger gradients can be selected to determine the key convolutional kernels. However, the combination of convolutional kernels in the model may have a greater impact. To improve the accuracy of key convolutional kernel location, the method provided in this invention adopts a coarse-to-fine approach. First, the ensemble gradient at the convolutional level is selected, and a certain proportion of candidate convolutions are obtained from the ensemble gradient values ​​at the convolutional level. Specifically, determining the key convolutional kernel based on the ensemble gradient of the convolutional kernels includes:

[0069] The ensemble gradients of each convolutional level are obtained, and multiple target convolutional levels are determined based on the ensemble gradients of the convolutional levels. Each convolutional level includes at least one convolutional kernel, and the ensemble gradients of the convolutional levels are determined based on the ensemble gradients of all convolutional kernels included in the convolutional level.

[0070] Key convolutional kernels are determined based on the target convolutional kernel.

[0071] A convolutional level consists of multiple convolutional kernels. The ensemble gradient of a convolutional level is the sum of the ensemble gradients of the kernels included in the convolutional level. A convolutional level with a higher ensemble gradient can be selected as the target convolutional level according to a preset ratio. The process of locating the target convolutional level can be expressed by the formula:

[0072]

[0073] Where P is the percentile, r k This is the preset key convolution kernel ratio, which can be set to 0.05, IG(θ) conv )=∑IG(θ filter ), It is the convolutional stage selected because it has a high ensemble gradient value, i.e., the target convolutional stage, IG(θ). conv ) represents the convolution level θ conv The integrated gradient, IG(θ) filter ) represents the convolution kernel θ filter The integrated gradient.

[0074] Determine key convolutional kernels based on the target convolutional level, including:

[0075] The target convolutional kernel is determined based on the ensemble gradients corresponding to each convolutional kernel in the target convolutional level.

[0076] Based on the cross-similarity of candidate convolutional kernels, the cross-similarity of candidate convolutional kernels reflects the similarity between candidate convolutional kernels and the similarity between candidate convolutional kernels and target convolutional kernels;

[0077] The supplementary set is determined based on cross-similarity, and supplementary convolutional kernels are determined in the supplementary set. The cross-similarity of the convolutional kernels in the supplementary set is lower than that of other candidate convolutional kernels.

[0078] The target convolutional kernel and the supplementary convolutional kernel are used as key convolutional kernels.

[0079] After determining the target convolutional level, ensemble gradient selection at the kernel level is performed. This kernel-level ensemble gradient further selects a certain proportion of key convolutional kernels for the entire model from the candidate convolutions. In other words, the ensemble gradient values ​​of the selected convolutional levels are compared again to locate the key convolutional kernels. This can be expressed by the formula:

[0080]

[0081] Where P is the percentile. It is a preliminarily selected convolutional kernel with a high ensemble gradient value, i.e., the target convolutional kernel, L={l1,l2,...,l c} represents the positions corresponding to the selected c target convolutional kernels. The parameters of the selected target convolutional kernels are only a small part of the entire model. Therefore, by locating a small number of key convolutional kernels for task t in the recovery model, we can further utilize them to learn new tasks through a convolutional kernel continuous learning mechanism, while achieving efficient utilization of model parameters.

[0082] In one possible implementation, the target convolutional kernel can be directly used as the key convolutional kernel. However, target convolutional kernels may have high similarity, leading to insufficient generalization. To address this issue, in embodiments of the method provided by this invention, diversity expansion is performed based on the target convolutional kernel, selecting an additional subset from the set of convolutional kernels that have not been selected as the target convolutional kernel. As a supplement, this part of the convolution kernel and The low similarity allows for the utilization of sufficient prior knowledge to enhance the performance of convolutional kernel learning. In convolutional kernel diversity augmentation, each convolutional kernel is divided into two parts based on whether it has been selected as a target kernel. The self-similarity and cross-similarity with the selected set are calculated in the unselected set, thus filtering a certain proportion of convolutional kernels as the diversity augmentation component. Specifically, the cross-similarity of candidate convolutional kernels can be determined by a third formula, which is:

[0083]

[0084] Where S represents the cross-similarity, S j Indicates candidate convolution kernels Cross-similarity, and This represents the set of target convolution kernels. Denotes the set of candidate convolutional kernels, and then, from Select supplementary convolution kernel:

[0085]

[0086] Where P is the percentile, r e This is the proportion of kernel diversity augmentation, typically set to 0.01. Then... This is used to generate the final key convolutional kernel corresponding to the current target degradation type.

[0087] After locating the key convolutional kernels, the model regresses to the baseline model before pre-training, freezing all non-key convolutional kernels (kernels other than the key kernels). The key convolutional kernels are then loaded into the corresponding positions in the model for forward inference and backpropagation. That is, the baseline model is trained using a second training dataset. During training, the parameters of the key convolutional kernels are updated, but the parameters of the non-key convolutional kernels are not. After learning the target degradation type, the key convolutional kernels corresponding to the target degradation type are added to the maintained convolutional kernel library for knowledge storage and training for the next task. In the image restoration model generation method proposed in this invention, only a small number of weight matrix parameters W and newly expanded learnable convolutional kernels need to be learned for each task. To update the key convolutional kernel This avoids designing and training a separate image restoration model for each degradation type. Furthermore, a convolutional kernel library is maintained, using kernels from previous degradation tasks as prior knowledge in the continuous learning process. Before training for subsequent tasks, the model is first regressed from the target model in the key convolutional kernel localization phase to the model after training the previous task, serving as the baseline model. This continuous learning mechanism, by preserving the corresponding key convolutional kernel parameters for each task, completely avoids the catastrophic forgetting problem with only a minimal increase in parameters.

[0088] When learning task t, the convolutional kernel learning process can be divided into two stages: kernel generation and kernel ensemble. Formally, the kernel generation process can be represented as:

[0089]

[0090] in and These are the reshaping operation and its inverse operation; square brackets indicate the join operation. The key convolutional kernel is generated for task t, and the subscript L indicates that it is learnable. After obtaining the key convolutional kernel from formula (8), the model can be assembled from the key convolutional kernel positions obtained in formula (5) to obtain a complete model for inference. In this way, new image restoration tasks can be learned with extremely high efficiency, catastrophic forgetting can be completely avoided, and new tasks can be learned effectively by retaining a small number of parameters during continuous image restoration.

[0091] Based on the above-described image restoration model generation method based on continuous learning, the present invention also provides an image restoration method, comprising: acquiring an image to be restored, inputting the image to be restored into an image restoration model corresponding to the image to be restored, and acquiring the restored image output by the image restoration model; wherein the image restoration model is generated based on the image restoration model generation method based on continuous learning as described above.

[0092] To verify the effectiveness of the image restoration model generation method based on continuous learning provided in this invention, comparative experiments were conducted with a series of existing continuous learning methods. In the experiments, the DIV2K dataset, containing 900 high-resolution images, was used. Degraded images were generated for different restoration tasks by adding rain streaks, Gaussian noise, and Gaussian blur to the original images. In each restoration task, 700 images were used for training, 100 images for validation, and 100 images for testing. Furthermore, the training and validation images were randomly cropped into 176×176 image patches, and the test images were randomly cropped into 256×256 image patches.

[0093] Quantitative analysis: Quantitative results are as follows Figure 3 As shown in the diagram. Upper-bound represents joint learning across all task data, indicating the highest achievable performance. SFT represents simply initializing the model parameters from the previous task and then fine-tuning them on the new task. EWC and LwF are regularization-based methods, while iCaRL is a classic replay-based method. PiggybackGAN is a conditional image generation method. Furthermore, two continuation learning methods in advanced vision are compared: ELI and SOIF, belonging to regularization-based and replay-based methods, respectively. ELI can be combined with other baselines, so three combinations are compared. Figure 3 The result is the result of continuous learning on the three tasks in the order of rain removal, noise removal, and deblurring. Figure 3 The results listed are the performance of the recovery model when it was trained up to the last task.

[0094] It can be seen that the method provided by this invention outperforms other methods in all three tasks. However, regularization-based methods such as EWC and LwF perform poorly, exhibiting some catastrophe forgetting and struggling to learn new tasks due to the constraints of the regularization terms. Replay-based methods such as iCaRL and SOIF show significant catastrophe forgetting on older tasks as the number of learning tasks increases, due to the decreasing representativeness of the buffered replay data. Furthermore, although PiggybackGAN also performs well, this comes at the cost of a large number of new parameters, resulting in low efficiency.

[0095] Qualitative analysis: Randomly sampled test set results from different methods are compared, and the results are as follows: Figure 4 As shown, Figure 4A visualization comparison of six randomly selected image samples from the test set is provided. Clearly, the regularized EWC performs poorly on new tasks, while the replay-based SOIF exhibits significant catastrophic forgetting on older tasks. The method provided in this invention achieves the best recovery performance across all compared methods for all tasks, thanks to its key convolutional kernel localization strategy and efficient convolutional kernel learning mechanism.

[0096] The image restoration model generation apparatus based on continuous learning provided by the present invention is described below. The image restoration model generation apparatus based on continuous learning described below can be referred to in correspondence with the image restoration model generation method based on continuous learning described above. Figure 5 As shown, the image restoration model generation device based on continuous learning provided by the present invention includes:

[0097] The data acquisition module 510 is used to acquire the baseline model and training set corresponding to the target degradation type. The training set includes a first training set and a second training set. The number of training data sets in the first training set is less than the number of training data sets in the second training set. Each training data set includes a sample image to be restored and the restored image corresponding to the sample image to be restored.

[0098] The pre-training module 520 is used to train the baseline model corresponding to the target degradation type based on the first training set to obtain the target model corresponding to the target degradation type.

[0099] The key convolutional kernel localization module 530 is used to determine the key convolutional kernel based on the target model and baseline model corresponding to the target degradation type;

[0100] Training module 540 is used to train the baseline model based on the second training set, update the parameters of the key convolutional kernels, and obtain the image restoration model corresponding to the target degradation type.

[0101] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a continuous learning-based image restoration model generation method. This method includes: acquiring a baseline model and a training set corresponding to the target degradation type, the training set including a first training set and a second training set, the first training set having fewer training data sets than the second training set, each training data set including a sample image to be restored and a restored image corresponding to the sample image to be restored; training the baseline model corresponding to the target degradation type based on the first training set to obtain a target model corresponding to the target degradation type; determining key convolutional kernels based on the target model and the baseline model corresponding to the target degradation type; training the baseline model based on the second training set, updating the parameters of the key convolutional kernels, and obtaining an image restoration model corresponding to the target degradation type.

[0102] The processor 610 can also call logic instructions in the memory 630 to execute an image restoration method, which includes: acquiring an image to be restored, inputting the image to be restored into an image restoration model corresponding to the image to be restored, and acquiring the restored image output by the image restoration model; wherein the image restoration model is generated based on the image restoration model generation method based on continuous learning as described above.

[0103] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the image restoration model generation method or image restoration method based on continuous learning provided by the above methods. The image restoration model generation method based on continuous learning includes: obtaining a baseline model and a training set corresponding to a target degradation type, wherein the training set includes a first training set and a second training set, the first training set having fewer training data sets than the second training set, and each training data set including a sample image to be restored and a restored image corresponding to the sample image to be restored; training the baseline model corresponding to the target degradation type based on the first training set to obtain a target model corresponding to the target degradation type; determining key convolutional kernels based on the target model and the baseline model corresponding to the target degradation type; and training the baseline model based on the second training set, updating the parameters of the key convolutional kernels to obtain an image restoration model corresponding to the target degradation type. The image restoration method includes: acquiring the image to be restored, inputting the image to be restored into the image restoration model corresponding to the image to be restored, and acquiring the restored image output by the image restoration model; wherein, the image restoration model is generated based on the image restoration model generation method based on continuous learning as described above.

[0105] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image restoration model generation method or image restoration method based on continuous learning provided by the above methods. The image restoration model generation method based on continuous learning includes: acquiring a baseline model and a training set corresponding to a target degradation type; the training set includes a first training set and a second training set, wherein the number of training data sets in the first training set is less than the number of training data sets in the second training set; each training data set includes a sample image to be restored and a restored image corresponding to the sample image to be restored; training the baseline model corresponding to the target degradation type based on the first training set to obtain a target model corresponding to the target degradation type; determining key convolutional kernels based on the target model and the baseline model corresponding to the target degradation type; and training the baseline model based on the second training set, updating the parameters of the key convolutional kernels to obtain an image restoration model corresponding to the target degradation type. The image restoration method includes: acquiring an image to be restored; inputting the image to be restored into the image restoration model corresponding to the image to be restored; and acquiring the restored image output by the image restoration model; wherein the image restoration model is generated based on the image restoration model generation method based on continuous learning as described above.

[0106] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating image restoration models based on continuous learning, characterized in that, include: Obtain the baseline model and training set corresponding to the target degradation type. The training set includes a first training set and a second training set. The number of training data sets in the first training set is less than the number of training data sets in the second training set. Each training data set includes a sample image to be restored and a restored image corresponding to the sample image to be restored. The baseline model corresponding to the target degradation type is trained based on the first training set to obtain the target model corresponding to the target degradation type; Key convolutional kernels are determined based on the target model and the baseline model corresponding to the target degradation type; The baseline model is trained based on the second training set, and the parameters of the key convolutional kernels are updated to obtain the image restoration model corresponding to the target degradation type. The step of determining key convolutional kernels based on the target model and the baseline model corresponding to the target degradation type includes: The key convolutional kernel is determined based on the ensemble gradient of the convolutional kernel; The ensemble gradient of the convolutional kernel is obtained based on the loss functions of the target model and the baseline model corresponding to the target degradation type.

2. The image restoration model generation method based on continuous learning according to claim 1, characterized in that, The determination of the key convolutional kernel based on the ensemble gradient of the convolutional kernel includes: The ensemble gradients of each convolutional level are obtained, and multiple target convolutional levels are determined based on the ensemble gradients of the convolutional levels. Each convolutional level includes at least one convolutional kernel, and the ensemble gradient of the convolutional level is determined based on the ensemble gradients of all convolutional kernels included in the convolutional level. The key convolutional kernel is determined based on the target convolutional level.

3. The image restoration model generation method based on continuous learning according to claim 1, characterized in that, The ensemble gradient of the convolution kernel is determined based on either the first or the second formula; The first formula is: The second formula is: Among them, IG i (θ,x) represents the ensemble gradient of the i-th convolutional kernel in the model. α∈[0,1], γ(0)=θ, θ represents the parameters of the baseline model, and θ represents the parameters of the target model. It is a loss function used to measure the distance between the output of the model with parameter θ for the input sample image to be restored x and the restored image corresponding to x, where N is a constant.

4. The image restoration model generation method based on continuous learning according to claim 2, characterized in that, Determining the key convolutional kernel based on the target convolutional level includes: The target convolutional kernel is determined based on the ensemble gradient corresponding to each convolutional kernel in the target convolutional level; Obtain the cross-similarity of candidate convolutional kernels, whereby the cross-similarity of candidate convolutional kernels reflects the similarity between candidate convolutional kernels and the similarity between candidate convolutional kernels and the target convolutional kernel; A supplementary set is determined based on the cross-similarity, and supplementary convolutional kernels are determined in the supplementary set, wherein the cross-similarity of the convolutional kernels in the supplementary set is lower than the cross-similarity of the other candidate convolutional kernels; The target convolutional kernel and the supplementary convolutional kernel are used as the key convolutional kernel.

5. The image restoration model generation method based on continuous learning according to claim 1, characterized in that, After obtaining the image restoration model corresponding to the target degradation type, the method further includes: The image restoration model corresponding to the target degradation type is stored as the baseline model for the next degradation type.

6. An image restoration method, characterized in that, include: Obtain the image to be restored, input the image to be restored into the image restoration model corresponding to the image to be restored, and obtain the restored image output by the image restoration model; The image restoration model is generated based on the continuous learning-based image restoration model generation method as described in any one of claims 1-5.

7. An image restoration model generation device based on continuous learning, characterized in that, include: The data acquisition module is used to acquire the baseline model and training set corresponding to the target degradation type. The training set includes a first training set and a second training set. The number of training data sets in the first training set is less than the number of training data sets in the second training set. Each training data set includes a sample image to be restored and a restored image corresponding to the sample image to be restored. The pre-training module is used to train the baseline model corresponding to the target degradation type based on the first training set to obtain the target model corresponding to the target degradation type. A key convolutional kernel localization module is used to determine key convolutional kernels based on the target model and the baseline model corresponding to the target degradation type. The training module is used to train the baseline model based on the second training set, update the parameters of the key convolutional kernel, and obtain the image restoration model corresponding to the target degradation type. The step of determining key convolutional kernels based on the target model and the baseline model corresponding to the target degradation type includes: The key convolutional kernel is determined based on the ensemble gradient of the convolutional kernel; The ensemble gradient of the convolutional kernel is obtained based on the loss functions of the target model and the baseline model corresponding to the target degradation type.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image restoration model generation method based on continuous learning as described in any one of claims 1 to 5, or implements the image restoration method as described in claim 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image restoration model generation method based on continuous learning as described in any one of claims 1 to 5, or implements the image restoration method as described in claim 6.

Images