Training method and apparatus for image restoration task, and electronic device

Abstract

A training method and apparatus for an image restoration model, and an electronic device, which are applied to the technical field of image processing. The method comprises: acquiring a raw image data set; performing a degradation operation on raw images in the raw image data set, so as to obtain different degraded images; inputting the degraded images into a generative network, so as to obtain first generated images, and on the basis of the first generated images and the raw images, performing a first model parameter update on the generative network; inputting the first generated images into the generative network that has been subjected to the model parameter update, so as to generate second generated images; on the basis of the first generated images and the second generated images, performing a second model parameter update on the generative network; and repeatedly executing the operation of performing a second model parameter update on the generative network until the generative network meets the requirements for an image restoration task. A visual operation interface corresponding to the training method is provided.

Description

Training method, device and electronic equipment for image restoration tasks Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image restoration model training method, device and electronic equipment. Background Art

[0002] In the field of image / video restoration, factors that lead to poor image quality include blur, noise, compression, and other factors, and their intensity and types vary greatly. In addition, the AI ​​development platforms or AI training platforms currently on the market all integrate common model training methods corresponding to a single type of image restoration method.

[0003] Summary of the Invention

[0004] To address the above technical issues, the present invention provides a two-stage self-knowledge distillation training method, device, and electronic device for image restoration, which can meet and adapt to the needs of complex and changing image restoration scenarios. The specific technical solution is as follows:

[0005] In a first aspect, the present invention provides a training method for an image restoration task, the method comprising:

[0006] Get the original image dataset;

[0007] performing a degradation operation on the original images in the original image dataset to obtain different degraded images;

[0008] Inputting the degraded image into a generation network to obtain a first generated image;

[0009] updating first model parameters of the generative network according to the first generated image and the original image;

[0010] Inputting the first generated image into the generation network after the model parameters are updated to generate a second generated image;

[0011] updating a second model parameter of the generative network according to the first generated image and the second generated image;

[0012] Repeat the operation of updating the second model parameters of the generated network until the generated network meets the requirements of the image restoration task.

[0013] In a second aspect, an embodiment of the present invention discloses an image restoration method, the method comprising:

[0014] Input the image to be repaired into the trained image repair model and output the repaired image;

[0015] The image restoration model is trained using the training method for the image restoration task.

[0016] In a third aspect, an embodiment of the present invention provides a display control method, the method comprising:

[0017] Select the original image for training the image restoration network;

[0018] In response to detecting an operation of selecting a degraded preview, displaying a degraded image corresponding to the original image in a display interface;

[0019] In response to detecting an operation of selecting to start training, displaying a generated image corresponding to the degraded image and a loss function curve during the training process in a display interface;

[0020] Wherein, the generated image includes a first generated image and a second generated image; the second generated image is obtained by training the first generated image through the training method of the image restoration task described in any one of claims 1-9; the loss function curve includes a first loss function curve between the original image and the first generated image and a second loss function curve between the first generated image and the second generated image.

[0021] In a fourth aspect, the present invention provides an image restoration model training device, comprising:

[0022] An image degradation module is used to perform degradation operations on the original images in the original image dataset using a preset degradation method;

[0023] An image generation module is configured to use a pre-set generation network model to repair the image input to the image generation module and output a repaired generated image;

[0024] A parameter updating module, used to update the parameters of the generated network model according to the training results;

[0025] An image segmentation module, configured to randomly divide the original image into a plurality of different regions;

[0026] The region selection module is used to select a region that needs to be repaired in the region.

[0027] In a fifth aspect, the present invention provides an electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0028] Memory for storing computer programs;

[0029] The processor is configured to implement any one of the method steps described in the first aspect and the second aspect when executing the program stored in the memory.

[0030] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any method step described in the first aspect is implemented.

[0031] The training method for image restoration tasks provided by the embodiments of the present invention can be adapted to various image restoration model training methods for low-quality images, enabling targeted restoration of different types of images, thereby improving the completion rate of image restoration tasks. Furthermore, the embodiments of the present invention provide a display control method that, in combination with existing technologies, enables visual operation and monitoring of image restoration model training tasks, facilitating the control of the restoration process by those skilled in the art when performing image restoration tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0033] FIG1 is a flow chart of the image restoration task training method provided by the present invention;

[0034] Figure 2 is a schematic diagram of the traditional algorithm degradation scheme for classic images;

[0035] FIG3 is a schematic diagram of a traditional video algorithm degradation solution;

[0036] FIG4 is a first image degradation method provided by an embodiment of the present invention;

[0037] FIG5 is a second image degradation method provided by an embodiment of the present invention;

[0038] FIG6 is a third image degradation method provided by an embodiment of the present invention;

[0039] FIG7 is a schematic diagram of updating the first model parameters of the generator network in the image restoration network provided by an embodiment of the present invention;

[0040] FIG8 is a schematic diagram of updating the second model parameters of the generator network in the image restoration network provided by an embodiment of the present invention;

[0041] FIG9 is a schematic diagram of the process of self-knowledge distillation;

[0042] FIG10 is a schematic diagram of a training scheme for a first image restoration model provided by an embodiment of the present invention;

[0043] FIG11 is a schematic diagram of a training scheme for a second image restoration model provided by an embodiment of the present invention;

[0044] FIG12 is a flow chart of a display control method provided by an embodiment of the present invention;

[0045] FIG13 is an initial display interface of the display control method provided by an embodiment of the present invention;

[0046] FIG14 is a schematic diagram of displaying a degraded image in a display interface according to an embodiment of the present invention;

[0047] FIG15 is a first degraded image display interface provided by an embodiment of the present invention;

[0048] FIG16 is a second degraded image display interface provided by an embodiment of the present invention;

[0049] FIG17 is a schematic diagram of displaying training results in a display interface according to an embodiment of the present invention;

[0050] FIG18 is a training visualization interface provided by an embodiment of the present invention;

[0051] FIG19 is a loss function curve visualization interface provided by an embodiment of the present invention;

[0052] FIG20 is a test mode interface provided by an embodiment of the present invention;

[0053] FIG21 is a schematic diagram of an image restoration device provided by an embodiment of the present invention;

[0054] FIG22 is a schematic diagram of a display control device provided by an embodiment of the present invention;

[0055] FIG23 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of the present invention.

[0057] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0058] The image restoration method provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0059] As shown in FIG1 , the present invention provides a training method for an image restoration task, comprising steps S101 to S104, wherein:

[0060] S101, obtaining an original image dataset.

[0061] In this step, the original image dataset includes multiple high-quality images or video frames. High-quality images or video frames refer to images or video frames that achieve the expected quality in the image restoration task. The expected quality refers to images with an appropriate resolution determined based on common knowledge in the art. For example, in an image restoration task targeting a 2K display device, the image resolution that achieves the expected quality is 2560×1440; in an image restoration task targeting a 4K display device, the image resolution that achieves the expected quality is 4096×2160, and so on.

[0062] Optionally, the above-mentioned original image can be a portrait image, a face image, an oil painting image, a multi-target image, a medical image, etc. Optionally, the above-mentioned image to be repaired can be a picture / video taken by a terminal, and the terminal can include, for example, a mobile phone, a tablet, a small camera, a large camera, etc. It is also suitable for surveillance videos, such as traffic monitoring, and indoor venue surveillance image processing. The content of the above-mentioned image to be repaired can include movie images, TV series images, variety show images, documentary images, sports event images, surveillance images, cartoons, art paintings, etc. The image restoration tasks mentioned in the present invention include all low-level computer vision training tasks, such as super-resolution, noise reduction, deblurring, compression restoration, face restoration, etc.

[0063] S102: Perform a degradation operation on multiple original images in the original image data set to obtain different degraded images.

[0064] Degradation involves applying random blurring, random upsampling and downsampling, random noise generation, and JPEG compression to a high-definition original image, resulting in a low-quality image. Common degradation schemes for classic images using traditional algorithms are shown in Figure 2. In this scheme, there are no restrictions on the choice of degradation scheme; either traditional degradation schemes or deep learning models can be used for degradation.

[0065] To extend this degradation scheme to video degradation, the original high-definition video can be first compressed and encoded using different encoders with different quality parameters / bitrate sizes, and then the compressed and encoded video frame information can be degraded according to a preset degradation scheme. A common degradation scheme for video is shown in Figure 3.

[0066] Here, different degraded images refer to images of varying degrees of degradation obtained by performing different degradation operations on the same original image. Optionally, the images of varying degrees of degradation may be images of varying degrees of degradation obtained by performing one or more of image compression with different parameters / bitrates, image blurring with varying degrees, upsampling or downsampling with varying degrees, or noise processing. The specific degradation operation type, processing steps, and sequence are not limited.

[0067] An embodiment S102a of a degradation operation proposed in this solution, as shown in FIG4 , inputs the entire high-quality image 10 into a degrader, performs different degradation operations, and outputs low-quality images 11, 12, and 13 of varying degrees; wherein the low-quality images 11, 12, and 13 may be obtained by processing using the same degradation processing means to varying degrees. For example, when the resolution of the high-quality image 10 is 4096×2160, the low-quality image 11 may be obtained by downsampling the high-quality image 10 to a resolution of 2560×1440, the low-quality image 12 may be obtained by downsampling the high-quality image 10 to a resolution of 1920×1080, and the low-quality image 13 may be obtained by downsampling the high-quality image 10 to a resolution of 1280×1024. The low-quality images 11, 12, and 13 may also be obtained using different degradation processing methods. For example, the low-quality image 11 may be obtained by Gaussian blurring the corresponding high-quality image 10, the low-quality image 12 may be obtained by downsampling the corresponding high-quality image 10, and the low-quality image 13 may be obtained by adding noise to the corresponding high-quality image 10.

[0068] This solution proposes another degradation operation embodiment S102b, as shown in Figure 5. High-quality image 20 is randomly divided into regions, and different regions are degraded according to different degradation operations. For example, region 2a can be obtained by downsampling the corresponding region of high-quality image 20, region 2b can be obtained by Gaussian blurring the corresponding region of high-quality image 20, region 2c can be obtained by Gaussian blurring the corresponding region of high-quality image 20, and image 2d can be obtained by inverse filtering the corresponding region of high-quality image 20. This degradation scheme allows for the representation of multiple degradation forms within the same degradation image, allowing for the addition of more complex scenarios during training, thereby improving the model's adaptability to various low-quality image restoration tasks.

[0069] In addition to the above image restoration scenarios, the applicant has also discovered that restoring large, high-resolution images consumes a significant amount of computing resources. In reality, not all content in an image requires restoration. For example, in images containing text, due to issues such as small text size, the text may not be accurately recognized, while other information in the image is relatively clear. Alternatively, in some photo restoration scenarios, issues such as focusing during shooting may result in different sharpness levels for different objects in the image. In these situations, restoring the entire image not only wastes computing resources, but also, due to the varying resolutions of different regions, applying a uniform standard to the entire image may result in poor restoration or over-restoration of certain regions. How to restore only specific regions, thereby achieving the goal of image restoration while minimizing computing resource consumption and improving image restoration efficiency, has become a pressing technical issue. To address this issue, this solution proposes another degradation operation, S102c, as shown in FIG6 . The high-quality image 30 is randomly divided into regions, and n randomly selected blocks are degraded, without degrading all blocks. Specifically, during training, a mask is added to the selected n blocks. When calculating the loss during training, only the loss for the masked portion is calculated, while the loss for the remaining ground-truth portions is not calculated. This degradation scheme improves the model's ability to distinguish low-quality areas in the image, effectively repairing the areas that need repair, and significantly reduces computing resources.

[0070] After obtaining the degraded image in step S102, step S103 of updating the first model parameters of the generation network in the image restoration network is performed according to the degraded image and the original image, as shown in FIG7 , which includes:

[0071] S1031, inputting the plurality of degraded images into a generation network to obtain first generated images, wherein the plurality of first generated images constitute a first generated image set;

[0072] S1032: Update first model parameters of the generation network according to the first generated image and the original image.

[0073] In this step, the degraded images obtained in step S102 are input into the generative network to obtain first generated images. The number of the first generated images is the same as the number of the degraded images input into the generative network. Compared to the degraded images, the first generated images are closer to the original images in the original image dataset in terms of image size, color, and brightness.

[0074] The generative network is used to perform the image restoration task. Image restoration refers to the process of restoring missing portions of damaged or missing image data through algorithmic means, making the image more visually complete and natural. Traditional image restoration methods, including texture synthesis and interpolation, typically use similarity algorithms to select image patches from other areas of the image for completion. Given a small texture sample, a large, visually similar image is generated.

[0075] Traditional image methods have limited learning capabilities and are unable to effectively repair relatively serious image blocks. Currently, ideas based on deep learning models are gradually being applied to image restoration. Deep learning models have stronger feature extraction and learning capabilities. They can capture the feature information corresponding to the image area to be restored from a large number of image samples, thereby filling the image area to be restored. Early image restoration methods based on deep learning models used convolutional neural networks (CNNs) to train the neural network's ability to extract image features through supervised learning. The labeled image was input into the model, and it passed through several convolutional layers and pooling layers in sequence to complete the shallow and deep feature extraction of a certain area. The fully connected layers were used to complete several classification tasks. Classic convolutional neural network models such as AlexNet, VGG, and Resnet can all be used for image restoration tasks.

[0076] Image restoration based on a generative adversarial network (GAN) architecture is suitable for learning and generating target region features from a small amount of data to complete image restoration. A GAN (Generative Adversarial Network) is an unsupervised deep learning model used to generate data through computer-generated learning. The model learns to produce better outputs through a game of mutual competition between at least two modules: a generative model and a discriminative model. A GAN consists of two key components: a generator (G) and a discriminator (D). The generator generates data through machine learning, aiming to "fool" the discriminator as much as possible. The generated data is denoted as G(z). The discriminator determines whether the data is real or generated by the generator, aiming to identify "fake" data as much as possible. Its input parameter is x, representing the data, and the output D(x) represents the probability that x is real data. A value of 1 indicates 100% authenticity, while an output of 0 indicates that it cannot be real data. In this way, G and D form a dynamic adversarial relationship. As training progresses, the data generated by G becomes increasingly similar to real data, while D's ability to discriminate against data becomes increasingly sophisticated. The GAN training process consists of two phases. In the first phase, the discriminator D is fixed and the generator G is trained. Using a well-performing discriminator, G continuously generates "fake data" and then gives it to D to judge. Initially, G is weak and easily discriminated against. However, as training progresses, G's skill improves, eventually fooling D. In the second phase, the generator G is fixed and the discriminator D is trained. After passing the first phase, the generator G is fixed and D is trained. Through continuous training, D improves its discrimination ability, eventually becoming able to accurately detect fake data. Phases 1 and 2 are repeated. Through continuous cycles, the capabilities of both the generator G and the discriminator D become increasingly powerful. Ultimately, a generator G that meets the requirements is obtained and can be used to generate data.

[0077] The generation network can select a mature image restoration network model in the prior art, which can be a traditional image restoration method or a method based on deep learning. The type of the generation network model is not further limited in this application.

[0078] Similarly, the image restoration method based on the deep learning model as the generative network can also select the loss function in the existing technology. For example, at least one loss function such as L1 loss (a mean absolute error loss function), MSE loss (Mean Squared Error Loss), and Charbonnier loss (an image reconstruction loss function) can be used to calculate the difference between the pixel values ​​of the restored image and the true value image.

[0079] After completing the first model parameter update of the generative network, step S104 of updating the second model parameters of the generative network in the image restoration network is performed according to the first generated image and the generative network after the updated model parameters, as shown in FIG8 , which includes:

[0080] S1041, inputting the first generated image into the generative network after the model parameters are updated to generate a second generated image;

[0081] S1042: Update second model parameters of the generative network according to the first generated image and the second generated image.

[0082] In this step, the self-knowledge distillation method is used to train the model. The flow chart of knowledge distillation is shown in Figure 9. Under complex degradation conditions, network training is difficult. Generally, it is easier to use the student model to learn the output of the teacher model than to use the student model to directly learn high-quality images. In the present invention, the generative network after the first model parameter update is completed is used as the teacher model, and the second generated image output by the teacher model is used as a benchmark for comparative learning with the first generated image. The use of the self-knowledge distillation training method is more conducive to the training of complex repair tasks. It can reduce the computing power loss of the training process while enabling the model to learn the features of the original image to the greatest extent, thereby achieving the purpose of lightweight training.

[0083] FIG10 shows an embodiment of a training scheme for an image restoration model of the present invention. Specifically, after the generative network generates multiple first generated images, a first generated image is selected from the first generated image set as the third generated image; wherein, the selection of the third generated image is determined by at least one of the following factors: the priority of the feature to be restored; or, the richness of the image features. The priority of the feature to be restored refers to the priority selection of features corresponding to the target that is the focus or needs to be restored in the multi-target image to be restored. For example, in an image with multiple animals, the animal we are interested in is a cat, then the image features of the area where all cats are located in the image have the priority for restoration; for another example, in an image that needs to recognize text, the image features of the text area in the image have the priority for restoration. The above embodiments are only used to assist in understanding the concept of the priority of the feature to be restored in the present invention, and do not represent all embodiments. Reasonable adjustments can be made according to needs during the specific operation process, and are not limited here.

[0084] After the third generated image is selected, a first loss calculation is performed on the third generated image and the original image; and a first model parameter of the generation network is updated according to the first loss calculation result.

[0085] The specific calculation method of the first loss is: based on the generated repaired image and the true value image, calculating the difference in pixel values ​​between the repaired image and the true value image as the first loss value;

[0086] Taking L1 loss as an example, the first loss calculation function formula can be expressed as: loss1=L1(out 11 ,G1)

[0087] Among them, G1 represents the original image, out 11 represents the third generated image.

[0088] The first loss calculation represents the difference between the third generated image and the original image. During model training, it is desirable for this difference to be as small as possible, indicating that the output image is closer to the true image. Specifically, when processing the degraded image obtained through the degradation operation embodiments S102a and S102b, the loss calculation is performed directly on the entire image. The loss calculation function using the L1 loss method is as follows:

[0089] Where C represents the number of image channels. A typical RGB image has three channels, so C = 3; a grayscale image has a single channel, so C = 1. H represents the image height; W represents the image width. f() represents the algorithm used in this patent solution, and y represents the corresponding true value.

[0090] For the degraded image obtained in the degradation operation embodiment S102c, when calculating the loss function, only a specific area needs to be calculated, so it is necessary to add mask information when calculating the loss. Define the mask as M, and the loss calculation function formula of the L1 loss is expressed as:

[0091] Where M represents the mask information, and its value is 0 and 1, representing no mask added and mask added, respectively; * represents the pixel-to-pixel product. The loss is calculated in the part where the M value is 1 (selected block), and the loss is not calculated in the area where the value is 0 (unselected block).

[0092] After obtaining the result of the first loss calculation, a first model parameter update is performed on the generative network to obtain a first generative network.

[0093] Then, the third generated image is input into the first generated network to generate a fourth generated image; the first generated images except the third generated image are respectively subjected to multiple second loss calculations with the fourth generated image. Taking L1 loss as an example, the general formula of the second loss calculation function can be expressed as: loss′2=L1(out 1n ,out′ 11 )

[0094] Among them, out′ 11 represents the fourth generated image, out 1n represents any one of the first generated images except the third generated image.

[0095] The model parameters of the first generation network are updated according to the multiple second loss calculations to obtain a second generation network.

[0096] After performing the multiple second loss calculations, it is necessary to sum the multiple second loss calculations as the total loss and use this to update the parameters of the first generation network. The total loss calculation function can be expressed as follows: loss2 = L1 (out 12 ,out′ 11 )+L1(out 13 ,out′ 11 )+…+L1(out 1n ,out′ 11 )

[0097] Among them, out 12 , out 13 …out 1n The first generated image excluding the third generated image is represented by [ 0 ]. The model parameters of the first generated network are updated according to the total loss calculation result to obtain a second generated network.

[0098] Here, the second loss calculation and the total loss calculation represent the difference between the fourth generated image and the first generated image excluding the third generated image. The second loss calculation and the total loss calculation are similar to the first loss calculation and will not be elaborated here.

[0099] Through this embodiment, the first generative network can be given the ability to capture the characteristics of a certain type of target in the image to be repaired to the greatest extent possible, so that in subsequent steps, the first generative network can meet the requirements of the teacher model and complete the self-knowledge distillation training scheme proposed by the present invention. At the same time, the first generative network can be given a good ability to capture the characteristics of the content that needs to be repaired first in the image to be repaired, thereby adapting to targeted repair tasks; wherein, the content that needs to be repaired first can be the main target in the image to be repaired. For example, in an image to be repaired containing multiple animals, the majority of the animal species are cats, and the position of the cat in the image can be the priority repair location.

[0100] It should be noted that the model parameter update operation is completed only after the difference between the generated repaired image and the real image reaches the preset requirements. It does not mean that the parameter update is completed by only calculating the loss once. The model parameter update is carried out according to the difference in pixel values ​​between the generated repaired image and the real image. It is an iterative process until the difference between the repaired image and the real image reaches the preset requirements. Generally, there is a threshold for the number of iterations, and this application does not limit the number of iterations.

[0101] Optionally, FIG11 shows an embodiment of another degradation scheme of the present solution. Specifically, after the generative network generates a first generated image set, a third loss calculation is performed on the images in the first generated image set and the original image respectively.

[0102] performing a third model parameter update of the generative network according to the third loss calculation to obtain a third generative network;

[0103] The third loss calculation includes multiple calculation results, the number of which corresponds to the number of first generated images. For example, after the original image passes through the degrader, three different degraded images are obtained. These three degraded images are input into the generation network to obtain three generated images, and the third loss calculation is performed on each of the three generated images compared with the original image. The third loss calculation is similar to the first loss calculation, both of which calculate the difference in pixel values ​​between the restored image and the ground truth image.

[0104] Taking L1 loss as an example, the third loss calculation function can be expressed as follows: loss3 = L1 (out 11 ,G1)+L1(out 12 ,G1)+L1(out 13 ,G1)+…+L1(out 1n ,G1)

[0105] Among them, G1 represents the original image, out 11 , out 12 , out 13 …out 1n The third loss calculation represents the difference between the original image and the plurality of first generated images. The third loss calculation also uses the L1 loss function. The specific calculation formula is similar to that of the embodiment shown in FIG11 and is not repeated here.

[0106] After obtaining the result of the third loss calculation, the third model parameters of the generative network are updated to obtain a third generative network, wherein the parameter updating method is similar to the embodiment shown in FIG11 and will not be described in detail here.

[0107] The images in the first generated image set are input into the third generation network to generate a second generated image set.

[0108] Among them, the first generated image is out 11 , out 12 , out 13 …out 1n For all images, the number of second generated images generated is the same as the number of the first generated images.

[0109] selecting a second generated image from the second generated images as a fifth generated image;

[0110] The fourth loss calculation is performed multiple times between the second generated images except the fifth generated image and the fifth generated image.

[0111] Similar to the embodiment shown in FIG11 , the selection of the fifth generated image in this embodiment is determined by at least one of the following factors: the priority of the feature to be repaired; or the richness of the image features. Also taking L1 loss as an example, the fourth loss calculation function can be expressed as: loss′4=L1(out′ 1n ,out′ 11 )

[0112] Among them, out′ 11 represents the fifth generated image, out 1nrepresents any one of the first generated images except the third generated image.

[0113] S1043b: Update the fourth model parameters of the third generative network according to the multiple fourth loss calculations to obtain a fourth generative network.

[0114] After performing the multiple second loss calculations, it is necessary to sum the multiple second loss calculations as the total loss and use this to update the parameters of the third generation network. The total loss calculation function can be expressed as follows: loss4 = L1(out′ 12 ,out′ 11 )+L1(out′ 13 ,out′ 11 )+…+L1(out′ 1n ,out′ 11 )

[0115] Among them, out 12 , out 13 …out 1n represents the first generated image excluding the third generated image. The model parameters of the third generated network are updated based on the total loss calculation result to obtain a fourth generated network. The fourth loss calculation also uses the L1 loss function. The specific calculation formula is similar to that of the embodiment shown in Figure 11 and is not further described here.

[0116] This embodiment differs from the previous one in that it can be used for situations where image quality is poor or the features of the target objects in the image vary significantly. In this case, using a restored image of a degraded image as a benchmark for the first model parameter update, the resulting teacher model is likely to have poor or incomplete feature recognition capabilities. Consequently, the student model trained with this teacher model will struggle to cope with the training task of multi-target complex images. Therefore, the use of this embodiment can maximize the teacher model's ability to capture the overall features of the image, and on this basis, further ensure that the model trained using the self-knowledge distillation method meets the requirements of the training task of multi-target complex images.

[0117] Based on the training method for the image restoration task, the present invention provides an image restoration method, characterized in that the method includes:

[0118] Input the image to be repaired into the trained image repair model and output the repaired image;

[0119] The image restoration model is trained using the training method for the image restoration task.

[0120] The image restoration method can be used to perform a variety of image restoration tasks, such as repairing damaged or missing image content, denoising an image, or performing super-resolution processing on an image, which are not listed here. The image to be restored can be a single frame or a multi-frame video, without limitation.

[0121] Based on the training method for the image restoration task, the present invention also provides a display control method for realizing the visualization and monitoring of the training process of the image restoration task. Figure 12 schematically shows a flow chart of the display control method provided by the present invention, which includes the following operations:

[0122] S1201, selecting an original image for training an image restoration network;

[0123] S1202, in response to detecting an operation of selecting a degraded preview, displaying a degraded image in a display interface;

[0124] S1203 : In response to detecting an operation of selecting to start training, displaying a generated image corresponding to the degraded image and a loss function curve during the training process in a display interface.

[0125] According to an embodiment of the present invention, the display interface may refer to a display interface of a client application process running in an electronic device on a display screen of the electronic device. The electronic device may be any device that supports human-computer interaction.

[0126] According to an embodiment of the present disclosure, the operation is implemented by triggering a task start control, which may be a control for indicating the start of an image generation task. The task start control may be represented as a key or a button on a display interface. A user may touch and click the corresponding key or button, or the user may control the mouse to click the corresponding key or button to trigger the task start control. When the task start control is triggered, the electronic device may generate an image expansion request so that the client application process within the electronic device responds to the image expansion request to expand the image.

[0127] The image restoration model training method provided by the present invention can be implemented using the display control method of the embodiment of the present disclosure. The method shown in Figure 12 will be further explained in conjunction with Figures 13 to 20 and specific embodiments.

[0128] Before responding to detecting the operation of selecting to start training and / or the operation of selecting a degradation preview, the display control method provided by the present invention further includes:

[0129] Selecting an image dataset for training an image restoration network; configuring training and degradation parameters, wherein the parameters include at least one of the following:

[0130] Degradation Count, High Quality Count, Degradation Scheme, Degrader Type, Generator Network Type, or Loss Function Type.

[0131] According to an embodiment of the present invention, the initial display interface of the display control method is shown in Figure 13. Multiple configuration controls are set on the display interface, including a degradation preview operation control and a start training operation control. In addition, it also includes a training data set selection control, a degradation scheme selection control, a degradation number selection control, a high-quality number selection control, a degenerater type selection control, a generated network type selection control, and a loss function type selection control.

[0132] According to an embodiment of the present invention, before responding to detecting the operation of selecting to start training and / or selecting the operation of degradation preview, an image dataset for training the image restoration network can be selected, and training and degradation parameters can be configured.

[0133] Among them, based on the triggering of the training dataset selection control and the degradation scheme selection control, the training dataset and degradation scheme adopted by the image restoration model training method are determined; the training dataset can be a public image dataset called from an open source website, or an image dataset downloaded in advance and deployed locally, which is not specifically limited here; the degradation scheme is obtained by different combinations of degradation operation means such as blurring, resolution upsampling and downsampling, noise or JPEG compression, which is not specifically limited here.

[0134] Based on the triggering of the degradation times selection control, the high-quality times selection control, the degrader type selection control, the generation network type selection control and the loss function type selection control, the parameters of training and degradation are configured; wherein, in response to detecting the triggering of the degrader type selection control and selecting the traditional degrader, the displayed degree coefficients of noise, blur, etc. are related to the set degrader. For example, the noise drop-down can select different types of noise such as Gaussian noise and Poisson noise, and the maximum and minimum values ​​of the noise degree of the noise are selected in the degree coefficient box. After selecting Gaussian noise and setting the completion coefficient, when selecting Poisson noise, the previous Gaussian noise coefficient will not be overwritten, that is, as long as the noise degradation method with the coefficient set is applied to the system.

[0135] According to an embodiment of the present invention, after the image dataset is selected and the training and degradation configurations are completed, in response to detecting a triggering of a degradation preview control on the display interface, a preview of the degraded image may be viewed. Specifically, the steps shown in FIG. 14 are performed as follows:

[0136] S1401, in response to detecting an operation of selecting a degradation preview, displaying a degradation image display area and a list of candidate images to be subjected to degradation operation;

[0137] S1402: Select an original image to be degraded from the list of candidate images to be degraded, and display the original image in the degraded image display area.

[0138] Figure 15 shows a degraded image display interface, which is divided into multiple display areas, including a list area for candidate images to be degraded, a degraded image display area, and a random parameter display area. The list area for candidate images to be degraded can display several unprocessed images. These images can use preset images or directly upload local images. An image to be processed is selected and displayed in the degraded image display area.

[0139] In addition, the random parameter display area is equipped with a random parameter refresh initiation control. In response to detecting the triggering of the random parameter refresh initiation control, different degradation parameters can be randomly generated. These degradation parameters include at least one of the following: noise, blur, degraded image size, or degraded image format. The degraded image display area displays the degraded image with the current parameters in real time. The degraded image display interface also features an original image / degraded image switching control. In response to detecting the triggering of the original image / degraded image switching control, the original image and degraded image can be switched for comparative viewing.

[0140] Figure 16 shows another degraded image display interface, which can simultaneously display multiple degraded images. As shown in Figure 16, the degraded image display interface also includes an area for listing candidate images to be degraded, a degraded image display area, and a random parameter display area. The degraded image display area can simultaneously display the original image and several different degraded images. There is no limit on the number of images displayed, nor on the order in which they are arranged. It only needs to be able to simultaneously display the original image and images corresponding to the original image at varying degrees of degradedness. The several different degraded images are obtained by degrading the original image according to different degradation schemes, including at least one of the following:

[0141] The entire original image is blurred, noisy or compressed to varying degrees; or the original image is randomly divided into regions, and the regions are blurred, noisy or compressed to varying degrees;

[0142] The original image is randomly divided into regions, and different degrees of blur, different degrees of noise or different degrees of compression are applied to the regions; the original image is randomly divided into regions, and different degrees of blur, different degrees of noise or different degrees of compression are applied to the regions;

[0143] Randomly dividing the original image into regions, and performing different degrees of blurring, different degrees of noise or different degrees of compression on the regions;

[0144] The random parameter display area is also provided with a random parameter refresh start control. In response to detecting the triggering of the random parameter refresh start control, the parameters can be refreshed once to provide another set of degraded image schematic diagrams.

[0145] After obtaining the degraded image according to the above method, the display interface shown in FIG13 is returned to. In response to detecting the operation of selecting to start training, the real-time training results of the original image dataset are displayed in the display interface, as shown in FIG17, specifically including:

[0146] S1701, in response to detecting an operation of selecting training, displaying an original image for training, a corresponding degraded image, and a corresponding first generated image;

[0147] S1702 : In response to detecting an operation of selecting a different batch of vectors, displaying multiple batches of second generated images.

[0148] The display interface shown in FIG13 includes a start training selection control. In response to detecting the triggering of the start training selection control, the display interface jumps to the training visualization interface. FIG18 shows the training visualization interface, which includes a display area and a batch vector selection area. The original image, its corresponding degraded image, and the corresponding first generated image are displayed in the display area. The degraded image is obtained by the degradation operation of step S1202, and the degraded image corresponds one-to-one to the first generated image. In FIG18, the high-quality image 1 corresponds to the original image, the low-quality images 11, 12, and 13 correspond to the degraded images, and the generated images 11, 12, and 13 correspond to the first generated images, respectively.

[0149] In response to detecting the selection of a different batch vector, different second generated images are displayed. As shown in Figure 18, batch vectors 1, 2, and 3 represent the number of iterations of the current training process. The interface initially defaults to batch vector 1, i.e., the first iteration of the training process. Generated images 11', 12', and 13' are the second generated images corresponding to generated images 11, 12, and 13, respectively. By selecting batch vectors 1, 2, and 3, the second generated images generated after 1, 2, and 3 iterations of the current training process can be viewed, respectively.

[0150] It is understandable that FIG18 shows a display interface provided by this embodiment, and does not further limit the number of degraded images, first generated images, and second generated images, the number of batch vectors, and the layout of each area display.

[0151] The training visualization interface also includes a loss function curve selection control. In response to detecting the triggering of the loss function curve selection control, the transformation trend of the loss function can be viewed, and the detailed loss value of the parameter update for each iteration during the training process can be displayed. In response to detecting the selection of the loss curve control in Figure 18, the user can jump to the display interface shown in Figure 19, which respectively shows the relationship between the loss value calculated for the first parameter update and the loss value calculated for the second parameter update and the number of iterations. Through the operation, the relationship between the loss value calculated for each parameter update and the number of iterations can be viewed, allowing the user to intuitively monitor the training process.

[0152] The training visualization interface also includes a mode selection control that can switch between training mode and test mode. The default setting is training mode, as shown in Figure 20. In response to detecting the triggering of the mode selection control, the training mode is switched to the test mode, and the list of images to be tested and the image display area are displayed. The image display area also displays the image restoration test results. The images to be tested are low-quality images that require image restoration, and the image display area displays the results of the trained model restoring the low-quality images that require image restoration.

[0153] In the image display area, an original image / result image selection control is further provided. In response to detecting the triggering of the original image / result image selection control, the image to be tested and the result can be switched for display.

[0154] In the above-mentioned scheme of the embodiment of the present invention, the restoration quality of image restoration can be improved. Furthermore, a method for training an image restoration model is provided, which provides an implementation basis for generating an image restoration model with targeted restoration. In addition, the display control method provided by the present invention can be used as an innovative design for model training in existing training platforms on the market. By simply configuring the relevant parameters, users can quickly implement the two-stage self-distillation training. The interface is simple, the parameter configuration is minimal, and it is convenient and quick to access the platform application. This solves the problem of the difficulty in monitoring the image restoration model training process, and is beneficial for those skilled in the art to adjust the training plan at any time according to the training results.

[0155] Corresponding to the image restoration method provided by the above embodiment of the present invention, as shown in FIG21 , an embodiment of the present invention further provides an image restoration device, the device comprising:

[0156] S2101, an image degradation module, configured to perform a degradation operation on the original image in the original image dataset using a preset degradation method;

[0157] S2102, an image generation module, configured to use a pre-set generation network model to repair the image input to the image generation module and output a repaired generated image;

[0158] S2103, a parameter updating module, is used to update the parameters of the generated network model according to the training results.

[0159] The image segmentation module performs different degradation operations on an original image in the original image dataset to obtain degraded images of varying degrees. The degraded images of varying degrees may be obtained by one or more of compression with different parameters or bit rates, blurring, upsampling or downsampling, or noise processing. The specific type of degradation operation, processing steps, and order are not limited.

[0160] Optionally, the image degradation module includes:

[0161] The image input submodule receives an image input to the image degradation module. The image is a high-quality image, which refers to multiple high-quality images or video frames in the original dataset. A high-quality image or video frame refers to an image or video frame that achieves the expected quality in the image restoration task. The expected quality refers to an image with an appropriate resolution determined based on common knowledge in the art. It is understood that in an image restoration task for a 2K display device, the image resolution that achieves the expected quality is 2560×1440; in an image restoration task for a 4K display device, the image resolution that achieves the expected quality is 4096×2160, and so on.

[0162] The above-mentioned original image can be a portrait image, a face image, an oil painting image, a multi-target image, a medical image, etc. Optionally, the above-mentioned image to be repaired can be a picture / video taken by a terminal, and the terminal can include, for example, a mobile phone, a tablet, a small camera, a large camera, etc. It is also suitable for monitoring videos, such as traffic monitoring and indoor venue monitoring screen processing. The content of the above-mentioned image to be repaired can include movie images, TV series images, variety show images, documentary images, sports event images, monitoring screens, cartoons, art paintings, etc. The image restoration tasks mentioned in the present invention include all low-level computer vision training tasks, such as super-resolution, noise reduction, deblurring, compression restoration, face restoration, etc.

[0163] The image degradation submodule performs varying degrees of degradation on the high-quality image input to the image degradation module, producing different degraded images. This degradation process involves applying random blurring, random resolution up-sampling and down-sampling, random noise generation, and JPEG compression to a high-definition original image, resulting in a low-quality image. Common degradation schemes for classic images using traditional algorithms are shown in Figure 2. In this scheme, the choice of degradation scheme is not limited; both traditional and deep learning models can be used for degradation.

[0164] This degradation scheme can be extended to video degradation. The original high-definition video can be compressed and encoded with different encoders and different quality parameters / bitrate sizes. Then, the compressed and encoded video frame information can be degraded according to the preset degradation scheme.

[0165] The different degraded images refer to images of varying degrees of degradation obtained by performing different degradation operations on the same original image. Optionally, the images of varying degrees of degradation may be images of varying degrees of degradation obtained by performing one or more of the following processes: compression with different parameters or bitrates, blurring, upsampling or downsampling, or noise processing. The specific degradation operation type, processing steps, and sequence are not limited.

[0166] The image output submodule outputs the obtained different degraded images.

[0167] Optionally, the image degradation module further includes:

[0168] The image segmentation submodule randomly divides the high-quality image into multiple distinct regions, which are then degraded using different degradation operations. This degradation scheme allows for the representation of multiple degradation states within the same degradation map, allowing for more complex scenarios during training, thereby improving the model's adaptability to various low-quality image restoration tasks.

[0169] The region selection module is used to select the areas within the image that require restoration. Specifically, during training, a mask is added to the selected n blocks. When calculating the loss during training, only the loss for the masked areas is calculated, while the loss for the remaining ground-truth areas is not calculated. This degradation scheme improves the model's ability to distinguish low-quality areas in the image, effectively repairing the areas that need restoration and significantly reducing computing resources.

[0170] Optionally, the image generation module includes:

[0171] The image restoration submodule includes a pre-configured generative network model, which receives the different degraded images output by the image degradation module and performs restoration on the different degraded images to obtain a first generated image. The number of the first generated images is the same as the number of the degraded images input to the generative network. The generative network can be a mature image restoration network model from the prior art. This application does not further limit the type of generative network model. For example, at least one loss function such as L1 loss (a mean absolute error loss function), MSE loss (mean squared error loss function), or Charbonnier loss (an image reconstruction loss function) can be used to calculate the difference in pixel values ​​between the restored image and the true value image.

[0172] A parameter updating submodule determines different loss values ​​according to the generated image output by the image restoration submodule and the high-quality image, and updates the parameters of the generation network model in the image restoration submodule according to the different loss values.

[0173] Optionally, the parameter updating submodule includes:

[0174] A first parameter updating unit updates first model parameters of the generative network based on the first generated image and the original image. Optionally, the first model parameter updating can be performed based on a result of a first loss calculation performed on a third generated image and the original image, where the third generated image is selected from the first generated image; optionally, the first model parameter updating can also be performed based on a result of a third loss calculation performed on all the first generated images and the original image.

[0175] The second parameter updating unit updates the second model parameters of the generative network based on the first and second generated images. Optionally, the second generated image can be obtained by inputting the third generated image into the generative network after the first model parameters have been updated. Optionally, the second generated image can also be obtained by inputting the first generated image into the generative network after the first model parameters have been updated, generating multiple images and selecting one from among them.

[0176] Corresponding to the image restoration method provided by the above embodiment of the present invention, as shown in FIG22 , an embodiment of the present invention further provides a display control device, the device comprising:

[0177] S2201, a degraded image preview module, configured to display a degraded image in a display interface in response to detecting an operation of selecting a degradation preview;

[0178] S2202, a training operation display module displays the real-time training results of the original image data set in a display interface in response to detecting an operation of selecting to start training.

[0179] Optionally, the display control device further includes a training set selection module and a parameter setting module, wherein the training set selection module is used to select an image dataset for training the image restoration network, and the parameter setting module is used to configure training and degradation parameters, wherein the parameters include at least one of the following:

[0180] Degradation Count, High Quality Count, Degradation Scheme, Degrader Type, Generator Network Type, or Loss Function Type.

[0181] The present invention also provides an electronic device suitable for an image restoration model training method. As shown in FIG23 , a computer electronic device 2300 according to an embodiment of the present invention includes a processor 2301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 2302 or a program loaded from a storage part 2308 into a random access memory (RAM) 2303. The processor 2301 may, for example, include a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (for example, an application-specific integrated circuit (ASIC)), and the like. The processor 2301 may also include on-board memory for caching purposes. The processor 2301 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0182] RAM 2303 stores various programs and data required for the operation of electronic device 2300. Processor 2301, ROM 2302, and RAM 2303 are interconnected via bus 2304. Processor 2301 executes the programs in ROM 2302 and / or RAM 2303 to perform various operations according to the method flow of the embodiment of the present invention. It should be noted that the programs may also be stored in one or more memories other than ROM 2302 and RAM 2303. Processor 2301 may also execute the programs stored in the one or more memories to perform various operations according to the method flow of the embodiment of the present invention.

[0183] According to an embodiment of the present invention, electronic device 2300 may further include an input / output (I / O) interface 2305, which is also connected to bus 2304. Electronic device 2300 may further include one or more of the following components connected to I / O interface 2305: an input section 2306 including a keyboard, mouse, etc.; an output section 2307 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 2308 including a hard disk; and a communication section 2309 including a network interface card such as a LAN card or modem. Communication section 2309 performs communication processing via a network such as the Internet. A drive 2310 is also connected to I / O interface 2305 as needed. Removable media 2311, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 2310 as needed, so that computer programs read from the removable media can be installed into storage section 2308 as needed.

[0184] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 2309, and / or installed from the removable medium 2311. When the computer program is executed by the processor 2301, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0185] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0186] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0187] For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 2302 and / or RAM 2303 described above and / or one or more memories other than ROM 2302 and RAM 2303 .

[0188] An embodiment of the present invention also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the image processing method provided by the embodiment of the present invention.

[0189] When the computer program is executed by the processor 2301, the above functions defined in the system / device of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0190] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 2309, and / or installed from a removable medium 2311. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0191] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, Python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect via the Internet).

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, as well as the combination of boxes in the block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or may be implemented using a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments and / or claims of the present invention may be combined and / or coupled in various ways, and all such combinations and / or couplings fall within the scope of the present invention.

[0193] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The scope of the present invention is defined by the appended claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which are intended to fall within the scope of the present invention.

Claims

1. A training method for an image restoration task, comprising: Get the original image dataset; performing a degradation operation on a plurality of original images in the original image data set to obtain different degraded images; Inputting the plurality of degraded images into a generation network to obtain first generated images, wherein the plurality of first generated images constitute a first generated image set; updating first model parameters of the generative network according to the first generated image and the original image; Inputting the first generated image into the generation network after the model parameters are updated to generate a second generated image; updating a second model parameter of the generative network according to the first generated image and the second generated image; Repeat the operation of updating the second model parameters of the generated network until the generated network meets the requirements of the image restoration task.

2. The method according to claim 1, wherein Performing degradation operations on the original images in the image dataset to obtain different degraded images, including: The images with different degrees of degradation can be obtained by performing different degradation operations on an original image in the original image dataset; The different degradation operations include at least one of the following operations: Different degrees of image blur, different degrees of noise processing or different degrees of image compression.

3. The method according to claim 2, wherein: Performing degradation operations on the original images in the original image dataset to obtain different degraded images, including: Input the entire original image in the image dataset into the degradator, and obtain multiple different degraded images through different degradation operations; or The original image in the image dataset is randomly divided into regions, and different degradation operations are performed on different regions; or, The original image in the image data set is randomly divided into regions, and a number of regions are randomly selected, and different degradation operations are performed on the several regions.

4. The method according to claim 1, wherein updating first model parameters of the generative network according to the first generated image and the original image; inputting the first generated image into the generative network after the updated model parameters to generate a second generated image; Updating second model parameters of the generative network according to the first generated image and the second generated image includes: Selecting a first generated image from the first generated image set as a third generated image; Performing a first loss calculation on the third generated image and the original image; According to the first loss calculation, updating the first model parameters of the generative network to obtain a first generative network; performing a second loss calculation based on the third generated image and the first generated image; Based on the second loss calculation, a second parameter of the generative network is updated to obtain a second generative network; wherein a first generated image is selected from the first generated images as a third generated image based on at least one of the following factors: The priority of the feature to be fixed; or, The richness of image features.

5. The method according to claim 4, wherein Performing a second loss calculation based on the third generated image and the first generated image includes: Inputting the third generated image into the first generation network to generate a fourth generated image; performing multiple second loss calculations between the first generated images except the third generated image and the fourth generated image; The model parameters of the first generation network are updated according to the multiple second loss calculations to obtain a second generation network.

6. The method according to any one of claims 4-5, wherein: include: The first loss calculation function is expressed as: loss1=L1(out 11 ,G1) The second loss calculation function is expressed as: loss2=L1(out 12 ,out′ 11 )+L1(out 13 ,out′ 11 )+…+L1(out 1n ,out′ 11 ) Among them, G1 represents the original image, out 11 Denotes the third generated image, out′ 11 represents the fourth generated image, out 12 , out 13 …out 1n represents the first generated images other than the third generated image in the first generated image set.

7. The method according to claim 1, wherein updating first model parameters of the generative network according to the first generated image and the original image; inputting the first generated image into the generative network after the updated model parameters to generate a second generated image; Updating second model parameters of the generative network according to the first generated image and the second generated image includes: Performing a third loss calculation on each of the images in the first generated image set and the original image; performing a third model parameter update of the generative network according to the third loss calculation to obtain a third generative network; Inputting the images in the first generated image set into the third generation network to generate a second generated image set; performing a fourth loss calculation based on the second generated image set; Based on the fourth loss calculation, a fourth parameter update of the generative network is performed to obtain a fourth generative network.

8. The method according to claim 7, wherein: performing a fourth loss calculation based on the second generated graph; Performing a fourth parameter update of the generative network according to the fourth loss calculation to obtain a fourth generative network includes: selecting a second generated image from the second generated images as a fifth generated image; performing a plurality of fourth loss calculations between each of the second generated images except the fifth generated image and the fifth generated image; performing a fourth model parameter update of the generative network according to the multiple fourth loss calculations to obtain a fourth generative network; The selecting of a second generated image from the second generated images as the fifth generated image is based on at least one of the following factors: The priority of the feature to be fixed; or, The richness of image features.

9. The method according to any one of claims 7-8, wherein: include: The third loss calculation is expressed as: loss3=L1(out 11 ,G1)+L1(out 12 ,G1)+L1(out 13 ,G1)+…+L1(out 1n ,G1) The fourth loss calculation is expressed as: loss4=L1(out′ 12 ,out′ 11 )+L1(out′ 13 ,out′ 11 )+…+L1(out′ 1n ,out′ 11 ) Among them, G1 represents the original image, out 11 , out 12 , out 13 …out 1n Denotes the first generated image set, out′ 11 Denotes the fifth generated image, out′ 12 , out′ 13 …out′ 1n Indicates the second generated image excluding the fifth generated image.

10. An image restoration method, comprising: Input the image to be repaired into the trained image repair model and output the repaired image; The image restoration model is trained using the image restoration task training method described in any one of claims 1 to 9.

11. A display control method, comprising: Select the original image for training the image restoration network; In response to detecting an operation of selecting a degraded preview, displaying a degraded image corresponding to the original image in a display interface; In response to detecting an operation of selecting to start training, displaying a generated image corresponding to the degraded image and a loss function curve during the training process in a display interface; The generated image includes a first generated image and a second generated image; the second generated image is based on the first generated image and is trained by the image restoration task described in any one of claims 1 to 9. The loss function curve includes a first loss function curve between the original image and the first generated image and a second loss function curve between the first generated image and the second generated image.

12. The method according to claim 11, wherein Before responding to detecting the operation of selecting to start training and / or the operation of selecting a degradation preview, the method further includes: Configure training and degradation parameters, where the parameters include at least one of the following: Degradation Count, High Quality Count, Degradation Scheme, Degrader Type, Generator Network Type, or Loss Function Type.

13. The method according to claim 11, wherein In response to detecting an operation of selecting a degraded preview, displaying a degraded image corresponding to the original image in a display interface includes: In response to detecting an operation of selecting a degradation preview, displaying a degradation image display area and a list of candidate images to be subjected to a degradation operation; Selecting an original image to be degraded from the list of candidate images to be degraded, and displaying the original image in the degraded image display area; Wherein, in the degraded image display area, an operation option for switching between original image and degraded image display is provided; In the degraded image display area, degradation parameters of the currently displayed degraded image are also provided, wherein the degradation parameters include at least one of the following: Noise, blur, degraded image size or degraded image format.

14. The method according to claim 13, wherein In response to detecting an operation of selecting a degraded preview, displaying a degraded image in a display interface further includes: Displaying an original image and images with different degrees of degradation corresponding to the original image in the display interface; The images degraded to different degrees are obtained by degrading the original image according to different degradation schemes, and the degradation schemes include at least one of the following: Performing different degrees of blurring, different degrees of noise, or different degrees of compression on the entire original image; Randomly dividing the original image into regions, and performing different degrees of blurring, different degrees of noise or different degrees of compression on the regions; The regions are randomly divided, and a number of regions are randomly selected, and different degrees of blurring, different degrees of noise or different degrees of compression are performed on the several regions.

15. The method according to claim 11, wherein In response to detecting an operation of selecting to start training, displaying a generated image corresponding to the degraded image in a display interface, including: In response to detecting an operation of selecting training, displaying an original image for training, a corresponding degraded image, and a corresponding first generated image; The corresponding degraded image is obtained by inputting the original image into a preset degrader, and the first generated image corresponds to the degraded image.

16. The method according to claim 15, wherein include: Display the number of training batch vectors and corresponding options; In response to detecting an operation of selecting a different batch of vectors, displaying a plurality of batches of second generated images, where the second generated images are obtained by training the first generated images using the training method for the image restoration task according to any one of claims 1 to 9; The number of batch vectors corresponds to the total number of batches of the first generated image and the second generated image.

17. The method according to claim 11, wherein include: In response to selecting the operation of viewing the loss curve, the first loss function curve and the second loss function curve are viewed; the first loss function curve and the second loss curve are used to characterize the relationship between the loss value and the number of iterations.

18. The method according to claim 11, wherein include: Display test options; In response to detecting an operation of selecting a test, displaying a list of images to be tested and an image display area; The restored image is displayed in the image display area.

19. The method according to claim 18, wherein include: In the image display area, an original image / result image switching option is provided.

20. An electronic device comprising a processor and a memory; wherein: The memory is used to store computer programs; The processor is configured to implement the method steps described in any one of claims 1 to 19 when executing a program stored in the memory.

21. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 19 are implemented.

Images