Image restoration method and apparatus, electronic device, and storage medium

Abstract

The present application relates to the technical field of image processing, and provides an image restoration method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring an image to be restored; and processing said image on the basis of an image restoration model to obtain a post-restoration high-definition image, wherein the image restoration model is obtained by means of training in a plurality of training stages, and a loss function of one of the plurality of training stages is a loss function aiming at optimizing structural information of an image. Therefore, the image restoration model can restore structural information in the image to be restored.

Description

An image restoration method, apparatus, electronic device, and storage medium Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image restoration method, apparatus, electronic device, and storage medium. Background Technology

[0002] Image restoration technology is being used more and more widely in the field of image processing. Image restoration (IR) is dedicated to improving the subjective quality of images that have been distorted by various forms of degradation. Image restoration tasks include image super resolution (SR), deblurring, noise reduction, and compression artifact removal.

[0003] The prior knowledge about real-world high-resolution images contained in diffusion models plays a crucial role in image inpainting. Existing diffusion-based image inpainting methods show good results in noise reduction and deblurring. However, these methods can generate additional, non-existent details during the inpainting process and are ineffective at repairing structural defects such as block noise and scratches.

[0004] Summary of the Invention

[0005] In view of the above problems, embodiments of this application provide an image restoration method, apparatus, electronic device, and storage medium to overcome or at least partially solve the above problems.

[0006] A first aspect of this application discloses an image restoration method, the method comprising:

[0007] Obtain the image to be repaired;

[0008] The image to be repaired is processed according to the image restoration model to obtain a restored high-definition image;

[0009] The image restoration model is trained through multiple training stages, and the loss function of one of the training stages is a loss function aimed at optimizing the structural information of the image.

[0010] A second aspect of this application discloses an image restoration apparatus, the apparatus comprising:

[0011] The acquisition module is used to acquire the image to be repaired;

[0012] The repair module is used to process the image to be repaired according to the image repair model to obtain a repaired high-definition image;

[0013] The image restoration model is trained through multiple training stages, and the loss function of one of the training stages is a loss function aimed at optimizing the structural information of the image.

[0014] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the image restoration method described in the first aspect of this application.

[0015] A fourth aspect of this application discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image restoration method described in the first aspect of this application.

[0016] The embodiments of this application have the following advantages:

[0017] In this embodiment, the image to be repaired is processed according to an image inpainting model to obtain a repaired high-definition image. Since the image inpainting model is trained through multiple training stages, and the loss function of one of these stages is a loss function aimed at optimizing the structural information of the image, the image inpainting model can repair structural information (such as block noise and other structural defects) in the image to be repaired. Thus, the repaired high-definition image obtained according to the image inpainting model can significantly improve the image repair quality in terms of structural defects, achieving high-quality image inpainting. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 is a flowchart of the steps of an image restoration method provided in an embodiment of this application;

[0020] Figure 2 is a schematic diagram of the structure of an image restoration model provided in an embodiment of this application;

[0021] Figure 3 is a schematic diagram of another image restoration model provided in an embodiment of this application;

[0022] Figure 4 is a flowchart illustrating a method for constructing a first training dataset according to an embodiment of this application;

[0023] Figure 5 is a schematic diagram of a sample high-definition image and an old photo style image provided in an embodiment of this application;

[0024] Figure 6 is a schematic diagram of training a style generative adversarial network provided in an embodiment of this application;

[0025] Figure 7 is a schematic diagram of the construction process of a target diffusion network provided in an embodiment of this application;

[0026] Figure 8 is a schematic diagram of a restored high-definition image provided in an embodiment of this application;

[0027] Figure 9 is a schematic diagram of an image restoration device provided in an embodiment of this application;

[0028] Figure 10 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specific Implementation

[0029] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] To facilitate understanding of the technical solutions provided in this application, the main technical concepts involved in the embodiments of this application are briefly described below.

[0031] Diffusion models refer to the process of gradually denoising data from purely noisy data through neural network learning. Diffusion models are a type of generative model; given a set of random noise, they transform it through a certain probability model to output data with semantic information (e.g., images, text). The goal of diffusion models is to learn methods to generate images from pure noise. This is mainly achieved by training a U-Net, which takes a series of images with added noise and learns to predict the added noise. Specifically, the training process of diffusion models is divided into a forward process and a backward / reverse process.

[0032] (1) Forward process:

[0033] x0 represents the original data. The noise increases with each step until, after T steps, it becomes a pure noise feature tensor. Since the noise is Gaussian distributed, the feature tensor after adding noise is also Gaussian distributed, as shown below:

[0034] From this, we can understand the transformation process from xt-1 to xt, and it is also easy to understand the transformation process from xt-2 to xt-1. Continuing this iteration, the feature tensor can be represented as: α t ∶=1-β t (Formula 3)

[0035] Where, β t is a hyperparameter, and z is Gaussian noise with a mean of 0 and a variance of 1.

[0036] (2) Reverse process:

[0037] If the forward process is the process of adding noise, then the reverse process is the denoising inference process of the diffusion model.

[0038] Where, ε θ (x t ,t) and σ θ (x t ,t) is the output of U-Net, and θ is the model parameters.

[0039] Specifically, the training process of the diffusion model includes the following steps:

[0040] Step 1: Extract x0 from the training data and randomly select time step t;

[0041] Step 2: Randomly generate Gaussian noise with a mean of 0 and a variance of 1;

[0042] Step 3: Calculate x according to the above (Formula 5). t ;

[0043] Step 4: Set x t The U-Net is input to t, and the U-Net predicts the output noise. The loss is calculated based on the predicted noise and the actual noise added. The network parameters are then updated based on the loss.

[0044] The sampling process includes: generating Gaussian noise x with a mean of 0 and a variance of 1. T And repeat steps (Formula 6) from T to 1 until x0.

[0045] The prior knowledge about real-world high-resolution images contained in diffusion models enables image inpainting. Existing diffusion-based image inpainting methods show good results in noise reduction and deblurring. However, these methods can generate additional, non-existent details during the inpainting process and are ineffective at repairing structural defects such as block noise and scratches.

[0046] Therefore, to overcome the limitations of related technologies, an image restoration method is provided. This method processes the image to be restored according to an image restoration model to obtain a restored high-resolution image. By introducing structural information of the image to optimize the objective, the image restoration model focuses on structural defects in the image during the restoration process, thereby achieving high-quality image restoration.

[0047] The image restoration method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0048] Referring to Figure 1, which is a flowchart of an image restoration method according to an embodiment of this application, the image restoration method provided in this embodiment may include steps S110 to S120:

[0049] Step S110: Obtain the image to be repaired.

[0050] Step S120: Process the image to be repaired according to the image restoration model to obtain the restored high-definition image.

[0051] The image restoration model is trained through multiple training stages, and the loss function of one of the training stages is a loss function aimed at optimizing the structural information of the image.

[0052] In this embodiment, the image to be repaired refers to a low-quality image, which may have problems such as blurriness, low resolution, or scratches. In some embodiments, the image to be repaired is an old photograph or a black and white photograph; in some embodiments, the image to be repaired is an image frame from an old video. If the image to be repaired is an image frame from an old video, the old video can be repaired by repairing the image frame. In some embodiments, the image to be repaired is a non-high-definition image captured by an image device.

[0053] Image inpainting models are a type of diffusion model. Because they are trained through multiple training phases, and the loss function in one of these phases is designed to optimize the structural information of the image, they can repair structural defects (such as block noise) in the image to be restored. Thus, the restored high-resolution image obtained using this model significantly improves the quality of the restored image in terms of structural defects, achieving high-quality image restoration.

[0054] In conjunction with the above embodiments, in one embodiment, this application also provides an image restoration method. In this method, as shown in FIG2, the image restoration model includes: a degradation removal network, a first control network, and a generation network; wherein, the degradation removal network is used to perform degradation removal processing on the image to be restored, so as to remove quality defects caused by various forms of degradation in the image to be restored; the first control network is used to generate a first intermediate feature based on the de-degraded image, so as to guide the generation network to generate a high-definition image based on the degradation information of the de-degraded image; the generation network is used to generate a high-definition image that meets the restoration requirements.

[0055] The step S120 above, "processing the image to be repaired according to the image restoration model to obtain the restored high-definition image," may specifically include the following sub-steps A1 to A3:

[0056] Step A1: Input the image to be repaired into the degradation removal network for degradation removal processing to obtain a de-degradation image. The degradation removal processing includes at least one of the following: deblurring, noise reduction, scratch removal, dust removal, brightness enhancement, and image super-resolution.

[0057] Step A2: Input the de-degradation image into the first control network for feature extraction to obtain the first intermediate feature. The first intermediate feature is used to guide the generation network to generate a repaired image based on the de-degradation information in the de-degradation image.

[0058] Step A3: Input the randomly sampled noisy image and the first intermediate feature into the generation network to generate the repaired image, and obtain the repaired high-definition image.

[0059] In this embodiment, the image to be repaired is input into a degradation removal network for degradation removal processing. For different images to be repaired, one or more degradation removal methods described above can be used as needed. After obtaining the degradation removal image, it is input into a first control network for feature extraction to obtain first intermediate features. These first intermediate features guide the generation network to generate a repaired image based on the degradation removal information in the degradation removal image. Thus, when generating the repaired high-definition image, the generation network retains low-frequency information consistent with the image to be repaired, without losing details, thereby ensuring high fidelity of the repaired high-definition image. In an optional embodiment, the degradation removal network and the first control network can be convolutional neural networks. Deep learning algorithms and convolutional neural networks are used to perform degradation removal processing on the image to be repaired, and feature extraction is performed on the obtained degradation removal image to obtain the first intermediate features.

[0060] After obtaining the first intermediate feature, the first intermediate feature and a randomly sampled noisy image are input into the generator network for image restoration. This involves fusing the randomly sampled noisy image and the first intermediate feature to obtain the restored high-resolution image output by the generator network. The randomly sampled noisy image conforms to a Gaussian distribution. In an optional embodiment, the generator network can be a plugin for a diffusion model. The generator network includes an encoder, a decoder, and a U-Net network, thereby combining the diffusion model (generator network) to fuse the randomly sampled noisy image with the first intermediate feature, and gradually obtaining the restored high-resolution image through the diffusion model generation algorithm.

[0061] In this embodiment, by configuring a degradation removal network and a first control network in the image restoration model, a first intermediate feature with degradation removal information is obtained. This feature guides the generator network to generate the restored image based on the degradation removal information. As a result, when generating the restored high-definition image, the generator network retains the low-frequency information consistent with the image to be restored, without losing details, thus ensuring that the restored high-definition image has high fidelity. In this way, high-quality image restoration is achieved.

[0062] In conjunction with the above embodiments, in one embodiment, this application also provides an image restoration method. In this method, as shown in Figure 3, the image restoration model includes: a degradation removal network, a structural information labeling network, a first control network, a second control network, and a generation network; wherein, the degradation removal network is used to perform degradation removal processing on the image to be restored, to remove quality defects caused by various forms of degradation in the image to be restored; the first control network is used to generate a first intermediate feature based on the de-degraded image, to guide the generation network to generate a high-definition image based on the degradation information of the de-degraded image; the structural information labeling network is used to label the structural locations in the image to be restored; the second control network is used to generate a second intermediate feature based on the labeled structural locations, to guide the generation network to generate a high-definition image based on the structural information of the image to be restored; and the generation network is used to generate a high-definition image that meets the restoration requirements.

[0063] The step S120 above, "processing the image to be repaired according to the image restoration model to obtain the restored high-definition image," may specifically include the following sub-steps B1 to B5:

[0064] Step B1: Input the image to be repaired into the degradation removal network for degradation removal processing to obtain a de-degradation image. The degradation removal processing includes at least one of the following: deblurring, noise reduction, scratch removal, dust removal, brightness enhancement, and image super-resolution.

[0065] Step B2: Input the de-degradation image into the first control network for feature extraction to obtain the first intermediate feature. The first intermediate feature is used to guide the generation network to generate a repaired image based on the de-degradation information in the de-degradation image.

[0066] Step B3: Input the image to be repaired into the structural information labeling network to perform structural location labeling, and obtain a structural location labeled image.

[0067] Step B4: Input the structural location marker image into the second control network for feature extraction to obtain a second intermediate feature. The second intermediate feature is used to guide the generation network to generate a repair image based on the structural information of the structural location marker image.

[0068] Step B5: Input the randomly sampled noisy image, the first intermediate feature, and the second intermediate feature into the generation network to generate the repaired image, and obtain the repaired high-definition image.

[0069] In this embodiment, the image to be repaired is input into a degradation removal network for degradation removal processing. For different images to be repaired, one or more degradation removal methods described above can be used as needed. After obtaining the degradation removal image, it is input into a first control network for feature extraction to obtain first intermediate features. These first intermediate features guide the generation network to generate a repaired image based on the degradation removal information in the degradation removal image. Thus, when generating the repaired high-definition image, the generation network retains low-frequency information consistent with the image to be repaired, without losing details, thereby ensuring high fidelity of the repaired high-definition image.

[0070] Since structural defects (e.g., block noise, scratches, etc.) in the image to be repaired can be identified, the image to be repaired is input into a structural information labeling network for structural location labeling to obtain a structural location labeled image. In some embodiments, the structural information labeling network includes a segmentation network and a masking network. The segmentation network segments the structural defects in the image to be repaired using an image segmentation algorithm (e.g., SAM), and then the masking network masks the segmented structural defects, that is, labels the structural locations, thereby obtaining a structural location labeled image.

[0071] After obtaining the structural location marker image, it is input into the second control network for feature extraction, yielding a second intermediate feature. This second intermediate feature guides the generator network to generate the repaired image based on the structural information in the structural location marker image. Thus, when generating the repaired high-resolution image, the generator network can specifically address the locations of structural defects in the original image, thereby repairing these defects.

[0072] After obtaining the first intermediate feature and the second intermediate feature, the first intermediate feature, the second intermediate feature and the randomly sampled noisy image are input into the generator network to generate the repaired image. That is, the randomly sampled noisy image, the first intermediate feature and the second intermediate feature are fused together to obtain the repaired high-definition image output by the generator network.

[0073] It is understandable that steps B3 to B4 can be executed before steps B1 to B2, after steps B1 to B2, or simultaneously with steps B1 to B2.

[0074] This application proposes a novel image restoration model structure. By configuring a degradation removal network and a first control network in the image restoration model, a first intermediate feature with degradation removal information is obtained. This guides the generator network to generate the restored image based on the degradation removal information. Thus, when generating the restored high-definition image, the generator network retains low-frequency information consistent with the image to be restored, without losing details, thereby ensuring high fidelity of the restored high-definition image. Furthermore, by configuring a second control network and a structural information labeling network in the image restoration model, a second intermediate feature with structural information is obtained. This guides the generator network to generate the restored image based on the structural information. Thus, when generating the restored high-definition image, the generator network can specifically process locations with structural defects in the image to be restored. In this way, high-quality image restoration is achieved.

[0075] In conjunction with the above embodiments, in this application embodiment, the image restoration model is trained according to the following steps:

[0076] Step C1: With the goal of optimizing the de-degradation information of the image, the image restoration model is trained in the first stage of degradation removal network training and the second stage of control network training according to the first training dataset to obtain the preliminarily trained image restoration model.

[0077] Step C2: With the goal of optimizing the structural information of the image, the image restoration model trained in the preliminary stage is fine-tuned in the third stage based on the second training dataset to obtain the trained image restoration model.

[0078] Each training data in the first training dataset includes: a sample high-resolution image and a degraded image; each training data in the second training dataset includes: a specific style image and a sample high-resolution image, wherein the specific style image represents an image whose style is consistent with the image to be repaired.

[0079] In this embodiment, the training process of the image restoration model is divided into three stages. In the first and second stages, the goal is to optimize the de-degradation information of the image. The model is trained using a first training dataset so that it can learn the de-degradation processing capability of the image to repair quality defects caused by various forms of degradation in the image.

[0080] Each training data point in the first training dataset includes a sample high-resolution image and a degraded image. In some embodiments, as shown in Figure 4, the construction of the first training dataset includes: constructing a sample high-resolution image set, which includes multiple sample high-resolution images; processing each sample high-resolution image twice according to the target degradation mode to obtain a degraded image corresponding to each sample high-resolution image; and using the sample high-resolution image and its corresponding degraded image as training data. The target degradation modes include four degradation modes: Gaussian blur, resizing, noise reduction, and JPEG compression. When degrading the sample high-resolution images, the processing parameters are randomly collected parameters.

[0081] The third stage aims to optimize the structural information of the images. It uses the second training data for training, enabling the image restoration model to repair structural information (e.g., structural defects such as block noise) in images of specific styles. Each training data point in the second training dataset includes a specific style image and high-resolution sample images. The construction method of the second training dataset is explained below.

[0082] It should be noted that U-Net in the image inpainting model was trained in the first stage of degradation removal network training, the second stage of control network training, and the third stage of model fine-tuning. The training method of U-Net in each stage can be referred to the "Training Process of Diffusion Model" above, and will not be repeated here.

[0083] In this embodiment, by training the image restoration model in the first and second stages, the model is able to repair quality defects caused by various forms of degradation in the image to be restored. By training the model in the third stage, it is able to restore structural information in images of a specific style. Thus, the high-definition image restored using this model significantly improves the restoration quality in terms of de-degradation and structural defect removal, achieving high-quality image restoration.

[0084] The following sections will explain each training stage of the image restoration model.

[0085] (1) The first stage is to train the degradation removal network in the image restoration model. In some implementations, with the goal of optimizing the degradation removal information of the image, the restoration model is trained in the first stage of degradation removal network training based on the first training dataset, including steps D1 and D2:

[0086] Step D1: Input the degraded image into the degradation removal network for degradation removal processing to obtain the first high-definition image.

[0087] Step D2: Update the network parameters of the degradation removal network based on the differences between the first high-definition image and the sample high-definition image.

[0088] In this embodiment, each training data in the first training dataset includes a sample high-resolution image and a degraded image. The degraded image is used as the input image, and the sample high-resolution image is used as the ground truth image. For each training data in the first training dataset, the degraded image is input into the degradation removal network for degradation removal processing to obtain the first high-resolution image output by the degradation removal network. Then, a first loss (i.e., the difference between the first high-resolution image and the sample high-resolution image) is calculated based on the first high-resolution image and the sample high-resolution image, and the network parameters of the degradation removal network are updated based on the first loss. After the training termination condition is met (e.g., the number of training iterations meets a preset number, or the accuracy of the degradation removal network meets a preset requirement), the trained degradation removal network is obtained.

[0089] Thus, after the first stage of degradation removal network training, the degradation removal network learns the ability to degrade images. Consequently, when using an image inpainting model for image inpainting, the image to be repaired, after passing through the degradation removal network, can eliminate the quality defects caused by degradation in the image to be repaired.

[0090] (2) The second stage is to train the control network in the image restoration model. In some embodiments, when the control network is the first control network, the second stage of control network training is performed on the image restoration model with the goal of optimizing the de-degradation information of the image, based on the first training dataset, including steps E1 and E2:

[0091] Step E1: Fix the network parameters of the degradation removal network, and input the degraded image into the image restoration model through the degradation removal network to perform image restoration and obtain a second high-definition image.

[0092] Step E2: Update the network parameters of the first control network based on the differences between the second high-definition image and the sample high-definition image.

[0093] In this embodiment, the network parameters of the degradation removal network are fixed, and the first control network in the restoration model is trained using the first training data. Similarly, the degradation-processed images in the training data are used as input images, and the sample high-resolution images are used as ground truth images. For each training data in the first training dataset, the degradation-processed image is input into the image restoration model through the degradation removal network (wherein the degradation removal network refers to the already trained degradation removal network) for image restoration.

[0094] Specifically, the degradation removal network degrades the input degradation image to eliminate various degradation-induced quality defects, resulting in a sample degraded image. This sample degraded image is then input into the first control network for feature extraction. The obtained first sample intermediate features are then input into the generator network to guide the generator network to generate a repaired image based on the degradation information, resulting in a second high-resolution image. Subsequently, a second loss (i.e., the difference between the second high-resolution image and the sample high-resolution image) is calculated based on the second high-resolution image and the sample high-resolution image. The network parameters of the first control network are updated based on the second loss. After the training termination condition is met, the trained first control network is obtained.

[0095] Thus, after the second stage of control network training, the first control network learns to extract intermediate features containing degradation information in the image, so as to better guide the generation of the image to repair the image based on the degradation information of the image. This allows the generation network to retain the low-frequency information consistent with the image to be repaired when generating the repaired high-definition image, without losing details, thus ensuring that the repaired high-definition image has high fidelity.

[0096] In other embodiments, when the control network includes a first control network and a second control network, with the goal of optimizing the de-degradation information of the image, the image inpainting model is trained in a second stage using a first training dataset, including steps F1 and F2:

[0097] Step F1: Fix the network parameters of the degradation removal network, and input the degraded image into the image restoration model through the degradation removal network and the structure information labeling network respectively to perform image restoration and obtain the third high-definition image.

[0098] Step F2: Update the network parameters of the first control network and the second control network based on the differences between the third high-definition image and the sample high-definition image.

[0099] In this embodiment, the network parameters of the degradation removal network are fixed, and the first control network and the second control network in the restoration model are trained using the first training data. Similarly, the degradation-processed images in the training data are used as input images, and the sample high-resolution images are used as ground truth images. For each training data in the first training dataset, the degradation-processed image is input into the image restoration model through the degradation removal network (wherein the degradation removal network refers to the already trained degradation removal network) and the structural information labeling network for image restoration.

[0100] Specifically, the degradation removal network degrades the input degradation image to eliminate various degradation-induced quality defects, obtaining a sample degraded image. This sample degraded image is then input into the first control network for feature extraction, yielding the first sample intermediate features. The structure information labeling network labels the input degradation image with structural locations, obtaining a sample structure location labeling image. This sample structure location labeling image is then input into the second control network for feature extraction, yielding the second sample intermediate features. The first and second sample intermediate features are then input into the generator network to guide it in generating a repaired image based on the degraded information and structural location information, resulting in a third high-resolution image. Finally, a third loss (i.e., the difference between the third high-resolution image and the sample high-resolution image) is calculated based on the third high-resolution image and the sample high-resolution image. The network parameters of the first and second control networks are updated based on the third loss. After the training termination condition is met, the trained first and second control networks are obtained.

[0101] Thus, after the second stage of control network training, the first control network learns to extract intermediate features containing degradation information from the image, and the second control network learns to extract intermediate features containing structural information from the image. Consequently, when performing image restoration based on the image restoration model, the first and second control networks can better guide the generator network to generate restored images based on the degradation and structural information of the image. This allows the generator network to retain low-frequency information consistent with the image to be restored when generating the restored high-definition image, without losing details. Therefore, the restored high-definition image has high fidelity, and at the same time, it ensures that the generator network can repair the structural information (i.e., structural defects) in the image.

[0102] (2) The third stage is to fine-tune the image restoration model in the image restoration model. In some implementations, with the goal of optimizing the structural information of the image, the image restoration model that has been initially trained is fine-tuned in the third stage based on the second training dataset, including steps G1 and G2:

[0103] Step G1: Fix the degradation removal network, input the specific style image into the pre-trained image inpainting model for image inpainting, and obtain the fourth high-definition image.

[0104] Step G2: Update the network parameters of the control network based on the structural information differences between the fourth high-definition image and the sample high-definition image in the luminance channel image.

[0105] In this embodiment, fine-tuning the image restoration model mainly refers to fine-tuning the control network within the image restoration model. The control network includes a first control network and / or a second control network. In some embodiments, when the control network includes a first control network, the first control network in the image restoration model is fine-tuned using a second training dataset. In other embodiments, when the control network includes both a first and a second control network, the first and second control networks in the image restoration model are fine-tuned using a second training dataset.

[0106] Specifically, the network parameters of the degradation removal network are fixed, and the control network is fine-tuned using the second training dataset. A specific style image is used as the input image, and a sample high-resolution image is used as the ground truth image. For each training data in the second training dataset, the specific style image is input into the pre-trained image inpainting model for image inpainting to obtain a fourth high-resolution image. Then, a fourth loss (i.e., the difference in structural information between the fourth high-resolution image and the sample high-resolution image in the luminance channel image) is calculated based on the fourth high-resolution image and the sample high-resolution image. The network parameters of the control network are updated based on the fourth loss. After the training termination condition is met, the final image inpainting model is obtained.

[0107] Both the fourth high-definition image and the sample high-definition image are RGB images. The luminance channel images of the fourth high-definition image and the sample high-definition image are obtained as follows: Both images are converted to YCbcr images (YCbcr is a color encoding scheme for images), and the luminance channel images are obtained based on the luminance channel information of the YCbcr images. For example, converting the fourth high-definition image and the sample high-definition image to YCbcr images can be represented as follows:

[0108] Y=R×0.299+G×0.587+B×0.114;

[0109] Cb=0.568×(BY)+128=-0.172×R-0.339×G+0.511×B+128;

[0110] Cr=0.713×(BY)+128=0.511×R-0.428×G-0.083×B+128;

[0111] Wherein, R, G, and B represent the red channel information, green channel information, and blue channel information in the GRB image (i.e., the fourth high-definition image and the sample high-definition image), respectively; Y represents the luminance channel information in the Ycbcr image; Cb represents the blue chromaticity information in the Ycbcr image; and Cr represents the red chromaticity information in the Ycbcr image.

[0112] In this embodiment, the specific style image represents an image with a style consistent with the image to be repaired. Therefore, after fine-tuning the image inpainting model based on the second training data, the image inpainting model can repair the image to be repaired under the specific style. Furthermore, the luminance channel image removes the influence of image color differences and can better reflect the structural information of the image. Therefore, based on the structural information differences between the fourth high-resolution image and the sample high-resolution image in the luminance channel image, the network parameters of the control network are updated to compensate for the structural loss after de-degradation processing. Thus, the image inpainting model obtained after the third stage of training can pay more attention to the structural information of the image, achieving high-quality image repair.

[0113] In conjunction with the above embodiments, in this application embodiment, the second training dataset is constructed according to the following steps:

[0114] Step H1: Construct a high-resolution image set of samples, which includes multiple high-resolution images of samples.

[0115] Step H2: Based on the target processing method, generate a specific style image corresponding to the high-resolution sample image.

[0116] Step H3: Use the high-resolution sample image and the specific style image as training data in the second training dataset.

[0117] In this embodiment of the application, since there is no real data for images of a specific style, the original real data (i.e., high-resolution sample images) is used to generate images of a specific style, and the generated images of a specific style and the high-resolution sample images are used as training data.

[0118] The specific style image includes any one of the following: black and white photo style, old photo style. In some embodiments, the specific style image is in the old photo style, and the image retouching model trained on the second training dataset can restore the image to be repaired in the old photo style scene; for example, restoring old photos or old videos. In some embodiments, the specific style image is in the black and white photo style, and the image retouching model trained on the second training dataset can restore the image to be repaired in the black and white photo style scene.

[0119] For example, Figure 5 illustrates the effect of constructing an old photo style image based on a high-definition image. In Figure 5(a), the image is from a high-definition image, and in Figure 5(b), the image is an old photo style image generated by the target processing method.

[0120] In conjunction with the above embodiments, in one embodiment, this application also provides a method for generating images with a specific style. In this method, step H2, "generating a specific style image corresponding to the sample high-resolution image based on the target processing method," may specifically include sub-step J1:

[0121] Step J1: Input the high-resolution sample image into a style generative adversarial network for processing to obtain a specific style image corresponding to the high-resolution sample image.

[0122] The style generative adversarial network is trained on training style-specific images and training high-resolution images, and learns the mapping relationship between the training style-specific images and training high-resolution images. The training style-specific images and training high-resolution images include training style-specific images and training high-resolution images that are not image pairs.

[0123] In this embodiment, the Style Generative Adversarial Network (Cycle GAN) learns the mapping relationship between training style-specific images and training high-resolution images. Therefore, when a sample high-resolution image is input into the Style Generative Adversarial Network, the network can transform it into a style-specific image consistent with the training style-specific image, i.e., the style-specific image corresponding to the sample high-resolution image, based on the learned mapping relationship. The training style-specific images and training high-resolution images used to train the Style Generative Adversarial Network do not necessarily have to be paired. For example, any training style-specific image (such as an old photograph) and a high-resolution image (i.e., the training high-resolution image) can be directly used as the training dataset for the Style Generative Adversarial Network.

[0124] Style generative adversarial networks (GANs) employ a bidirectional cyclic generation structure, enabling style transfer from the source domain to the target domain. This requires the GAN to both fit the pixel probability distribution of the target domain image and preserve the corresponding features of the source domain image. Specifically, as shown in Figure 6, given two domains X and Y, where X represents the source domain image and Y represents the target domain image, the goal of the GAN is to learn two mapping functions: G: X→Y and F: Y→X. This ensures that transforming an image from the X domain to the Y domain via G and then back to the X domain via F preserves the original image; similarly, transforming an image from the Y domain to the X domain via F and then back to the Y domain via G also preserves the original image. This cyclic consistency training process helps the GAN learn the mapping relationship between the two domains.

[0125] Specifically, the loss function used for training style generative adversarial networks is expressed as:

[0126] L cyc (G,F,D x D y ) = L GAN (G,D Y ,X,Y)+L GAN (F,D X ,Y,X)+λL cyc (G,F);

[0127] Among them, L GAN (G,D Y (x,y) represents the adversarial loss generated by X to produce Y, L GAN (F,D X Let λL(Y,X) represent the adversarial loss generated by Y to X. cyc (G,F) represents the cycle consistency loss of generators G and F in a style generative adversarial network, where λ is the loss weight.

[0128] In this embodiment, a training image of a specific style is used as the source domain image, and a training high-resolution image is used as the target domain image. The style generative adversarial network is then trained according to the above training method so that the style generative adversarial network learns the mapping relationship between the training image of the specific style and the training high-resolution image. Based on the learned mapping relationship, the style generative adversarial network can convert the sample high-resolution image into a specific style image.

[0129] In conjunction with the above embodiments, in one implementation, this application also provides another method for generating specific style images. In this method, step H2, "generating a specific style image corresponding to the sample high-resolution image based on the target processing method," specifically includes sub-steps K1 and K2:

[0130] Step K1: Based on the style of the specific style image, perform color correction on the sample high-definition image to obtain a color-corrected image.

[0131] Step K2: Degrade the color-corrected image to obtain a specific style image corresponding to the sample high-definition image. The degradation process includes at least one of the following: blurring, noise reduction, scratch removal, dust effect processing, and brightness reduction.

[0132] In this embodiment, color correction is performed on the sample high-resolution image to ensure that the colors of the color-corrected image are consistent with the colors of the image in a specific style. For example, if the specific style image is a black and white photograph, color correction on the sample high-resolution image converts it into a black and white photograph style image. As another example, if the specific style image is a vintage-style photograph with a yellowed, aged feel, color correction on the sample high-resolution image converts it into a vintage-style photograph with a yellowed, aged feel.

[0133] For example, converting a sample high-resolution image into a black-and-white photo-style image (Gray) can be represented as:

[0134] Gray=R×0.299+G×0.587+B×0.114;

[0135] Converting a high-resolution sample image into a vintage-style image with a yellowed, aged feel can be represented as:

[0136] Among them, R * G * B * The red, green, and blue channels of the retro-style image are represented respectively, while R, G, and B represent the red, green, and blue channels of the GRB image (i.e., the high-resolution sample image), respectively.

[0137] After obtaining the color-corrected image, various forms of degradation processing are sequentially applied to the color-corrected image to impart various quality defects caused by degradation to the resulting image of a specific style. These degradation processes are implemented in the following ways:

[0138] Blur processing: Different blurring methods are loaded using OpenCV, such as Gaussian blur and mean blur. In Gaussian blur, the required parameters (e.g., kernel size) and standard deviation are set to random numbers within a certain range.

[0139] Noise processing: Different types of noise are implemented using OpenCV (Open Source Computer Vision Library), such as Gaussian noise and salt-and-pepper noise, to create a flickering effect on the pixels. The mean and variance of the Gaussian noise are also set to random numbers within a certain range.

[0140] Scratch processing: Randomly select certain areas of different shapes and sizes, set the pixels in the area to simulate scratches as high-brightness uniform pixels, and perform smoothing filtering in the area.

[0141] Dust effect processing: Randomly select certain areas of different shapes and sizes, set the pixels in the area to simulate scratches as low-brightness uniform pixels, and apply a smoothing filter to the area.

[0142] Brightness processing: Adjusting the brightness of an image to reduce its saturation and contrast.

[0143] In this embodiment, each high-resolution sample image is color corrected and degraded in the manner described above to obtain a specific style image corresponding to each high-resolution sample image, thereby obtaining training data based on the high-resolution sample image and the specific style image.

[0144] In conjunction with the above embodiments, in one implementation, this application also provides another method for generating specific style images. In this method, step H2, "generating a specific style image corresponding to the sample high-resolution image based on the target processing method," specifically may include sub-step L1:

[0145] Step L1: Input the high-resolution sample image into the target diffusion network for processing to obtain a specific style image corresponding to the high-resolution sample image.

[0146] The target diffusion network is obtained by modifying the network parameters of the diffusion network using a first matrix and a second matrix. The first matrix is ​​randomly initialized according to a standard normal distribution, and the second matrix is ​​initialized to 0.

[0147] In this embodiment of the application, the diffusion network refers to an existing open-source diffusion network with image generation capabilities (e.g., Stable Diffusion, SD model). In order to quickly obtain a target diffusion network that can generate images of a specific style, the diffusion network is fine-tuned to obtain a target diffusion network that can generate images of a specific style.

[0148] Specifically, as shown in Figure 7, the network parameters of the diffusion network are fixed, and two low-rank matrices (i.e., the first matrix and the second matrix) are multiplied together and added to the weight matrix of the diffusion network (i.e., the network parameters of the diffusion network). This achieves fine-tuning of the diffusion network, that is, the parameters of the cross-attention layer in U-Net can only be modified by the first matrix and the second matrix.

[0149] In this embodiment, the training data for the target diffusion network can be existing images of a specific style (e.g., existing old videos, old photos, etc.). Finally, the trained target diffusion network is used to process each sample high-definition image to obtain a specific style image corresponding to each sample high-definition image, thereby obtaining training data based on the sample high-definition images and the specific style images.

[0150] For example, Figure 8 illustrates the effect of image restoration based on the image restoration method in the embodiments of this application. Figure 8(a) is the image to be restored, and Figure 8(b) is the modified high-definition image obtained after restoration. It can be seen that after image restoration, the restored high-definition image significantly eliminates the quality defects caused by degradation, and also repairs the structural defects in the image.

[0151] This application embodiment also provides an image restoration device. Referring to FIG9, FIG9 is a structural schematic diagram of an image restoration device provided in this application embodiment. The device includes:

[0152] Module 910 is used to acquire the image to be repaired;

[0153] The repair module 920 is used to process the image to be repaired according to the image repair model to obtain a repaired high-definition image;

[0154] The image restoration model is trained through multiple training stages, and the loss function of one of the training stages is a loss function aimed at optimizing the structural information of the image.

[0155] In one optional embodiment, the image inpainting model includes: a degradation removal network, a first control network, and a generative network; the inpainting module includes:

[0156] The de-degradation image generation module is used to input the image to be repaired into the degradation removal network for de-degradation processing to obtain a de-degradation image. The de-degradation processing includes at least one of the following: deblurring, noise reduction, scratch removal, dust removal, brightness enhancement, and image super-resolution.

[0157] The first intermediate feature generation module is used to input the de-degradation image into the first control network for feature extraction to obtain the first intermediate feature. The first intermediate feature is used to guide the generation network to generate a repaired image based on the de-degradation information in the de-degradation image.

[0158] The first high-definition image generation module is used to input the randomly sampled noisy image and the first intermediate feature into the generation network to generate a repaired image, thereby obtaining a repaired high-definition image.

[0159] In an optional embodiment, the image inpainting model further includes: a second control network and a structural information labeling network; the inpainting module further includes:

[0160] The structural location marking module is used to input the image to be repaired into the structural information marking network to mark the structural location, thereby obtaining a structural location marked image;

[0161] The second intermediate feature generation module is used to input the structural location marker image into the second control network for feature extraction to obtain the second intermediate feature. The second intermediate feature is used to guide the generation network to generate a repair image based on the structural information of the structural location marker image.

[0162] The second high-definition image generation module is used to input the randomly sampled noisy image, the first intermediate feature and the second intermediate feature into the generation network to generate a repaired image, thereby obtaining a repaired high-definition image.

[0163] In an optional embodiment, the apparatus further includes a model training module for training the image restoration model, the model training module comprising:

[0164] The first training module is used to optimize the de-degradation information of the image by sequentially training the image restoration model in the first stage of degradation removal network training and the second stage of control network training according to the first training dataset, so as to obtain the preliminarily trained image restoration model.

[0165] The second training module is used to perform a third-stage model fine-tuning on the initially trained image restoration model based on the second training dataset, with the goal of optimizing the structural information of the image, to obtain the trained image restoration model.

[0166] Each training data in the first training dataset includes: a sample high-resolution image and a degraded image; each training data in the second training dataset includes: a specific style image and a sample high-resolution image, wherein the specific style image represents an image whose style is consistent with the image to be repaired.

[0167] In one optional embodiment, the first training module includes:

[0168] The first-stage training module is used to input the degraded image into the degradation removal network for degradation removal processing to obtain a first high-definition image; and to update the network parameters of the degradation removal network based on the difference between the first high-definition image and the sample high-definition image.

[0169] In an optional embodiment, when the control network is a first control network, the first training module includes:

[0170] The second-stage training module is used to fix the network parameters of the degradation removal network, input the degraded image into the image restoration model through the degradation removal network for image restoration, and obtain a second high-definition image; and update the network parameters of the first control network according to the difference between the second high-definition image and the sample high-definition image.

[0171] In an optional embodiment, where the control network includes a first control network and a second control network, the second-stage training module is further configured to:

[0172] The network parameters of the degradation removal network are fixed, and the degraded image is input into the image restoration model for image restoration through the degradation removal network and the structural information labeling network respectively to obtain a third high-definition image; the network parameters of the first control network and the second control network are updated according to the difference between the third high-definition image and the sample high-definition image.

[0173] In one optional embodiment, the second training module includes:

[0174] A fixed input module is used to fix the degradation removal network and input the specific style image into the pre-trained image restoration model for image restoration to obtain a fourth high-definition image.

[0175] The update module is used to update the network parameters of the control network based on the structural information differences between the fourth high-definition image and the sample high-definition image in the luminance channel image.

[0176] In an optional embodiment, the apparatus further includes a construction module for constructing the second training dataset, the construction module comprising:

[0177] A high-definition sample construction module is used to construct a high-definition image set of samples, which includes multiple high-definition images of samples.

[0178] A specific style image generation module is used to generate a specific style image corresponding to the sample high-resolution image based on the target processing method;

[0179] The training data module is used to use the sample high-resolution images and the specific style images as training data in the second training dataset.

[0180] In one optional embodiment, the specific style image generation module includes:

[0181] The first processing module is used to input the sample high-definition image into a style generative adversarial network for processing to obtain a specific style image corresponding to the sample high-definition image.

[0182] The style generative adversarial network is trained on training style-specific images and training high-resolution images, and learns the mapping relationship between the training style-specific images and training high-resolution images. The training style-specific images and training high-resolution images include training style-specific images and training high-resolution images that are not image pairs.

[0183] In one optional embodiment, the specific style image generation module includes:

[0184] The second processing module is used to input the high-resolution sample image into the target diffusion network for processing to obtain a specific style image corresponding to the high-resolution sample image.

[0185] The target diffusion network is obtained by modifying the network parameters of the diffusion network using a first matrix and a second matrix. The first matrix is ​​randomly initialized according to a standard normal distribution, and the second matrix is ​​initialized to 0.

[0186] In one alternative embodiment, the style of the particular style image includes any one of the following: black and white photo style, old photo style.

[0187] This application also provides an electronic device. Referring to FIG10, FIG10 is a schematic diagram of the structure of an electronic device provided in this application embodiment. As shown in FIG10, the electronic device 1000 includes: a memory 1010 and a processor 1020. The memory 1010 and the processor 1020 are connected via a bus communication connection. The memory 1010 stores a computer program, which can be run on the processor 1020 to implement the steps of the image restoration method described in this application embodiment.

[0188] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the image restoration method described in this application.

[0189] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image restoration method described in this application.

[0190] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0191] This application describes embodiments of methods and apparatus according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal equipment to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal equipment, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0192] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0193] These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable terminal equipment, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0194] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0195] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0196] The above provides a detailed description of the image restoration method, apparatus, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image restoration method, wherein, The method includes: Obtain the image to be repaired; The image to be repaired is processed according to the image restoration model to obtain a restored high-definition image; The image restoration model is trained through multiple training stages, and the loss function of one of the training stages is a loss function aimed at optimizing the structural information of the image.

2. The image restoration method according to claim 1, wherein, The image restoration model includes: a degradation removal network, a first control network, and a generation network; based on the image restoration model, the image to be restored is processed to obtain a restored high-definition image, including: The image to be repaired is input into the degradation removal network for degradation removal processing to obtain a de-degradation image. The degradation removal processing includes at least one of the following: deblurring, noise reduction, scratch removal, dust removal, brightness enhancement, and image super-resolution. The de-degradation image is input into the first control network for feature extraction to obtain a first intermediate feature. The first intermediate feature is used to guide the generation network to generate a repaired image based on the de-degradation information in the de-degradation image. The randomly sampled noisy image and the first intermediate feature are input into the generation network to generate a repaired image, resulting in a repaired high-definition image.

3. The image restoration method according to claim 2, wherein, The image inpainting model further includes: a second control network and a structural information labeling network; the method further includes: The image to be repaired is input into the structural information labeling network for structural location labeling to obtain a structural location labeled image; The structural location marker image is input into the second control network for feature extraction to obtain a second intermediate feature. The second intermediate feature is used to guide the generation network to generate a repair image based on the structural information of the structural location marker image. The randomly sampled noisy image and the first intermediate feature are input into the generation network to generate a repaired image, resulting in a repaired high-definition image, including: The randomly sampled noisy image, the first intermediate feature, and the second intermediate feature are input into the generation network to generate a repaired image, resulting in a repaired high-definition image.

4. The image restoration method according to claim 2 or 3, wherein, The image restoration model was trained according to the following steps: With the goal of optimizing the de-degradation information of the image, the image restoration model is trained in the first stage of degradation removal network training and the second stage of control network training according to the first training dataset to obtain a pre-trained image restoration model. With the goal of optimizing the structural information of the image, the image restoration model that was initially trained is fine-tuned in the third stage based on the second training dataset to obtain the trained image restoration model. Each training data in the first training dataset includes: a sample high-resolution image and a degraded image; each training data in the second training dataset includes: a specific style image and a sample high-resolution image, wherein the specific style image represents an image whose style is consistent with the image to be repaired.

5. The image restoration method according to claim 4, wherein, With the goal of optimizing the de-degradation information of images, the restoration model undergoes a first-stage degradation removal network training based on a first training dataset, including: The degraded image is input into the degradation removal network for degradation removal processing to obtain the first high-definition image; The network parameters of the degradation removal network are updated based on the differences between the first high-definition image and the sample high-definition image.

6. The image restoration method according to claim 4, wherein, With the first control network as the target, and aiming to optimize the de-degradation information of the image, the image inpainting model undergoes a second-stage control network training based on the first training dataset, including: By fixing the network parameters of the degradation removal network, the degraded image is input into the image restoration model through the degradation removal network for image restoration, resulting in a second high-definition image. The network parameters of the first control network are updated based on the differences between the second high-definition image and the sample high-definition image.

7. The image restoration method according to claim 4, wherein, In the case where the control network includes a first control network and a second control network, with the goal of optimizing the de-degradation information of the image, the image inpainting model undergoes a second-stage control network training based on a first training dataset, including: By fixing the network parameters of the degradation removal network, the degraded image is input into the image restoration model through the degradation removal network and the structural information labeling network respectively to perform image restoration and obtain a third high-definition image; Based on the differences between the third high-definition image and the sample high-definition image, the network parameters of the first control network and the second control network are updated.

8. The image restoration method according to claim 4, wherein, With the goal of optimizing the structural information of the image, a third stage of model fine-tuning is performed on the initially trained image inpainting model based on the second training dataset, including: By fixing the degradation removal network, the specific style image is input into the pre-trained image inpainting model for image inpainting to obtain the fourth high-definition image; The network parameters of the control network are updated based on the structural information differences between the fourth high-definition image and the sample high-definition image in the luminance channel image.

9. The image restoration method according to claim 4, wherein, The second training dataset was constructed according to the following steps: Construct a high-resolution image set of samples, which includes: multiple high-resolution images of samples; Based on the target processing method, generate a specific style image corresponding to the high-resolution sample image; The high-resolution sample image and the image with the specific style are used as training data in the second training dataset.

10. The image restoration method according to claim 9, wherein, Based on the target processing method, a specific style image corresponding to the high-resolution sample image is generated, including: The high-resolution sample image is input into a style generative adversarial network for processing to obtain a specific style image corresponding to the high-resolution sample image. The style generative adversarial network is trained on training style-specific images and training high-resolution images, and learns the mapping relationship between the training style-specific images and training high-resolution images. The training style-specific images and training high-resolution images include training style-specific images and training high-resolution images that are not image pairs.

11. The image restoration method according to claim 9, wherein, Based on the target processing method, a specific style image corresponding to the high-resolution sample image is generated, including: The high-resolution sample image is input into the target diffusion network for processing to obtain a specific style image corresponding to the high-resolution sample image; The target diffusion network is obtained by modifying the network parameters of the diffusion network using a first matrix and a second matrix. The first matrix is ​​randomly initialized according to a standard normal distribution, and the second matrix is ​​initialized to 0.

12. The image restoration method according to any one of claims 9-11, wherein, The style of the specific style image includes any one of the following: black and white photo style, old photo style.

13. An image restoration apparatus, wherein, The device includes: The acquisition module is used to acquire the image to be repaired; The repair module is used to process the image to be repaired according to the image repair model to obtain a repaired high-definition image; The image restoration model is trained through multiple training stages, and the loss function of one of the training stages is a loss function aimed at optimizing the structural information of the image.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the steps of the image restoration method according to any one of claims 1-12.

15. A computer-readable storage medium having a computer program stored thereon, wherein, When executed by a processor, the computer program implements the steps of the image restoration method according to any one of claims 1-12.

Images